Why 94% of Companies Fail at AI ROI | Sven Peters (Atlassian)

Only 6% of companies see a real Return on Investment from AI. The rest buy licenses, burn through tokens, and confuse faster individual coding with actual business value. Speeding up an individual developer is meaningless if team workflows and PR reviews stay completely unchanged.

On the MarsBased podcast, Àlex Rodríguez Bacardit joins Sven Peters, AI Evangelist at Atlassian, to tackle why most corporate AI implementations fail. Since every company has access to the exact same underlying LLMs, your true competitive edge relies on internal context, documentation, and codebase history, not simply upgrading to newer models.

They explore how active leadership usage makes engineering teams four times more likely to adopt AI long-term, while highlighting how Atlassian measures genuine DevX impact instead of tracking wasted token budgets.

Follow Sven Peters on LinkedIn: https://www.linkedin.com/in/svenpeters73/

Transcript

The Hard Truth About AI ROI

SPEAKER_00 0:00

What we found out is that just like six percent of organizations see a return of investment of their eye. Everyone else just like tries AI on the individual level and makes the individuals faster, but faster doesn't mean better or return of investment because it's just like faster. If leaders are living and breathing AI and using it themselves, it is four times more likely that also the team is using AI. Models are super intelligent, but they are nothing without the context. And it's your competitive advantage. Your competitor can be using the same models as you. Um, but your context is actually the differentiator. It's not how many tokens you burn, it's not how how much how much your people use AI, it is the impact on the business.

SPEAKER_01 0:59

Don't worry, I will not be using any more German words.

SPEAKER_00 1:02

Any more German words?

SPEAKER_01 1:04

No.

Meet Sven And The Evangelist Role

SPEAKER_01 1:06

Sven, welcome to the show. How are you doing? Welcome to Life on Mars.

SPEAKER_00 1:09

Yeah. Hello, Alex. Thanks for thanks for having me. Um it's it's it's it's great. I I'm a big fan of of the show, actually.

SPEAKER_01 1:17

Uh you're a fan of the show? I did I didn't know.

SPEAKER_00 1:20

Yeah, yeah, yeah. Surprise, right? No, but you just like pinged me and then I watched a couple of episodes, and then I said, Oh wow, this is really cool. I need to put that into my my podcast list, actually. Great interviews. Nice.

SPEAKER_01 1:33

Yeah thank you. Well, actually, uh comes to uh comes as a little bit as a surprise because normally our biggest part of the show, the show is split in two, right? We got the the the feed in Spanish, the feed in English, but both of them have been growing a lot. And we started the year with only a thousand subscribers. Now we're past 12,000. So it's kind of like taking off now. So yeah.

SPEAKER_00 1:53

Awesome.

SPEAKER_01 1:54

Big part is because we bring great speakers like you, and so I'm pretty sure this one is gonna uh reap a lot of views as well. So well, um I want I wanted to kind of like kick it off by by asking you because uh by and large, as a consultancy, we get to see how different companies operate and how different companies organize internally, uh small startups, SMBs, large star uh scale-ups, and now even corporations work for, but there are several companies we don't know because we don't we we don't work for, and and some of them are pretty secretive about how they work, some others have got more culture of sharing. In your case, having having this this role of evangelist, the death rail as well. Some companies are more prone to having that and sharing more. So um, as you have extensive background in this kind of roles, what uh what what are you actually liking about this role? How has it transformed in the last 12 months?

SPEAKER_00 2:53

Thanks to AI. So I speak to a lot of um also CTOs of of companies and see how they how they actually use use AI. Um and that's actually my my role. So just like talk to CTOs of larger companies. Mostly these are big enterprises, like big banks, big insurance companies, all of all of these companies. And um just like help them with with their adoption. Now, as you as you know, like there's a lot of a lot of struggle, especially in this enterprise, to get to get uh AI adoption right. Um and it's it's my role actually also to just like say uh I've seen that in in this one company and it worked there. So might be might be good applying that to your company too. So I've I see see a lot of a lot of patterns there and try to help help corporations to adopt AI in this new new era.

SPEAKER_01 3:57

But I see like when you talk to CTOs and uh potentially to establish some sort of partnership and whatnot, they might want to know more about that. And you're like, yeah, Cynthia's another company, but you cannot tell who the company is or what they do exactly, the exact details of the implementation. How do you go about this? Then sharing just the right amount of information.

SPEAKER_00 4:16

Yeah, so I'm just like an a door opener. So I can I can make connections, um, obviously. Um but these CTOs are are pretty cool with just like sharing their information with with other with other companies. I obviously am not a consultant and can't just like sit with them and help them with AI adoption, but I can give them ideas um and then show them show them the way. Uh, and then someone else has to take over and help them actually. Or they actually have to step up and say, okay, let's try these these three things uh to adopt AI, um, to have a great AI adoption and have our employees use AI on a daily basis. Um This is just what I what I can do. I can just inspire um and then maybe give them connections to others to just like dive deeper into the mix, into the into the wheels. Agree.

SPEAKER_01 5:09

You mentioned that you've got big opinions on AI. So let me challenge that. Let's start. Let's start with the big

Why Most AI Use Stalls Out

SPEAKER_01 5:15

one. Um my observation as a as a as an external consultant, right, has has been that we've seen a lot of people and companies saying that they use AI. And what they've been using of AI is kind of like they're buying licenses, they're spending credits like there's no tomorrow, and they're creating a lot of AI slot. But very few of them are actually using AI for something transformative, or say like, well, that's a good example, I can replicate somewhere else, right? I've yet to see a lot of these cases. I've seen one or two or three or four, but definitely not every company has shown me something that's inspirational, something that I can adopt in my own company. Where you you're probably talking to more and bigger companies. So if you can single out three or four cases in which you've seen this is transformative. And the second part of the question is like, are people out there really using AR or are they just saying that they use AI? Um question two one.

SPEAKER_00 6:14

Um let me let me put it that way. Um we actually have the same observation about the return of investment. We we are running a uh what we call State of Teams report where we ask companies, knowledge workers in in different different organizations. And what we found out is that just like six percent of organizations see a return of investment of their of their eye. Right. Um that's that's a big that's just like a big number. Now, taking that, these are corporations of all kinds of sizes, um, mostly enterprise corporations. Um and they just like start. Some some some companies are just like start with with AI right now. Um and then of course, yeah, it takes time to educate your employees, to really identify your your your use cases until you can see a return of investment that that actually takes time. Um and others are just like on the other side of the spectrum where they just like run every single genetic automation, blah, blah, blah. Um so we we see just like a big spread right now in the market. And we will see more people being on the return of investment, investment side. But what we also found out is um in the survey, what what are those companies are actually doing differently? Um and um they apply AI at the team level, which if you if if you think about it, makes sense, right? Because everyone else just like tries AI on the individual level and makes the individuals faster, but faster doesn't mean better or return of investment because it's just like faster. And the other ones are applying on team level, and we know that since years. I mean, I've been before I became the AI evangelist for Atlassian, I was the team evangelist for Atlassian. So teams are actually thriving and driving in organizations, and they make the difference in organizations. Good teamwork makes a difference in organizations. Um and we know that since years. So applying AI at the team level really, really helps. Now, this sounds very theoretical, right? Um we we just like want to have some some you probably want to have some practical examples here. Um and we also looked into this what are those those those actually doing differently on on a practical level and identifies three things.

Context Workflows Culture Explained

SPEAKER_00 8:47

First, right context matters. If AI doesn't have the right context, it can't really, really, really help you, right? Um so giving AI the right context and not just your team context, but the whole organizational context uh really really helps. So these these have figured it out how to give them context. Um, also how employees give AI context. So we we all know that, right? The first question that we ask uh in Chat GPT, we get an answer, and then uh it's not not the right thing that I gonna ask you, uh, and then people give up. But giving learning how to give AI the right context to get the to the right answer or to the right result requires some some training, some experimentation, some um just like experience with with AI. So also this context matters. That's also what we found out. Like these are uh using AI for for longer, they actually get the context, they get the courage to use AI. So this is this is number one context. Number two is workflows. Now we are known as the the the the workflow company, uh process company Dira is all about workflows and processes. Um and also here what we see is just like we encourage our customers actually to look at their exist existing processes and look closely at the process and see where is actually the human making steps in this process where actually AI can fit in. For example, um Mercedes-Benz, um, they looked at their bug report process and found out actually that they have a lot of duplications in their backlog. Now, finding duplications is not a good job for humans, obviously. Um, looking at all your backlog items and finding, oh, this is a duplicate bug report. Um it's it's just like a very good job for AI. So having those small little uh agents, and and they they uh they fix their their duplication problems by having an agent looking at all new created created work items or created bug reports, and let them just like, okay, this is a duplication of this, and we just like close that automatically. Um so that that that is that is that is working pretty pretty good for for Mercedes Bands. Um But all of these small little helpers can help you just like accelerate and in your workflows. Now, acceleration as as I said, like faster doesn't get us to a return of investment and getting better. Um you also need to see how you how you rework your workflows. So the second is workflows, and the last one is is the culture. Um you need to build an AI culture that works or that concentrates on, everyone just like concentrates on business impact and not just using AI to getting faster, but really to create business impact. And it should be baked into people's minds that okay, every time I use AI or I want to build an agent, what business impact does that have? Um and that's a lot of I just like talk to you about just like okay, training and experimentation, that's important. That's this bottoms-up movement, but you need to also have a top-down movement from leadership. Um and we we we also have have stats around that, that um if leaders are living and breathing AI and using it themselves, it is four times more likely that also the team is using AI and and continue to use AI. You need to also look at the continuation. So also there needs to be a top-down mandate. Obviously, um AI culture means business orientation, which means you need to measure the impact, not just your tokens that you burn. Um and that that that's just like a metrics that I a metric that I that I see in a lot of organizations. Just like they look at, okay, how many tokens did you burn and encourage people to burn more tokens. That's not that's not a good good usage of or just like metric to use AI. You need to really nail it down to the business uh impact that that you have um and that that you want to want to encourage.

SPEAKER_01 13:14

There's a lot to unpack here.

Leadership Makes Adoption Stick

SPEAKER_01 13:16

This is super clear on the on the three fronts. Uh uh let's just start from from the very end, because uh I didn't expect this, but uh when you mentioned that it has to some s somehow trickle down, right, from leadership teams. So we know that incorporate innovation, perhaps the uh the the involvement of C-levels and the executive team plays a big role in whether in the chances of a project being successful or not. Um But um I I I have I'm struggling to see how that works in AI, especially in such large corporations where you you know are you how do you actually measure that the impact of leadership, some sort of like inspiring the bottom layers of the company in companies that perhaps have got like 17 or 70 different layers? How do you ensure that it trickles down to every department?

SPEAKER_00 14:13

So get this straight, right? Um there are companies that have been thriving and driving before AI and that change changed constantly their way they work and constantly know where our way we do create business impact. These companies are already set up for success, right? They just like use, okay, now it's AI, just like to the mix. Um then there are companies there where where things are going a little bit slower, um, or where it's not baked into the culture that you change the way you work constantly. Um and those companies are are the ones probably that you mentioned that are just like it's it's it's just like how do you get those those CEOs want to have the impact of AI. They want to see the impact. They want to just like, yeah, please use AI. But that's an easy thing to say, right? Uh and I think a lot of companies just like here are the tools, here are the tokens, here's your budget, um, here's your training, here's your experimentation day or whatever with AI, your playground, your sandbox. Um and now success. And then that's not happening. Make no mistakes. Yeah, right. Now it's on you. We're pushing it to you. You know, now it's your your fault if it doesn't succeed. Um but we have seen that also, and I'm I'm just like I'm a long-term Atlassian um employee, and I joined 2011 when we were all in the agile transformation, and people were coming to me and saying, hey Sven, you know, we have we have bored Jira. We have certified our Scrum masters, and we also run sprints. We are agile, but we can't see the results. And said, duh, right? It's not, it's a cultural change, right? It's like constantly change the way you work and iterate. And it as it's the same now with AI. It's probably the same with every change, but it's just like obviously it's the same now with AI. We need to look at the measurements and and and leadership needs to be accountable for the success. All kinds, all down the ladder. And I think this is this is this is the top-down uh that you want to drive, right? Leadership needs to just like we have a goal at Alassian, just like to drive impact with AI and change our ways of working. And that's it just like company level goal that we just like drive. Um, to just like see which parts of the business can be changed to just like have an impact for our which workflows need to change or which which processes need to change. Um, and we measure the impact. Say, okay, we we built those agents, we this is our uh this is our idea. We just like look at where are those bottlenecks in the organization, where where are the signals that we can just like measure to assume that if these signals go up or down or whatever the direction is, will show us, will lead us to the success, right? And then we we leadership is accountable for just like make those AI use cases and show that there's business impact. But it needs to be seriously from the organization driven down. It can cannot be just like, okay, here's AI and use it, right? The leadership needs to be accountable for the success of AI. And this is what I mean with with top-down. And it needs to be baked into the culture so everyone is aware of okay, once we just like build an agent, how do we measure the impact? If it's an let's say if it's a bigger agent that a whole department uses, right? What is the impact of of those agents?

SPEAKER_01 17:49

What we've seen in in companies adopting AI, like we've seen two different approaches, right? In one of them is precisely the one you mentioned is like, oh, we want everyone's licenses. Uh, now you can experiment with AI. Go and do your magic thing, and um letting them run wild. Uh surprise, at the end of the year, no one produced tangible things, or maybe just isolated teams, they did it right, but the rest of the organization just went chaos and anarchy and just spending mo uh tokens and like maybe even the the they learn the wrong way. Like if you learn how to drive on your own and you you get like bad postures and like things you shouldn't do because you haven't learned the um the canonical way. And the other approach, which to me is much more um much more positive and maybe has yielded better results, in which the the leaders of the companies, and perhaps this happens in smaller companies, they have been running workshops or sharing stuff. Like, for instance, we got the example of Shopify. You got uh Toby uh from Shopify sharing everything he does with the team. Uh like that's super instructional, that's super um, that's super um transparent. That inspires people off the team, right? So I don't know, like what what what approach have you done internally and has it been consistent across the board, or have you have a little bit of both at the lassian?

Internal Experiments That Drive Change

SPEAKER_00 19:14

Um we actually we we we do a lot of these things that you just like said. We are sharing, uh we are just like run an experiment, for example. We we had a uh company company-wide um uh product builder week where we actually encouraged um our product managers and designers to just like come together for a week and build something something with AI. And that has actually um has has actually had a bigger bigger impact, actually. Um because now people are sharing their their AI ideas. We're coming together across um across department corporation call collaboration. We we we saw there that that's that's a big impact. Um but doing this seriously and just like also concentrating on the impact that you have and not just running an ex running run running also what I see just like people are just like running in a sandbox that has nothing to do with what they do in their business, but just like trying things out. That's not experimentation for me. It's really with business problems and just like coming together and solving as a cross-functional team this business problems and experimentation. What we also do is what we what we have is channels, Slack channels where we share actually we in the marketing team, there's probably there's a channel called uh Marketing AI Hacks. Uh, and it's it's really it's really good. Um because sometimes you're not aware of just like that. AI first can't do that, and second, oh, this use case is just like fascinating. So for example, last I think it's just like two months ago, um, that I saw someone chat, like, oh, I'm showing all my um we use we use Loom uh note taker for just like taking notes and in meetings um that summarize the meetings on a page for us. Now, these pages are AI generated, and who reads through those pages? It's a lot of AI slob, right? It's huh. Um it's still good information in there. So what what we what what what I actually was the AI hack was to create a confluence whiteboard out of the meeting notes. So out of the transcript, create a whiteboard. And sometimes I can't attend meetings because they are we we are we are Sydney based uh our headquarters in Sydney, we have people in the US, so time zones are things, so I can't attend every meeting. And then I read the summary of the meeting, or we need to watch watch the recording. But what I do is now just like I take the recording, throw it into our robo and say, hey, just like create me, create me a confluence whiteboard out of out of this meeting. And then visually it builds a whiteboard, and it's just so much better. Than just like reading through a wall of text. It's so much better. And just like it it changed actually how I consume those meetings. So this is just like one example how we just like share our learnings. And it's part of everything. It's the training is important. Don't don't I mean, I think if we need to learn also how to use AI in an ethical way and all of this, just like how to use what are the different models and when to use what. The experimentation is important, just like to concentrate on business problems, to see what's what's possible, what's not, and also what's not possible with AI, um, and where we need still the humans. And then sharing the success stories across the organization also super important. I think it's all good practices, and you need to see in your organization where are you in your on your journey and which which is the best practice to adopt. Um don't throw everything at all at the same time at people, they get overwhelmed. It's like start step by step. But um that's just like also part of what we're doing, actually.

Champions And Uneven Team Adoption

SPEAKER_01 23:06

In fact, let's start at the team level, because you mentioned that you earlier on your career you worked as a team evangelist, if I got that correctly. So um at team level, one of the things that we have seen is that teams, uh, especially large organizations, they adopt AI differently, right? Or different paces. Some of them adapt, some don't. Therefore, it's not fair to compare them, right? Because you're comparing pairs to apples. But how have you dealt with the transformation or adoption of AI in Atlassian with different teams that probably some of them were more skeptical, some of them were not as tech savvy? How have you approached the best adoption?

SPEAKER_00 23:41

Um, so different thoughts about that. So, first, skeptical, I think well, what we see is actually and what also what our state of teams report point out is actually that people want to use AI. What they what they are afraid of mostly is not that they're just like afraid of losing their job, but they are afraid of not using AI and then getting left left behind. Um so it's it's for those companies, right? Just like let people use AI and let's let them let them play play around um with it. Um sorry, what was your question? I just like I got carried away. Sorry.

SPEAKER_01 24:24

Yeah. Um so how to keep up with different adoption paces because the adoption is not um homogeneous.

SPEAKER_00 24:33

Yeah, so uh obviously our our um people in in in the engineering organization, they are very tech tech heavy. Uh savvy. They they just like want to use AI, they they just like try things out, they build stuff with AI. Um so these this is not our our big big issue, right? If we just like allow them to, they just like build something, they build awesome things, we share that, we just like try to try to platformize it. So it's it's it's just like in a big organization, 6,000 engineers, you need to platformize it and just like run it, you know, run a run a uh golden gold pass with it. But anyway, um for the other ones, uh what we what we did is we actually looked at um who's who's using AI. Um and we identify champions. So people in legal, in sales, in marketing that are using AI every day, multiple times, maybe even built agents uh with AI, and using those people, we use those people to go to them and say, Hey, great job, but you are the go-to person in your organization to spread it out and to help people adopt AI. Because and then it was clear from from our leadership that we want to use AI. It is something that we need to bake into our our system. And then having those champions that encourage people to use AI, and then also having someone just like in in your same same department that you can go to and say, ah, I have this legal problem, and then this person understands this legal problem and understands AI, just like perfect match, right? And you can go go to those people, and that helped us also in in this part of the organizations to implement AI. Now, obviously, engineering is our number one AI use use use uh persona. Um, surprisingly, it's legal. Um, so they build a lot of agents, just like to review contracts and all things. If you think about it, it makes sense, right? Um, but we got them to that they're really adopting AI and they're building a lot of a lot of agents to help them through their daily work. And then we need to see just like how we how we encourage more the salespeople, the marketing people, um, that are a little bit further away from from the technology. But um, yeah, that that that is our approach to to do that. And I saw I saw that also in a lot of enterprise organizations that identifying those champions, seeing who's who are the ones that are using heavily AI, and making them um the go-to persons to help. And then giving them, okay, they run trainings, um, they can that we give them time so they can run trainings, all of this, um, helps actually to spread AI, AI adoption. And not just short term, um, because that's also what we saw. Like, if you just like run trainings, people use AI for one week, for two weeks, but then it drops. But using it um sustainably and uh constantly, that's that's that's also a challenge and that you solve partly with those champions, but also with Slack channels or or Teams channels where you just like share best practices.

SPEAKER_01 28:00

Ideally, um I think that if the champions that you identified overlap 100% with your tech leaders. So let's just talk about let's focus on tech here because um mainly our audience is mostly tech heavy. Sure. Um we can go to other departments afterwards. But uh on the tech department, if you got your leaders who overlap 100% with the champions, that's ideal. Because then everyone who is in like in the bottom layers of the of the organization, they look up to the leaders and they're like, oh, I respect that person. I learned from him, I learned from them. Um and I I want to be like him or her, and I want to be in her position. Therefore, if they are they are telling me that I have to use AI, I will use AI. What happens in organizations where the tech leaders are not the champions of AI or they don't use them, they don't use AI as they are supposed, which we've seen that also happen. And like maybe the other layers are like, ah, I somehow they don't trust that person anymore. For AI, maybe for tech, I respect them. But for AI, they're not the best example. And I'm better suited now to do that role of champion.

SPEAKER_00 29:10

It's

When Leaders Don’t Model AI

SPEAKER_00 29:11

a problem. Um, definitely. Uh and as I said, like lead leaders, if leaders actually use AI, it's four times more likely that the team just like Thrive and Drive in this world. Um so how can we how can we solve that? It's just like we need to encourage those leaders to just like use AI so they understand also what's happening down at the at the at the layer. But it was always a problem. Like people, leaders that don't understand the code, but the team is very close to the code. It's really hard, right, to understand this technical, technical architecture questions and answer that and respect respect that. You probably want to have AI leaders or tech tech. I mean, like, like you have people manager and then you you have your your tech leads. And probably you need to also have your tech leads that people look up to, being also the AI lead um in the in this organization. Now there's there's another nuance here with with AI adoption, because if you look at the what we saw, like junior people are not that experienced, but they use AI like they are AI native, right? They just like live and breathe AI and they think AI are that. Now, with the seniors, that's a that's just like okay, they just like, oh, it's a new technology, and let's see how we can apply that. And it's not, it's not, I mean, I've been doing things for years this way. Why should I use now AI? It just like doesn't help me. Um But what we also saw is that um if if seniors use AI, they they they they are more effective because they can just like validate if the output of AI is the right output, or if I just like have to ask AI another question or lead AI in another way. Um juniors not so effective because they don't have the experience. But bringing those those those two together is also a perfect match, right? So having the still the innovation brain of a junior person um and having the experience of a senior person and bringing those together in a team is like magic, right? And then then it happens just like the the the the experience, people can just like juniors can get that. Um but uh and and seniors can learn from juniors. So perfect here. But as I said, right, um finding those tech leaders and just like let them use AI, that's that's that's key. I don't know how else we can we can solve that. It's uh it's a human problem, um, not a technology problem.

SPEAKER_01 31:48

Yeah, 100%. And we we've seen that also extensively, where some tech leaders, like sometimes it's like the VP Engineering or a certain tech lead and stuff like that, engineering manager. They're like, they're using AI because they've been forced to, but they really don't want, and so they do a sloppy job. And uh that also trickles down because the people below them, they just don't see them as a reference anymore, right? And so there must be some sort of like KPI. Like they use it, but maybe they're kind of like, oh, I'm just copy-pasting the output of what the AI says here and just like tap-tap tap through things, which is not perhaps the right thing to do. So speaking of a team level, what sort of KPIs are you using to measure AI adoption and the ROI in technical teams?

Adoption Metrics Versus Real ROI

SPEAKER_00 32:33

So I mean, AI adoption is probably how much how much they use AI on a daily basis, how much agent they build, how much, how much they they uh use AI prompt. That's AI adoption, that's how how we how we uh measure it. Return of investment is something super different, and you should measure it on the team level, um, or even even on the department level, whatever your layers are, right? Um and it's always it's it's it's it is still the good old um developer experience metrics that you you run, right? PR throughput is is is one metrics that you can just like see. It's it's a signal, I would say. It was not not really like the outcome is just like, oh yeah, PR throughput. But if you just like see how much how much pull requests come through your your systems, that's that's a measurement um that that you see. And what we also saw is like it's not just the and you need to look not just at the coding bit. Um our our own uh team from DX did a survey, and that what they see is AI adoption is just like rising, but um actually the the um the effectiveness or how how how productive developer teams are s still stays at 10 to 15 percent um in just like in the last year. And we don't get and and we know the models get better, the tools get better, but we're still stuck in here. So we need to also look at the developer going back to the developer experience metrics and not just apply AI at the coding level, um, but also apply AI at all kinds of pockets of the software development lifecycle. Um and that's that's just like brings us brings us higher in the in the in the productivity. Um and that's a good old devx metrics. Um just like identify the the problem, uh have signals, right? So we run regular surveys, for example, to identify areas where the team struggles, um or where where where where not I wouldn't say struggle, but where the team is not satisfied, right? Um execution independence, shipping code, quality code fast enough, all of these things, right? And we just like look at those pockets and then identify the the the pocket basically. Then we look inside of this and just like unfold it and just like talk to developers and say, what is exactly what you're what what you're struggling with here? And then we look at what are the signals that we can measure that we are successful, and then we just like try can AI just like remove remove these bottlenecks or help us, and then we measure the signals, and at the end we have the survey again if this area improves, right? Um, this is a typical DevX um metrics where you just like say quantitative data, qualitative data, and that's basically also giving you the return of investment of AI. It's not how many tokens you burned, it's not how how much how much your people use AI. It is the impact on the business. At the end, you build something and you want to increase customer satisfaction or whatever. Um, but that's that's a metric that is is uh just like the lacking metric, and it's sometimes hard to see how this influence. So we just like relying on the signals that we believe will influence the the lacking uh metric that we have, lacking indicators um that we have. But yeah, this is how how how I think like we don't have to throw everything out because now AI, right? Um we we just like good go go back to good old practices that we have learned along the way that still true. Um so we don't just like say, oh, it's AI, and now we just like use it for we use clawed code and it's it's just like coding, coding helps us with the coding, but now we're just like getting faster. No, no, no, no. We just still need to look at what makes sense, where's the metric, and how do we influence the business impact.

SPEAKER_01 36:38

Exactly, because uh PR throughput, like it's it's not a far cry from like the old old uh lines of code, right? It's just like a little bit further down, a little bit more elaborated, but it can be tricked, it can be gamed, right? Uh it could be like some people just sending like PRs left, right, and center, just because it's an incentives game. That's where uh that's where I wanted to go. So if you tell people like, look, what you have to do is this, uh of course it's not burning tokens. It's like, oh, we want you to ship uh good PRs. Like, well, okay, let's define what is a good PR. Like a good PR can be something that is aesthetically aesthetically pleasing for like improving the UX, but that doesn't improve the performance of the of the code base. Or I'm removing a lot of technical depth, but that that is not shipping a new feature. Therefore, the business value of this is hard to calculate, right? So do you have any sort of sort of like scoring or metrics to quantify uh and and qualify whether the PRs are good

Pull Requests And AI Review Agents

SPEAKER_01 37:36

or bad? Because uh it's a very subjective thing. How do you classify them?

SPEAKER_00 37:41

Oh, that's a good question. Um yeah, I mean, definitely uh what we will look at is PR size is one one metric that you can track, right? It's like if you have just like a huge pull request, uh we see that people are opening it and close it again. So keeping keeping the PR size as a at a at a level. Um but with pull requests, it's it's it's now very um very difficult because we also use AI reviewers, um, and we need to just like see which part of the code or is part part of of of this uh or which part of the code to change. Does it need a human to review this? We are experimenting with with that too, right? And I see that also with a lot of companies that just like say, okay, if it's if it's part of our financial software that checks credit card, blah, blah, blah. I we should have humans to review that. But is this just like a small user interface um change? Probably AI can review that and merge that automatically. It's not a big risk in there. And I see that um now that code we we we use AI for creating more code. The bottleneck is pull requests, so we need to see just like how to use AI there. But what is a good pull request, right? That's different from organization to organization. It's it's probably the um the size, the quality of of the code. But AI can help us to have a pre-review of the pull request. This is how we started, like get it, get a pre-review, get a comment in, and then add the senior developers to look at that at that pull request. But most things are already ironed out. Um, most obvious things are already iron out. But yeah, um I think there's a lot of movement right now, also with having we are also applying different agents to pull requests. So security reviewer, um, architecture reviewer. So we just like depending on um the code that has changed, there are different um agents looking at it or multiple agents looking at the pull request. Then obviously you need to balance that a little bit out with just like token usage. You just like don't want to burn all the tokens because you just like throw all the on the small pull request all the all the reviewers. So there's a lot of a lot of things going on right now in the in the whole software development lifecycle that we probably need to rethink. Um But we are, I think a lot of companies are right now in the figuring out phase how that looks like. And there's no playbook written right now. Um and you shouldn't wait until a playbook is written. You just like need to try to solve it your way because once it's written, maybe it changes again in this in this whole era of AI. It's just like running so fast. Um But I I think it's it's a good opportunity now to look at your software development lifecycle and see where can AI help and how do we want it how do we how do we want to change it so it helps us to have better outcomes?

SPEAKER_01 40:54

We got we got 10 minutes left, and uh I wanted to focus on something that you say. Um and I wanted to

Knowledge Graphs And Competitive Context

SPEAKER_01 41:00

challenge you on that. You say that uh context, not models, are the ultimate AI accelerators, right? And so um coming from a company that has betted so much on documentation, like Confluence is a big product at Atlassian, right? And I'm a huge proponent of documentation, yet in the development world, that is one of the most hated jobs there are to do. It's like no one likes updating the documentation because on the one hand, it is really useful, but creating it and updating it and maintaining it has always been very cumbersome and little rewarding. Therefore, documentation has always known something as something that gr it grows stale, therefore, let's not do it. However, with the advent of AI or these like new AI models, now the creation and updating of documentation is virtually free, right? And that's one side, then the same happens for context, right? However, I think that a lot of people are still not doing it. Just because it's so cheap to do it, they forget, right? It's like, no, with the AI, I don't need documentation anymore. It just figures it out. It's like, yeah, but you have to reinforce the context, you have to update it, you have to prune it and all of that. So how are you maintaining the context layers in Atlassian?

SPEAKER_00 42:21

Um to be honest, to be frank with you, I don't know what what we do with uh with with documentation um uh at at Atlassian. But I see a lot of companies. Not only documentation, yeah context as well, like both of them. Yeah, yeah, yeah. So um I mean context is so important. Um I mean give just like think about it. If you have a coding agent that should change something, um it should it needs to read all the code um to just like make sense out of that. Um and see where where where where the change is made. This is a lot of token token usage, right? Instead of just like generating documentation from the code and put that in a in a in a in a in a graph, uh, we call that the teamwork graph. Um and there's all the information is inside this team teamwork graph. And it comes with your cloud subscription, you get a teamwork graph. But then it is inside a graph and it burns, doesn't burn so much tokens if you request that that data. And I feel like code documentation is now not just written for humans, but also for especially for agents, um, which which is super cool. But think about it, um and and and that that's a great concept. If you just like have not just a graph of documentation or put the documentation in your in your teamwork graph, but put also your document uh all all your documents, right? Your your requirements, um and not only this, also your your your meeting notes, um, or even just like your meeting recordings, because today the transcript is just like it uh just like a test. To the recording anyway, it gets richer and richer, right? There's a lot of information that you can tap into and that agents can actually tap into. And that's a it's a huge advantage. So our teamwork graph has more than 150 billion connections, right? And then if you not just have Atlassian data in this in this teamwork graph, but also data, we open the teamwork graph to can add your Slack data, your conversations into the teamwork graph, your Google Docs, your SharePoint, all of all of this, right? It's getting richer and richer. There are other knowledge graph technologies out there, I just like don't want to pitch for. But I want to just like say that what we saw actually, when you use a graph, um a a knowledge graph like this, or we just like make make our our measurements on the Twitter graph, we see a f we we we actually see a 44% more accurate uh results. Because the connections are already there, right? So you have have the connections. Plus, we see uh 48% fewer token usage because you don't have to navigate through through all of this, through your Jira tickets, to the to the uh requirements documents, maybe to a meeting or whatever, right? Because it's already there and it just like doesn't burn any any any any lot of tokens, right? And we know that tokens usage get up and up and up. So we need to we need to mix and match these these technologies, and the teamwork graph is a is a great example of of this, how we just like need to balance um the AI token usage with just like other technologies that that are complementary. Um models are super intelligent, but they are nothing without the context. Um I I think like 24, if 24 and 25 were the year of models, new models coming, right? 26 is the year of context. I hear that everywhere, uh, that context is important and it's your competitive advantage. Your competitor can be using the same models as you, um, but your context is actually the differentiator. So the more context your LLMs have, right? The the better are the results and the the better you thrive and drive um drive there.

SPEAKER_01 46:27

It's uh it's good that you throw out the the concept of internal documentation like being uh less token consuming, because that's actually one of the good uses we have found in the company. Is also you can direct your LLMs uh better with you know, like for you're probably familiar with the concept of LLMs.txt. That's something that you put at the base of your project. For web, it's not so adopted widely, but for internal documentation, it's very good because you send your agents there, it reads the LLMs.txt, and then you can scope um app like a little bit more, like larger or smaller, depending on you can direct it's kind of like giving the cloud.md for that specific project. So that's a really good use for this uh for this technology in particular. So I really want to give it like a shout-out to this thing you mentioned. The second part is, and uh maybe like the last question before we wrap

Who Owns Context Inside Companies

SPEAKER_01 47:22

it up, too. I want to be sensitive of your time. Is like, is there any specific profile in the company that takes care of nurturing the context and just making sure that everybody applies, gives it gives the context of every team, every department, every project the right amount of information, just you know, updates it, um, puts the guardrails and stuff like that. Who takes care of the context? Is that specific role or everybody does it across the board?

SPEAKER_00 47:49

Um I think it's it's first it's probably the role of everybody because just like a person can't just like get all the contacts. It's think it's coming back to the culture um that we we we need we need an open culture to share everything and and open open our systems um for giving AI the right context. Now, there is probably also a role um with with within organizations that needs to look at their look at the knowledge um and say, okay, what is this old knowledge, is this new knowledge, is this relevant knowledge? So there is something like a role like a knowledge architect um in this world and how does this how AI reachs reach that knowledge? Um we're we're still this whole AI um change transformation brings us new roles, um new organizational um changes, right? Where does AI live? And we we're currently figuring that out, right? Does it live in the in the in the in the in the IT department? Or is that just the technology? Or does it live we we have also driving AI from the HR department because we think like it has something to do with ways of working? And that's what our HR department is is actually, our talent ed department is actually responsible for. So we just like have have it have it there. Um but I think organizations are different and we need to figure out where where where where to just like put this AI. It needs to live first in everyone, um, um all employees all over the or over the um organization. But then who owns what is probably something that we need to figure out. I mean, we have been driving technology for a long time, IT departments, right? We had IT departments who just like installed all the technology, was working for for a while. Um, but with AI, it's it's a it's a mix of um ways of working and technology. And it's and companies are uh struggling at the moment to just like where where should we put that? Should we have an AI central team? Should we put AI into every team? Where do we put that? Um and that's something that we probably need to figure out in the next next few years. Um yeah.

SPEAKER_01 50:08

We'll see if there is uh a new role coming up, Chief Context Officer for the companies. We'll see. We can discuss that in a year from now. Sven, uh, we're we're out of time.

Closing Challenge Share What Works

SPEAKER_01 50:18

Really last question. This one. Uh thank you for your time. How can we help you back? How can the community of Mars-based life on Mars help you or Atlassian?

SPEAKER_00 50:27

Um I would I would just like I I don't I don't know. Uh I I just like would say um just like use AI, adopt AI, try, try, try it out, um, go go on. We are on this journey, you're on this journey, share your stories, share your successes, share your failures. Um, I think this is the best way how we all can learn in this world. Is it is first so fast driving, right? Things that are true half a year ago are not true anymore. Uh I think we we all need to open up and and and share more than than ever before because uh there we we just like can learn from each other. Um and it's new, it's fast, and it's exciting. And let's just like share, share the stories. I think this is the best way you can you can help us and uh yeah, give back.

SPEAKER_01 51:24

I thought you were gonna say use Jira. Use more confluence. Come on. No, no, thank you very much. Vielen Dank, Sman. It's been a real pleasure. We learn a lot from you and thank you for your time.

SPEAKER_00 51:36

Thank you.