SPEAKER_00 1:18

Can you actually prove AI is working at your company? Sure, everything feels faster, and hopefully outputs have improved, and maybe more work is getting done. But is there ROI? And don't just give me vibes, give me receipts. What's the actual return on your investment for AI at your company? Chances are you can't answer that. And the reasons are actually very simple. It's because businesses are using the same digital transformation playbook they've always used when it comes to AI. And well, that playbook is useless in 2026. So there's a 99% chance you're not actually measuring ROI, but I'm going to simplify it for you on today's show and tell you why the ROI on AI debate is absolute nonsense. And I'm going to give you seven simple steps on how your company can fix it today. All right, let's get measuring. So uh here's what you'll learn on today's show. You're going to learn the basics of measuring Gen AI ROI and why most companies never actually do it. You're going to learn why a piece of MIT marketing, disguised as a study, misled markets in what five larger studies actually show. You're going to learn how to use a four-metric scorecard covering quality, cost, reliability, and risk, and how to run a seven-step evaluation sprint that actually solves your ROI. So, what the heck is this? Well, this is everyday AI number one, but this is our start here series. I didn't have a good answer when people first subscribe to the podcast and they're like, Jordan, there's 700 episodes. Where do I start? I'm like, I don't know. Well, now I do know you start here with the start here series. This is the essential podcast series to both learn the AI basics and to double down on your AI knowledge. So make sure to go to starthearseries.com. Uh, that's going to give you free access to our inner circle community. And then you can go in the Start Here Series space and listen to now all 11 episodes of this series. Uh, so last episode, which was a banger FYI, um, we talked about from AI chatbots to autonomous workers and how consumer AI has changed and what's next. So make sure you go uh check that out. It's episode 723, but uh volume 10 of the Start Here series. And today we're gonna be measuring ROI and tell you why your company is doing it wrong and these seven steps to fix it. Let's start here. Talked about this once or twice, but let's talk about this benchmark from OpenAI. It's called GDP Val. So this benchmark tests AI on real-world deliverables across 44 jobs in nine different GDP sectors. Essentially, this is a unbiased benchmark that pits AI models against expert humans, experts in their fields, right? They each have the same task, which is generally uh to produce something of value, right? That's why it's called GDP Val, to produce something of economic value. And then there's expert judges, expert human judges, and they don't know which output came from the AI. They don't know which output came from the expert human. And right now, top AI models either tie or win 70% of the time in these blind evaluations against professionals that have an average of 14 years of experience. I mean, my gosh, we should just end the episode now, right? There's your ROI on AI, right? Done in under four minutes. There's a lot of issues here. Number one, even what I just said, right? Do you know how to use today's models to, you know, one shot personalize, synthesize information, create, you know, spreadsheets, uh, PowerPoint decks, and use your own company's information without you having to do anything and verify outputs? Well, probably not, because companies aren't educating their people, and it's hard to keep up with uh today's capabilities of AI models. But what this did show us is that the whole debate on ROI is absolute rubbish. Yes, there we go. Uh, for uh listeners across the pond, look at me. I'm uh multilingual here. Uh it's garbage. This whole discussion on does artificial intelligence give you a return on your investment, it's it's honestly a very dumb question, if I'm being honest, right? Uh go go read the GDP BAO study. Um, I keep saying, oh, I'm gonna do a show soon. I I will actually do a show soon uh because it's worth it. So not only did it show that, well, today's AI models are exponentially better than expert humans when judged by other expert humans, right? But it said that in many tasks they do it a hundred times faster. So again, I am not a mathematician, but if the AI model is the same or better 70% of the time and it's a hundred times faster, that's the math. So why are we still even debating this concept of is there return on investment? Well, that's because of an infamous piece of marketing from MIT. So uh, and yes, I'm calling it a piece of marketing or study in very heavy quotes, right? Like the uh uh what is that, the the Austin Powers uh, you know, a billion dollar. Yeah, huge quotes on the word study. Uh, right. So MIT claimed uh in their wildly viral study that 95% of enterprise AI pilots delivered zero ROI. And this was in August of 2025, right? And I mean the markets moved, right? Nvidia stock tanked three and a half points, Palantir went down uh nearly nine percent. There's actually billions of dollars of losses across the uh not just the tech sector, just the business sector, because right, AI has essentially propped up an otherwise struggling uh US and at times global economy. Um, and then you know, this study came out and said, Oh, AI's, you know, there's no ROI. Well, uh most people, right? As a former journalist, I understand this, right? There is one study that came out, and then essentially every other uh journalist hit the copy and paste, uh, you know, more or less. Hey, that headline sells, uh, you know, 95% of AI pilots don't uh produce any ROI, right? That's great. Let's go ahead and run that. But I don't think any journalists actually read the story, or very few actually did, uh, right. That's because this quote unquote study was based on 52 qualitative interviews. There's no quantitative piece to that 95%. And it was uh directionally, right? Uh, this was a directional study. So it's more or less this was a vibe. This was a vibe study. I'm gonna talk to 52 people, you know, nothing quantitative. I'm just gonna give my vibe on this. But the stat came from 52 qualitative interviews that measured PL impact in under six months, which there's no such thing, right? When you talk about digital transformation, go back and look at the internet, right? You could have shown the PL impact in under six months on using the internet, right? Because if you could have, you could apply that exact same methodology to AI. Anyways, well, why this is a piece of marketing and not actually just uh a bad study. I would call it a bad study, but it wasn't. It was marketing because in the end they were selling their own product. They said, Hey, uh, right, all these AI pilots fail because you're not using our agentic AI product. And they were, you know, selling access to it. Um, yeah, I'll stop there. If you really want the down low on that one, make sure to go listen to episode 597, where I, like a human, uh read it, like a human with a brain, uh, read the study multiple times, and it's uh, you know, laughable. Anyways, if you look at the real data, right? Companies that talked to thousands and got quantitative data, obviously the ROI for AI is there. Uh, right. The IDC, the International Data Corporation, found a $3.70 return for every dollar invested in AI. Wharton, a three-year study, found that 74% of enterprises report positive ROI on gen AI. Google Cloud's 2025 annual AI study found 74% of executives reported achieving ROI just within the first year of generative AI deployment. Also, Deloitte's 2025 AI survey found 84% of those investing in AI said that they were gaining ROI. Um, this I think the this this debate is still going on because humans are innately lazy. Right? That's the reason why we don't want to actually sit and measure, right? Again, my my background, I have a little bit of marketing and advertising background, so I understand the importance of measuring something before and after to see if there's an impact, right? Running millions of dollars of ads for clients over the years. There's this thing called return on ad spend, right? You see how much uh you know how much money was spent on ads. You have to be able to attribute and track the revenue that it brought in, and then you do some simple math, right? Return on ad spent. Well, there is also a simple equation that I'm gonna give you guys here in a little bit, the same thing, return on AI. So here's the dirty little secret because you're probably thinking, okay, Jordan, well, if every company's using AI, well, why isn't everyone just seeing exponential growth? Well, uh, I think you would see that a little bit more if only 5% of uh businesses were using AI, right? But most studies say more than 95 or 97% of enterprises. So uh the playing field has just gone up, right? Uh the minimum entry is no longer like, oh, you're using AI. That's great. That's such a differentiator. No, uh, no, it's you're using AI. Okay. I've been saying since 2023. It's the same thing as, oh, our company's using the internet. So, right. Uh, but here's what's happening. Workers are pocketing it. That's where the ROI is going, uh, right, because true productivity and true ROI, well, it requires results-driven metrics, not time-based management. Most companies haven't, you know, gotten this figured out yet. Uh, and I think a big portion of this is that remote and hybrid workers are using AI, whether it's approved AI or uh, you know, whether it's uh, you know, AI sprawl, uh, you know, dark AI, whatever you want to call it, and they're just pocketing it. It's this invisible productivity, right? I've literally talked to countless people. I won't say hundreds because I don't know if it's actual 200 or more, but I've easily talked to more than a hundred people over the past few years in very successful, right? Even one of my um, you know, one of my good friends uh told me this like uh maybe one or two years ago. Pretty, pretty high up uh at a company that was uh public company when he worked there, said he automated about 95% of his job with AI, right? No one knew because he's at home. I'm like, what are you doing with all this time? He's like, ah, you know, chelling golfing a lot, uh, right? That's the reality. You know, uh so many companies, so many employees, I think, are just pocketing this time savings, right? As as we've gone to this uh, you know, remote hybrid workforce. And I think that's one of the reasons why last year uh so many companies had this, you know, big return to work push. Uh and right now, according to workday, 89% of organizations haven't even updated their roles to reflect AI, right? So they're still using uh you know 2026 tools uh inside of job structures made 10 years ago. Um, right. So why does that matter? Well, again, hiring and roles in today's enterprise are still pre-AI. So what that generally means, well, expectations or output is measured in a pre-AI way, which isn't necessarily the right thing, right? And in the same way that I've encouraged um you all for years to not upskill or reskill, uh, that's that's a waste of time when it comes to AI, right? You need to unlearn and rebuild, been saying that for a long time. Um departments and companies need to do the exact same thing, right? You can't just, oh, let's add a little bit of AI to this job role, or uh, you know, hey, let's let's try to think a little more uh AI natively here at this company. No, you you gotta tear it apart in the same way that job roles haven't changed, right? That means outputs aren't going to change either. But if you're using AI, if you've trained your people on AI, if they have the right AI, they're just banking that productivity. So here's how finding ROI is done. It's simple. Uh ROI can mean a lot of different things. It can a lot of people just think, oh, that means you're bringing in more revenue. No, I think mainly it's time saved, right? But I think ROI also means just cost reduced, revenue increased, or risk avoided. It's not just prompts sent, right? I think that's what a lot of people they're looking at. Uh the number of hours, uh, you know, your utilization rate if you're using, you know, an enterprise tool like ChatGPT, right? Everyone's like, oh, you know, we need to increase our utilization from you know eight percent to 20%, right? And then companies, you know, hire us to, you know, train hundreds or thousands of employees. And it's like, okay, utilization is not uh the metric that pushes it. It's are you saving time? Are you um increasing revenue and avoiding risk? Uh the formula, well, simple math. It's the time saved or uh increased revenue. Uh, but let's look at time saved. So it's just time saved multiplied by fully loaded hourly rates minus any any monthly AI subscription costs. Simple formula, right? Um, and right now boards and CFOs just care about cost per task, uh cost per task throughput and error rates, right? Not usage dashboards, not utilization. But we're seeing the same reasons for failure, right? Still years later, right? Um, through my own conversations and conversations with others, I can say when it comes to measuring or not measuring ROI, I still think that there's five patterns or five main reasons why either individuals or departments aren't able to do this. So, number one, not having a pre-AI baseline. Number two, you know, doing these slow year-long pilots. That's a recipe for disaster. Number three, celebrating one lucky run and making that your AI strategy. Number four, when vanity metrics become the focus. And number five, shiny object syndrome, pulling focus uh away before real company-wide implementation can actually start, right? Oh, you get a little, you get a little ground, you show ROI, and then you're like, oh, what about this model? What about this? What about this? What about this, right? Before you actually, you know, uh implement it company wide. And these aren't technology problems, right? This isn't standard digital transformation. These are we as working human beings, as knowledge workers, uh, we don't have a playbook to follow, right? So all we've been doing is following the same playbook that we've always done, right? Those slow pilots. You know, you get one thing right and you're like, oh, this is what we do now. It's the wrong way to do this. The other big reason, well, training. Right. Uh the training actually leads to those five other failures. So it's more of a of an underlying uh foundational issue, right? But right now, 49% of enterprise leaders cite recurring, uh, sorry, cite recruiting AI talent as their single biggest challenge. All right. AI, you need documented workflow steps and company data. And that process documentation is part of the ROI investment, right? Education, training, ongoing learning, it is required. Uh, right. So many, I think when um when AI, uh when an AI strategy or actual AI implementation looks to looks like you know, pushing top down, you know, either C-suite or board pushing, you know, a certain tool top down and not training or educating, that's when it's so easy to fail, uh, right. I use AI all day, every day. And I'll I'll I'll tell you this. If I take two weeks off, I'm gonna fail. That's the that's the pace, right? People are always like, okay, well, Jordan, you clearly don't know what you're saying here. If AI is a hundred times faster, well, shouldn't we just cut all these jobs? No, I think jobs are gonna look very different. I've been on record as saying this since the very first episode. AI is going to uh, I think, quote unquote, take away more jobs than it will create, but it will create many roles that we don't even understand. I've always said most enterprise organizations, they need a lot of people like me, that all they do all day is they keep up with AI and they apply it to that business, right? Very few businesses have that because they're like, okay, that's a crazy role. We're not gonna let you know, Bill over in IT or you know, Deborah over in marketing just spend all day uh, you know, tinkering with AI and seeing how to apply it. No, right? No, but yes, that's exactly what you should be doing. Uh, and here's all right, uh, this is usually something I think we only uh give away to companies that uh pay us a couple dollars. But let me just go ahead and give you some of the secrets here. All right. Here's an acronym, BASE. You need a base. This is the baseline rule. So this is BASE stands for baseline assessment of standard execution pre-AI. All right, so before AI touches any workflow, you got to get that base, right? That's your time multiple employees completing that exact task without it. You need to time it. You need to record the average time, the error rate, the rework cycles, and cost per completed task as the baseline. You can't go back and collect this retroactively, right? Because usually what happens, well, if you've already, you know, sprinkled AI in the process and you've been doing it for uh a year, right? Let's say it's a 10-step process and hey, number two and uh step two and step six, we you know uh we molded those a little bit around AI, right? You it's it's it's too late, right? You need to do it before you implement AI. You need to measure how long it takes humans to go through and do these certain projects or do these certain tasks, right? Think of it like an internal GDP valve, right? You are gonna go through and have a human with no AI go through and do this task, document every single step of the process painstakingly, because you need to see okay, uh what other humans need to be involved in this? Are there other meetings? What about the person checking the work? What about the communication going back and forth? You need to measure it all and document it all every single step. And then what's the completion rate? What's the error rate? What does quality look like? You need to establish those baselines and then redo the entire process, right? Blow it up, measure it first. That's your base, right? Baseline assessment of standard execution. That's your base, blow it up, do the same thing with AI, right? Not your first iteration, iterate on it, and then you measure. And that number, right there, you need that because that is the base, quite literally, for how you're going to ultimately measure your ROI. So now let's give you that seven-step guide. All right, I'm delivering. Here we go. Well, let me just uh bring my notes up here because you know, in typical live stream, unedited, unscripted fashion, uh, I didn't put my slide up here with my uh seven steps. But don't worry, I got my notes. All right, there we go. Notes on the screen. So here's the seven step blueprint, right? Step one, we already talked about it. That's the uh well, actually comes a little bit before the base. So step one, you have to define, you have to define. What the heck is it you're doing, uh, right? And you have to be very rigid. You can't be flexible because you need to get um an accurate before and after. So that's defining the rubric uh rubric, uh, defining the success criteria and KPIs before you even begin testing. And then step two, like I talked about, that's the base. You need to measure the human baseline. And then you're gonna uh, you know, time multiple employees doing that and then record the averages. Uh step three, get messy, right? Uh you have to build 20 to 40 real messy work examples, including drift cases, right? These aren't easy things for humans or for AI. All right. And then step four, you need to configure the exact production workspace. It's the same plan, the same model, and the same uh permissions. So you're not, you know, flip-flopping uh, you know, between different people, different accounts. No, right? It has to be controlled like any experiments. Uh, and it has to be repeatable and scalable, the exact same criteria when you're taking this to a production run. Uh, step five, you're gonna run every three times. Okay. So, sorry, run each test three times with memory off and also require proof artifacts. So, here's what I mean by that. We're doing this on the front end. Okay. Um, if you listen, if you've listened to this podcast at all, you know I'm a big believer in bringing as many of your processes over uh to front-end large language models. We actually did a dedicated episode uh on that in the start here series, talking about an AI operating system. So, yeah, this is all going to be doing things on as an example, chat gpt.com, claw.ai, gemini.google.com, right? But doing it with memory turned off and doing it usually in a temporary chat if your uh you know AI operating system of choice uh lends itself to that. All right. So essentially there, you have steps three through six is that's your internal GDP valve, right? Uh you have to get the the correct use uh use cases. Um, and then from there uh you grade blind, right? So same thing. You have the the series of people do it uh three times, those 20 to 40 use cases using the same AI model. Uh, and then you have humans do those same exact things as well, multiple humans in the same way. Uh I would suggest, you know, three different times um running the same case and then three different sets of humans. And then you grade blind, you standardize the output format, you have to agree on, you know, what's a pass, what's a fail, what's the grading scale, et cetera. But then at that point, well, you have the input and the output, right? At that point, after step six, when you can have your grading criteria, uh, you know, you run through and you do the test three times, um, you know, those 20 to 40 use cases, three times uh in a large language model, memory off, uh, temporary chat. You have your your grading rubric, you have your humans do it, uh, right? The same 20 to 40 tasks, three different humans. You have it right there. You have the time, right? You multiply uh the time that it takes the humans to do it on the AI side, all right. Minus out, uh, you know, so take, let's say it's a hundred human hours, take their hourly rate, times it, minus the uh cost of whatever AI tools that you're using. Uh, all right, there's your uh augmented cost, and then you compare it to your human-only cost, right? And obviously the human-only cost is gonna be much higher. Uh, right. The same thing if you are using any, you know, paid non-AI tools uh for the human-only cost, right? Maybe there's uh some, you know, I don't know if you're using the Bloomberg terminal or what right, whatever. Uh you know, a certain SaaS, and maybe you don't need to use that SaaS application if you're doing it AI native. The same thing, you need to look at the total human and software costs. And there you go. That's your return on investment. Uh, and then step seven, you need to retest this monthly after every model update. All right, and then track a three-month rolling average. Costs are going to go up and down, right? You think that, oh, well, they're just always gonna go down. Well, no, sometimes uh, you know, the Frontier AI labs will actually roll out an update that's under the radar. It's not like going from a GPT-5.2 to a 5.3, right? A lot of times there might be five, 10 different versions of a GPT-5.2 until there is a GPT-5.3. And, you know, whether it's OpenAI, Anthropic, Google, Microsoft, etc., sometimes, you know, one of those under-the-hood updates actually might make things worse. All right. Uh, so that's why you need to retest monthly or after any major model update, and then keep a three-month rolling average. There you go. All right, I'm gonna go through it quickly now with no commentary. Step one, define the rubric and success criteria. Step two, measure the human baseline. Step three, build the 20 to 40 uh real messy work cases. Step four, configure the exact production workspace. Step five, run uh with three times uh AI models in three sets of humans. Uh step six is grade blindly and standardize the output format and criteria. Uh, calculate your ROI. And then step tab step seven is retest monthly. So there you go. That's how you measure return on investment in AI. But let me just tell you this, and I'm gonna be uh very direct when I say this. You don't need to do this. You absolutely should, right? You absolutely should. You don't need to. This is this is gravity, right? AI is gravity at this point. Um, it is the all-encompassing force, and there's no denying it, right? Real studies with quantitative data that asked thousands of business leaders all overwhelmingly show that the ROI in AI is real. Three dollars and seventy cents for every dollar invested. The GDP valve, right AI models without training, right? Without iteration, in a single, in a simple shot, blind test, the AI model is the same or better than the expert seventy percent of the time and a hundred times faster. So I'm gonna leave you with this. Yes, you need to configure and figure your ROI on AI. But the new ROI question is not did AI work? It's how much did we lose by not educating and measuring sooner? That's the reality. And that is the challenge to you today, dear listener. Stop looking at ROI on AI like it's something tricky, like it's something we can't all achieve. It's as simple as being meticulous in your measurement, and that's it. And when you are meticulous in your measurement, you will undoubtedly see in insanely high return on investment of your AI. So no longer a question mark, no longer does AI work. The question is, and the way you need to change, is how much are we losing by not measuring sooner and educating our teams better? All right, that's a wrap, y'all. That is volume 11 of the Start Here series. I hope this was helpful. And if it was helpful, well, number one, please uh like and follow the podcast. I'd appreciate that. But then when you're done, go to starthirseries.com. That's going to give you free access to our inner circle community. And then you can go listen to the entire Start Here series all right there and connect with others who are trying to grow their company and their careers with generative AI. And hey, while I have you, make sure if you haven't already, go listen to the 2026 AI prediction and roadmap series. That's episodes 712 and 713. So thank you for tuning in. Hope to see you back tomorrow and every day for more everyday AI. Thanks, y'all.