What a Chief AI Officer Actually Does, and Why Every Enterprise Needs One

The Chief AI Officer is one of the fastest-emerging roles in enterprise technology, and most organizations still don't know what it actually requires. In this episode, Shawn sits down with Ed Keisling, Chief AI Officer at Progress Software, to pull back the curtain on what the function looks like in practice and why some version of it is becoming critical even for organizations that can't add a C-suite seat. They dig into Ed's Vanguard and Champions model for scaling AI culture from the inside, why a 20-page AI policy kills adoption faster than any technical limitation, what shadow AI really signals about your tooling gaps, and how to think about model routing and inference budget now that "free" is officially over.

What You'll Learn:

• What a Chief AI Officer actually does day-to-day and the signals that tell you your organization needs one
• How Progress Software's Vanguard and Champions model builds AI culture from the inside out
• Why rewriting your AI policy from "don't do this" to "here's what you can build" changes adoption overnight
• What shadow AI is really telling you about gaps in your tooling strategy
• Why the pilot-to-production gap has more to do with coordination than technology
• How to think about AI technical debt before it comes due in production
• Why inference costs going from virtually free to unpredictable is the budget problem nobody planned for
• How to use model routing and context management to keep AI spend under control as you scale

Transcript

SPEAKER_00 0:12

Hey everyone, welcome to the Amplitude of Tech Podcast. I'm Sean Corner Chief Marketing Officer of Amplex. Today we have Ed Keisling. He's the Chief AI Officer at Progress Software. This was a great conversation, maybe one of my favorites on the podcast. It was wide-ranging. We talked about a lot of areas of AI and practical tips that will help you uh get AI adopted and scaled in your business. Listen to this conversation. It's a really good one. Hope you enjoyed it as much as I do.

unknown 0:45

All right.

SPEAKER_00 0:46

Ed Keisling, welcome to the podcast. Thanks. Great to be here, Sean. Great to have you. Why don't you just take a few seconds and talk about who you are, where you come from and what your company does.

SPEAKER_01 0:55

Sure thanks. So I work at uh Progress Software. I'm currently the chief AI officer there. I've been in that role for about uh a year and a half now. I have about 30 years in the industry. Uh, has spent about 17 or 18 years at Pega Systems, which is uh located in Burlington Mass. Uh, spent uh six to seven years at a small healthcare robotics company uh where I ended up being uh the acting CIO for a bit and uh have spent the last nine years or so at Progress leading different product and engineering teams. So what Progress does is we enable organizations to deploy, manage, monitor, configure sort of rich uh digital experiences and applications. I think one of the more interesting things that we we have been doing more of late is uh really trying to help organizations unlock all of the unstructured data that they have within their organization through solutions such as like agentic rag.

SPEAKER_00 1:44

Well, that's a perfect place to start because you've got one of the sexiest new acronyms in the industry, which is the CAIO designation. How did Progress decide that it was time for a CAIO?

SPEAKER_01 1:58

Yeah, so so it's it's funny, right? I I I sometimes kid with people, I say, well, I was I was prior the uh the chief crypto officer, but that didn't work out so well. But uh I wanted to jump on the next uh the next trend. But I think it at Progress we had a very specific problem to solve. So I think uh the thing that probably brings most clarity to folks is that we we don't actually have a chief technology officer. And we we actually have a fairly complex business uh in that we have multiple products that are sort of organized into different BUs. And we needed, I think, a lot more sort of intentional orchestration across how each how AI was being used across each of those different BUs. And so I'll I'll give an example, which is that uh, you know, I think one of the great things of progress is that the teen really leaned in heavily into AI very early, uh, but that created some interesting challenges, which is that you know, the minute that MCP comes out or the minute this new technology comes out, we sometimes have seven or eight or nine different sort of flavors of that same technology. And that sort of dilutes sort of the impact in terms of being able to sort of do that. And so my role is really to try to figure out how we can better coordinate uh sort of those activities across all the different business units, uh, really sort of drive reuse and sort of uh better enablement, but also you know, product capabilities uh within the different products and uh really trying to identify some of the common components as we build AI systems uh that can be used across.

SPEAKER_00 3:27

So it's really a multifunction but also cross-functional role, and you're uh spending a lot of time within the business identifying use cases, either internal or you know, customer facing. What exactly are like the day-to-day responsibilities of a CAIO? I keep wanting to say CIEIO, but Yeah, exactly.

SPEAKER_01 3:46

My uh my friends say like uh chow chow, right? But uh I I think uh, you know, so I I have a small team uh that is sort of oriented around uh sort of enablement for the folks within engineering and also some of these product uh cross-product capabilities, right? And so uh for reference, we have about 1,200 people that are leveraging um co-pilot studio um or GitHub co-pilot studio at Progress. And so my role is really to try to figure out how we can best enable and sort of drive adoption within that cohort. And so uh, you know, what we do within my team is that we have folks on my team sort of focused on enablement, sort of finding finding and identifying and enabling the AI champions within the organization. We've launched an initiative that we call sort of the Vanguard program. So these are sort of like the front of the pack in terms of our most sort of forward-looking sort of AI evangelists in terms of getting them together as a cohort so that they can grow and learn uh from each other. We do sort of bi-weekly training with Microsoft in terms of uh any of the new capabilities that they're coming out with. So, you know, we'll have several hundred people a week be jumping on one of these sessions that we have uh to learn more about you know some of the product capabilities that uh that Microsoft is doing. Uh and then, you know, I mentioned we're we're sort of focused on reuse. So that the challenge is that if somebody builds something in a silo, how do we get visibility into that across the organization? And how do we be more intentional about making sure that if somebody does actually build something, that they've gotten uh sort of input from some of the other stakeholders from across the organization so that it could be reused uh if they were to build it, right? So I think that that's one portion in terms of how we do that. We have a uh an AI capability board that we run things through. Uh and then it's also about sort of the intake, right? Which is it's it's a discovery challenge, which is that, you know, like most organizations, I think that there are all these different assets uh that are being created. Uh, how do you actually make it so that if I'm saying, hey, I want to build an MCP server that's gonna connect these two systems together, I want to discover artifacts in terms of how to manage, monitor, govern that, right? And I don't want everybody sort of rebuilding some of those components from the ground up. Uh and so my team sort of helps uh sort of enable that as well, too.

SPEAKER_00 6:10

Got it. I love the idea of the Vanguard thing. So it just if I understand this, you you've got these kind of uh pioneers, I'll say, or early adopters or power users of the AI internal tools that you've rolled out. And so you use them as a success story and uh social proof amongst their peers, create an environment of learning and kind of cross-pollination. Is that right?

SPEAKER_01 6:34

Yeah, and so the the cohorts have maybe sort of slightly different sort of focuses. So what when we are thinking about sort of our vanguards, uh we're really trying to say, like, how fast could we actually go with AI? Like what where are the limits in terms of how we could take AI and apply it to our organization? And, you know, maybe if we were to relax some of the constraints around inference or access to tools or governance, right, uh can we create environments for them to sort of do really sort of wild experimentation in sort of like a safe, controlled way uh where they have access to IT or myself or leadership in order to sort of support and drive some of that change, right? Uh these are, I think, really much more, could be potentially much more disruptive uh initiatives that we are sort of partaking it progress in in how do we fund and sort of create an environment there. And and I think what this also allows them to do is that this is a very small cohort. They can when they tech typically have questions, nobody else can answer these questions, right? Because they're all they're off way on the horizon, but like together, uh they can sort of find answers and sort of uh advance the ideas in terms of what they're trying to accomplish. And so, you know, for instance, the the the Vanguard cohort in our in our organization, as you might expect, has the largest inference budget in terms of AI credits that are being allocated out, uh, which is something that we just put into place. And so that that's sort of the intention of there. Just breaking, you know, sort of extending this out. So, what do the other people do? I mentioned that we have a champions cohort. Uh, the intention of the champions uh is really to try to figure out how we can scale some of the AI best practices across the organization. And again, we have sort of a hub and spoke at progress. So if we identify a change or a capability or some new tool that we want people to use, the challenge that we have is getting that to every one of the 1,275 people that might be on GitHub Copilot, right? And so the champions sort of understand within each of their different regions, because we're a global organization and we have offices in Bulgaria, you know, the Czech Republic, uh, India, Ireland, et cetera, right? So how do we take sort of the general guidance and make that sort of bespoke for the different regions uh and make that meaningful for them? And then to also make sure that we have follow through in terms of whatever initiative that we're rolling out, right? So the two two different sort of sides in terms of how do we approach that.

SPEAKER_00 9:00

So Vanguard is the DARPA and champions are the info.

SPEAKER_01 9:05

We used to work with DARPA at uh what if I previous companies, but yes.

SPEAKER_00 9:08

Oh, is that right? Oh, that's a that's a different podcast. We're gonna like you guys have really focused a lot on building a culture around AI. And I know that's one of the areas where uh adoption can go flat, whether we're talking AI or we're talking software as a service or you know, whatever initiative you're rolling out, right? It's all about getting buy-in and adoption. So um how how did you approach building this kind of culture?

SPEAKER_01 9:29

Yes, so we uh we started this journey probably a little over a couple of years ago, and uh we we started with a a smaller cohort, right? I think we were sort of testing the waters to try to figure out in in I think everybody is still in the same boat, like what is real, what is actual organization. But we we started with you know some of the thought leaders that we we sort of trusted their opinion and gave them access to the tools and and started to do some experimentation, uh, do some hackathons, do some sort of controlled experiments. And I think once we started to realize that there was some real value there, we started to scale it out across the organization and you know, tried to make it fun in terms of hackathons. And uh we didn't set up any of these crazy sort of inference leaderboards that you're you're hearing about today, but uh really tried to make it so that like this was a tool, uh, another tool that they could have in their tool belt uh that would hopefully make them more effective uh with their job. Uh I think the reality is that like there was like a lot of doubt, right? Because there there's obviously this headwind in terms of jobs and what this is gonna mean for your job, and and is AI gonna replace engineers? Um I would say that we got over that probably a little over a year ago. Uh and the reason that we got over that and really started to sort of see exponentials in terms of adoption and and utilization and I think outcomes and and we were literally on exponentials because I look at these these metrics every day and they they terrify me, is that I think that the the engineers got to a point where they realized that like I am the person, like I am so critical for this AI to be successful that like there is no way you could do this without me. Right. And I think they sort of once the point that they realized that they were indispensable in the process, uh albeit that maybe the way that they would go about their work and their role in terms of how they were doing their working was shifting. But like that all of those really critical decisions and the architecture and really sort of like what makes software enterprise grade, which is what we really care about at progress. Once they sort of got over that hump, then like it was off to the races, right? And so we just start to see more and more people, I think, lean in. And again, the as I'm looking at some of these charts, right? The the problem is now that we are using AI very, very extensively, I think within engineering, and then uh, you know, now it's like how do we how do we more responsibly use AI?

SPEAKER_00 11:52

I think a lot of enterprises are kind of in a position, you know, let's say uh mid-market enterprises are in this position where AI adoption is happening, but it's happening in different places within the organization. There's kind of an atomization of AI initiatives, shadow AI, you could call it, maybe not not always, right? But but often that's that's happening. And uh and different departments maybe kind of operating in silos. And so there's probably a need for this role in every organization, just for the coordination between stakeholders and and discovering use cases, but not every organization is in a place right now where they can carve out uh another C-level position. So, what advice would you give those technology leaders to kind of uh put that function in place without the title?

SPEAKER_01 12:40

Yeah, it is interesting. I just came back from a uh a conference uh and uh you know a lot of my peers there were program managers, right? Uh program managers that were maybe leading up agile transformations at their organization that were now leading up sort of AI transformations. And I think that that is not a bad spot for this to land because I think when you think about the types of problems that you're trying to solve with AI, it's really not about AI, and there's too much damn hype around AI, right? Saying that even with my title. But the I think the core of it is just trying to sort of understand what are the types of problems that we need to sort of solve, what are the value streams that we need to look at as an organization? How do we drive top line? How do we reduce our expenses? And like the program managers, or at least the the many of the program managers and the and the transformation uh folks that are heading up the transformations, like that's what they've been looking at all along, right? It's not like trying to drive efficiencies within organizations is like a new idea, right? So as long as I've been in tech, you're always trying to figure out how you can sort of make something a little bit faster, a little bit cheaper, right, with higher quality. Uh it's just that we now have like a different tool in our toolkit uh that dramatically accelerates our ability to sort of do that across the organization, right? And so I think the program managers I think can drive a lot of this. Um Zapier actually has sort of an interesting sort of phrasing for this, uh, which is one of the conferences that I was at recently. And uh they sort of frame this through the lens of sort of spotters and builders. And so I think the the program managers they don't have to build the solution, right? But I think they can help spot the problem and then you can resource how you actually implement that within IT or engineering or marshalling some resources from somewhere else.

SPEAKER_00 14:26

So if you're gonna bring a program manager into that kind of role, do you think that there needs to be some sort of uh standardized education across those those roles so that they can effectively spot something that AI can solve?

SPEAKER_01 14:40

Yeah, I I would say it's it's like a a split mind uh that they need to bring. So it is a program manager, but it's sort of a program manager plus, right? I think uh what what I've always sort of shared is that like you you need to sort of approach this transformation two ways, right? And so one way is sort of what we were just talking about from sort of program manager perspective, which is that um we have all these value streams that exist within the organization, right? So in concept to cash or like different flows that basically drive revenue for our organization. Um, I think the first place that you need to start in order to create room to sort of really transform your organization, right, is just by taking sort of some of the low-hanging fruit out of that process, right? Um, and making that a little bit more efficient. I don't think that that's where you want to end, but I think the reality is that so many people are so damn busy, right, that like when you ask them to take on something else, they're like, I already have the work of 700 people, right? Like now you're asking me to do AI, you're asking me to reinvent this process, like what the hell, right? Uh so I think if you can like eliminate some of that load, right, by finding 15, 30 minutes a day within their workflows uh so that they can breathe a little, right? Uh that I think reduces some of the barrier that many of them have in terms of like, okay, like now I have a like a at least a little bit of time to sort of get my head around that. Or you can say, hey, use that 15 or 30 minutes a day and now go play with these tools that we've sort of purchased you, right? Uh but I think in parallel, what what needs to happen is that like really the opportunity is to sort of reinvent how you do your work, right? And sort of take a more declarative approach to this is the outcome that we're trying to drive. Uh, given this outcome that that I'm trying to drive, what are the bits of data information process that need to sort of occur in order to drive that? And could I do that in a fundamentally different way with agents, or could I eliminate steps, right? A lot of times the the the best AI is no AI, which is that like you just decide I don't need to do that anymore, right? Or that that step was a result of something that we introduced 15 years ago. Something really, really bad happened. My boss said, like, if that ever happens again, you're gonna get fired. And so, you know, for the 15 years since then, everybody has been doing that same dance step, and we we have no idea why. Right. And so I think uh, but but I think that that's where sort of like a lot of the gains can come in terms of like really trying to have a parallel stream in terms of like what does that look like if you were to truly repeat.

SPEAKER_00 17:11

Yeah, it's interesting. A lot of the conversations that I'm having in real life and on this podcast now, people are are starting to bring up you don't need AI for everything. And that's counter to what those conversations were like last year. Last year was like you need AI for everything, right? So it there seems to be a a shift in in the way the world.

SPEAKER_01 17:29

I'm in camp uh you you definitely do not need AI for everything, and it is not the best tool for everything, right? And it's maybe a worse tool for everything, right? In some cases, right?

SPEAKER_00 17:38

But uh yeah, yeah. Uh and I think we could talk about the cost impact of that later. But um I I wanted to ask though, is there like an obvious tipping point where a business can see these signs or signals and say, now's the time we need this chief AI officer?

SPEAKER_01 17:52

What when you need a chief AI officer? So I I think uh definitely when you start to see overlap in terms of uh multiple people building the same thing, right? Uh or sort of uncoordinated hackathons across the organization. Um, you know, I think one of the challenges with AI is that like if if you don't provide consistent enablement, there's a there's a challenge of people sort of like chasing shiny objects and and you know potentially being counterproductive in terms of what you're trying to achieve as an organization. Uh and so I think that that would be sort of a good time to sort of introduce AI as well too, right? And so so what do I mean by that, right? Which is that I think um AI is such a natural amplifier, I think, of the best people that we all have in our organizations, right? And I think it's very easy for all of us to get infatuated with how well and how amazing these people are sort of going out and building capabilities within the organization, right? But but what we have, you know, Progress has 3,000 plus employees, right? So for larger organizations, the challenge that we are facing is sort of a diffusion problem, uh, which is that we need everybody not to be as good as the best people, but like to be knowledgeable in terms of what the tools are capable of and be able to sort of navigate that and make smart decisions with AI. And this is sort of a tangent, right? But the the way that I sort of think about this is that like I think we've all seen this, which is that you you give sort of somebody AI and they have an idea, but they don't have like the full idea in terms of what's going on, they can very rapidly you know cycle through a bunch of different prompts and arrive at a completely wrong conclusion, that could be very disruptive for your business or just outright incorrect, right? And and the way that I sort of think about that is like this 80% rule, right? So if if if I ask AI something and I ask it to create a very detailed plan and it gets 80% of it correct, uh, and for whatever reason I, as an end user, fail to detect that there are some like meaningful errors in terms of the output, uh, and this is very easy to do when AI is hemorrhaging, you know, sort of like text back at you. You can think about like multiple turns. If I just do two or three turns of 80, I'm at basically a coin flip in terms of whether or not that output is actually meaningful at all. Right. And so you you need to give people enablement. And and I think to your point is what is the signal, which is that if you're saying, hey, like, why am I not seeing the benefits from AI? It's like, well, you you may have a a challenge where like a lot of this AI is going to solve problems that you don't that don't actually drive the top line or for your business, right? So yeah.

SPEAKER_00 20:36

It almost sounds as like this needs to be part of the the strategy and the roadmap for AI, because I would think you wouldn't want to go in and ask the board for budget for AI, and then they come back eventually because they will and say, Where's your ROI on this? And you're like, it's coming, but I need to get this really expensive headcount in order to get that ROI squeezed out, right? So you should probably have that baked in. Exactly. Yeah. So if someone was listening to this conversation that is not in tech, they would come away with the impression, I think, thus far, that every business is up to their elbows in AI. Are there still businesses out there that have not, and I mean, you know, mid-cap uh and larger businesses that have not started to adopt AI and are still greenfield, not including shadow AI.

SPEAKER_01 21:23

Yeah. So so I think, you know, I think that this pivot occurred for most organizations many years ago, which is that all companies started to become technology companies whether they sort of realized it or not, right? And so uh what we saw this with Tesla, where like we thought we were in you can sort of think about how that is sort of extended to to other industries as well, too. So I think technology has always been a differentiator in terms of the types of customer experiences uh that you could deliver for people. But I think that there was always a ceiling in terms of what people could sort of deliver. As a result of that. And so I think with AI, like it just opens up the possibilities in terms of like how you can sort of approach and solve and sort of serve your customers. And it gives you a very big, meaningful lever in order to sort of do that. My sense is that like most organizations have have entered sort of the experimentation points where they are adopting tools. I think there's been a huge run on anthropic and claude code and claude work over the last six months. I've I've certainly heard a lot more chatter uh from some of my peers in terms of like, you know, our organization has said, just go get Claude and figure it out, right? You know, you have the budget. Let's just go make it happen. And so I think we've reached that tipping point where probably the middle majority, right, is is going off in adopting. Um I think for the folks that have not started yet, it is going to be really difficult, uh, unless you're a very small organization, uh, to pivot because these other folks have sort of started. So for reference, though, so at the conference that I was at, we we did ask this question, and there were maybe 30 of us in the room. They said, how many of you have not gotten a mandate from your CIO or not CIO, from your CEO to basically use AI? Uh there was only one person out of the 30 that had not stepped up, right? So there's there are still pockets, and I still hear this when I talk to some consultants, that there are still some organizations, still some very large organizations that are just starting to give access to the tools. Um and again, there's a diffusion problem, right? Giving access, it's you you are still going to talk months, years before these people are comfortable with the tools.

SPEAKER_00 23:38

Total tangent, but you you mentioned Tesla. Are you a car guy by chance?

SPEAKER_01 23:42

I am a car guy, yes, yes, absolutely. So yeah.

SPEAKER_00 23:44

Did you see Ferrari's new Lucha in the EV that they put on?

SPEAKER_01 23:48

I have not, but I'm I'm sure my kids have. We went to the Ferrari museums actually a few years ago, and uh my kids were in uh in heaven.

SPEAKER_00 23:56

So it's a complete disaster, man. It's it's a very ugly, very un Ferrari design, and they're they want six hundred and forty thousand dollars for it. And you know, if you're a Ferrari purist, what you want is the the Yeah, I I want a Ferrari, yeah, you know, like I only want one car, yeah.

SPEAKER_01 24:17

Uh it's uh it's interesting. So yeah. I'll have to check it out. I'm sure my kids know about it.

SPEAKER_00 24:22

So you know the the car world has suffered two great losses here because Jaguars going all EV and and they totally destroyed the design. I mean, it's just it's a nice one.

SPEAKER_01 24:31

I I am in the camp save the manual, so all of my you know, my my my daily driver is the automatic, but everything else that is fun has a stake. Right. So I'm in the camp of like I actually like to drive cars.

SPEAKER_00 24:44

So yeah, um I think BMW, uh the new M3 is coming out with a manual. Anyway, so back to AI. It's it's rare that I get to talk about anything but AI on this podcast. I'm gonna rebrand it to the AI podcast because uh it's uh you can you I literally don't know what to talk about that doesn't touch on AI anymore, you know? Like I think it's so uh so much chatter about it, right? Like it's yeah, overwhelming. And like you have to talk about you know, it's not like I uh anyone's gonna listen to an episode on POTS replacement. So very few people have not ventured into AI adoption, and that means that the majority have at some level. You just were a panelist at our event that we had in Cape Cod or Technology Leadership Summit, and the whole point of that event and and the uh theme of it was an assumption and an acknowledgement that people have started adopting AI now. It's not where we were two years ago, and they're saying, where do we start? So, you know, they're already you know on first or rounding second, but they don't they never put the things in place that they should have at the beginning, right? So they took a shotgun approach and they don't have the governance in place, they might not have the security measures in place, they might not have the training and education for their employees in place. So the idea of the event was where how do you go back now and start from the beginning? So what is the difference in the approach of someone that's starting from scratch versus someone that's maybe skipped some of those steps? And if they did skip those steps, what are the most important ones that they need to go back to?

SPEAKER_01 26:25

Yeah, so so I I would say that um th this is an organizational transformation, right? Uh and I think it needs to have clear leadership support uh from the top in terms of why are you undergoing this transformation and why is it important to the business, right? I think too often uh it can be perceived, right? Because we we've all heard these stories before in terms of like, you know, management goes and they read a book and now your company strategy has changed, right? But uh with with AI, I think a lot of people sort of hear that like, hey, we want to roll out AI because we want to drive greater profits for the organization so that I can go buy a bigger boat, right? Uh and I don't think that that's like a super inspiring challenge uh for people to hear. Uh but I do think that most organizations, you know, I'm sure there are some companies that are unfortunately like that, but I think most organizations they they have an existential crisis in terms of like there is a big risk in terms of the types of services, the types of products that you provide being significantly disrupted by AI. And that like I think it's very important for leadership to clearly articulate exactly what are those risks uh and what are the potential opportunities and what is in it for not only the employee, but potentially the organization, right? And make it much more of sort of a collective approach. But I think that is good, right? You obviously need to provide the budget and access to the tools and and and you know policies and other things to sort of enable people to act. Uh, I think a lot of the mistakes, so if one one of the mistakes that we certainly made was that like we had an AI policy, and I I think it was like 20 pages long, right? And uh granted there was some filler at the beginning and end, but like people would read the AI policy and they would be so damn afraid to do anything that the safest because at the end of the policy is like, well, if you don't follow this policy, you're fired, right? It's like, and then like it and I you know I'm I'm paraphrasing, but I think that this is probably not dissimilar to many organizations, right? Is that like you get so up and like what you cannot do, right? Don't do this, don't do this, worry about this, data privacy this, data privacy that, right? Is that uh I think the better approach and what we actually ended up doing at Progress uh 18 months ago is that we we simplified that and tried to sort of be like, this is what you can do, right? This is what there are controls for you to sort of experiment safely with. Um, you know, this is the process for sort of getting additional help. And and granted, some of those processes I think are still evolving, but you know, more in terms of like what are the possibilities, right? So now leadership has sort of created that space where people can sort of feel like it's safe to experiment, it's safe to make mistakes, et cetera. But but then it's a challenge of sort of finding those vanguards and those champions, right? The people who can sort of grow this from the inside out and create the wins, showcase the wins, build on the momentum, right? Really sort of celebrate success in terms of uh people make making it a dramatic change. Uh we were attempting to, although this fizzled out, but I want to try it again in terms of like highlighting the failures, right? Like if failure Friday, I think was what we were sort of uh calling it at some point, in terms of just like, it's okay if you tried something and it failed, but like what did you learn, right? Um and uh try to make that sort of fun from sort of a culture perspective. But I I think uh again, it's just an organizational change in terms of like how can you make, how can you articulate why this is important and what they can grow from and and why this is important for their their careers and sort of you know their jobs and and for the organization, but then really try to just find the people and get out of their way. Uh you you had talked about sort of shadow AI. I wanted to touch on that because it's just sort of interesting. Is it like obviously shadow AI is a concern. We we we have shadow AI at progress that we are we're constantly monitoring. I think getting some visibility into sort of what the tools the teams are using is actually really interesting because it may be actually identifying where you have gaps in terms of what you're providing in terms of tooling to your organization, right? To me, they're like clues as to like we're we're giving everybody access to X and we think that this solves their problem, but yet everybody is going off and using Y. Like, why are they using Y? Like, I I think there's information there that is a signal of like there may be misalignment between what you've provided and what they need, right? Uh in trying to sort of get people in in a governed way to sort of where they need to be, I think, is is uh the opportunity that sort of comes from the shadow AI. Like you can obviously you have to block access to Deep Seek, you have to block access to some of the Chinese models. Like you have to do obviously the uh the smart things to sort of protect your business. Uh but I I think that there's some places at the edges where you can use it as signals.

SPEAKER_00 31:25

Yeah, that's a great call out if you, you know, let let's say you invested in something like uh, you know, Zoom revenue assistant. Right. Uh but people are still signing up for Gong or some other note taker or something, right? Like maybe there's something that ZRA is not doing, uh revenue accelerator not assistant.

SPEAKER_01 31:43

But uh Yeah, and it's the question that you were asking before, too. What is the signal like when you need a chief AI officer, right? Which is that like something is happening there, right? Like either and we see see this with copilot 365 all the time. People like, oh, I need anthropic to do X. And then we ask, like, well, have you tried it in Copilot 365? And they're like, No, we haven't yet, right? Or they haven't, they they didn't know how to sort of do the second or third prompt and get like a 95% result in in co-pilot. And so it's like either the tool sucks, right? And we actually do need to get a new tool, or we actually fail to enable you on the tool that we gave you, and you don't actually know how to take the most advantage of it, right? And I think that that that's sort of the discovery that needs to occur.

SPEAKER_00 32:25

Yeah, that that that's the uh shining light, uh blinking light that you need to inspect that a little bit closer. Um and I want to point out that you what I keep hearing from you is culture. You you've really gone out of your way to build a culture around AI and around AI usage. And I love all the talk of enablement and and uh I love the idea of kind of flipping a script and not saying don't do this, but rather you can do this, because you know, humans weight negativity more than uh positive things. Like it it it hurts more to lose 20 bucks than it feels good to gain 20 bucks, and that's yeah, exactly. That's a uh evolutionary thing, you know. So people are already a little bit nervous about it. If you start filling their head with all these negative things and how scary it can be, then it really is going to slow down adoption. Yeah. So I think when we were doing a planning call, you brought up an example of a top 10 accounting firm that had to start over. They had uh they had rolled out some adoption, they had adopted some AI initiatives and it was early, and they have found that these were not the right things, uh, and they had to kind of start from scratch. Can you talk about that experience a little bit?

SPEAKER_01 33:37

Yeah, I I so uh what the the way I sort of think about this is uh sort of the builder and sparter challenge, right? And and we actually made uh I almost did this at Progress as well, too, and I and I pulled back on it, uh, which is that again, I got so or we got so enamored with the builders, right? With all of the incredible demos that people were building, uh the new dashboards, the agentic workflows, like the the automations within engineering. We said, Oh, everybody, we want everybody to be doing that. And and then I very quickly realized that like, no, we we don't actually need everybody to be using clawed code and sort of going out and vibing everything at the same time, because uh that's really counterproductive. Because what what we'll end up having is um a whole bunch of people solving a perceived problem that they have that's very tactical and not maybe thinking sort of strategically enough in terms of the problem to be solved. And you have a whole bunch of bespoke systems now that like may not be governed or sort of registered that you now you know are probably not managing. And so it goes back to sort of like the right problem to solve, right? I think there has to be sort of intention in terms of these are the types of problems that we think are going to drive the most value for our department, for our business. Uh how can we, in a controlled way, get a few people to try to solve that problem, right, uh, and build an application that many people can then use uh instead of trying to have everybody sort of go and automate their inbox, you know, 10 different ways, right? And so I think what what happened at the accounting organization was that like they just realized that there were too many applications that were being built by people that had incomplete information in terms of what was the goal to be achieved that was leading to disruption because you know, if you think about this arrows all pointing in a direction, like a bunch of those arrows were pointing in the wrong direction, uh, and were actually creating drag uh in terms of the velocity. And in we we had talked about, I think, the MIT study at the uh at the conference that you guys ran. Um I used to think that the MIT study was like complete crap. Uh and in fact, it still is complete crap, right? I think it was not a very good study, it was a very small study. Uh there may have been ulterior motives in terms of why MIT was publishing that study. Uh, you know, full disclosure, I do a lot of work with MIT and uh but uh but a terrible study, right? But uh but but I think the premise of it is that like uh when you're trying to scale this across an organization uh and you're trying to connect value streams and you have all of these different parts of your organization going a slightly different direction, they are failed projects, right? Like it's it's it's a failed investment of time in terms of they didn't accomplish the intended outcome. They took time away from something to do something that wasn't needed when they could have been doing something else, right? And so I think it's maybe they have learned something, right? But I think you could argue that what you really need to do is to create another venue for them to learn and to engage with AI uh that was more meaningful in terms of what your organization was trying to accomplish, right? And so, but I I think that a lot of these IT projects uh just ended up sort of going sideways. The other thing that I would say with that MIT study, again, you didn't ask this, is that like most IT projects fail. Like if you're running those types of experiments, you're gonna expect a high failure, right? Right. That is uh if you're not failing, you're not trying, right? The whole point of what we're trying to do is to innovate and be at the edges. And so you would expect some some portion of failure.

SPEAKER_00 37:13

Two things that I think are positive about that study that it identified are one of them may be self-serving for us, but it's it's it really shine a light on the uh pilot's scale gap. And so even if a pilot is not necessarily intended to scale, maybe it's just an experiment, but you should be going into these experiments with scale in mind, right? So planning for what that next step is going to be and budgeting for that scale as well is probably something to think about as you're designing these pilots. And then the second thing, which is the self-serving thing, is they identified that teams that worked with a partner like Amplix or you know, third party that had expertise in AI, they had a higher success rate of scaling and realizing ROI.

SPEAKER_01 37:55

Yeah, a hundred percent. And so I actually just wrote a I could not agree more. I just I just wrote a LinkedIn article that uh was sort of saying that's echoing, I think, many of the same sentiments, uh, which is that and so the premise of the article that I just wrote was was that you know SaaS isn't dead, we're we're being misled. That was basically the title. But that underlying all of this AI hype, uh, when you actually try to bring a lot of these things to production, it is really flipping hard and complicated and made even more complex by the fact that you are trying to implement and manage and monitor uh non-deterministic systems, right? It is a whole new world that people need to navigate that is being like completely glossed over uh by all of in that the way that organizations are sort of like masking this is that you know that they're all going crazy in on you know forward deployed engineers, right? So forward deployed engineer is like the new hottest job. But you know, I was sort of arguing that a forward deployed engineer is just really sort of professional services in disguise, right? Uh in your masking that like something that I should be able to do on my own with your tool is admittedly like beyond the capability of what I could potentially do, right? If if we need all of these forward-deployed engineers, in that that it's it's a strong signal that like maybe it's not as easy or turnkey as what you see on x.com or on YouTube, right, from all these influencers to saying, I reinvented my workflow overnight. It's like, well, Mike, maybe you did, but like you're not actually running a business and you're not in an enterprise, right? Uh but the answer for for Amplex, right, is that like how do you how does an organization solve that? Is that you partner with people that have solved problems for you before, that deeply understand your business, that deeply understand your pain points, that have a vested interest in your success, right? And and you don't get that from some random forward deployed engineer that's going to be on your team for three to four months and then walk out the door with all the knowledge, right? And so so that that was sort of my thesis for sort of SaaS is not maybe dead, right? Which is that like you have all these services organizations, you have all these software companies, like those are your forward deployed engineers because they have in your forward deployed organizations, because they have the customer relationships, they have the understanding, they have the the domain expertise, right, to actually do these things at scale, enterprise great, you know, uh for their customers.

SPEAKER_00 40:30

Yeah, that's what I always tell people is you know, if you're if you're trying to get stronger and you go into the gym and you do one rep and leave, you're not gonna get stronger. But is this my problem? Is that man? That's you're supposed to call me out at the uh that and get your protein, you know.

SPEAKER_02 40:47

Yeah.

SPEAKER_00 40:48

But you know, if you've got someone that's doing the reps day in, day out, then that muscle starts to grow. And so, you know, when when we I don't want this to be a shameless plug for Influx, but when when we approach a project, we've got thousands of reps behind it, right? So, you know, we've been through the learning curve, we know the the landscape of things. And that's especially important right now with AI because I I mean really nobody knows what the hell's going on here, right? I mean, yeah, if if if someone comes across as an expert, they're lying to you. They've got an ulterior motive to your point about the influencers.

SPEAKER_01 41:20

You know, and this this is my I don't put this in the article, but like so they're trying to hire tens of thousands, hundreds of thousands of forward deployed engineers, right? So these are the people who are like supposed to be experts to help you implement AI. Where the hell are these people coming from? Right. Like, like you're like it just it doesn't the math doesn't math, right? Like in terms of you're gonna create the same problem, which is that a lot of these forward deployed engineers may have no understanding of your business. And so there's potentially gonna be unintended consequences in sort of other places, right? I I'm I'm sure there are plenty of very qualified forward deployed engineers. I've met many of them, right? But uh but I think when you're hiring at that scale with that speed, right, for these mission critical things, there are gonna be gaps in terms of you know what they're promising versus what is actually gonna occur.

SPEAKER_00 42:11

Yeah, use your critical thinking skills, right? Consider the source. I mean, uh Sam Altman just came out this week and said maybe there's not gonna be a job apocalypse like we say, right? You know, I think everyone on the ground, I just said this on another podcast recording, but everyone on the ground that I've talked to that are actually involved in enterprise grade AI on either side, as a supplier or a vendor of AI or a consumer, you know, customer of AI, nobody saw a job apocalypse coming. They everyone says it's a it's an enablement play, right? It's gonna make your workers more productive, more powerful. And uh yeah, some jobs will be lost, but those jobs will be shifted to other jobs that are created by AI.

SPEAKER_01 42:52

There's certainly no shortage of uh jobs building data centers at this point, right? Like but it is a shift of jobs.

SPEAKER_00 42:59

Yeah. So technical debt. It seems like AI is ripe for creating a whole new type of technical debt, which is really what we were just talking about with uh you know having to start over, right? Or to protect yourself from having to start over because you again, it's that atomization issue, right? You've got people that are doing tactical things and maybe not communicating across uh the organization. So how do you view AI technical debt and like how big of a problem do you think it's going to be five years out or two years out?

SPEAKER_01 43:31

So so I'm I'm looking at this through sort of like the enterprise lens, right? So I think your or an organization's tolerance for technical debt, I think it depends on uh sort of their definition of what is good enough for what they are trying to accomplish and maybe how much risk uh they're willing to sort of take uh with with any given system that they're implementing, right? By definition, I would say all of these applications that are vibecoded, particularly by people Who are non-engineers have technical debt in them, right? The technical debt is almost built in because they are not thinking about performance implications. You know, none of these models lead with, unless you're creating skills, you know, sort of uh least privileges or zero trust sort of principles in terms of mine, right? Uh so they have, you know, sort of things that probably need to be cleaned up inherently as part of being built, right? And I think that that is the risk of sort of the democratization in terms of development, which is that like one of the things that engineers think a lot about are all of sort of the supportability, extensibility, you know, scalability, secure security, et cetera, right? Uh, in that we have checks and balances for all of those things, uh, likely on our pipelines to make sure that these things are at a reasonable level of quality with all the right code sniffs so that it can be supported by someone else in the organization, et cetera. And so I think with a lot of these applications that are being built, right, they're not going through uh sort of a similar level of rigor because I think in some cases people just simply don't know the right questions to sort of ask. And so I think that there is some technical debt there and also just a lack of sort of the platforms necessary in order to manage, monitor, govern a lot of these types of agentic systems. A lot of these frameworks have really only come out in the last three to six months. And so your ability to sort of really get a single pane of glass in terms of what's going on in your organization across all the different dimensions, particularly if you have tens of thousands of agents, it's difficult. It's not impossible. Again, if you have four deployed engineers and you have enough enough money and you're a large enough organization, I think uh you can sort of work around that. But but but I think the point is that at some point this this debt is going to come due, right? Which is that you're gonna need to make changes, there's gonna be an event in production, you're gonna have a security breach. And because it is a hundred percent vibe-coded tool, you're you're sort of at the um, you know, risk of just you you don't know what was actually built or why it was built. It can't go through an architectural review. Uh, so you're sort of leaving it to the AI to sort of find and remediate whatever issue uh that sort of could come up. Uh so I so I think it's it's an interesting, I think it's going to become an interesting challenge, particularly as as these things continue to scale. Uh I would look at it though from sort of the flip side, which is that there's a huge opportunity. This is something that we're doing at Progress, but I'm I know it's not unique to us, which is uh AI in all these agents are an incredible opportunity for us to start automating to address some of the technical debt that we've sort of taken on as an organization over the years, right? And so we have agents uh that we are working on uh to sort of pull security issues from Black Duck or pull third-party libraries that are maybe out of date uh out of the Black Duck system, pass them to agents and see what type of uh PR that could be created uh and to see if we can actually start to mitigate and resolve a good number of issues just fully agentically. Uh, and you're not going to get 100%, but if you get a small percentage of those things, right, you can start to sort of work those uh vulnerabilities down. And I think that that's a great opportunity, right? And I think Mythos and a lot of these other sort of uh models that are coming out, GPT-5.5, which is apparently nearly as good, right? While they are creating additional technical debt in terms of identifying things that need to be fixed, right, I think it also gives you an opportunity to then go and mitigate some of these risks as well in a more automated way. But there's also the challenge of triaging, not not all of these things are you get a lot of false positives and and noise in terms of any of these security scans that creates a little bit of chaos because yes, it may find a hundred issues, only 10 may be relevant for your organization, and there may only be a handful that actually follow the right path where they could actually get fully exploited, but you have to look at each of those, right?

SPEAKER_00 47:58

Yeah. We talked about scale a few times in the conversation here. So I think planning for scale requires understanding the economics of these models and the underlying cost structures. So right now, pricing is fluid. We've seen Anthropic take away the all-you-can-eat model and start metering inference costs and and uh token usage. And we're we're seeing AWS increased the cost for GPUs. And you just told me as we were getting on the podcast here that Microsoft increased the cost for Copilot. So like how do you deal with this? How do you start to think forward about what the budget implications could be for these projects?

SPEAKER_01 48:40

Yeah, so uh so I would say right now is that if anybody has an answer on this, that I would love for them to sort of call me and you can uh help me work this out, right? But I think my my general sense is that uh we're we're all sort of in the same boat trying to sort of figure this out, right? Which is that I think uh it would be naive, I think, for any of us to not see this day coming uh where they basically switch from uh subsidizing the the models to doing more of a profit approach. Uh, but I think the speed at which this has transacted uh is taking, I think, a lot of organizations aback. And in some cases, um, you know, they don't have the controls in place or the reports in place to be able to monitor and forecast these expenses sort of moving forward, right? And so that this is certainly what is happening uh with Microsoft. So uh so for those who are not aware, right? So Microsoft, a GitHub co-pilot, uh has always worked on this uh premium request model. So every single time that you asked the AI to do something, you would get assigned some number of premium requests and it would go off and sort of do the work for you and come back. As the models have gotten better and better and have sort of been able to sort of support more complex agentic workflows, the amount of work that you can do for a single premium request has exploded. Right. And I think what this was, and again, this is all you know conjecture because nobody has actually, I think, officially said this right. But I think the the what this created a situation is that I think a lot of the the plans that Microsoft were offering became instantly unprofitable, and particularly when you had power users, you know, wildly unprofitable for the organization. Um, you know, to the point where they are losing money, right? And I think this was the same pattern that I think emerged with anthropic uh in a lot of the people that sort of figured out how to do their open claw with their Macs plan by uh sort of doing a back door in terms of how they could actually secure the inference, right? And so I think a small portion of the user base really was having a very, very outsized impact on Google, on anthropic, on Microsoft, et cetera, and that in that they very quickly had to make a change, right? And so why are they making the change, right? Obviously, a lot of these organizations are trying to go public, right? And so uh I think they needed to start to think about sort of the true economics in terms of the inference. But what the challenge that it's presented is that like we now have an enormous amount of uncertainty in terms of forecasting what our inference costs are going to be from a development perspective in terms of funding the engineers. So and you have sort of a very difficult base of data to work from because all the data that I have in terms of the usage patterns for the engineers were them using AI that was basically virtually free. And so, you know, when I look at our top 150 users, they're all what I was saying, like YOLOing on like Opus 4.7 and 4.6. Um, and the reality is that I think in most of those use cases, they probably didn't need 4.6 or 4.7, right? But because it was so cheap, they were using it. Right. And so now I'm trying to figure out, or we're trying to figure out what is that right water level in terms of like what is the appropriate level of usage if you were to use something like Codex or Sonnet 4.6 as your daily driver, going back to the cars, right, in terms of uh you know, cost basis uh and then trying to be much more intentional around the economics of like what models are we using for what tasks, thinking more about like if I'm starting up a project or a new repository, am I assigning a pool of AI to fund that development? You know, uh thinking about asking questions when you have users that are spending thousands of dollars a month on AI, what exactly are they doing? Right. Uh so all all these questions are sort of coming up at the same time. And uh the data is uh challenging to say the least in terms of forecasting this, right? So I think I think it's gonna be a very it's gonna have a whole bunch of, I think you could have a whole new show on just the economics of AI, right? Uh, which is that this is gonna cause ripples across the industry in terms of I I don't I I would sort of predict that there would be some impact in terms of the adoption and growth of the anthropic and and some of those vendors when people actually start to see the bills, they're like, oh, I want anthropic. I was like, I don't think you understand what you're getting into in terms of like the bill, right? And so the these trade-offs I think are gonna become front and center at some point.

SPEAKER_00 53:22

Yeah, there's so much to unpack there. Well, first of all, with these businesses, you know, they they they use the drug dealer model, right? They get you hooked on something and then they they start charging it for it. And we've seen that many times, particularly with social media, right? That's one that comes to mind. And but the speed at which the price increases have hit the market is really surprising to me because since when does Wall Street actually care about profitability anymore, right? Capital keeps flowing to these companies, it's all speculation and betting on their their future and that eventually they'll be able to, you know, be profitable. So it's really surprising to me. It must be loads and loads of cash that they're that they're losing.

SPEAKER_01 54:01

Yeah, I think that they're losing stunning amounts of money, right? And so uh but but I think the economics are really like when it's free, yeah, we work in a global economy, right? And so that there is uh less expensive labor throughout the world that can do many of these tasks with people, right? And so a lot of these agentic automations, I think, are not going to pass sort of the sniff test in terms of and you see this with open claw and all the people posting about their open claw bills and what they're it's like, well, do you really need to spend all of that money to like manage your inbox and your calendar and some of those other things? So, you know, it's sort of based off of this assumption that the the inference was going to continue to get cheaper, and I think that that that big assumption that we all made maybe 18 months ago is no longer true. And you have sort of this mismatch, right? Which is that it's great that Opus 4.8 came out. Do I really need Opus 4.8 for 90% of what I do? Like, do I need an all-knowing model that speaks every language, that you know, can decode the Rosetta Stone? You know, and like probably not, right? Like, I probably need a much smaller model. Uh in it, and it's interesting to me as to how the economics start to shift as organizations start to move to some of these smaller open source models uh to offload some of their processing and don't just go with like the the default, which is like the most expensive model, right? Or the one that sort of gives you the best answer because it's the best at uh sort of deciphering the ambiguity in terms of what you asked it to do, right? Versus like you being better about like this is what I need to do, this is what my success career criteria looks like, this is what I want the output to look like. Like I think if you do that with some of the less expensive models, you'll get the same result as, or similar results as some of these more expensive bottles.

SPEAKER_00 55:52

Yeah, here at Amflix, we rolled out an enterprise claude account, and I got credentials to it. I'm one of the you know few users that are officially allowed to use it. And within four hours of getting access to it, I had used all of my allocation. And you know, so one thing I learned from that is which model are you using for these tasks, right? But another thing I learned about it was the context window and making sure that I I minimize that, right? Because if it's going back and referencing the full conversation every time, we're just we're wasting money there. So but these are things that I think you need to learn these things. And I know you can probably control it at an enterprise level for you know, if you've got a particular you know initiative that is using Claude, you can map it to the right model. But what about the individual users, right? How how do you educate them and get them to comply with that so that they're more efficient?

SPEAKER_01 56:49

So I think maybe we we have sort of a multi-approach to this, right? As we're sort of thinking about this through the lens of the engineers, which is that I think previously when a new model was made available, we would make it immediately available to the engineers and they could all go to start experimenting with that. I think what we're gonna be moving towards a model is you know, we're gonna let these models sort of soak for a week to try to see if the hype behind them actually matches to the reality in terms of what and so probably the more recent example of this was like Opus 4.7, uh, which is that you know, whenever these models come out, they're the greatest and best models ever, it's gonna change everything. You know, I built a business overnight that's making me $600 a day. And then like, you know, a week later you could you actually start to see what where the the edges really are. Uh and I think with 4.7, I think the majority of our engineers use 4.7 and they were like 4.6 is better, right? It follows the instructions better. It's giving up again, going to the sort of scale problem. If we if we had rolled 4.7 out and we have now 1,200 people playing with 4.7, uh 4.8, I'm sorry, there is a meaningful economic impact there, right? And so again, I was talking about that the top users at our organization were all using OPUS under the new billing. You know, they were incurring, they would be incurring hundreds of dollars of OPUS charges per day. Right. And so I can't have a thousand people experimenting with 4.8 to see if it's fit for purpose for their job at several hundred dollars per day. Right? Uh we need to basically have some way where we say these are the business cases where we think that Opus is going to be meaningful, run some experiments, and then ultimately be able to say for these projects, this is where we think, or for these repositories, this is where we think Opus is going to drive the most value. And here is the budget that you have. Microsoft today does not offer any capability uh for uh doing that type of budgeting, right? So I cannot assign OPUS to just the architects in my organization, right? Uh it's an all or nothing, at least the way that our organization is currently set up, which is that if I assign OPUS to one person, all 1,275 get it. And I can't, so so that limits sort of the flexibility in terms of how we think about this. Uh the the other way that we're sort of talking, you were talking about sort of context product uh caching. Um we actually built a harness that wraps around these uh coding agents uh that actually does some management of the context for the users, right? So what the harness does, which is sort of, I think, just sort of a best practice for everybody, is that you want to minimize the number of turns that you're going into the agents with, right? Because it's repassing this context in and out. And we talked about sort of this 80% problem before, uh, which is you know, can you create a plan that is much more complete in terms of like what you're asking for, what success looks like, so that there you're eliminating the ambiguity for the model, right? And I think that that has the biggest impact in terms of how much value you get out of that particular turn uh that you are sending to the model. So we have uh a product that we've been testing in-house uh that basically helps build that context for the users and also does model routing so that you're using the right model for the right task uh at the point in time. And and I don't have to think about like, oh, I'm writing unit tests, I can just use Hi Cool here, right? Uh and and we'll help manage some of that expense. So we we have sort of like a multidisciplinary approach, I guess, uh to sort of try to manage this for users. But it is a a lift, right? There is a lot of uh just talking about tokens, you're talking about caches, like it it gets complex pretty damn quickly, right?

SPEAKER_00 1:00:27

Yeah. When when is the right time to introduce something like that product that you're talking about? Because I it sounds to me like as you're starting to scale, you really do need something like that that is going to route to the right model, keep it cost effective, you know, contain the context. I mean, that that sounds like an early-on thing that you should be adopting.

SPEAKER_01 1:00:50

Yeah, I I think it's uh again, it's it's it's a scale problem, right? So if you're managing smaller teams of engineers, it's it's easier for you to get them all in the same room and sort of have a conversation around what models we're going to be using for what, and and you can sort of uh communicate that maybe during your daily stand-ups or in Slack or whatever mechanism that you're using, right? I think for us the challenge is that you know these models, the model uh acceleration and the rate at which they're sort of coming out with new capabilities and Claude Code or or Copilot CLI is coming out with new capabilities, right? They're they're iterating what feels like maybe multiple times a day at this point, right? Uh there is no way that you can diffuse that information in a way that people can sort of absorb it across lots of people. Yeah. Right. Um and some of the solutions that the vendors have provided, right? I think there's an auto mode now with Copilot CLI. Uh, for those who are not aware, so Copilot CLI is Microsoft's uh sort of agentic harness that is similar to Codex or to Claude Code. For the Microsoft bashing that I've been doing, it it is actually probably the best product uh that nobody knows about that they have. And it uh prior to some sort of them uh raising the rates was certainly the most cost-effective way to sort of bring agentic workflows uh to your organization, and I think will still be a very compelling um alternative to cloud code with all the governance that you need on the back end in terms of making sure that you're not pulling open source code and things like that. But but I think it's a challenge, right? So they have model routers that you can see like auto mode, it'll it'll attempt to determine based off of what you're asking what is the best model to use. And and they do some model splitting in the back when you have all these subagents that'll assign different models to do different tasks. Uh but I think that that's good, but I think ultimately organizations are going to want to be more opinionated about what tool they use for what task and not leave it to the model to sort of make those decisions for you. So to answer your question, there are products like this that are starting to emerge. Like I said, we we have one that we've been playing with internally, maybe going through some discussions whether or not we want to do a market test with that. But uh it it feels like an opportunity, right? Particularly now that these costs have exploded.

SPEAKER_00 1:03:08

Yeah. Uh well, sign me up for the beta, please, because I think excuse it. Uh what I've taken to do and probably a terrible practice, and I shouldn't admit this on a podcast, but anything that's sensitive, whether it's got PII or IP, I'm running through the business instance, and I'm just using my personal one for anything that is public. So uh if I'm editing copy for the website that's going on the website, I don't really care if that gets out in the public domain, right? So I I just do it through my personal cloud account.

SPEAKER_01 1:03:38

Yeah, that there's a lot of clues. Uh, you know, we I think most organizations have a data privacy policy, right? And uh I think the data privacy policy drives a lot of the behavior in terms of what where you use AI, right? And uh I do the same thing, which is that like this is stuff that everybody knows already. I'm just gonna put it in my corporate GPT account and what it's like, right?

SPEAKER_00 1:04:02

Yeah. Advice for people listening to this. I mean, everyone's gonna be at a different stage. Is there any broad blanket advice that you could give to people to help them uh implement effectively, adopt effectively, keep it cost effective, keep it secure?

SPEAKER_01 1:04:18

Well, that's a lot, right? I would say I so so first of all, I think AI is a participatory sport, right? Like I think the only way to sort of learn this is to actually do it, right? And so a lot of times I find that we are in meetings where we're talking about how we could govern it or talking about how we could deploy it, and it is just way friggin' easier to just go do it and see what happens and learn off of that, right? I think that that you you learn by doing uh and you learn by failing. And I think it's it's critical that you figure out how to reduce that cycle time so that you can run those experiments faster uh and communicate out to the organization. The the second thing that I would say is that like really explain the why uh and try to have the organization help you solve the problem, right? And so the example that I would give there is that, you know, again, Microsoft dumped these pricing changes on us. We're looking at, you know, what could be five to seven X in terms of our monthly bill. And I was just on a champions call this morning and I said, like, I really need your help, like figuring out how we roll this out, right? This is the problem, this is the constraint, this is how much money it's gonna cost, this is what we can do from sort of a governance perspective, this is what we can't do, this is what I'm seeing. Like, how should we approach this? Right? Because like we're all in it together, right? We're all figuring this out. And so I think if you adopt that more broadly at an organization perspective, and again, this goes back to culture, uh, then I think you have a chance in terms of like rolling this out and and and getting some some velocity in terms of the adoption.

SPEAKER_00 1:06:04

And I think that's a good place to wrap it up. I could talk to you all day, but not even my mother could listen to me for more than an hour. So we're gonna wrap it up here. But uh thank you so much for joining us and for your time and expertise.

SPEAKER_01 1:06:14

Great to be on Sean. Thanks again.