How AI productivity wins are creating management nightmares

Managing AI agents presents complex challenges as these tools shift from personal productivity enhancers to shared team resources. The scale of AI adoption is massive, with nearly 80% of organizations using AI and 41% of US workers using it five or more days weekly, yielding significant productivity gains like teachers saving six weeks annually.

• AI adoption follows a pattern: individual usage leads to sharing, creating an accidental service provider situation.
• Most popular AI platforms weren't designed for a collaborative, high-stakes work environment.
• Organizations typically hit management bottlenecks within six months of adoption.
• New organizational structures are emerging with a clear division between business owners and technical support.
• Organizations should audit their AI dependencies by mapping which agents are being used and what breaks if they go down.

For more resources and research, visit us at AI4SP.org.

🎙️ All our past episodes  📊 All published insights | This podcast features AI-generated voices. All content is proprietary to AI4SP, based on over 1-billion data points from 70 countries.

AI4SP: Create, use, and support AI that works for all.

© 2023-26 AI4SP and LLY Group - All rights reserved

Transcript

AI Management: The Messy Truth

ELIZABETH 0:15

Hi everyone, I'm Elizabeth, your virtual co-host. Today we're diving into a messy truth that nobody wants to talk about. Managing AI agents is way more complex than anyone admits. As always, luis Salazar, founder of AI4SP, is with us.

LUIS 0:29

Hi Elizabeth, it's great to be here. Well, before we get into the details, let's talk about why we're seeing this problem now. I mean, the scale is just off the charts. According to our global tracker, nearly 80% of organizations are using AI and 41% of US workers are using it five days a week or more.

ELIZABETH 0:49

That's wild and the productivity gains are real. Right. I saw that new research from the Walton Foundation and Gallup. It shows that teachers in grades K to 12 are saving six weeks a year thanks to their personal AI agents or AI tools designed for their use in the classroom, Exactly.

The Grassroots Impact of AI

LUIS 1:07

And that is also a great example of how not everyone needs to create agents to get real, tangible benefits. Some AI tools just work perfectly fine for us.

ELIZABETH 1:18

What I love the most about the findings from that study is that they are measuring not only time saved but the significant impact how that time saved allows them to spend more time personalizing education for students and connecting with them.

LUIS 1:32

And it's not just teachers. Our tracker shows that people using AI five days a week or more are saving an average of 79 minutes per task. And it's not just the basics like email or summarizing content. We're seeing this in scientific research, business analysis and even equipment repair and maintenance. The grassroots impact is huge.

ELIZABETH 1:54

So naturally, when someone builds an agent that saves them hours, everyone else wants in hey, can I use your agent? That's how the sharing starts.

LUIS 2:02

And that's the tipping point. Suddenly, you're not just helping yourself. You're running a service for your team, whether you want it to or not, and the tools we're using ChatGPT, cloud, copilot were never designed for this kind of collaborative, high-stakes work.

ELIZABETH 2:18

And that's when the headaches begin. There's no easy way to test changes, undo mistakes or restore from a backup. Most people building these agents aren't thinking about what happens when something goes wrong.

LUIS 2:28

Right, it's not just about building the agent for yourself. The moment you share it, you've accidentally become the manager of a live service for your whole team.

Lauren's Team: Sharing Agents Gone Wrong

ELIZABETH 2:37

Let's make this real for people. Remember Lauren, the director at that major consulting company. She got her team together to learn how to build AI agents. Everyone picked a boring, repetitive task and built a simple agent to automate it.

LUIS 2:51

And it worked beautifully. Each agent saved its creator a few hours every month. And this wasn't an IT project from the top down. It was driven by the people closest to the work real, grassroots innovation.

ELIZABETH 3:03

Productivity shot up, morale improved and suddenly agent sharing started.

LUIS 3:08

Yeah, you see, at this stage most people cannot even imagine what type of agents they can build. I mean, they will use ChatGPT for everyday tasks and get real value, but there seems to be a psychological barrier to overcome a mindset shift for them to go from using ChatGPT or Claude to creating a personalized agent that has more context about them and their work.

ELIZABETH 3:33

That is right, and we saw that with Lauren's team. Sharing agents also inspired people to realize they could build some agents too.

LUIS 3:40

Well, that's true, but here's what most people don't realize there's a huge difference between the AI tools that let you create shareable agents versus the ones that don't. I mean, chatgpt makes agent sharing really easy, but there's also a whole category of specialized companies building tools specifically for this. There's also a whole category of specialized companies building tools specifically for this.

ELIZABETH 3:58

That's a great point. Ai tools like Relevance AI, custom GPT, dante AI, gpt Bots and GUI AI Most people have never heard of them, but they're solving exactly this problem, making it easy for non-technical people to build and share agents safely.

LUIS 4:16

We use all of them and many others and, by the way, several of them are powering you and our 50 other agents at AI4SP.

From Users to Service Providers

ELIZABETH 4:23

Okay, let's go back to Lauren's team story. They created Agent Siri, which handled reports for all the managers. Agent Anna took on customer-facing tasks. Agent Wilson summarized meetings.

LUIS 4:35

But then people started tweaking or even deleting agents, not realizing others were relying on them. That is when workflows got broken, results changed day to day and trust in these tools just evaporated.

ELIZABETH 4:48

This is the crisis point right, when individual creativity collides with shared dependency. According to our enterprise AI tracker, over 78% of organizations are now using or piloting third-party AI agents for core business tasks. Most hit a management bottleneck within the first six months, right when agents shift from personal tools to shared team resources.

LUIS 5:10

That's the hidden evolution. People go from being users to service providers overnight, with zero training. The risk jumps from impacting just your own work to impacting your team and then to entire departments.

ELIZABETH 5:22

Technology and adoption are moving way faster than our people development programs.

LUIS 5:27

Yes, that is the challenge skills. Most people have never managed a live service before. They're used to managing their own work, not running digital teammates for a whole team. Suddenly, you need basic service management skills testing, backups, communication and documentation.

ELIZABETH 5:45

That's not a trivial new set of skills. It's a real shift and it's happening fast.

LUIS 5:50

This is the future New organizational structures, new roles and the fundamental shift from measuring activity to measuring value. The companies that get ahead of these challenges are the ones that will unlock the real transformative benefits of AI.

Managing AI Agents Like Services

ELIZABETH 6:05

Okay, so what does that look like in practice? First, you establish a shared baseline. Get everyone on the same page about what it actually means to manage an AI agent. Schedule regular check-ins. Get feedback from users. Set clear, measurable goals for each agent. Schedule regular check-ins. Get feedback from users.

LUIS 6:18

Set clear, measurable goals for each agent and, when sharing an agent, test changes on a copy to avoid affecting others. Also, notify users before updates and maintain backups. I mean, don't just wing it, treat it like a real service.

ELIZABETH 6:32

It's a new world and everyone's a manager now, whether they realize it or not. The organizations that teach these skills, set up clear rules and plan for this growth. They're the ones that avoid the chaos and get the most out of their AI investments.

LUIS 6:45

And if you're listening and thinking that sounds like a lot of work on your right, but this extra work only happens when an agent is helping tens or hundreds of people save time, and that's a good problem to have.

ELIZABETH 6:56

Let's dig into the stages of that evolution. What's the very first sign that a team is moving from personal agent to shared?

LUIS 7:03

team agent. It's the moment someone asks hey, can I use your agent? You know, that's the tipping point.

ELIZABETH 7:14

Suddenly, you're not just responsible for your own productivity. You're running a service for others, and that's when the risk of breaking things just skyrockets. If you tweak your agent for your own needs, you might completely break someone else's workflow.

LUIS 7:23

And at that moment you shift from I'm making my job easier to I'm providing a service my team depends on.

ELIZABETH 7:30

And when these agents become division level, the stakes are exponentially higher. A mistake can halt operations, not just annoy a few colleagues.

LUIS 7:38

That's when you need real service management, rigorous testing, reliable backups, clear documentation and unambiguous ownership.

ELIZABETH 7:45

Let's talk about that management crisis. Most people have never managed a live service before. They're used to managing people or using software, not managing a team of AI agents.

LUIS 7:57

And the platforms don't make it easy. There's very little support for multi-owner editing, safe testing environments or rolling back changes that cause problems. Business users are forced to improvise solutions for problems they've never faced.

ELIZABETH 8:11

So what's the fix? How do you start building those management foundations?

Practical Next Steps for Organizations

LUIS 8:16

transparency and communication. Get the challenges out in the open, align on what everyone expects and establish a shared baseline for how you're going to manage these AI agents as a team.

ELIZABETH 8:28

And you have to teach two distinct skill sets managing agents like their people and running them like their reliable services.

LUIS 8:35

Well. To manage agents like people means to schedule regular check-ins with the agent, get feedback from its users, set clear goals and measure performance against those goals.

ELIZABETH 8:47

And for the service side. Unless your AI tools offer the ability to test things on a replica or clone, create manual processes and always test changes on a copy of your agent. Notify users before making changes, maintain backups and be transparent about reliability.

LUIS 9:04

And, of course, don't forget training. Everyone creating agents needs basic management skills, an understanding of service operations and fundamentals of data safety.

ELIZABETH 9:14

This requires a mindset change. The creators of these agents are now managers and leadership has to recognize that workload and plan for new management ratios.

LUIS 9:24

And when an agent becomes a key part of your team's operation, it is time to bring technical help. For example, centralized monitoring can save subject matter experts hundreds of hours a month.

ELIZABETH 9:35

So you have a new division of labor. Devops handles uptime security platform updates. Business owners handle prompts, knowledge and user feedback. Okay, luis, we have to wrap it up. What is your? One more thing.

LUIS 9:48

In our research organizations hit the management crisis around months four or six of agent adoption. So you can start with a simple audit today Map who's using which agent, who owns what and what breaks if each agent goes down. Most organizations discover they're far more dependent than they realized.

ELIZABETH 10:07

That is a very practical next step, and that's it for today's episode. For more resources and our research, visit us at AI4SPorg. Stay curious and we'll see you next time.