How to Benchmark Readiness Before You Scale GenAI

Technology Journalist, Francesca Roche, sits down with Hideki Hashimura, CRM and CX Strategist at redk.

With MIT research showing 95% of enterprise Gen AI pilots failing to deliver measurable ROI, this conversation cuts through the hype to unpack what's really going wrong – and what genuine AI maturity looks like in customer service. If your organization has launched an AI chatbot and wondered why nothing has really changed, this one's for you.

Main Description

Most enterprise Gen AI pilots don't fail because of bad technology; they fail because of everything surrounding it. Hideki Hashimura of redk shares a no-nonsense breakdown of where organizations go wrong and how to build AI-powered customer service that actually scales.

The isolation problem: Most AI deployments are siloed – a chatbot here, a translation tool there – with no contextual learning or connection to wider service processes, which kills ROI before it starts.

AI maturity is an org problem, not a tech problem: True readiness means aligning people, process, and technology together. Agents are evolving into service architects, and closing that skills gap is the most critical step leaders are currently missing.

False positives are everywhere: Deploying an AI bot that handles surface-level queries isn't transformation. If customers still can't change a flight or request a refund, the foundation underneath the bot simply isn't built.

Pilot success must be tied to business KPIs: Define your north star metrics first – NPS, resolution rates, handling time – then build the pilot around impacting those specific numbers. Speed without strategic alignment is just expensive experimentation.

Find out more about redk's AI-readiness workshops here.
For more Customer Experience tech news visit CX Today.

Transcript

SPEAKER_01 0:08

Hello everyone and welcome back to CX Today. My name is Francesca Roche. I'm a technology journalist today uncovering why so many enterprise Gen AI pilots fail to translate into measurable ROI and what it really takes to assess AI maturity and scale Gen AI sustainability in customer service. Now enterprise teams are moving faster Gen AI, but the gap between experimentation and real-world impact is proving bigger than many expected. When AI is introduced into customer service, it immediately touches high-stake areas, customer trust, data quality, compliance, knowledge management, and the day-to-day reality of agent workflows. So value only shows up when the underlying operating environment is ready to support it consistently at scale. Today I'm joined by Hadiki Hashimura, a CRM and CX strategist called Red K, an award-winning transformation consultancy, helping organizations align strategy, technology, and execution, partnering with brands such as ZenDesk and Salesforce to help support teams in building AI-powered CX frameworks with the right foundations in place. But before we get into the discussion today, Hadiki, how are you doing?

SPEAKER_02 1:22

I'm doing great today. Thank you. Thank you for having me.

SPEAKER_01 1:26

That's great to hear. Now, previous MIT research has suggested that 95% of enterprise Gen AI pilots fail to deliver measurable ROI. What's the biggest reason you see behind that?

SPEAKER_02 1:40

It's really interesting what the MIE MIT report uh there was a report that came out last year, and they studied a series of organizations, and they found patterns on what works and what doesn't work, right? But to boil it down to you know something simple, the way organizations are adopting uh AI today are not uh they are not considering the full context of the organization. So they are looking at very isolated pieces like uh a chatbot, for instance, or a piece that's that does translations or text generation. Um, but the the key is on the lack of learning. The AI systems that are set up are not uh really understanding the context in which they deliver the activity, uh and because of that, it's not a complete answer. They are also very disconnected uh uh functions from the rest of the organization, so they are not streamlined into the process of service. But to build this is a lot more challenging than uh initially was uh perceived, and for these initiatives to deliver transformational uh impact, they need to be approached in a completely different way. Um I'm happy to share the report if anybody wants to reach out.

unknown 3:04

Yeah.

SPEAKER_01 3:05

Thank you very much. Uh now AI maturity is a practical way to assess how ready a service organization is across people, processes, data, and governance to use AI reliably in day-to-day operations, you know, not just in isolated environments. Hadiki, how would you define AI maturity in customer service? You know, what capabilities separate the early from the advanced?

SPEAKER_02 3:31

So I think the main capability has nothing to do with technology. It has to do with how ready the organization is to adopt the these type of frameworks. Um it's all about combining the three pieces that build the capability. So that would be process design, uh, people, and technology, right? So how the three of them uh come together to deliver what you need for your team. Uh, something that we are seeing, for instance, is the change on the functions, the roles that people play on in this case in customer service, uh in the customer service teams, and how they interact with technology. When you bring a technology that can accelerate accelerate processes and deliver things for for that before used to be manual, you have to change the way people work. So the the uh people are gonna be using new tools, they're gonna be uh making sure that they have access to data, uh, etc. etc. This changes substantially when you incorporate AI, right? So not only the practices like knowledge management, um AI architecture, you know, people move from being agents to being um service architects, understanding how the all the pieces of the technology framework come together to deliver the function. Uh the the roles of people are changing substantially. And I think this is the most important gap that organizations need to fulfill before they understand how they're gonna make uh the most out of the AI tools that are out there today.

SPEAKER_01 5:18

Yes, I agree. I think a lot of teams feel close to being AI ready also because they've modernized part of their stack or launched a few automation initiatives. But readiness can be misleading if underlying services, the service foundation aren't consistent across channels, teams, and knowledge. What are the most common false positives that make organizations think they're AI ready when they're actually not?

SPEAKER_02 5:44

Yeah, I think the easy the easy thing to identify is when people are have implemented some sort of uh AI bot or AI assistant that is servicing clients at a very superficial level. Because what you realize is that they they are not having a major impact on the customer experience. If anything, in some cases it's even the opposite, is uh frustrating clients that they first have to go through these steps before they speak to an agent, right? Um I think that's a pretty big false positive that uh organizations are able to, they don't know it, but they feel it, you know. Yeah, we've implemented it, we have something going on, but it hasn't changed our lives. It hasn't changed the lives of our customers. Um, and I think this is because usually uh organizations are approaching this question in the wrong way. They are asking the question of what is the best AI uh chatbot or AI agent out there. Let's implement that. The AI chatbot is only the tip of the iceberg. Everything that you need for that uh chatbot to have a transformational impact on both the customer experience and the organization sits underneath. Data integration, uh uh the knowledge management, uh, changing the capabilities and role and in the roles and functions of the teams, the skill sets, implementing skills in the AI tools, uh, contextualizing by integrating the customer service platform into the digital ecosystem. So I think that's a a big uh false positive, like you call it, you know, the uh having an AI bot that answers some questions. It generally doesn't have actionability. It doesn't, like let's say for instance, you you are chatting with a chatbot on an airline. Okay, you may get faster to opening a ticket, but you're not able to, you know, change your flights, change your reservation, canceling it, requesting a refund, or for a hotel chain, or for in the case of uh you know retail, uh understanding how to action completely the request. You know, AI bots alone are not going to deliver a full cycle of trying that is gonna impact the customer experience, right? You need to see underneath what's underneath.

SPEAKER_01 8:14

Absolutely. I think buying an AI add-on doesn't necessarily equal AI ready, especially if your knowledge-based data quality and service processes aren't properly set up for success. How do you think leaders should define pilot success criteria so that impact is measurable?

SPEAKER_02 8:32

So I think you have four layers of understanding of how to build technology capabilities. So you have the strategic layer, and this is the one that actually guides or is the North Star for any type of these projects. Um things in that layer, you'll have things like what are the business objectives that you want to hit? You want to uh reduce the amount of uh time you take to respond to clients, uh your your total processing time for uh queries, uh the your NPS, your CSAT, uh what are the key KPIs that you want to impact? So you can actually start with a fairly small uh you know technology project, but you're trying to impact one or two very specific KPIs and then build the business case around it. So AI technology is not cheap, right? Um so to not only the technology licensing alone, but the implementation services, the uh getting the organization to change the way they work, uh to embed this technology. Um so to make it successful, you need to tie the results to uh to that top layer, right? So you have strategy, the operations layer, so how the organization works, the tactical layer, how people use the different tools in their day-to-day to perform their work, and the technology layer. But if you go from the top down, you're obviously going to try to change the way you work at these bottom three levels. Uh, but if you go from the top down, you're gonna understand what you want to impact. One of the things that we're working on is trying to get the highest amount of automated resolutions from AI. So if we're saying, you know, 80% of the uh customer service requests that an uh an organization gets is about fairly simple stuff. How can we go after that? And often we start with a fairly low percentage, 30 to 40 percent of uh tickets get sold automatically because the AI is well built to deliver that. And by doing that, we we increase the amount of available time from the agents to have more meaningful conversations, but also for them to start understanding and moving into the roles and functions that they need to increase the capability. So it's all part of it's all part of a uh a journey. So to answer the question, the pilot success criteria needs to be business criteria. You know, it needs to be how are we tackling these very specific KPIs? And if you pursue that, you should be able to deliver value to the business.

SPEAKER_01 11:18

And once success criteria is set, you know, once they've managed to align to those four uh layers, what does a good Gen AI pilot operating model look like? What has to be in place to move from pilot results to repeatable scale?

SPEAKER_02 11:35

Okay, so everybody wants to have quick wins and everybody wants to move fast on this space, um, but you you have to balance speed and quality, right? If you're thinking long term, or and when we say long term, you know, 18 months, then we cut down to midterm, then we put down to uh to short short term. A good Gen AI pilot could be solving, uh you know, going after the the easy-to-solve use cases where Gen AI can have an impact of solving things, you know, delivering automated resolutions to a high percentage of the of the tickets that you open. But if you want to create scale, you need quality, you need robustness, you need you know, well-integrated data, well-integrated uh ecosystems. Uh you need people that are working in the teams to really understand how to use the new tools that are out there, right? All of these uh uh capabilities that are provided, like uh from technology vendors, are not just about you know getting people to use uh uh a ticketing platform. There's a lot more to it. And the the roles and functions of people when they they are trying to scale customer service operations uh change substantially because they are now being able to do what in the past you needed a software developer to change in your platform. Now the system architects, which are you know ex-customer service agents, are able to redesign the flow of information, the flow of the ticker resolution because you have AI tools. The AI bots not only service the customer, they also service internally the user, and they assist to the user on how to speed up certain things. They also help the administrators and the developers. They also help create things like uh reports, you know. Instead of going into uh the report function and having to spend two or three hours creating a report, you prompt the system. But you need to prompt the system understanding how it works and understanding how the data is structured. If you need to create an integration, you can prompt the system to develop the integrate and integration kit, but you really need to understand that, right? So the the if you want to scale after you do your your uh proof of concept pilot, you need to work on the foundation. Foundation is very important because that's how you're gonna achieve transformational impact.

SPEAKER_01 14:19

Absolutely agree. I think foundation is definitely the key here. And just before we close, from a future perspective, or in your case, in the next 18 months, what capabilities do you believe will separate organizations that merely pilot Gen AI from those that can scale it sustainably?

SPEAKER_02 14:36

Yeah. So the capabilities are what's gonna happen over the next 18 months is that the technology is gonna evolve. So you keep things are gonna become better. There's gonna be a few mistakes here and there, some fails. Uh vendors always overpromise, you know, a little bit more than they are capable. Um what's not uh what's uh so this technology is gonna change fast. So if you want this to be successful in your organization, you have to focus on the human team because it's not about replacing the positions in the customer service team, it's about enabling them to work with new tools, new technology. So there is a is a it's definitely a step up in terms of skill upskilling and providing the training and the mentoring to move these people from uh being service agents to understanding the organization and structuring the platforms to uh contextualize the uh actions, you know, the service delivery, customer service delivery function. You know, how does the this organization, this part of the organization, work with outside parties, like you know, in the case of retail, for instance, logistic vendors, uh uh the weather, you know, uh things that they cannot control. How do they work with the rest of the organization? Finance, uh, back office, warehousing, uh, refunds, e-commerce, etc. And how do they work with the client, right? They become a lot more insightful into how all these things come together, and they now have the tools to action and change the way they operate to deliver better service, faster, higher quality, more effective, right? So if you're looking towards the future, don't think, oh, what technology I'm gonna get in 18 months, because you don't know. We don't know what technology is gonna come up in 18 months. We have an idea, right? But what we do know is that regardless of anything, your team needs to be ready, and you have to have that foundational piece of understanding how to build this architecture, right? Is how how do your process take place when you have certain technologies, how the roles that people have and the functions and how to deliver those functions and what skills they need for that, and the technology that supports that, that is you know under underpinning that that capability. So focus on building a holistic approach to the capability. And this is exactly what we're doing with clients. You know, we're we're trying to do an assessment, understand what they are, what are the short-term needs, what are what are their mid-term uh needs, and what is the long-term vision of what they should be able to achieve to create this hybrid uh capability of humans and technologies.

SPEAKER_01 17:37

Absolutely, I definitely think we're gonna see a lot more investments in that human in the loop idea. Now, unfortunately, that is all we have time for today, but I would like to thank Hadiki for joining us. It has been really great to get a better understanding on what it takes to move Gen II in customer service from pilot to measurable scale impact. So thank you so much for joining me.

SPEAKER_02 17:59

Thank you very much. I hope it's been useful and feel free to reach out.

SPEAKER_01 18:03

And from all of us at CX Today, thanks for watching. Goodbye.