Contact Center AI Is Only as Good as the Data Behind It – So Why Are We Ignoring the Data?

Rhys Fisher, Associate Editor at CX Today, sits down with Dave Rennyson, CEO of SuccessKPI, to tackle one of the most pressing yet underexplored challenges in modern contact center operations: what it actually takes to build an AI-ready data foundation.

As AI agents become a fixture of the contact center, Dave makes a compelling case that the real risk isn't the technology, it's the lack of governance, measurement, and rigor underneath it.

If your organization is deploying AI without asking how you'll manage it, this is essential viewing.As contact centers race to deploy AI, most are skipping the hard work that makes it sustainable. Dave Rennyson pulls no punches on what's going wrong and what leaders need to do differently.

AI agents actually produce more data than human agents, including failure signals and turn-taking data that human conversations never generate. The opportunity is significant, but only if you have the architecture to capture and act on it.

"Ground truth" is the step almost everyone skips. Dave breaks down what it means to establish a reliable baseline for AI performance, why it demands real scientific rigor, and what model drift looks like when you ignore it.

Agentic AI is genuinely different from legacy IVR. The removal of linear flow constraints opens up a new design space, but only if organizations build the right orchestration and monitoring layers on top.

Rather than declaring surveys dead, Dave argues that generative AI can now appraise every single conversation at scale, turning a historically biased metric into something far more powerful.

For more Customer Experience tech news visit CX Today.

Transcript

SPEAKER_01 0:08

Hello and welcome to TF Today. I'm Reese Fisher, Associate Editor, and today I'm delighted to be joined by Dave Renneson, the CEO at Success KPI. Dave, thanks for joining me. How are you doing today?

SPEAKER_00 0:22

I'm great, Reese. Thank you for having me here.

SPEAKER_01 0:24

No, absolutely. I'm uh yeah, I'm happy you've joined us. I'm looking forward to the chat. You know, we're going to be talking about what I think is a pretty prominent challenge in the contact center space right now, which is, I guess, data governance in this new era of AI. So I suppose just to kick things off, leading on from that, uh within the contact center, you know, now with AI especially, we see that we're getting a lot more data and we're still finding situations where that data is often siloed across platforms. How does that problem change or maybe potentially even worse than when we add AI agents into the mix?

SPEAKER_00 1:03

Well, I think an AI agent is actually becoming more and more like a human agent in many ways. They're more conversational, able to understand and comprehend more complex terms and conversations, etc. I think ultimately what we've seen with speech analytics and the ability to get into the actual conversation, understand what is being said and what is being felt in human-based conversations, a lot of these technologies are going to be able to be applied to these AI-based conversations quite seamlessly. So in many cases, they're still the same. Now, what's interesting is a human doesn't really eject any data when they don't understand something, right? There's no input, no match, or uh confusion moments in conversations that are recorded. But uh an AI agent actually not only can be listened to, but also can put out sets of data about how it is proceeding and turning, taking in conversations. So in some ways you'll get more data in a more complete picture. In others, it's very much the same.

SPEAKER_01 2:06

Yeah, yeah, that's interesting. That that aspect. I'm not sure many people talk about that, that kind of additional data layer that the AI agents are able to pick up. I was just wondering, obviously, like you said, it's it's far more prominent now, the, you know, the they're pretty much in every contact center now. What in your opinion, what does a kind of a genuinely AI AI ready data architecture look like within a contact center? And how far do you think perhaps most organizations are away from having that?

SPEAKER_00 2:36

I I find that when we walk in and assess AI readiness for almost all of our clients, we find that they haven't even really thought about how to manage and monitor the AI agents. In many cases, there's a excitement about being able to reach a higher level of automation and the potential to be able to achieve business objectives that were almost impossible to do with previous generation IVRs or IVAs. Um, but there's a lack of thought about well, how is it working, why is it working, and is it really working? And so what I've found is we're we're heading into a place where the bullishness and excitement about these new conversational AI tools is maybe outstripped the ability to manage them. I mean, just take an example. If you were putting in a BPO for the first time, right, you you wouldn't all of a sudden outsource 20% of your calls and say, well, that's great, solve the problem, and not wonder if you need to check behind to make sure that those conversations are proceeding as well as they were with your agents on staff.

SPEAKER_01 3:41

Yeah, I think definitely this is AI for AI's sake is something that's coming up a lot in these conversations and kind of trying to almost educate, I guess, organizations on on how to properly introduce those new tools. I guess building off the back of that, you know, when you are building an AI-driven performance management across uh like a hybrid workforce, what is the concept of ground truth and why why is establishing that so often skipped?

SPEAKER_00 4:11

I think it's really hard. Um, you know, I think if you were to take a series of real conversations and transcribe those and then put those through an AI bot and be sure that they were able to handle the similar equivalent situation, you could record the responses and watch the machine perform and see that it can do that task on what I would consider a go-right path uh through the entire um transaction and benchmark that. And it takes a lot of work. You have to transcribe the actual conversation, you have to then synthetically replay that conversation, then you have to observe that conversation being played forward manually. So you're doing a lot of manual work effectively to make sure that the AI is working, right? When you're done though, you know that the AI is competent and capable of handling that task without hallucination. And as you move forward, um, you know, one of the big things we have to remember is the underlying technology is getting better and better. So as the models improve, you have to ensure that you don't have model drift where maybe the agent was great at this task when it was maybe a little less smart than it is three releases later. Does it then try to take on more or overstep and get outside the bounds of expectations? But if you record in a ground truth conversation a completed transaction, and then when you bring out each new release, you test those ground truth transactions against the body of work, you'll know that at least uh you have a certain consistent benchmark. On top of that, then you put a surveillance type of speech analytics and conversation analytics management program in place on top of the platform, and you kind of can get it from both ends. But I I think honestly, the the reason why uh it's not done frequently is because it's hard. And it takes a lot of thought and a lot of work, and you have to, you know, use effectively the scientific method to go through all of these steps in order to make things safe. Uh but the the cost of not getting this right is is significant. Um, you know, and and it's important work, but it is hard, and I think that's why a lot of people don't get to it.

SPEAKER_01 6:28

Yeah, yeah, it makes a lot of sense, perhaps a little bit. Treatment has a little bit of a shortcut, I guess. Trying to save save time now, but like you say, costs them further down the line. Um I wanted to mention, you know, and I I said this from the start, we're in this kind of what I refer to as a kind of the era of AI, but obviously AI has been you know around the contact center for a long time and automation more generally, you know, we've had uh IVRs, chatbots, virtual assistants. And the results have been I I would say mixed. I think that would probably be fair to say. What is, in your opinion, genuinely kind of different this time around when it comes to agentic AI and what are the perhaps the conditions or the parameters that make it really deliver?

SPEAKER_00 7:15

I think the biggest difference is the linear flow is no longer required, which opens up a lot of design constraints. You know, the voice medium is in general very hard to work with. It's one way in direction, it's difficult to rewind. If you're looking at a web user interface, you can see all over the screen, you can pause, you can scan, you can think, you can pause. But when a voice interface is coming at you linearly, it's hard to go backwards without creating confusion in the conversation. Um, as humans, especially in face-to-face conversation, even digitally like we're having now, it is very easy for people to take verbal and nonverbal clues to make these conversations more fluid and decide when to interject, when to interrupt, when to nod, right, and when to have these conversations proceed. When you're dealing with voice, it's a very constrained domain. And the fact that these conversational AI in this generation are more resilient to less linear flows opens up the design space a bit more. You know, just take an example in an IVR. I think many of you know I ran a SAS IVR, an ACD early CCAS player many years ago. Um, you know, we had to have VUI designers that were building a flow to get five or six tasks done in series. And if any one of them broke, the entire flow would have problems. Now, if you know you need to complete these five tasks, the conversational AI bot can really do them in any order. So you need an orchestration layer that is aware that it needs to pick up, let's say, your credit card number and expiration date, order quantity, order type, product category, et cetera, but not need to do those in any sensible order if it doesn't need to. And even if the customer doesn't have their credit card handy, we'll be resilient enough to say it seems like you're having trouble. Do you need a few more minutes to get the card and kind of almost surmise what's happening there based on an even unexpected response on, oh my gosh, I don't have my credit card here, hold on a second. And the bot be able to respond to a completely what I would call unexpected in the previous generation, no match condition, um, and globally be able to have a set of handlers designed to deal with resilience in human conversation. So there's a lot of difference in how this next generation can work. That said, will you perceive that as a positive conversation? Will it have a great sentiment? The variances of interaction with these bots will be nearly the same in complexity and variance as what we have with humans. And so we will need to learn how to gather feedback about these interactions from a third-party observer point of view in order to ensure that we are providing the level of customer experience and automation that we hope to achieve.

SPEAKER_01 10:06

Yeah, that's really interesting. How do you think companies can go about kind of get harvesting that feedback that you mentioned there?

SPEAKER_00 10:16

Yeah, it's funny. I saw an article that said that uh the CSAT survey is dead, and another that NPS is dying, and everyone's predicting the demise of firms like Medallia and Qualtrics. You know, ultimately, the net promoter score, um, which came out of a uh uh an article in a Harvard Business Review study that basically showed that customers that perceive companies to be over a certain level of net promoter score tend to grow more quickly, right? I think there's a Fortune magazine article or Fortune magazine article that it's the only number you need to grow. And it caught uh you know wildfire because, like, well, if we if we can simplify surveys down to the simple bit. But there is some purity in the essence. Um, whether you're talking to um a team member on in your team of just saying, hey, what's the one thing we could do to make things better around here, right? What's the one thing we could do to improve? That type of feedback is extremely helpful because often there is only one or two major things that need to get work done. But I would imagine a survey, you know, much more simply, uh appraising an MPS score universally and you know, letting a generative AI bot sit in the persona of the human consumer and reuse that concept, but do it for every phone conversation. The real problem with CSATs and surveys in general is this barbell approach that you end up with the haters and the lovers. But with such scalable technologies at our fingertips, we can appraise any and every conversation consistently every day with bots that don't get tired. And perhaps the the real secret for how to use generative AI is in this appraisal and survey situation.

SPEAKER_01 12:02

Yeah, yeah, it makes a lot of sense. I guess just the the final question I had to a few, Dave, was I suppose putting myself in a position, you know, for a contact center leader who's perhaps you know, they're convinced that AI implementation is the right decision, but uh perhaps they're not sure where to start. What do you think is the first decision or first decisions that they they really need to focus on and really need to get those right to really improve their chances of a successful implementation?

SPEAKER_00 12:33

I would probably start by saying, how good am I in my automated appraisal of my human agents today? Because you're not gonna be able to scalably appraise conversational AI or any uh generative AI agentic system in a manual fashion. You're not gonna be able to keep up. So if I had a strong automated quality management process in place first, that would be my first layer. The second would be to use that auto QM layer to determine what the most likely successful transactions would be for automation. And then I would move from there to turning some of those transactions on in the front end and automating those, and then using my auto QM system to test that I have indeed automated them and not just maybe half automated them and then pass them off to agents, um, but also that I have had a reasonable customer sentiment, customer experience, you know, along the journey of those automated transactions.

SPEAKER_01 13:33

Yeah, thanks a lot, Dave. That's a really kind of a really nice thorough breakdown. I think that's gonna be really, really useful for our audience. And yeah, thank you just more generally for the rest of the chat. I think we've kind of covered some really interesting areas there, which I think don't often get covered in these uh discussions around AI, where quite often discussions are a little bit surface level. So yeah, thank you very much for your time today.

SPEAKER_00 13:56

My pleasure. Thanks for having me here.

SPEAKER_01 13:58

Absolutely. I would also like to just quickly thank our audience for tuning in as well. If you enjoyed this, please do remember to like and subscribe to the channel and head on over to cxtoday.com for more stories like these. Until next time, thanks for watching.