Trust Becomes the KPI: What the EU AI Act Forces CX Teams to Prove

 Trust is moving from a soft signal to a hard CX KPI, and the EU AI Act will accelerate that shift. In this CX Today roundtable, Kane Simms, Nerys Corfield, Dr Scott Allendevaux, and Nick Holme explore how the legislation changes the day-to-day reality of deploying AI in customer journeys.

The panel breaks down what ‘high-risk’ can mean in practice, why organisations need a clear view of where AI is actually in use, and how documentation, transparency, and accountability will increasingly shape buying decisions. They also discuss the operational knock-on effects: tighter governance across CX, legal, and data teams, more structured testing for conversational automation, and clearer rules for when customers must be told they are interacting with a bot.

The takeaway for CX leaders is simple: compliance is not just a legal project. It’s a trust strategy. Teams that can prove control, explain decisions, and show robust oversight will protect customer confidence and reduce risk as AI becomes more embedded in frontline service. 

Transcript

SPEAKER_04 0:07

Hello and welcome to uh CX Today's Round Table, and uh today we are talking about the EU AI Act. Um and we are uh gathered by a great uh gang here today. I'm just going to do a quick uh guest list roll of introductions, um, and I'm gonna work from my screen around the table, so in no particular order, we're joined by Kane Sims, who is the founder and CEO at Booksworld. Thanks for joining us, Kane. Pleasure. Thanks for having me. Uh Naris Caulfield, Director at Injection Consulting. Hi, Naris. Hi. Dr. Scott Allendevo, who is the practice lead law and policy at Allendevoe and Company.

SPEAKER_00 0:52

Hello.

SPEAKER_04 0:53

And Nick Holm, uh, he is the director of growth at Capital. Hi, Nick.

SPEAKER_03 0:59

Hi, great to be here.

SPEAKER_04 1:01

So um we've got a really good mix here of people who can talk around the technology, people who can talk around operations uh and the impact on the kind of people side of stuff and the way it's gonna be this impact of this uh new AI Act. Uh and we also have um uh Scott, you are obviously the um the expert in the room from a legislative perspective, uh regulatory perspective. Um you're gonna keep us honest and true, which is uh is good. Um tell us what we're gonna do. A really nice mix of people with the right kind of uh knowledge sets to kind of paint this picture. Uh and we're gonna be looking at this EU AI app through uh the CX Lab. Um and and and I think the the theme here is pretty much about trust. Um trust is quickly becoming uh a CX KPI, I think, with with everything in mind. So we're gonna cover that. We're gonna look at what uh market shifts look like, we're gonna look at what it means operationally for those CX teams out there, uh, and importantly, uh the customers themselves as well, um, alongside those frontline teams that actually have to deliver that experience to them. So um that's what we're gonna cover. And to get started, I'm gonna kick off um straight to Scott. Question to you. Um just kind of set the scene if you can a little bit. Um, what's the kind of one thing you expect CX leaders uh most need to understand about where this EU AI Act is heading and what it means for them?

SPEAKER_00 2:34

I think it's important for leaders to understand uh what's happening inside their organizations in terms of their intended use of AI or their actual use of AI. I was at the IAPP conference, which is the International Association of Privacy Professionals last week, and um we were being spoken to by a number of regulators and attorneys in the room, kind of giving us a broad overview of what's actually happening uh throughout the worldwide landscape. And while we're focusing on the EU AI Act, they they made a comment that was actually just uh a little bit shocking to hear it, and that was in the US states alone, just in the first three months, over 1,000 AI bills have been introduced into state legislatures. And it's very difficult, I think, for the average cloud services company or UCNC entity to really get a handle on all this movement that's happening all over the world. And so when we talk about the EU AI Act, the one good thing is it's kind of like um the golden piece of legislation right now that was really put together well. Uh, and it was put together in a way that many other pieces of legislation are looking to it. Uh, but what's happening, I think, throughout organizations, and I hear this being echoed by uh even sessions I attended, like with the ICO, where they were talking about how important it is to understand what's actually happening in the back office of organizations, not even understanding or realizing how many different AI tools are being used. Um, I did an exercise about two weeks ago with an organization, and they took an inventory to find out which of the main tools were being used, and they were surprised to find out that 70 different AI tools were being used, and they were being used with customer information to process customer information, and you talk about trust. It's so difficult to establish trust with customers when you don't even know what's happening in your organization. So I think really getting your mind around what we need to do to understand and comply with the EU AI Act is first understanding what's happening in the enterprise with AI that's being used in a shadow way, uh, and what's happening with AI in a planned way.

SPEAKER_04 4:56

Wow, that's a lot. I can't even I can't even name those seven. Kane might be able to name them, but I can't. Um so okay, that that's wow, that's mind-blowing. Uh, and actually, it's it's a really good point around just because it's the EU AI Act, um, it doesn't mean that that's not going to impact other regions, either, like you say, from a best practice piece, but also from if you're gonna do business with Europe, you need to have you need to know what's going on and you need to be aligning with it. So it's a good shout to call that out. Um, so so just moving over that from a kind of looking at the the delivery uh perspective, Nick, um from a supplier or an ecosystem uh angle, what changes first this year? How how are programmes going to be planned? Is that gonna be like where this changes, or is it more around governance or measurements, that kind of thing in the operational side?

SPEAKER_03 5:47

Yeah, and I think it it for me it's kind of in line with what Scott's just been talking about. So the governance aspect I would say is going to be the biggest thing that's gonna be prioritised because it's it's forcing the BPOs out there, the providers, to really put that in the forefront of the conversations to give confidence that whatever we're delivering or selling or so forth is has that governance piece behind it and it's been considered right from the get-go. So the risks are in line with what the act is expecting, and there's a true understanding. So I I would say on a on a governance perspective, that's going to be the biggest noticeable change in the conversations. It's now not going to be something that's talked about down the line once the idea of AI is discussed, it's now going to be at the forefront of conversations for me.

SPEAKER_04 6:31

Do you see that happening already? Is it starting to creep in or is it a bit too early for that yet?

SPEAKER_03 6:36

More more for us, yes. Um, more that's because the the approach that we're taking through growth is to do that initial open conversation about um you know what what is it that you're looking to change as opposed to what product are you looking to deliver. Um so those conversations are naturally happening earlier down the line, but you know, I could probably get in hot water for saying this, but I don't think that's a um uh a topic that's said broadly across the industry as yet. It's still, you know, you only have to go on LinkedIn, you see that the product is still being sold first rather than the full picture of the governance piece around it as well. But I think businesses now will be asking those questions right from day one.

SPEAKER_02 7:16

Do you see that translated into the enterprise as well? On the one hand, you've got kind of like the from the sales side and the vendor side, it's product, product, product, isn't it? Great AI, AI, AI. I also see the same happening from the enterprise side, which is just build, build, build, rather than what are we trying to achieve, how is the best way to achieve it, how are we going to measure that we've actually achieved what we aim to achieve and then how we're gonna kind of operationalize this over time? It's like this tool's cool, let's build something now, you know. So it's kind of like what what the AI Act does, I think, a little bit for me is like if you read it, it it's a bit like it reminds me a bit of the you remember when accessibility kind of came in in the EU and it was mandatory, a legal requirement that your website has to meet certain accessibility standards. And it turns out that actually by building an accessible website, you don't just make it accessible for those that need those accessibility requirements and features, you actually make it better for everyone. And that's kind of the same thing here, I think, is it the EOI act that you read through it and you think, oh my god, that's gonna be hard. Transparency, what you mean we've got to have traceability across the whole stack, like what vendor are we gonna use is gonna give us that? And it's like it's a this the little things in there that are quite challenging. But if you think about it and you you go through it, you think you actually think actually, well, doing that will make our solution have more integrity, it'll give us more control, it allows us to do things better and more more kind of robustly. So it's yeah, it reminds me a little bit of that accessibility sort of initiative, which is like it's all kind of good practice, but some of it is going to be more difficult to achieve than other parts of it, I think.

SPEAKER_01 8:49

That's so true, and I think that's a great example when you know speak to vendors about WCAG2, which is that accessibility requirement. Some will have really clear um clarity, they will have real clarity on how their solution addresses that um requirement, and then others not so much. So I think that's a really pretty with HECA, it's the same with PCI, it's the same with Offcom. All of these regulations then are integrated and consumed by the businesses to a greater or lesser extent, and it's the practical application. Practical application that is the really important thing. But you're right, Kane. When you really look at the foundations, you're like, this makes absolute sense. Why wouldn't you do this? Creates transparency, robustness, and all that. But the reality is with all of those regulate regulations, enterprises are on different sides of the scale in terms of their interpretation and then their deployment.

SPEAKER_02 9:47

You know, there's certain parts of it that you could argue add more value than others. Yeah, if you're getting bogged down in bits that seem to be a bit more of a regulatory tick box that don't necessarily contribute great value to the end user, you're in in the realms potentially of you know, a lot of resources required to do stuff that hasn't it meets a regulatory requirement but doesn't necessarily add to the user experience, so to speak. So it's kind of a yeah, I don't know. It's yeah.

SPEAKER_01 10:13

Yeah, and it's taking it to the nth degree. It's like when you ring those organizations, particularly in the FCA, and they'll say in the financial services industry, and they'll say, press one if you don't want your call recorded. That's taking the legal interpretation to an extreme level, that's adding clunkiness to the customer experience. It's not necessary, but the DPO has determined no, this is very necessary because that's my interpretation of GDPR and what needs to be done to adhere to that law. And it's like, well, if that were the case, everybody would be doing it, and that they aren't, and they've all made their own determination, so yeah.

SPEAKER_04 10:49

It actually brings me back that um uh analogy to a period of time where I was running voice of customer programs, um, and when GDPR came out, I was working for an organization who basically said they were no longer going to run their voice of customer uh surveys and questions because they felt they didn't have the right to reach out to their customers. A consultant had told them that. And it's just kind of like it's the it's dangerous like interpretation of these things that can be a can be critical, right? Yeah, exactly. Just on on your point, Ken Grant, just if we if we kind of go to look at vendors again, AI vendors, but specifically conversational AI and those platforms. Um for our audience's benefit, can you kind of just kind of lay out what what's going to be really kind of important here? What's kind of like must-have table stakes as this is really key, versus what is going to disappear and it's just gonna be the noise that we're hearing a little bit about now.

SPEAKER_02 11:46

Yeah. So I suppose it depends on on the vendors. There's some vendors who have a kind of, I suppose, a bit of a quick and dirty approach to getting clients up and running, which is important because you know, in the NLU days before generative AI, it would take you six to eight months to build an AI agent. Now generative AI brings that cycle time right down. Um, but the challenge is that it's easier to build a prototype, it's a lot harder to get to production and have all the controls in place that need to be in place for it to be reliable at scale. And one of the things that many vendors do is they'll do things like scrape the internet for company data and they'll build your first initial kind of rag agent off the back of that. They'll use your website, publicly available data, all that kind of stuff. Um and one of the kind of parts of the act is that essentially you can't have any bias in the in the data that's used behind these models, it's all got to be kind of accurate and and all that kind of stuff. There's a lot more requirements around the governance of the data that you're using, either to train models or sitting behind the models. And so the days of just give us your data and we'll build a bot from it, I think those are gonna be long gone. Which, to be honest, if you again, if you follow the proper process, if you assess your user needs, if you then curate the right content and rewrite it and re-architect where it's where it's needed to fill the gaps and stuff like that, and curate the data that you need to use for your agent, and then build your agent, you're gonna cover a lot of the act anyway. It's just the laziness that gets in that that it kind of comes up because people want to get there faster, they'll just take any data that they want to. So some of it's also down to the it's not so much the vendors, although some of the vendors will do this because some of the vendors will do the implementation for their clients, but a lot of it is also down to the the design and the implementation of the agent as well. So things like you know, you've got to have transparency over whether you're talking to an AI or not. That's one of the kind of things that's absolutely critical. Now, some vendors do not do that and are quite principled about about not doing that. They'll essentially just you'll start a conversation with their agent, it'll just start talking, it'll sound really, really, really realistic. And maybe you finish the conversation not having even realized that you're talking to an AI. That that will change. You're gonna have to declare that you are speaking to an AI. But the thing that I think will be required for many vendors, that some of them do, some of them don't, is that whole sort of transparency piece around visibility in the stack. If your AI agent makes a decision or gives a customer a response or an answer that is incorrect or the wrong uses the wrong data, or uses you know, somebody else's data, or whatever the case may be, you're gonna have to be able to go and figure out exactly why that happened and how you can put controls in place to prevent that happening again. And some of the vendors at the moment are a bit of a black box. You write a prompt and everything else is taken care of, and that's not necessarily going to be um good enough, certainly for high-risk use cases.

SPEAKER_04 14:35

So on that, Kane, um, if we look at uh AI vendors uh that we're talking about there, and most importantly conversational AI vendors and platforms, what becomes the the kind of really key important table stakes that are going to have to remain? And what do you expect might kind of quietly disappear?

SPEAKER_02 14:56

So the the things that need to um remain or improve is mostly I would say down to not a technology uh problem but more of a design problem. So, for example, you know, a lot of vendors in order to get up and running quickly will just ingest content. So they'll just you know ask you to send them all of your content, they'll scrape your website, they'll scrape publicly available information on the internet, and they'll ingest all of that and they'll build you your first agent. And the value of that is it you will have an agent built in a week rather than half a year as it used to take. But the problem with that is that you're not vetting that data means that who knows what's in it, who knows what kind of biases are in it, who knows whether it's accurate and up to date and all that kind of stuff. And the EUII Act has a very strong kind of mandate in there to make sure that when you're training an AI model or you're ingesting data into an AI model, you're doing so in a way that it doesn't include any bias, that the information you can vouch for, and then there's a whole bunch of kind of governance requirements around the data that you're using. So getting up and running quick and dirty is not going to be something that you're gonna be able to do. Now, that's not the best practices anyway. It's like I was saying before about most of this stuff is just the best practices that you should be doing. So if you're gonna build retrieval augmented generation systems, you need to assess your user needs, you need to find the data that marries to those needs, you need to re-architect it and rewrite it so it's in the right kind of format to make light work for the models and all that kind of stuff. So all the stuff that you should be doing, you can still do. It's just that sometimes vendors want to get to market quickly, so they'll try and cut a few corners. That won't necessarily be doable anymore. The other thing is around being transparent, so you're gonna need to now declare that you are an AI or the user is speaking to an AI agent. And there's a lot of vendors out there today that don't do that, and they kind of pride themselves a little bit on not doing that because their assistants are so good that it can convince people it's a human and all that kind of stuff. That will need to change. Um, the the other thing is around kind of like things like human escalations and human in the loop. Like there needs to be a human, at least human oversight, and also the ability for humans to intervene. So when you design an agent, you're gonna have to have the option for human escalation, certainly in high, high kind of risk use cases. So it's things like this, and and the big thing that that will need to change from the vendor standpoint is that you're now gonna need to have visibility into what decisions your solution is making and why, what data it's using, and if it's got something wrong, why has it got it wrong? So, what some of the vendors have done over the last few years in order to make things really accessible and easy to use is essentially they they I suppose hide all of the complexity of the tech stack and the front end is basically just a prompt. Now, the problem with that is that you've got no visibility into any of that stuff I'd just mentioned. So those really accessible, really easy to use tools that are starting to really grow will become quite a problem in terms of adopting it at scale for high-risk use cases because you have no visibility into what's going on under the hood. So that might even cause some difficulty in some vendors being able to sell in the EU. Um, but most of it for me, you know, it's it's a case of good practice, start right, don't ship it and fix it later, start right, adopt best practices, focus on building the right kind of thing and making sure when you do build it, you've got visibility into your into your stack. And so if vendors can support you in that journey, as I'm sure many of them can, then that's a thing to be looking for. Stay away from the black boxes that are a bit sort of prompt and pray type of systems, you know.

SPEAKER_04 18:32

That's uh it's really interesting though when you talk around the need to be able to tell make sure people know that it's AI that's handling the conversation because not so long, it doesn't feel like that long ago that everyone could say, oh, you can tell it's AI. It's not like it's dead obvious, and there's so many kind of things that you can spot that tells you when you're dealing with AI. And now we're getting to the point where it's transitioned and improved to the point where we're actually going to have to kind of tell people just so you know this is AI. That that just shows how fast and significantly it's improved.

SPEAKER_02 19:03

Well, yeah, I can share one one actual real life kind of um story of this in practice, basically. And we didn't really think about meeting the AI Act, it was again just good practice, which is we did we recently did a pilot with a debt collection company, it was an outbound voice AI agent that calls people that have fallen behind on their payments, and it's just to check if everything's alright and see if they can make a payment and stuff like that. Now, obviously, it needs to escalate to an agent, so we had that feature in there, and it introduces itself as an AI assistant. But the problem is that on a voice channel, people's short-term memory is very short. So even though you say, Hey, I'm an AI agent, I'm calling on behalf of so-and-so-and-so, that doesn't necessarily sink in. Maybe the company name that we're calling from is the thing that syncs in. So, in this one particular case, um, I was preparing for a talk that I'm giving next week, so I was going through all the recordings, and in this one case, we got through the authentication piece, confirmed the person's name, you know, address, all that kind of stuff, got them through the authentication bit, retrieved all their case details, told them how much they'd missed the payment, asked them if they can start making a payment and get it back on track, etc. This was about a minute and a half into the conversation, and the user then said, Is this automated? And the agent said, Yeah, yeah, I'm a I'm an AI agent, I'm calling on behalf of so-and-so. I'm I'm here to try and help you get your things back on track. And as soon as he found that out, he said, Can I just speak to an agent? And sure enough, the bot said, Yeah, absolutely, yeah, no problem. I'll put you through straight away, and it put them through to an agent. And so it's just interesting how you're right, even though you declare that you're I'm an agent calling, it didn't really sink into the user. They got you know a good way through the call before they clocked on, hold on, is this is this automated? And then when they realised it was automated, it was like, Oh no, I don't want to talk to this anymore, I'm gonna get out.

SPEAKER_01 20:41

And that in and of itself is quite interesting, isn't it? Our assumption is that in terms of transparency, it's really the job of the marketeers to think about what how they're going to create a persona around their bot and make it really evident that it's a bot both on the voice channel and on the digital channels, because we're seeing bots sending emails out now within seconds of you sending an email. So there really needs to be, and and using the right language as well that we know consumers understand because saying this is AI doesn't necessarily resonate with the wider public just because we totally get what that means. There's an example there, the language being used by is this automated, whereas so I think you've got to really carefully work with your marketing teams around that as well. What are you going to create, your bot persona, and how are you going to make sure that the consumers really truly understand and get that transparency box is ticked?

SPEAKER_04 21:35

Just to flip that over as well, Naris, looking at it from the buyer's perspective from their side, how is how is that changing things in what kind of contact centre leaders are looking for when it comes to RFPs or or demos with vendors and that sort of thing?

SPEAKER_01 21:50

Well, funny enough, I was I was primed to be able to say the real the practical reality is not an awful lot. There's starting to be some tentative questions. Certainly around sort of you know the data side of things and data sovereignty and data residency, but in terms of that explainability around the black box decisioning, that hasn't I haven't really seen that come through in RFPs, and maybe I'm just not reading the right, you know, RFPs. I'm sure there are some out there. But this morning, um, a financial services organisation was asking lots of questions about an AI um in terms of is it defensible for the advisors? Because I think one of the things that is high risk is auto QA. So auto QA, particularly with no human intervention, which I definitely know some customers, despite my consultation, are going, no, no, no, we don't need we don't need the QMs anymore. We'll just bot can do it all and send it out. Well, as we all know, there's things in there which could be determined as sentiment um grading, like rapport, empathy, and those things which are a bit nuanced. That is high risk. That is not um going, you know, that is not allowed. So I'm sorry to answer the question. I'm starting to see customers dig in a little bit more around understanding um, yeah, what's going on? They don't really have the language to ask the right questions in the RFP yet, but they're starting to tentatively dig in that area. And I guess I work with a lot of UK, so we're not totally there on the EU I that even though we know we're going to align to it by and large.

SPEAKER_02 23:36

Do you think you'll see Neris? Because do you remember when like the when GDPR came around? And I I was doing a lot of work with government at that time, and they they actually appointed in in in various government organizations, they appointed GDPR people or a person in the business whose responsibility is to make sure that everything's GDPR compliant. Do you think we'll see the same thing as far as the AI Act is concerned? Someone whose responsibility is to align the business around that.

SPEAKER_01 24:01

Well, I think until it's really my honest view is, and it's interesting, obviously, in terms of the ICO, who will, who is going to go to govern it? Um, because I think until the clay GDPR dictated that you definitely needed somebody to govern it and understand it. I suspect that the product managers within the organization would be the right people to do it. Um but until the fines are clear, then the investment in that role and resource to make sure that the audit all AI tools are auditable, all AI tools have that transparency and explainability, I suspect I I barely see, I barely speak to many DPOs, right? Even though they are supposed to have them if they're at this certain criteria.

SPEAKER_00 24:45

My perspective is while it's possible that we might see a separate role uh be allocated in enterprises for responsibility of AI processing and AI compliance. I'm watching the market and I hear discussions about expanding the DPO's role, expanding their competency, expanding their understanding, and expanding their remit, uh, because a DPO must have uh many of the backgrounds that's required, such as they have to be an expert uh in legal matters, uh, they have to understand uh personal data and the processing of personal data under the GDPR. And so I'm I'm watching, and I'll be surprised if we don't see an expansion of the DPO's role as having this remat responsibility within organizations as well.

SPEAKER_01 25:32

Yeah, that's interesting. I just think in practical reality, it opens them up into the vendor landscape and perhaps conversations that they have haven't been sort of uh embarked on. But yeah, there's a learning curve there, isn't there? It makes sense that it sits with the DPO because it's part of that regulatory compliance.

SPEAKER_00 25:51

I have another perspective too. We talk about AI being a black box. And if you think about it, I I don't know any other technology that would be as much of a black box as AI. It's it certainly looks like a black box to users and to uh uh purchasers that are that are using AI at companies. I was sitting in a session uh and the speaker was uh the ICO commissioner, and and he was talking. And he was talking about the EU AI Act and the importance of it, but he also pointed out to us about the GDPR in Article 35 of the GDPR, and he said if there was ever a relevant article, it would be Article 35 of the GDPR. Because whenever you hear about data being processed by AI, that goes in effect and it should be highlighted because the idea of Article 35, which is to conduct a data protection impact assessment, is to ensure it's not a black box. It causes, like you're saying, the product development group needs to actually architect the data flows and show information entering a system, going to other systems, being defined, being inventoried, the algorithms that are being used that needs to be determined, and all the different processes of training data, if biases are being measured, if discrimination is coming out of the model, even pen tests that are conducted. And I'm not talking technical pen tests, I'm talking AI pen tests, where you ask the AI system questions and then you measure the ethicalness of what you're getting back from the AI system to see if it passes the pen test according to the boundaries that are set. So I hear us talking about black boxes, and it's very real. And I think one of the big challenges that organizations are going to have, and they have a statutory responsibility to do it, is to bring about that transparency. And people should be able to ask an AI supplier to supply and provide them their DPIA, which shows what's really happening behind the scenes.

SPEAKER_04 27:53

It's interesting that we've we've gone into talking around the GDPR a lot with this conversation, and there are a lot of parallels, I think, in terms of that and this new act. Another one would be when you think about GDPR, you have different roles in terms of responsibilities as a who's an owner of the data, who's a processor of the data, and the the many different ways of kind of splicing those up. That's the same here, right? It's it's a little the accountability, uh, Scott, is a little fuzzy, right? Vendor, deployer, business owner, who who owns what, who should be responsible for what?

SPEAKER_00 28:29

Absolutely. Like in the GDPR, you'll have controllers and processors and subprocessors. While there's that idea, there's different terminology that's often used in AI. And I think a very good publication because we talk about people don't know the language in in companies to even talk about AI. I like to use ISO 22989, which is all about what is the vocabulary of AI? What are the words so that we can communicate with each other? What are the definitions of those words that we are using and what is the context of the words that we are using so that we can talk? That way, when we when we say things like uh transparency, we have we have an AI definition or the standards uh definition with that, uh, or we say robustness or reliability or resiliency or controllability or explainability, all of these things are defined. And especially if you're going to implement an AI management system, then it's important to have the definition of those words. But when we talk about the roles, like you just asked, there is the AI provider, and we don't have enough time today to really get into this, but just as a high level, uh, there's the AI provider, there's the AI producer, there's the the AI customer, there's the AI partner, and then there's the AI subject. Um, and those are all very specifically defined terms, and we need to have an understanding of who they all are, how they interact with each other uh in this new uh AI conversation.

SPEAKER_04 30:02

It sounds like a rabbit hole that needs to go down, but we're just not gonna be able to do that today. We might need to do another thing on that uh uh in another call. But um, Kane, just listening to that, if you were if you put your like CX leader's hat on today and you were going, right, we need to move fast, but we've got to stay safe with all of this stuff to consider. Uh trying to make it a bit more simple for those guys, what are the kind of non-negotiables that they need to start with before they start thinking about scaling things up?

SPEAKER_02 30:33

You've got to assess where where are you exposed, essentially, currently. So a lot of companies have had AI solutions in place for a long time, machine learning solutions, generative AI solutions now starting to come kind of to the fore. And I think that if you've got use cases that are, you know, concerned with finances, if you're financial services, debt collection, stuff like that, or it's concerned with people's kind of health and well-being, healthcare insurance, and you're dealing with systems that either make predictions or decisions that are going to affect people's lives, those are the sort of use cases that you want to kind of assess and figure out, okay, are we following the things that we need to follow? Do we have the right infrastructure, the right processes in place? If you don't have anything and you want to get started, again, it's not necessarily a case of following the act, it's a case of best practices. What you want, what you don't want to do is start by trying to build the biggest, most complex, most high-risk solution imaginable. You want to start by finding something that's low risk, low impact, fairly straightforward, you know, minimal kind of um risk exposure and start there because that's gonna do a number of things for you. One, it's gonna allow you to get to market and get things live that ultimately are not gonna have a big impact on your customers and your business's reputation. But crucially, at the same time as doing that, it allows you to build the capabilities that you're gonna need to scale. It allows you to start thinking about stuff like testing. How do we test this solution? You know, Scott, you reference one type of testing, which is essentially are the answers ethical, but there's a whole suite of testing required. Not only are the answers ethical, are they accurate, are they consistent, are they reliable, um, are they contextual? And then in the terms of a conversational AI, when you've got a back and forth exchange with a customer, uh is the user even achieving their outcome? Is the system really ambiguous? You know, there's a whole bunch of stuff that needs to be done, not just to test that you're kind of like being transparent and being accurate and all that kind of stuff, but also are you actually serving the end user, you know? So I think that's one of the things that absolutely you should that you should do is start with something that's that's less risky. The other thing is, you know, again it's just best practice, but you've always got to have an out. One of the misconceptions of these generative AI systems is that because they've got such an enhanced language understanding capability, that they're gonna solve everyone's problem. But that's not necessarily the case. And in fact, in fact, if you were to diligently design your solution, which you should do, you've got to identify places where you don't want the AI to be involved, you want to get this through to a human straight away, and that might even be a subsection of one use case. For example, let's say that you have a simple use case, which is a change of address use case. You might think, well, I can automate that 95% of the time. But maybe, you know, let's say that you're a debt collection company and one person has changed their address four times in the last three weeks, that might be a vulnerability signal, therefore, you might need to escalate to an agent. So it's about really diligently designing where do you want people to be involved and also always give the user an out. If they say, I want to speak to an agent, don't get in the way, give them, put them through to an agent. Um, so it's things like that, really. Again, for me, it's all just kind of best practice. And and lastly, I suppose once you've done that level of testing, is when in production you still need to keep that monitoring going, having flags for things like hallucinations and things like that. Uh, a lot of companies and teams do this manually right now. The tooling on this is a little bit immature, but I think we'll see tools. We're gonna see loads of tools sprout up just to support uh a lot of this stuff, I think, over the next over the next few years.

SPEAKER_04 34:02

That's a good point. Yeah, and that often is the case. There's a need for something that we find ways of uh fulfilling the need. Um but just listening to all of that, and I guess that's kind of stuff that you're familiar with. Um what what though it's all very possible, all this stuff that we need to be doing, but what is actually the kind of harder things to operationalise when it when it comes to the real world? So is it the documentation side of it? Is it the monitoring or the production? What what about human oversight? Because that's important uh as well. So is it all of the above or is there anything that we should be prioritizing?

SPEAKER_03 34:39

Yeah, and I think in the interest of not upsetting anyone who's accountable for those different aspects of this from documentation and so forth, then you know I think each state, I mean we talked about already in terms of the role of the DPO and so forth, everyone will have their challenges of trying to be in line with what this act will bring in play. But you know, from my position, you know, I'll definitely naturally lean towards the um the you know meaningful human oversight because at the moment, rightly or wrongly, I think there is a trust in the the AI, and we need to change that behaviour to be more involved, not so much human loop, but human very much in that middle piece that is in control, very much alluding to what Kane was was talking about. So I think that the you know it's going to be an evolving piece as people adapt their behaviours around it, how businesses approach AI, how the agents use their AI from um the agent assist tools that are available now, even down to what Naris was talking about with Auto QA, you know, how that's utilized, making sure there is strong human intervention in that to make sure the right things are being done. So, you know, from my perspective, I think that the you know genuine human in oversight is going to be the biggest challenge to change those behaviours, but I don't think we can rule out any aspect of what's already been discussed to this is not important and a challenge for the individuals involved in it.

SPEAKER_02 35:59

And again, the the human oversight, I'd be interested to get your perspectives on on this, Nick. My my kind of theory around around this is that human in the loop is a short-term consideration because of the probabilistic nature of the technology. Like nobody can guarantee that a generative AI model is going to do the same thing every single time, no matter what vendor claims what. So I'm not sure and and and uh thankfully it's in it's in the act because uh again, best practices is that you want humans in the loop to be able to make sure that everything is, you know, going as best as it can be. But my my kind of like you know, I don't know, devil's advocate kind of hat is like is that only because we don't trust the technology? We talk about trust a lot, and I think sometimes the cut the the cut the the narrative around trust is that the end user has to trust the system. But the reality is that the end user really couldn't care less and doesn't really know anything about it. Like if you were to if I was to ask my mum, who would be a prime candidate to use a voice bot if she called up Asda to get a refund and a voice bot answered, she would probably just use it because what other choices she got. But if I was to ask my mum, hey mum, what's a hallucination? She'd probably be thinking it's something that happens as a side effect of taking LSD. You know, so like she hasn't got a clue what it is. It's not that she doesn't trust the system. For me, the concept of trust is about the business having trust in the solutions it is creating so that it can guarantee as best as it can that it's doing the thing that it should do all the time. And the human in the loop thing, it's not an opinion I've got, it's just a question of whether this might be the case, is that the human in the loop might be to satisfy our lack of trust in the technology.

SPEAKER_00 37:41

Yeah, I think so. I think so. I think the outcome of that is really to bring about um when when you talk about the HITO, the human in the loop um perspective, they're looking for non-maleficence, of course, uh to make sure that there's no harm to people and to societies. They're looking for beneficence. Like, is this really benefiting people uh by having this agentic agent uh you know uh in a system that a person is using because it's going to be more efficient uh to be able to talk to an AI agent and sit on the line and wait for a very long time for some agent to answer the phone. And so we talk about these other things about discrimination and bias and monitoring all those, but I think all those boil up to making sure that there's no non-malfeasance and that we actually have a state of beneficence for our users.

SPEAKER_01 38:31

Yeah, I I guess it's interesting, Kane when you started talking about your mum there and then just o overlaying um Yeah, the fact what I was thinking about when I think about users and and where the act is really focused on is on the workplace users as well. Um yes, there's the transparency about the box and the explainability um of you know what's happening and the processing decisions. But for the employees who are now, you know, there's I'm I'm spending so much time on auto QM discussions, demos, um, proof of concept and that side of things. But it it's in that that they that that those those QM tools cannot now suddenly become the manager of those resources, which impacts their bonuses every at the end of every month, every quarter. It it it will have a material impact on their employability and where they are on the score, you know, on the score against their peers. So you you you can't leave that to a bot in and of itself. So that's not a beneficiary requirement. That's a you you can't ask AI to suddenly start managing and determining whether employees are doing the right or wrong thing. You have to have human intervention. That's my take on it.

SPEAKER_02 39:53

And is is that is that a short-term requirement, do you think? Or do you think that that's kind of the question again? It's not an opinion, it's just a question is is it because the technology is not good enough yet, or that we it's not reliable enough yet? And at some point maybe it will be, maybe at some point it will be good enough to have completely automated WA.

SPEAKER_00 40:12

I don't know. I I do think a hundred years from now, will that still be as important when all kinds of things are ironed out? Uh and I think uh it's uh should always be about us. So maybe it should be, but we'll have to see if it is.

SPEAKER_04 40:27

I think there's also a bit of a hangover, a legacy issue from what is not uh generative AI, but uh the box of old that basically uh didn't do a great job and just created a lot more frustration in terms of the customer's experience and therefore there's a PR piece that hasn't happened yet in terms of what the art of the possible is with because it we've talked about it already, but this technology is moving so fast and improving so fast that someone who like 18 months ago was you know really kind of put off using bots because of uh any amount of uh companies having a really poor uh performing bot. And we've all heard the horror stories of you know swearing and telling people to what you know, there's a list of those. Um that whole piece has impacted people's perception of of what a bot is, and they don't they don't get the engine behind it, so they don't know that this new generative AI that sits behind the new bots is is far more advanced. So there's there's there's a possibly a gap there as well. Um so that kind of PR piece needs to happen, or or we just need to prove that it is better now.

SPEAKER_01 41:38

But whether it is or whether it isn't, the fact is irregulatory rise now, it's not optional. You have to have a human in the loop, and that's what the new EOI Act dictates. That you have to have a human in the loop, whether that's like you have to have a get out, or you can't have everything determined as being contained within automation, which is interesting when you think about what you do out of hours and overnight, and you're putting bots in to service um customers that you actually weren't able to service before, so you're extending your service for them. But actually, now what does that look like? And certainly, certainly, you know, we've all well, certainly me, um Rob and Nick have spent a lot of time doing QM scorecard, um, sitting side by side with advisors. It has to be defensible, it has to be um, you know, uh contestable, they have to be able to contest it and say, well, what's the decision? When you do a side-by-side of QM versus human monitoring against the scorecard, particularly when there's inferences about mood, um, which is you know what the act bans, you will get very similar. Bots would perform very similar to the way the human was, and not I sometimes argue actually there's less bias because that advisor turned up 15 minutes late and they've really annoyed me because they're walking around with you know the wrong shoes on or whatever. Um so my bias is like, oh, I'm gonna score them down. The bot doesn't have that, but the reality is it doesn't matter what what feels right or wrong, the rather is the EUI Act dictates that you have to have a human in the loop and you can't pretend that your AI is there to manage their quality, and certainly when it comes to determining rapport building, authenticity of connection, soft skills, that side of things.

SPEAKER_02 43:25

Yeah, where do you think or or what do you think the impact of of this then is on innovation? Because if I I think about like yeah, this is one of those moments when I realise I might be old, because I think about the kids of today uh who are who who don't have the baggage that we have, who don't have the same concept of quality that we have, which is everything has to be right, that that are kind of unshackled by previous experience, that just see these tools and think, oh my god, I can build an automated sales manager, I can build an automated marketing manager, I can I can do all this kind of stuff. And maybe for that generation. They're quite okay with ninety-five percent of it's gonna be great, five percent of it's gonna be rubbish. But if I can reach a hundred thousand, two million, a hundred million, a billion people, and ninety-five percent of them are gonna have a great experience, maybe we just have to deal with the five percent. And so it's kind of uh I don't know how much you think that and again this isn't an opinion, it's just a question, which is like will something like this end up getting in the way if and when new innovations come around, or if and when the technology gets better that perhaps places less when when we get more trust in it places less reliance on people needing to have human in loop and stuff like that. I just wonder whether you think it, yeah, do it's I I I completely agree with everything that's been said so far, which is that right now this is best practice and this is a good idea. But in ten years' time, if this act doesn't move along with it, is it gonna end up actually causing problems?

SPEAKER_00 44:51

I think there's an argument um, you know, behind the law is that AI will be more sustainable and more trusted if it's developed with cleared clear guardrails. And I think businesses may find compliance demanding, um, but I think the alternative is an environment where powerful AI systems are deployed without enough oversight, and that could ultimately undermine trust in the technology itself. So yeah, I would say the act is best understood as not attempting to stop AI, but as an attempt just to make it governable.

SPEAKER_01 45:26

Yeah, a defensible.

SPEAKER_04 45:28

Yeah. I I I want to move into some positive stuff because you've touched there, Kane, on actually there's that that there is the potential for things to do things really well and for AI to actually do have some really positive impact. So, Nick, you are kind of in the operational side of things. Have you seen and can you share examples of actually where AI is having a great uh improvement in terms of providing better experiences rather than just where it's creating friction? Talk about friction if you want to as well. Um, but I want to kind of understand what positives are we seeing out there at the moment as well.

SPEAKER_03 46:06

Yeah, and I I think it's it's well publicised in terms of the impact AI is having in terms of improving you know the classic KPIs of AHT reductions, um, first-time resolution improvements, either through um enabling agents with um auto summaries and so forth, though those aspects of AI, even down to more personal experiences where you know we can put in previous customer information, give prompts in terms of what's best to suit this customer validation flagging and so forth to improve that that adherence to support more of your customer demographic and support them. I think if you go into the lines of where does the challenges come, I would from a personal standpoint is when AI involvement or delivery is rushed and not really considered. I spoke about it on a post last week where you know there's facts going around as of saying that 20% of AI delivery is it is successful, 80% fails. And rightly some of the comments were AI typically fails when it's not been considered with true due diligence in terms of the impact it will have across the wider business, not just on the bottom line, but in terms of the impact to your customer demographic, your agents that will be using it, your managers that need to manage alongside it, very much touching what Neris said on on QA. Um, all these things that AI comes in, you know, it can't be rushed in my opinion. It can be very tempting when you see a shiny chatbot or or uh something that's going to improve uh your speed to service customers. But um, for me, that the greatest way of success is when we sat down with clients in a room and said, you know, what is the the outcomes you're looking to achieve, and very much not leading with product, but leading with you know understanding what the challenges are and then try and build a solution around that. That I would say is as much as it's more time consuming, it's a much more successful way to use AI because AI, the AI typically doesn't fail, it's the way that it's been delivered, I would say, is the uh the biggest challenge.

SPEAKER_04 48:04

Yeah, I don't think uh I don't think Kane will argue with you on that one. Um Neris though, um let's look at uh this. We've not we've not looked at this for the employees uh side, the frontline, the guys on the front the team uh delivering those experiences. What is going to change for those guys now? What's the biggest kind of uh risk to the to the employee experience that we need to consider and we need to not underestimate, I guess?

SPEAKER_01 48:34

This is the same as where when you're deploying a new solution and have been, you know, I've been involved with that for 11 years now, deploying new tech solutions, and it's the same with AI. It fails when you haven't, you know, sort of informed and and embraced the end user experience. So a bit like what Nick was just saying then, but with the users, you know, I see people who've had got auto summarization deployed, but the agents of there's a cohort of agents that won't use it because they don't trust it, and they are their ways of working and their sort of essence of value comes from doing the due diligence and the summarization, and they've always prided themselves on that. And and AI deployments miss that when they are just a technology deployment and pushing it out to everybody. You have to secure your superusers, you have to involve the end users in those discussions to determine what is the right thing for the way that the auto-summarisation works. What value, and this is really critical because obviously the subtext of any II deployment to the front line is they are going to get rid of us, they are making way to reduce volumes and therefore get rid of us. Now, in in some instances, there's definitely a case where they're not recruiting, you know, they're they're doing more with the same, and that's you know, that's really critical. What they have to understand is what the business intends to do with the breathing space and the headroom that the AI has given them. And some businesses are choosing to make much more proactive contact and engagement with their customers. Some are focusing on that FCR, which is supported by um you know the knowledge and the support assets that the agents are getting through AI. But I think there's just a lot that means that you have to have to include that front line in any deployment of AI and bring them on the journey with you and not just do to them because it will backfire.

SPEAKER_02 50:39

I think the two the two the two main things I think you could summarise both sides up in terms of the customer face and stuff and the employee face and stuff, the only two things really that matter is value and ease of use. There has to be value, so it can't be AI is coming for your job, it has to be AI is helping me. It can't be AI is stopping you from speaking to someone, it has to be AI is getting my problem resolved. There has to be value. And next it has to be ease of use. Now, the vast majority of AI systems today, not all of them, because a lot of them are triggered via your APIs and stuff like that, and I think we'll see a lot more of that agent-to-agent stuff, but most of them, when they when they meet an end user, the interface is a conversational user interface. What's the point of a conversational user interface? There's only one point of a conversational UI, and that is to make the thing easier to use. It's as simple as that. We don't have to use the computer's language and use the mouse and the keyboard and all this rubbish. The computer uses our language to understand what we want and get stuff done. So those are the only two things. It's exactly the same as anything else in life. If you want to lose weight, the value is you lose weight. What's the easiest route to losing weight? Well, you've got to find a place that's easy to adopt exercise. Maybe you leave your trainers by your bed when you wake up, so the first thing you do is put them on. You know, maybe you put it by the by the door, whatever it might be. You you create things that make things easy to adopt in order to see the value. And that's exactly the same here. Path of least resistance in terms of adoption has to be easy to use and there has to be value at the end of it. And you know, when when you said Nick, you know, start by figuring out what the problem is and solving the right problem. It's unbelievable how many knee-jerk reactions to AI there is based on a vendor selling something and a business implementing it because they want to do AI. And as basic and as fundamental as it sounds, those three things are missing. Path of least resistance to adoption, ease of use when using it, and value at the end of it. That's the only things you need to worry about. And it's surprising how much of that is completely missing.

SPEAKER_04 52:29

So I I I I could go on forever about that. I think we could all talk about this forever, but we're running out of time. Um, I do want to kind of I want to put you all on the spot a little bit, just as kind of around around the table, one last kind of thing. And I want to say before we finish, give us just a quick sentence, um, a takeaway for CX leaders out there. Um, and what does trust as this new KPI mean in practice from from your different perspectives? Uh, and again, we'll go around from my screen. So Nerish, your first, then Scott, Kane, and next, please.

SPEAKER_01 53:08

Um, I guess AI is a very strong augmenter of your experience for your customers and your employees. But when you are deploying it, you need to be able to make sure that you trust it and you understand how it's built and what how it's making decisions so that it's if the regulator comes with their um sort of uh you know, black book and asks you questions, you it's defensible and you know um how to explain why it's delivering the value that it's delivering and how it's delivering the value it's delivering.

SPEAKER_00 53:46

For businesses, I think the real challenge is not simply understanding the text of the law, but it's building internal disciplines to know where AI is being used, like what risks are are being created, you know, who's going to be accountable for this? How can these systems be governed uh responsibly uh over time? And I think when we do this, when we're transparent, when we build a sustainable foundation uh for ongoing uh innovation in a way, the AI Act is not just about avoiding fines or getting all of this right, it's about showing that AI can be deployed in a way that's credible, it's sustainable, it's responsible. And I think we can say it's also worthy of um public confidence.

SPEAKER_02 54:28

For me, we we spend most of our time on kind of customer-facing AI applications. So where there is a user that needs to accomplish something, and the AI helps them accomplish that via conversational automation. And in that space, the thing that leaders need to be thinking about sooner rather than later, and and doing this will essentially allow a lot of this other stuff to fall into place, which is testing. At the moment, testing is being done manually, which means it doesn't cover half the tests that you need to do. There's so much inconsistencies in it, it takes up so much time and effort, and it is necessary, but it is lacking in so many respects. There are tools now emerging. We have one which is pretty cool, which essentially simulates those tests. You can run all different types of scenarios, difficult customers, easy customers, you know, escalated conversations, complex conversations, all the different scenarios you need to handle. You can test that and you can test the integrity one of the interaction. Are we on track? Is anything going wrong? Is anything kind of bad happening? And two, are the customers getting the outcomes that they that they want? Because ultimately that's all people care about. So the thing to focus on, everyone's working on AI, everyone's rolling out AI solutions, face staff facing, internally facing, externally facing, all that kind of stuff, everyone's doing it, but no one has really got the proper testing in place that can give you the confidence to say, when we put this live, we know it's good, we know it's safe, we know it's accurate, and we know it's actually helping people.

SPEAKER_03 55:51

And uh at the risk of repeating myself in the last one, you know, for C for leaders in general, it's really take the time to look at what the operation is telling you, ask the questions from every level, build a true awareness, you know, draw your process maps, your customer journey, and have a true understanding about where your bottlenecks are, your pain points, uh, what's driving emotion internally in the business that could be driving attrition, and then from there we have a clear understanding about what the holistic business is. Then you can start to look at where AI could help, but don't be uh distracted by you know the the promises of the ads um that could just come in and fix everything when it might not even be a problem you need to fix because it's not generating a pain point. Look at where your true pain points are, where can you have your wins? They don't need to be quick wins, it could take time, but just make sure it's actually good for your business.

SPEAKER_02 56:43

What about the big shiny AI button? What about just pressing that? It's tempting, yeah.

SPEAKER_04 56:54

And and I think that is uh a great way to finish, Nick. That's a really good point, and Kane. You've left us all with and our audience with the image of the big button and that decision as to whether to press it or not. So hopefully um our session today will have helped the audience uh decide whether they're ready to press that button or not. Um if not, there might be some future uh episodes of these round tables that will help you get even closer to making that big decision. Um, but uh but for now, unfortunately, that is all we've got time for. I just want to thank my great panel here today for their amazing contributions. It's been absolutely fantastic. I think we could all carry on for another hour, but uh I don't think anyone will watch it. So anyway, thanks to everyone, and um we'll leave them wanting more. Uh and and yeah, until the next time, I've been Rob Wilkinson, and this is CX Today. Thanks for the thanks for watching.