Kustomer: CX AI Needs Outcomes, Not Tokens

 AI is everywhere in customer experience, but CX leaders are under pressure to prove it is moving the needle. In this CX Today interview, Rob Wilkinson speaks with Brad Birnbaum, CEO and Co-Founder of Kustomer, about the shift from measuring AI by deflection rates and activity to measuring it by outcomes that matter to the business.

Birnbaum explains why the right metrics depend on the company, from revenue uplift in retail to upsell and expansion in B2B SaaS, and why traditional CX measures like CSAT and handle time still matter but cannot be the full story. The conversation also digs into AI governance, observability, and trust, including how teams can evaluate non-deterministic models and reduce risk as automation expands.

Finally, Birnbaum shares practical guidance for rolling out CX AI without chaos, including why consolidating fragmented tools and starting small beats trying to boil the ocean. He also outlines where human-in-the-loop remains essential and why the goal should be turning humans from creators into reviewers. 

Transcript

SPEAKER_01 0:08

Hello and welcome. I'm Rob Wilkinson, and today we're taking a closer look at outcome-driven AI in customer experience and why CX leaders are demanding proof rather than promises these days. So if you're challenged with making AI pay off in real customer loyalty or if you're exploring automation without damaging the customer relationships, stay with us because you're going to get some practical takeaways you can use in your organization this quarter. Because today I'm joined by Brad Bernbaum, CEO and co-founder at Customer. He's very much an expert in this space. And his role gives him a very unique vantage point, really, on the decisions and risks and even opportunities, I guess, that CX leaders are seeing uh in this space today. So welcome, Brad. Thanks very much for joining me.

SPEAKER_00 0:56

Oh, so good to be here today, right? I really uh really appreciate you having me here.

SPEAKER_01 1:01

So recently uh there was an announcement from customer around your new uh architect uh element of the platform. So and in that announcement, um you talked around how um CX AI has been being measured by kind of the wrong standards. Just to kind of before we we get into the detail of that though, can you kind of break that down and set the scene for our audience? What is what is it that's broken about uh deflection rates and handling time as success metrics nowadays?

SPEAKER_00 1:31

Sure, no, um happy to. So as the world of AI has been evolving and it's certainly evolving quite rapidly, um, I think initially people were happy to see AI do anything it could it could do for them, right? If it could, if it could help a little, if it could deflect, which is a word we don't really like around here, um, but provide some form of perceived value or or not perceived, actual value. Um as people have been doing more and more of that on on our platform, on other platforms, just generally in the industry, um, we're noticing, and we we we certainly feel it as well here at Customer, that the shift now needs to go from using a lot of AI to actually making sure you're getting proper benefit from the AI, which of course would be the outcomes, right? So, not just to use AI for AI C, because that um you know, that that doesn't get too much value, right? We we we see in the world people, when AI first started, everybody's like all about token maxing, right? Like that's about token max. And people are realizing quickly, like, just because you spend $5,000, $8,000 per person per month on tokens, as maybe a developer or or or somebody doing marketing, it's it's not about the use of the tokens, it's about the value of the tokens, it's about the outcomes that those tokens are going to drive you. And the same maps to CX. And that's really where this next generational shift needs to go is focusing not around having AI do something for you, but getting the value of that of that something, um, getting the outcome of that something. And that that's really where we're where we're helping drive, uh hopefully helping drive the industry's mind shift towards.

SPEAKER_01 3:13

It does make sense. And and we really do have to be mindful of this, because you mentioned the how fast this space is is evolving. And with that evolution of technology comes evolution of people's roles and for this kind of conversation, those metrics that drive everything or that kind of monitor everything. And sometimes, you know, are delivering you know, the board pack that says this is the ROI we're getting off investment. So we really shouldn't um we really should uh let those evolve as well. So it's really, really important. Um I guess um you you you talk to different organizations, different sectors all the time, and you see how they're doing this. So can you kind of share a bit of insight into what you're seeing? If if we're swapping out uh the kind of uh the old school ticket metrics for want of a better phrase, uh to these more fo like outcome-focused um metrics, what would be the kind of the three measures that you would say this has got to be in the board pack for a CX leader today?

SPEAKER_00 4:15

For sure. Um, well, first and foremost, it completely varies on the type of company that you are, right? So if you are a retailer, um you might care about increasing revenue. You might care about if somebody is returning um a sh a black shirt, you maybe want to try and get them to buy the gray shirt instead, instead of just processing the term, right? So um there's there's revenue could be a very big piece of it. But again, it depends on the on the type of business you are. Um but I think the best way that I would distill it is the businesses need to know what's most important to them, right? What's what's their north star? Um, and we're seeing that CX is helping drive non-conventional CX North star, you know, not non-conventional CX North Star. So um historically, CX was all about CSAT and and handle time and how quickly did you get a response out. Those are still very important metrics, to be clear. But now as we're moving towards outcome-driven metrics, it's how can you help the business, right? So it's not just um the conventional metrics anymore. It's it's how am I helping grow revenue? How am I helping customers think about retaining their subscriptions for subscription-based business? If you're a B2B SaaS business, um, how are we upselling other products along the way? How are we recommending other products, right? So there's there's many different ways to think about business outcomes that then get layered on top of conventional CX metrics and ways that you know those of us have been doing this a long time. The call center, there's they're still important, those metrics, but but the outcomes layer on top, and the AI should help drive that. So um it's a hybrid, I would say, of those conventional and and these these newer ones that I would just say I can't give you three because it's really depends completely on your business. You know, if you're if you're if you're in travel, um, you know, we have some travel customers. If you're in if if you're in travel and a flight gets canceled, um the outcome should be to get you on another flight, right? Um, one that works best for your timing and your location and hopefully the cost, right? So like the outcomes are completely dependent, in my opinion, on the business and what's most important to the business. Um, but really it's it's marrying um improving customer satisfaction and what customers desire, which is immediate, amazing responses, hopefully with the resolution that they desire, um, but coupled with now how the business can also benefit from that great experience. And that that's really how I would think about it.

SPEAKER_01 6:57

It's a it's an interesting uh approach that to monitor those two things. And I'm glad that you didn't have you know three stock ones to to throw in the mix because it's not always you know the the right thing to do. And I think we're all learning, aren't we? So the fact that um these uh these outcomes are going to be more closely measured and they're gonna be done in the way that you suggest that that's that's only a good thing. Um and it's complementary to um kind of the more uh more traditional stats that we still monitor, um but but that obviously they can't they don't they don't translate into the world of AI, so they've got to be kind of uh you know kept over there where they belong. So um something that we do see a lot of uh organizations looking at now is um observability. So um what does that basically what does AI observability look like in CX work? We've got leaders all the time talking around, yeah, but you know, how does the AI uh sorry, how does the AI uh reach the outcome? How does it come up with the answer? And and can we trust it to do what it's done? Uh you know, it's it does that kind of especially for new people to this, there's there's a kind of a concern and a risk attached to that, isn't there?

SPEAKER_00 8:10

Yeah, look, I think if you pick the right product, the right partner in your AI transformation journey, they will help you with that and they'll they'll they'll provide you a lot of the tooling that you do require to focus on trust, right? So in a customer, trust and and now AI governance is really a part of trust, is something we're we're very, very focused on. And AI governance is uh you know a new buzzword, right? Everybody's talking about AI governance. Um, because you do have to to um make sure that it's behaving correctly and doing the right things. And you know, there's there's examples out there on the internet of of AI doing kind of wacky things, right? So I think the way that we think about it is one, from the ground up, we've always believed that you know trust must be woven into all facets of delivering CS experiences and AI in particular needs it needs it you know all over the place. So for us, you know, there's a lot of ways to think about trust and governance in our platform. Um one is is just simply as you're creating uh what we call our procedures, um we have an incredibly complex and sophisticated uh evaluation platform and framework that will make sure that the AI delivers the results you expect every time, right? So we'll you'll you'll run it through and run many, many, many, many iterations because AI is non-deterministic, right? You might get this slightly different answer for something you ask virtually the same way every time, right? It's it's it's theoretically possible. So you need to tune and make sure your prompts do that. So we've got a really rich evaluation framework that will run against that. Um, and then we use AI to help build AI. It's kind of inception-like, but you know, so we have um assistants that are a part of our evaluation framework that evaluate um evaluate the evaluations. That's an interesting one, but evaluate the evaluations and make recommendations on how to tune um your your procedures uh to get even better out output and better, sorry, better out uh better responses. So, you know, that's just one way that we see this happening. Um, you know, other other ways are moving to uh sticking within certain boundaries from a governance perspective, making sure AI doesn't veer off that way or that way and stays keeps you on the straight and narrow path of what it is that you want to talk about and not allowing you to go this way or not allowing you to go that way. So there's a variety of ways that you can do it. But yes, it is important. Um it's important that you find a partner who understands that. It's important that you find a partner who um whose products you know will enable that and and put it first and front and center.

SPEAKER_01 11:01

So it's it's it's exciting to hear that um the you know that with the evolution of AI, we now have bots monitoring bots and fine-tuning and helping them to improve and to ensure that that kind of uh that trust is there and that that observability is in there. So it's really yeah, some people might kind of historically, and I think in a couple of like 18 months ago, two years ago, it was kind of people's like big brother scenario. If they'd have heard about that, that they'd be they'd be worried. But it's it's it's I think uh everybody's seen how uh robust now uh the this AI is and uh some of these things are only possible now because it's uh the technology's shifted, it's improved, and and it's so much stronger than it than it was even just a year ago. And and the speed at which it's continuously improving is is is quite hard to keep track of. So it's uh yeah, it's it's really interesting to hear. So thanks for that. I think we've gone into uh I think we've got a nice grasp on kind of the operational impact and and I want to talk a little bit more around the kind of the broader strategy, the market, the wider market kind of let's look at um look at things in a in a bit more detail. So if we've got messy stacks and we've got inconsistent knowledge, how can you recommend an approach to rolling out CXAI without any chaos attached? Um where should people start and what's the thing that they've got to do or got to have to be true first?

SPEAKER_00 12:32

Sure, sure, sure. No, it's a great question. We are asked this all the time here at Customer. Um, we obviously have a point of view um that we're the way that we think is correct. But when we see, and this has been true for the contact center for for decades, frankly. It's it's how customers started almost 11 years ago. Um we believed having a variety of disjointed tools to try to service the one the one effort of providing amazing support experiences makes it really challenging. Um and that's what customer has always been about since day one. Um being the centralized platform to match your customer, to um to take care of all the exchanges, all the interactions, all the data, and orchestrate all of it together, right? So not having 15 or 20 different tools to manage things because when they inevitably try and work together, you get a little crisscross, a little confusion, um, probably a suboptimal experience. Well, um, as it relates to AI, we see that the same way. So there's a lot of you know, bolt-on tools out there, we call them, um, where they will sit on top of something else and they'll they'll they'll do their best to do a good and fine job. But when you have everything in a cohesive system and a product that can interrogate everything and all everything we know about the customer, the CRM capabilities, um, coupled with all the abilities to talk to third-party systems via MCP, via our app platforms, app integrations, via OpenAPI, those two things in unison really do provide a rich experience to help avoid the chaos and help provide the cohesive data set that the AI can build on top of and take full advantage of. Even simple things like your knowledge base, having your knowledge base inside of the AI platform has value because AI needs, they're all the models are trained on the same general public data. Um, they're not trained on your company-specific data. So therefore, you know, and nor should it be, by the way. Um, so it's gonna rely on all the data sources it can it can use to create those proper responses and to give the customers the best answers. So when you have all of the knowledge coming from all the different places, that's how you're honestly going to get the best outcomes.

SPEAKER_01 14:53

Right. Yeah, it's like anything. We we all work this base in this industry, we we evolve all the time. We have new technology all the time. I don't think there's a better place for technology in than contact centers and CX. Um and I think we've in the past we've actually managed really well to adopt those technologies, uh, and we do it by getting the foundations right, don't we? And it's always the data needs to be right and the foundations need to be in place. And I guess you're saying it's that's the same now with this. These are the kind of fundamental things that just because AI is really clever and powerful, it's not a miracle worker. Yeah, you put rubbish in, you get rubbish out, don't you? So yeah, that context is so important. Um and yeah, if you've got too many uh tools talking to each other, the risk of that context getting muddy is is is too high, really. Um there's such a lot of noise around AI agent dry now. Um so I want to try and cut through that a little bit. Um CX specifically then, where should uh leaders allow autonomy today? Um and where should they be saying actually human in the loop, absolute necessity, non-negotiable?

SPEAKER_00 16:09

Sure. Um again, every business is gonna be different, right? If you're um selling an incredibly high-end expensive product, you might need a little bit more human oversight um than a than a more modestly priced product. So I do think it depends on every business. I think it depends on the use cases and severity of your of your business and your outcomes. So I'd preface it by by saying that. Um, but what I what I would say is AI is getting very good. RAI is great, just in general, it's great. And the models, we all live off the same frontier models, and they are incredibly powerful and getting more powerful by the day. Um, and we're just gonna continue, you know, in our opinion, those are gonna continue to exponentially get better and better and enable all of us to just provide even better and better and better outcomes. So we're going to see containment rates going up, we're going to see outcomes improving. That's, you know, to me, that's that's a given at this point in time, right? Our our products and our applications will continue to become better, the models will become better. So that's gonna happen. However, we're definitely not in a world where AI can handle 100% of everything. I I don't I don't see that yet, right? Maybe certain use cases, yes, but generally no. I mean, we're seeing at customer we're seeing really high containment rates of you know 70% for some of our customers. So like we are able to see some of that magic truly happen. Having said that, um, we generally believe in a world of of AI doing a lot of work, but humans also in the loop, right? We we we we still see that. Um there's a variety of reasons. Um humans have empathy, right? Uh AI has less empathy. I'm hearing stories where it's it's getting better actually, but like it's not there yet, right? So I think there's a there's a fine line. So I think one, there's just operational things you may say that like, oh, any return under $100, yeah, I can always handle. Any any return, you know, between $100 and $1,000, you know, you can have a few boundaries. And any return over $5,000, then you know, human must approve, right? So like you can define those those guidelines, those boundaries, if you will. And I and I do see some companies doing that. Um, other examples though, where I see AI and humans working together, um, maybe sometimes goes back to the empathy example. Let's say I want to return, you know, a shirt, a simple, a simple item. I want to return a shirt and the company's return period is 30 days, and I try and do it at six months. And the AI is just not gonna allow it. It's gonna, it's generally gonna say, you know, look, our return policy is 30 days, you know, we're really sorry. Um, you know, thanks for your loyalty, but you know, it's 30 days. And if the customer continues to push inside, say, like, well, no, I had extenuating circumstances, or this happened, or that happened. Um we can actually have AI say, look, we should give this to a human to look at. So it'll it'll move from an AI conversation to a human that will look at it. And where they go, where they work so so well together is the human may take a look and be like, oh wow, like Brad's extenuating circumstances were real. Um, and Brad's a really loyal customer, and Brad spends a lot of money with the business. We should just approve that. But where they where where the the humans in the loop work really well is, and we're one of the few companies that I think does this very effectively, is it goes from AI to human, but the human can look at it, could intervene, or the human could look at it and say, you know what, I'm just gonna approve this thing, approve this exchange, return, whatever it is, and it'll go back to the AI agent to actually process it and say, Oh, all right, like, yeah, we're gonna make that exception for you. So you're letting the AI do the more routine and mundane tasks, and then processing the return or exchange and collecting the information, like a human doesn't need to do that. So it can actually move between agentic human back to agentic, and customers really, really appreciate that about our platform because it it lets the human do, you know, the the parts that it needs to think about. But yeah, it can do most of most of heavy lifting. And um, you know, we kind of have a phrase around here, a slogan, if you will, uh buzz is is just our our objective is to turn humans from creators to reviewers. And that's just an example of of that, right? So rather than having to do all the work and create the response and create the actions, and they would review the suggestion, they'd review the recommendation. Um, and in doing so, one, you know, AI is doing the heavy lifting and the mundane tasks, but two, you're able to just be that much more effective. And when you are, um the companies are happy because they're reducing costs, the customers are happy because they're getting responses right away. Everybody wins, right? Um, and it helps businesses scale. So, you know, we're that is how I see AI moving. That's that's customers' point of view for how we see AI moving in the in the contact center is um conventional agent roles are gonna convert from from creators to reviewers.

SPEAKER_01 21:03

I like that that phrase, and it's it's very um very appropriate for the the the evolution of that role, isn't it? Um and you know they need to be supported, they need to be uh you know trained to deal with these different types of uh inquiries more often than how to be resilient because they're not gonna have the password reset easy things to do and uh their days are gonna be more intense. But I think everybody understands that. Um I am interested in what you think though, is if we were to say that's today and that's kind of where we're at, um how long is it gonna take for AI to be able to do the reviewing as well? Because it's gotta be. I mean, if you'd have asked us if AI could do what he can do today two years ago, you'd have said no. So do you think there's a possibility that that that that role is is is down the line?

SPEAKER_00 21:57

I do. Um I do think. That there will be other models or other prompts that might actually work hand in glove and potentially do some of those reviews as well. But you could also argue that the creation of it should be so robust it doesn't need that. But I'll give you a slightly different analogy in the coding world. Um, and and we do a lot of AI coding, pretty much all AI coding a customer at this point. Um we've converted um all code always needed human reviews in the past, right? We we made sure of that to make sure that we're we're we're maintaining trust and security and writing good code, et cetera. It's a common practice in software engineering, but we now have multiple third-party products that are doing AI code reviews. So it's in the in the engineering world, there's AI creating the code through tools like Claude and Cursor, et cetera. But then we have other tools, once that code gets pushed up before it merges in, other tools that are then reviewing that code and providing feedback. And it is AI working together. So, like in the coding world, that's very much happening today. We we do that currently. Um and I joke because sometimes we we have multiple products doing the reviews because we want to have the best code we can, and sometimes they fight with each other a little bit. One will say do this, the other will say do that, and then it'll do that, and then the other, and we'll see a little bit of the fighting between them go back and forth. But generally speaking, it does an incredible job, right? Um, and I think as products and models continue to get better, they're gonna they're gonna do more. So I do think there's worlds where that may evolve into. Um I do actually. I do think that there are worlds of that. And I do think that we have two, I shouldn't say I think I know that we have tooling within the customer platform that is constantly reviewing how some of our AI is working and suggesting recommendations, suggesting ways to optimize it. So, to a certain extent, that is reviewing your config on a on a regular basis and saying, hey, we notice customers are talking about this, you may want to optimize some of your prompts or the procedures to do that. So we are seeing that up today in our products, and I do think that we're gonna see that more um at the end user level at the at the at the customer to agent level.

SPEAKER_01 24:07

Realm it's real-time constant customer kind of continuous improvement in in real time. It's kind of amazing. Um so we're we're running out of time a little bit, but I just want to kind of um before we wrap up, I want to kind of have a look at your just get that give our audience the benefit of you know your your knowledge and what you what you've seen in the space. And and I want to my I want to put the TX leader hat on for just for one minute and I want to think about what are the right things to be doing. In fact, sort of a sensible step-by-step plan um to prove value from uh this uh this technology, um, kind of without overpromising to my business that the you know that we're gonna fail to deliver on those promises.

SPEAKER_00 24:54

Yeah, no, um, first off, I really do think that the world of AI is is amazing, right? I I often you know call it magic, right? Because it it is it is evolving so fast. The the tooling, the application, the models is it's is truly in in very interesting time to be in this world. Um and you know, I often I often tell people like my crystal ball is currently broken, like I no longer know where where things are going, right? We have a vision, of course, a customer, but like I can't see five years out anymore, right? I feel like I used to be able to, and I I don't know that I can because I can't predict the the power of these models and and the evolution of where they're going. Having said that, um I do think that there's really interesting ways to get started, right? I do think that um you should have reasonable expectations. I don't think you should expect that you're going to one one hour in, you know, handle 100% of everything perfectly. Like I don't, I don't, that's an unreasonable expectation on our product, or frankly, any product, in my opinion. But I think what you should start with is defining some of the most important business outcomes. I think you should think through what is what is most important to your business. And it might, your initial business outcomes might actually just be hand, you know, uh handling customers faster, right? Like getting to them right away. Because we've seen over the years, um CSAT directly correlates, conventional CSAT directly correlates to response time, right? If you get an immediate response, your CSAT levels are generally higher than if you had to wait a while, right? So um, and it seems obvious, but it's it's true, right? We have we have data behind that. So, you know, some people may want to just say, let me take a couple of my most important business outcomes, let's get started on that, let's get those up and running really, really quickly, let's learn, let's figure out how to go, and then we're just gonna keep adding and adding more ways that AI can help. Um, and that's generally what we see our our customers doing. Um, you know, boiling the ocean is probably not the way to go. Um, it's an evolution. So let's get you started, let's get you up and running. We get customers up and running very, very, very quickly. So to be clear, it's not a very long process. Um, basic things like uh basic RAG, retrieval of method generation off of our knowledge base, that's up and running in minutes or maybe an hour. So like you're already getting a lot of value where you're talking to your knowledge um, you know, via chat and email, etc. So that that's like instant value. But then going further and making these more advanced procedures to take these actions to drive those those next level outcomes, you figure them out, you keep evolving them, you keep tuning them, and you keep adding additional ones as you're seeing them, as you're seeing the benefit, as you're seeing what's working. Um, yeah, pay attention to trust and governance. Make sure that you're writing proper evals, that you're not going off the rails. But you know, a good partner like us will help you with that. Um, we provide great tooling for it. So you're not writing code, but we'll provide you great tooling, but find a partner that's that's gonna that's gonna help you with it as well.

SPEAKER_01 27:59

That's fantastic. I think um, yeah, I think I think there's a lot to unpack there. So thanks so much, Brad. That's uh that's that's really useful stuff. Um unfortunately it it is all we've got time for. So thanks again for joining me and answering all those questions. Um really enjoyed it. Um for anyone who's watching this though, uh uh maybe they want to kind of explore this in more detail, maybe look at what you guys are doing. What's the best way for them to do that?

SPEAKER_00 28:26

Oh, um there's a tremendous amount of information on our website, www.customer.com with a K, customer with a K. Um and love for you to check it all out, check out our products, um, learn more, and then happy to schedule, you know, schedule time with people who want to learn more, see demos, understand how they can use our product and the value of it. We've got a great process to to show you how how we can provide benefit to your company. That's great.

SPEAKER_01 28:52

Um, and don't forget, you can also find a wealth of related resources, uh, stories and other videos just like this one at cxtoday.com. Uh, but uh for now that does wrap everything up. Uh, I'm Rob Wilkinson for CX Today. Thanks very much for joining us.

SPEAKER_00 29:07

Thank you.