Token Economics: Why It's So Hard To Track AI Spending At Large Companies

Enterprises around the world are struggling to predict and track their AI spending. What makes this a hard problem to solve, and what can CFOs do now to get a handle on token spending? This episode is ideal for anyone at an enterprise today who is trying to manage these costs. 

Speaker: Ravi Kuppan

LinkedIn: https://www.linkedin.com/in/ravikuppan/

Yarken.com 

Yarken is a Betatron Venture Group portfolio company. 

Transcript

Copy Transcript Copied

Arshad 0:00

Joining us today is Robbie Koupan, who is the CEO and founder of Yarkin, a company that is helping some of the biggest companies in the world track their spending broadly, but he zeroed in on a particular problem, which is token economics. And that's something we're going to talk about today. So, Robbie, thanks for joining.

Ravi 0:20

Thanks, Arshad. I appreciate uh the invite in. And yeah, looking forward to talking more about everything about token economics and what are these tokens?

Arshad 0:29

Let's start from the very, very beginning. So, who really does need to worry about their token usage?

Ravi 0:36

Well, I think it's an enterprise problem, but I think the challenge we've got is the CFO in particular is the person who has to the bit someone's gonna have to pay for this. And someone has to forecast it. So we've got people in IT and IT finance who are having to build a forecast model. We just heard about Uber burning all their tokens within three months their entire budget, Salesforce spending 300 million just on Clawed. Most people are using Copilot. It ships with 35. Great. But then I've also layered that cost with Claude. But ultimately you may be paid multiple times for the same product. And are you getting any value?

Arshad 1:21

What does it mean multiple times for the same product?

Ravi 1:23

If we think about um Copilot, it's highly prevalent. It comes with my 365 subscription. Most organizations have it. But then I've also got Clawed. And I might be using Clawed for AI design, Clawed to write my emails, I might use Claude code. But then I've also got GitHub, which is shipped with Microsoft as well. So what we're seeing is this entwined overlap of slightly different LLMs, but ultimately, if I'm not a superpower user, do I need all these different things? Really? Do we need that speed? Um it's kind of like having a Ferrari when your, you know, your Nissan would be fine. Um but what we're seeing is a fundamental shift happening where CFOs are worried. I was speaking to a large telco the other day, and they've just suddenly said, well, at a board level, oh have we got this under control? And these are the questions I think which are coming up for large enterprise now. And unless they understand the multiple doors of AI, I think this becomes an even more challenge.

Arshad 2:33

Tell me about that. I mean, if you're in enterprise, you might be paying for tokens directly through Claude or ChatGPT or Microsoft's tools, but then there are lots of hidden token usage buried inside the features of other companies that now all have some sort of AI. How is it even possible to look through and figure out what the total cost of AI is if some of it is hidden inside of other apps?

Ravi 3:01

And and I think this is where this becomes increasingly difficult. Is we call it uh AI total cost of ownership. What you need to start to understand from a business enterprise perspective is I need to bring all my cost into one platform for AI. And I need to stop breaking down what I am paying for and how do I pay for it. And and that's one of the things we try and solve at Yarkon. We bring this all together into one platform where we connect the contracts and we connect into the GL so we can start to understand what you're paying for, but then also from a usage basis, what are you doing? How are you using it? And obviously, this is material, it's all about materiality. You don't you're not really interested in a large enterprise for you know a thousand dollars, but when you spend five hundred thousand dollars on your Salesforce and it's not based on users, you're suddenly kind of thinking, well, what's changed here? And it becomes highly variable. And I think Arsha, that's the big piece we're seeing in for these organizations, why this becomes important, because I used to pay per user, uh, but that's gone, and I've got these variable spikes on AI and how I use AI. Because coming back to our first question, it's tokens. Everyone's got to pay in these tokens, which we can't fully see. And ultimately, it is about this energy we're utilizing back in some data center. We've got a university as one of our customers, for instance, and being a research-based university, they've decided we're gonna actually co-locate and create our own on-premise data centers with AI, with our own chips to help support the academic staff. But that's a huge cost, and then suddenly their energy costs of just for one rack gone and double the entire data center costs. So this TCO complexity we're seeing with AI is something we've never seen before with cloud, let's say, which was kind of quite isolated. I'd get my bill from one of the providers, but there are many doors now for AI costs. And unless a company gets on top of this quickly, it can soon spiral. And that's could be a potential cost blowout. This is what we're seeing because you can't forecast this.

Arshad 5:18

Yeah, makes sense. So your customers are very large companies. What advice do you have for the CFOs at those companies for the upcoming year in uh say 2027 and beyond in terms of managing their spending?

Ravi 5:32

Well, I think most companies, 75%, I would say, were exceeding their AI spend budgets. What I would say for them is they need to establish that ownership immediately. They need to think about the multiple avenues of where that spend's coming from. One of the big things we're seeing is AI gateways. It's critical to put in a gateway now. So you can throttle, uh, not just throttle, maybe that's the wrong word, you can control all access to AI. And then you can start to track it. What is the AI gateway exactly? So an AI gateway sits within your network and you proxy all, and think about it like a proxy server. You can actually control all network traffic, it gets put through this proxy, and then you can actually see which model they use, what do they do it for, what were the queries. There's a lot of information here you can start to capture. And that starts giving you this incredible transparency because step one is who's using it, which user, who's driving up my spend, which business unit. And so I think we're gonna get to a world very quickly. I want chargeback, I want to charge back as a CIO, back to my other consumers within a large enterprise, because I can't just flip this bill. But I'm also gonna be thinking about is it producing me revenue? And I'm gonna be getting more interested into productivity or revenue generation, and I can tell a story. So we were talking to um pretty large organization all around uh the lottery, and what they see in all of these organizations is suddenly on betting day, we've got Powerballs, and their infrastructure costs just spike right up what's driving that cost. How is that relate back to ultimately a revenue piece? And so that's what we're gonna see with AI, I think, as well. It's gonna be increasingly important to track those other business outcomes as opposed to just a cost. So I would suggest to every CFO and CIO now, for your large initiatives, you need to build that in and you don't allow anything else to go live or in pilot, unless you've got a very strong governance metric.

Arshad 7:40

Is there a set price for tokens? Like is this token priced at OpenAI and what it gets you in terms of compute the exact same as what you might get at Claude?

Ravi 7:51

And unfortunately not. It's not like for like, and I think that's what makes it really hard. Token costs uh at the moment are very difficult for us to fully understand and benchmark. What is a token? I mean, ultimately, they are a way to quantify some level of compute, effectively. And if we just think about compute, large language models use a lot of different elements, but compute's probably one of the biggest pieces. When we type something in, it's either an input or it's an output when we get a result back. And it's really a way to meter the cost. So it's like put in petrol in our tank and we can go a number of miles, depending on how big that engine is. And the question we ask, we get a different type of output. And what we've got is you know, small language models. Um, we've got Chinese models which are open sourced. You could run them on your own hardware as well. So it's highly variable. This is why I think building transparency around token costs and token economics is going to be really vital because it's so dynamic. But the good news price is going down.

Arshad 9:04

So it sounds like the CIO's CTOs are now well aware of the inflating costs of AI and the questionable attribution to outcomes in their companies. Are you seeing American or Australian or Singaporean companies starting to adopt the Chinese models?

Ravi 9:24

I think a lot of the US companies are still very wary about using Chinese models, or they'll bring it all on premise and host it themselves. We're going to see a market where trust is and security are also very, very important pieces here. Where's this data go? How do I create trust around this information? And can I lay a security around this? So building harnesses is going to become increasingly important. Because at the moment we create MPC servers, we have this lovely ability just to connect everything into these models. But where's the data going? Who's got access to it? So sovereignty of data and AI is going to be increasingly important. And we're actually just talking to a couple of companies in the UK around this at the moment. And I think that's going to become a vital piece for every Western country. No one wants to rely on someone else.

Arshad 10:20

You know, I would have thought by now companies would already have tools to maybe track their AI spending and all the token spending, but it sounds like that's not really the case. Sounds like the existing tools aren't good enough. Talk to me a little bit about that.

Ravi 10:34

Yeah, no, look, it's surprising. I was just talking to a Fortune 50 company earlier this week, and they don't know what their AI costs are. People do not know because it's so scattered. And this is where I think the ability to be able to link information from multiple sources and also think about the complexity of that value question is going to become increasingly important as we spend more and more money in this area. Yakin means to be prepared in Middle English, and uh we really want to make our customers prepared for the AI world and ready for this AI wave. And I think what we're seeing from organizations is everyone is struggling. And now everything's on demand, and then we get the oh, you've run out of tokens. What am I gonna do? I had this the other day, and I was producing some proposals and I ran out of tokens. And my team were back, I was on the other side of the world, I couldn't access anything, and I was like, Oh no, I've got to wait now until 1 a.m. to reset. And it's like, wow, this is the new world. I need a little slot machine in my computer.

Arshad 11:48

Yeah, right. That's right. Put some coins in, coin operated access to AI. So that might be where we're heading. Thinking about the future, what's your prediction? Five years out, will we still be worrying about token spend?

Ravi 12:01

Tokens will still be around. This is this is fundamentally how the industry has created a unit of measure. But we will get to a world where we'll understand well, if I buy X, what value do I get? And we'll become a bit more educated over well, I don't need the Ferrari every day. And I can do with my my Nissan. So I think token costs, that awareness, actually will grow.

Arshad 12:30

With all the new AI data centers being built around the world, uh, what kind of impact will those data centers have on token economics? Do you expect it to drive the costs further down? And what's the impact for the end user like the enterprises?

Ravi 12:46

Yeah, great question. I do think with the scale of this investment, which we've never seen before in our lifetime, costs won't go down. I don't fundamentally believe that we'll just use it more. So someone's gonna have to pay for this. And because you're gonna you're gonna use AI, of course you are, but if you can't track it, then you can't measure it. And that's one of the key things we we really want to focus in on. And and I think we're gonna start to see that conversation around those two parts. Is it revenue generating or is it reducing my costs? And it's those dimensions of AI, I think, will govern investment in a much more judicious way going forward.

Arshad 13:30

Really an illuminating conversation. I think what you're working on is incredibly relevant and timely, and a problem that companies around the world are struggling with. So good luck to you as you are out in the world and uh and and and closing new contracts. Thanks so much for joining today.

Ravi 13:46

Thanks a lot, Mr. Chat . Appreciate it.