Ep. 60: What if AI is too Expensive?

What happens if AI turns out to be more expensive than we think?

For years, large language models (LLMs) have been treated like low-cost cloud services. But what if today's pricing is heavily subsidized? What happens when AI costs rise and organizations are forced to rethink their strategies?

In this episode of Talk To Me Petey D, I explore 10 predictions for a future where AI and LLM inference becomes significantly more expensive, including:

✅ Moving from frontier models to lower-cost alternatives
✅ Shifting from RAG (Retrieval Augmented Generation) to fine-tuned smaller models
✅ Increased use of model routing and intelligent cost controls
✅ A return to traditional software and "good old-fashioned AI"
✅ Better AI harnesses and domain-specific architectures
✅ Edge AI and on-device inference
✅ More human-in-the-loop systems instead of autonomous agents
✅ Greater budget governance and AI spending controls
✅ Increased focus on AI ROI and business outcomes
✅ Re-evaluating automation projects as costs change

I also discuss why the economics of AI may be fundamentally different from traditional software and cloud computing, including the concept of rival vs. non-rival goods, and why GPU-intensive AI services may not scale economically in the way many assume.

Whether you're a business leader, technologist, policymaker, or AI enthusiast, this episode offers a practical framework for thinking about the future economics of artificial intelligence.

Connect With Me
📖 Book: People Management: Ground Up for Aspiring Leaders

💼 LinkedIn: Pete Dempsey

🤖 Course: Partnering with AI Agents for Business Success

🌐 Website: PeterDempseyWrites.com

📰 Newsletter: Subscribe Here

🦋 Bluesky: Follow on Bluesky

📺 YouTube: Talk To Me Petey D

🍎 Apple Podcasts: Listen on Apple Podcasts

🎧 Spotify: Listen on Spotify

Transcript

SPEAKER_00 0:00

The past few years have seen the rapid proliferation of large language model-based AI solutions. LLMs have been treated like existing cloud services, which are low cost, even at a large scale. Recent developments are shedding light on the fact that LLMs might be quite a bit costlier than many believed. Today I'll dive into my 10 predictions of what will happen if LLM costs rise significantly. I'll give you these predictions first, and then I'll deep dive into a few areas to discuss how LLMs have different economics than most software solutions. This is the Talk to Me PDD podcast, where we discuss all things knowledge work, leadership, and of course, lots on AI. I'm your host, PDD. Please take a minute to like and subscribe to support the channel. Check out the resources in the show description. Alright, here we go. This is episode 60. What if AI is too expensive? So here are my top 10 predictions about what may happen if the costs of AI for large language models specifically rise in the changes that we'll see. 1. People are going to use more cost-effective models when possible. Two, a move from RAG or retrieval augmented generation to fine-tuning small language models. And ten ROI changes. So that's a list. I know it's a lot. I'm excited to deep dive into each of these and a couple more topics. So go back in and take a look at each of these and really see what we're thinking here. So the first one, use cost-effective models. This is a pretty easy change. The different labs have different models that are accessible via APIs. There's also lots of open source models, smaller models. So in most solutions, swapping that around, at least from a code change perspective, is pretty easy. So if there's starting to be cost pressures and you have cheaper models that you can use, that's that's an easy change that you can make. Now, that being said, the the change is easy. The effects may or may not be desirable. So there's some testing that's going to need to happen there. But when everything is low cost, uh, I think lots of people are just dropping in whatever the latest frontier model is because they're not paying the full costs for it. So if those costs go up, which I'll talk about kind of after I get through this list, then naturally people may change and swap out the models that they're using. Uh the second one with with RAG to fine-tuning, I think this will be interesting to see if this happens. So RAG is putting traditional search and prompting large language models together. Um, what's really great about that is the content in your search doesn't have to be in the model itself. You can quickly update content into a search index and then it's available in the same prompting interaction that you would have with a large language model. But the disadvantage here is that you're running this search beforehand and then you're taking all of that content and appending it, making it part of the prompt that you send into the large language model. So naturally that's larger. And since most of these offerings are moving to a pay per token model, you're naturally going to be paying more there. So if you have the opportunity to take a model, take the content that you have in your search index and fine-tune the model and bring the content in, then you no longer have to pass in those additional tokens from search to get the information back. Now there's some more that's involved there, but I think there's also an opportunity to use smaller language models with fine-tuning on that content to achieve those goals. So that's something that I expect we'll see. The third one is model routing. And you see this in a lot of offerings now, where instead of picking a specific model to go to, you often have an auto mode. Umai was the the first company that that offered this in Chat GPT, where they'll do some analysis on the prompt and determine if it needs to go to a more expensive, complex model or a cheaper one. Um this is also ways to regulate different types of accounts where if somebody's using too many tokens or too many services, then you can downgrade them to a cheaper model. Um, so that's already out there. Um, but I think there may be um kind of more offerings here, or um companies and organizations may use their own routing to make similar decisions, um, either based on logic or costs or consumption or things like that, but it'll allow more fine-grained control around what models are used and what the cost of those are. Now the GoFi uh number four, the good old-fashioned AI, I think this is a really interesting one and interesting opportunity. If you think large language models are free, right, just go ahead and use them for everything, which is a lot of what we've seen in the design. Um, but you don't necessarily need that. And I think there's opportunity to translate solutions or parts of solutions that were built on prompting large language models into traditional software or more old-fashioned AI rules-based systems, expert systems, that type of thing, where you have deterministic behavior, and then your costs are really traditional computing costs, which are incredibly low. Um, so I think people will, if they're seeing more costs than they want within a LLM-based system, they can go through and parts of that can be replaced with more traditional software, um, which are lower cost and then often are beneficial because they're entirely deterministic in their outcomes. Uh, the fifth one, this harness and model pairing, this is what we've seen in a lot of coding solutions where yes, there's been progress on the LLM model that they use, but a lot of the gains have been in the harness, which is really just the traditional software that is the interface to the large language model. And because you're putting specific rules there in a domain-specific way right now for coding, um, you're able to really direct the behavior, get a lot of improvements. And I think we'll start to see that in in more domains than just coding. Um, and having a better designed harness and a harness paired with a specific model can be much more efficient in terms of the costs and the interactions that need to happen. And then, you know, going back to that number four, the GoFi, that you can put some of that code, that deterministic code in your harness so you don't have to go to the large language model for everything. Uh, number six is edge inference, and this is simply running the models on device, which could be you know a PC, it could be a mobile phone. Um, and that way you're taking um kind of the costs off of the cloud infrastructure, hosting on big GPU farms. Now, of course, you need a device that can run the model, which is likely going to be smaller models, but increasingly seeing this, and then um the provider is no longer bearing the the platform cost of providing uh inference for the large language model. Uh, seven is more human in the loop versus versus agents. For a while, autonomous agents that could run uninterrupted for for days or however long were really the rage, but that's consuming massive amounts of tokens and therefore cost in the background. Um, so while I think that will will still be a thing, um, there may be more ways that humans are consulted or directing the path of that, because if you have that that interaction, um, you can drastically reduce sort of the the exploration space that those agents are going in and the number of tokens that they can consume. Um so that's number eight. Uh number nine is around budget governance. We're we're already seeing this in a lot of organizations. You previously you had this concept of token maxing and leaderboards for people that were using the most tokens within their organization. Meta is the company where this was sort of most well known. Um, they've since walked that back. It's sort of like, do you want to have a contest to burn as much money as as possible? Um so now there's going to be, I think, more focus on what are what are the costs, how do you constrain that cost, who gets access to those resources, um, and having more fine-grained controls within these solutions about where the spend is going. Um number 10 related to the cost and spend is ROI, the return on investment, um, really measuring not just the activities that AI can do or sort of the amount of emails scanned or messages sent or things like that, but translating that into what actually matters for the business. Um, a great example of this was Uber talking recently about um, you know, their AI spend and not necessarily seeing that translate into significant features and improvements that were shipping in their code. So ultimately, businesses and organizations will need to connect their investments and costs in in AI spend into um not just activity measurements, but into business metrics that that really matter for their investment. So, yeah, there you go. That's kind of a detailed dive into my list. It's always fun to predict the future. We'll we'll see what happens. Um, and then I would just I'm gonna touch on a couple other topics related to you know potentially rising costs in large language models and AI. Um, a lot of the labs and vendors are switching from a sort of per seat model where you pay for a certain kind of amount of features and then have unlimited usage to a pay per usage or pay-per-token model. And I think there's really a different psychology for users here. Um, I think it's kind of unprecedented in for most people using computing, especially on the consumer side. You're used to sort of unlimited use of these features. And even if it's not a massive cost, it changes, I think, the the way that you approach the interaction and the product. A lot of these chatbots are designed to have long, kind of continuing conversations or having people brainstorm or having a voice conversation. Um, and maybe as a user, I think you look at it differently. If the longer your conversation goes on, the higher your your bill is. Um Ed Zitron's done a ton of great research on the actual costs here. And who who knows what's true? But he's he's estimated that for every one dollar of spend, whether it's a user or an organization, that the model large language model providers are having costs of roughly eight to thirteen dollars per $1, which is which is a huge difference. And one of the things I think is helpful to illustrate this, if let's say you were in the kind of the early days of large language models and you you had a five-person team that did a certain task, each of those people cost you around 80K a year. So let's say like 400k per the team to do this task, and you might be able to put in an AI system that could automate this work, and let's say it costs you 50,000 a year to do that, to automate that work. Well, that's a 350k savings, so kind of no-brainer for a lot of businesses to go and make that change. Now, if those costs were to true up sort of based on that estimate, and let's take somewhere in the middle, let's say you they go to that $10 per one, which is you know roughly in the middle of that of that range, you're then going from an AI solution that costs $50,000 a year to one that costs $500,000 a year. You're actually spending $100K more on that AI automated solution than you were for that five-person team, which likely had some other capabilities and things that you could do. So that'll be really interesting because if these cost changes come into play now, they're going to have significant effects, effects on previous projects that you know showed real ROI at the time, kind of based on the current cost, but now will no longer have the same ROI. They may even have negative um impact. Um, so not only will it impact sort of future-looking projects and growth, it may potentially impact ones that are already in place and have companies rethink or need to redesign those those solutions to manage costs more effectively. And that may even mean going to you know more human-based solutions in some cases, even for ones that were previously automated. Um kind of back to Uber as an example. A lot of people have have talked about Uber sort of winning their market initially by subsidizing costs and tried to compare that to what um AI labs have done now by subsidizing the cost of their models. Um, but there are there are some differences here. Um, I think you know, one is that that backwards cost impact that I just mentioned with with costs going up. You're not as a rider, if you rode Uber during its early subsidized days, two years in the future, you don't have to go back and pay more um for those rates, where effectively that's you know what might happen here if if LLM costs go up. Um Uber was also trying to drive other businesses out of the out of the market, and there's a lot more barrier barrier for entry there, overhead costs, things like that. Whereas AI automation is displacing human workers in a lot of places where there's not the same barriers to entry. If the if the cost trade-off adjusts, those people aren't going away. They they still exist. It's not necessarily the same as a large business with barrier to entry. Um, so if the costs of automation rise and surpass what humans would have cost, I think we'll quickly see an adjustment and more people coming back in and displacing those solutions. Um one of the other differences with with Uber is they're really providing sort of just kind of the platform layer, not the underlying infrastructure of the cars that that drive people around and things like that. Whereas labs are having to spend massive capital expenditures building out the AI data centers and the GPUs to be able to run these things. So there's a much bigger investment in infrastructure, not just sort of the coordination layer on top of people that are providing this infrastructure. Um, you know, lastly, I'm going to go into a little bit of economics and a term from economics that I've found helpful to understand LLM costs and sort of thinking about them differently than traditional software costs. And this is the concept of a rival versus a non-rival good. So a non-rival good can be consumed by multiple people simultaneously without reducing anybody's ability to consume it. That's really the model that we've had for most software, software as a service. If you know I'm going to a cloud hoster or using a website or things like that, it really has no impact on other people's ability to do so or drive up costs or availability or anything like that. Now, a good is rival if one person's use of it prevents or limits another person's use of the same same good. Um, and that's really the case with AI GPUs, especially for large models where each prompt and interaction of inference takes up a significant portion of that, and there are real limits there. Um, and that's that's one of the differences. So there is a lack of availability for these goods, and there are significant costs for a lot of the these large models the way that they are today. Um, and that's quite a bit different than how most other software and cloud software um has existed in our history. So I think it's it's challenging to shift our mindset when we're so used to thinking about a particular availability and financial model for software. So that's it for today. I hope you enjoyed. Um, you know, let me know what you think the future holds for large language models and AI in general and solutions. Uh, you know, please like and subscribe the channel. Um, check out all the resources in the descriptions, and I'll see you soon on the next episode of Talk2Me PDD. Thanks.