When AI Goes Off The Rails I 30th May

A mystery company just blew half a billion dollars on AI in a single month, while Boston Children's Hospital quietly saves lives with the same technology. From Groq's massive pivot to military AI training and the great mathematics debate, today we explore what happens when artificial intelligence meets human judgment - or the lack thereof. Plus, California's governor steps in as companies get dangerously AI-obsessed. This is the story of AI's growing pains in 2026.

Transcript

SPEAKER_01 0:00

There are two ways this story could end. In the first version, companies learn to deploy AI thoughtfully, with proper controls and human oversight, creating genuine value while protecting jobs that matter. In the second version, we get decision makers drunk on AI hype, burning through hundreds of millions of dollars while laying off a quarter of their workforce for digital agents they don't understand.

SPEAKER_00 0:23

And early reports suggest we just got a very expensive preview of which future we're heading toward. Half a billion dollars expensive.

SPEAKER_01 0:31

The question isn't whether AI will transform business. It's whether businesses will survive their own AI transformation. And based on what we're seeing today, that's not a given.

SPEAKER_00 0:41

Yeah, because when you're spending clawed money faster than some countries' GDP, you've crossed a line from innovation into something much more dangerous.

SPEAKER_01 0:52

You're listening to Build by AI. I'm Alex Shannon. And what you just heard, that's the sound of 2026's biggest AI reality check happening in real time.

SPEAKER_00 1:02

And uh I'm Sam Hinton. Today we've got uh everything from a mystery company's half billion dollar oops moment to actual life-saving AI deployments that are working exactly as intended. Plus Google's latest demos, military AI training, and why one startup just raised six hundred and fifty million dollars to completely change direction.

SPEAKER_01 1:23

It's honestly the perfect snapshot of where we are right now. AI doing incredible things in some places while creating absolute chaos in others. Let's dive in.

SPEAKER_00 1:32

Starting with that mystery company, because honestly, how do you even explain this to your board?

SPEAKER_01 1:43

Alright, so according to early reports from Tom's Hardware, an unidentified company, and we're talking completely anonymous here, spent five hundred million dollars on clawed AI usage in a single month. Half a billion dollars. The reason? They failed to put usage limits on employee licenses, so people just went wild with it.

SPEAKER_00 2:03

Dude, this is like giving every employee a company credit card with no limit and being shocked when someone books a private jet to lunch. Except worse, because at least with the jet you get frequent flyer miles.

SPEAKER_01 2:18

But wait, how does this even happen? Like, are we talking about a Fortune 500 company that just didn't read the fine print on their anthropic contract?

SPEAKER_00 2:27

That's exactly what I think happened. Look, Claude is incredibly powerful, but it's also incredibly expensive when you're doing heavy usage. If you've got thousands of employees all using it for everything, drafting emails, writing code, analyzing documents, having philosophical debates with an AI. Those API calls add up fast, like really fast.

SPEAKER_01 2:50

Okay, but I have to push back here. How do you not notice $500 million leaving your bank account? This isn't a rounding error. This is like multiple quarters of revenue for most companies.

SPEAKER_00 3:02

Right. And and that's what makes this so terrifying. It suggests this company either has absolutely terrible financial controls, or you know, they're so big that half a billion dollars can slip through the cracks. Either scenario is pretty alarming when you think about AI governance.

SPEAKER_01 3:24

And think about the psychology here. If your employees are burning through that much AI usage, what were they using Claude for? Were they just talking to it all day, having it write their entire jobs for them?

SPEAKER_00 3:35

Aaron Powell That's the million-dollar question, or in this case the $500 million question. Because if people were using Claude to literally do their entire workday, that tells us something pretty significant about either how easy their jobs were to automate, or how much they didn't understand the cost structure.

SPEAKER_01 3:53

And this ties into something bigger we're seeing. Box founder Aaron Levy is calling this kind of thing AI psychosis. Decision makers using AI to replace jobs without understanding those roles. ClickUp just laid off 22% of their workforce for AI agents.

SPEAKER_00 4:09

Exactly. And um here's the thing: if you can accidentally spend $500 million on AI, you probably shouldn't be making decisions about which humans to replace with AI. This is the corporate equivalent of texting while driving a bulldozer.

SPEAKER_01 4:24

But let's think about this from a business strategy perspective. What kind of company even has access to unlimited clawed usage? Are we talking about a major tech company? A financial services firm, maybe a consulting giant.

SPEAKER_00 4:46

Or it's a tech company where engineers were using it for everything from debugging to documentation. The scary part is without usage limits, there's no feedback mechanism to tell people they're being expensive.

SPEAKER_01 4:59

And the ripple effects here are huge. If one company can accidentally spend half a billion on AI, how many smaller companies are accidentally spending millions? This might just be the tip of the iceberg.

SPEAKER_00 5:11

Absolutely. And here's what's really concerning. Like this probably happened because someone at the C level, you know, said give everyone access to the best AI tools without understanding what unlimited access to Claude actually costs.

SPEAKER_01 5:30

So what's the takeaway here for companies looking at AI deployment? Because clearly the move fast and break things approach has some serious downsides when things cost half a billion dollars.

SPEAKER_00 5:41

Start small, set hard limits, and for the love of all that's holy, monitor your usage. This isn't just about money, it's a it's about proving you can handle AI responsibly before you scale it up. Because if you can't manage Claude costs, you definitely can't manage Claude replacing your workforce.

SPEAKER_01 5:59

And maybe just maybe understand what your employees are actually doing with the AI tools you give them. Because $500 million buys a lot of human employees who actually understand their own cost structure. Speaking of massive amounts of money, early reports suggest chipmaker Grok is seeking to raise $650 million in internal funding. But here's the interesting part. They're pivoting their entire business strategy. Instead of focusing on hardware, they're shifting to AI inference optimization, which is basically refining how AI models respond to user prompts.

SPEAKER_00 6:33

This is huge, and I think people are missing why. Grok built these incredible inference chips, like stupid fast for running AI models. But now they're saying the real value isn't in the chips themselves, it's in optimizing how AI thinks and responds.

SPEAKER_01 6:51

But wait, isn't that a pretty dramatic pivot? They've spent years building hardware, and now they're essentially saying actually, software is where the magic happens.

SPEAKER_00 7:00

Yeah, but think about it, NVIDIA just did that $20 billion not quite acquisition we mentioned, and the whole chip space is getting incredibly competitive. Meanwhile, inference optimization is this massive unsolved problem. Every company using AI is struggling with getting consistent, reliable responses from their models.

SPEAKER_01 7:22

So they're basically betting that knowing how to make AI models think faster and better is more valuable than making the chips that run them.

SPEAKER_00 7:30

Exactly. And honestly, they might be right. Hardware becomes commoditized eventually, but if you can consistently make AI models give better responses faster, that's something every AI company needs. It's like going from making race car engines to teaching drivers how to win races.

SPEAKER_01 7:49

But here's what I'm curious about. If Grox hardware is already optimized for fast inference, don't they have an advantage in understanding how to optimize the software side too? Like they've seen both sides of the equation.

SPEAKER_00 8:02

That's exactly why this pivot makes sense. They understand the hardware limitations better than anyone. So they they know exactly where the software optimizations can have the biggest impact. They're not just guessing at what will make models faster. They've been measuring it.

SPEAKER_01 8:21

The timing is interesting too. This comes right as we're seeing companies struggle with AI deployment, like our mystery company burning through clawed budget. Better inference optimization could actually help with cost control.

SPEAKER_00 8:34

Right. If Grok can can make AI models more efficient and predictable, that directly addresses the chaos we just talked about. Companies need AI that doesn't just work well, but works predictably and affordably. You know, and that's worth six hundred and fifty million dollars if they can pull it off.

SPEAKER_01 8:52

And think about the market timing here. Every company is trying to deploy AI right now, but most of them are struggling with consistency and cost. If Grok can solve both problems simultaneously, they're not just selling a product, they're selling peace of mind.

SPEAKER_00 9:07

Exactly. But plus inference optimization is something that gets more valuable as AI models get more complex. Like if GPT-5 or Claude 4 are even more powerful but also more expensive to run, Grok's optimization could be the difference between profitable AI deployment and burning money.

SPEAKER_01 9:25

I'm also wondering if this signals something broader about the AI infrastructure space. Maybe we're moving past the throw more compute at the problem phase into the make compute smarter phase.

SPEAKER_00 9:38

Raw computational power was the bottleneck for a while. But now that we have these incredibly powerful models, the bottleneck is figuring out how to use them efficiently. Grok might be betting that efficiency is the new frontier.

SPEAKER_01 9:56

Keep an eye on this because it might signal a broader shift in the AI space from let's build bigger, faster chips to let's make AI actually usable for normal businesses.

SPEAKER_00 10:07

And honestly, if they can help prevent more half billion dollar AI accidents, that alone might justify the investment. Sometimes the best innovation is just making existing technology less dangerous to deploy.

SPEAKER_01 10:23

Now, while companies are burning money and pivoting strategies, let's talk about AI actually working exactly as intended. Boston Children's Hospital has implemented open eye technology and successfully diagnosed over 40 rare disease cases. We're talking about real kids getting real diagnoses for conditions that might have gone undetected.

SPEAKER_00 10:46

It's not about replacing doctors, it's about making doctors superhuman. Rare disease diagnosis is this incredible puzzle where you need to connect symptoms that might seem unrelated, and AI is brilliant at pattern recognition across massive data sets.

SPEAKER_01 11:05

But how does this actually work in practice? Are doctors just feeding symptoms into Chat GPT and hoping for the best?

SPEAKER_00 11:13

No, it's way more sophisticated than that. The AI is probably trained on medical literature, case studies, and diagnostic criteria for thousands of rare diseases. When a doctor inputs a complex case, it can suggest conditions that a human might not immediately think of, especially for diseases that show up once every few years.

SPEAKER_01 11:34

And here's what's fascinating. The contrast with our first story is pretty stark. Here's a hospital using AI thoughtfully to save lives, while some mystery company is accidentally spending half a billion dollars. What's the difference?

SPEAKER_00 11:47

Purpose and process. Boston children clearly defined what they wanted AI to do, help with diagnosis, and they built systems around that goal. They probably have doctors who understand both the technology and the medical context making decisions. It's the opposite of just throwing AI at everything and hoping it works.

SPEAKER_01 12:10

But let's think about this from the hospital's perspective. Implementing AI in healthcare is incredibly risky from a liability standpoint. If the AI suggests the wrong diagnosis and someone makes a treatment decision based on that, who's responsible?

SPEAKER_00 12:24

That's exactly why this implementation is so smart. The AI isn't making diagnoses. It's giving doctors additional perspectives to consider. The human doctor is still making the final call, but now they have a tool that can say, hey, have you considered this rare condition that matches these symptoms?

SPEAKER_01 12:44

And it's improving both patient care and operational efficiency according to the reports. So we know AI can be deployed successfully when done right. But what makes a hospital better at this than a tech company?

SPEAKER_00 12:56

Hospitals are already used to high-stakes decision making with multiple checks and balances. They have protocols for everything, they understand risk management, and they know that failure can literally be life or death. Tech companies often optimize for speed and iteration. Hospitals optimize for safety and accuracy.

SPEAKER_01 13:16

That's such a good point. And when you're dealing with rare diseases, accuracy is everything. These are conditions where a misdiagnosis might mean years of unnecessary treatment or missed opportunities for actual treatment.

SPEAKER_00 13:30

Exactly. And here's what's beautiful about this: every rare disease diagnosis potentially helps the AI get better at the next case. These aren't just wins for individual patients. They're building a knowledge base that could help kids worldwide.

SPEAKER_01 13:48

So we're talking about a virtuous cycle here. Better diagnoses lead to better training data, which leads to better AI performance, which leads to even better diagnoses. That's the kind of AI deployment that actually makes sense long term.

SPEAKER_00 14:02

And think about the cost-benefit analysis here. Instead of spending $500 million accidentally, Boston Children's is probably spending a fraction of that to literally save lives and improve patient outcomes. That's ROI that actually matters.

SPEAKER_01 14:15

This should be the template for AI deployment everywhere. Clear objectives, expert oversight, measurable outcomes that actually matter. Not just AI will solve everything, but AI will help us solve this specific problem better.

SPEAKER_00 14:29

And notice how they're not replacing doctors. They're making doctors more effective. That's the sweet spot for AI deployment. You're not eliminating human expertise. You're amplifying it with tools that can process information at scale.

SPEAKER_01 14:43

Shifting to something that's both fascinating and concerning. Early reports indicate CBS News got access to US military training for AI deployment on the battlefield. We're not talking about science fiction here. This is apparently happening right now.

SPEAKER_00 14:57

This was inevitable, right? If AI can help doctors diagnose diseases faster, it was always going to end up helping military personnel make tactical decisions faster. The question isn't whether this would happen, it's how responsibly it's being deployed.

SPEAKER_01 15:14

But there's a huge difference between AI suggesting a rare disease diagnosis and AI making decisions in combat situations. The stakes are completely different, and so are the consequences of getting it wrong.

SPEAKER_00 15:27

Absolutely. And that's probably why CBS got access. Like the military wants public oversight and discussion about this. They're not hiding it. In warfare could be anything from better logistics and supply chain management to real-time threat assessment and target identification.

SPEAKER_01 15:46

I have to say, part of me is worried about the same AI psychosis we talked about earlier showing up in military contexts. Decision makers who don't understand the technology-making choices about life and death systems.

SPEAKER_00 16:00

But military procurement is usually way more rigorous than corporate AI adoption. They test everything extensively, they have clear chains of responsibility, and they understand that failure isn't just expensive, it's deadly. That might actually make them better at AI governance than most companies.

SPEAKER_01 16:20

But think about the speed of warfare versus the speed of AI development. Military systems are typically built to last decades, but AI models become obsolete in months. How do you reconcile those timelines?

SPEAKER_00 16:33

That's a really good point. You know, the the military might need to completely rethink how they approach technology procurement instead of buying a system that that works for 20 years, they might need to buy a platform that can continuously integrate new AI capabilities as they develop.

SPEAKER_01 16:50

On the other hand, if our mystery company can accidentally spend $500 million on Claude, what happens when AI military systems have similar oversight failures?

SPEAKER_00 16:59

Right. Uh which is why the training aspect is so important. If they're investing heavily in teaching people how to use these systems properly, that suggests they understand the risks. The biggest danger would be rushing AI into combat without proper human understanding and oversight.

SPEAKER_01 17:16

And there's the ethical dimension too. Even if the eye works perfectly, there are questions about autonomous weapon systems and where you draw the line on human control over life and death decisions.

SPEAKER_00 17:27

Exactly. But notice that they're emphasizing training. Training implies human operators who understand and control the systems. That's very different from fully autonomous weapons. It sounds more like they're building AI-assisted decision making rather than AI replacement decision making.

SPEAKER_01 17:47

Which actually brings us back to the Boston Children's Model. The AI provides information and suggestions, but humans make the critical decisions. That seems like a much safer approach than fully autonomous systems.

SPEAKER_00 17:59

Right. And if they can maintain that human-in-the-loop approach, AI could actually make warfare more precise and reduce civilian casualties, better intelligence, faster threat assessment, more accurate targeting, all of that could save lives on both sides.

SPEAKER_01 18:19

Keep watching this space because it's going to force conversations about AI governance that we probably should have been having already. Military applications have a way of clarifying the stakes for everyone else.

SPEAKER_00 18:31

And honestly, if the military can develop good AI governance frameworks under life or death pressure, maybe the rest of us can figure out how to avoid accidentally spending half a billion dollars on chatbots.

SPEAKER_01 18:47

Alright, let's hit some rapid fire updates. Google showcased nine demos of their new Gemini Omni and Gemini 3.5 models that were announced at Google iPlash O 2026.

SPEAKER_00 18:59

Classic Google announce at I.O. Then flood us with demos to prove it actually works. But honestly, nine demos suggest they're really confident in these models. You don't put out that much content unless you think you've got something special.

SPEAKER_01 19:14

And the timing is interesting with Grok pivoting to inference optimization. Everyone's trying to solve the same problem, making AI responses better and more reliable.

SPEAKER_00 19:24

Yeah. And you know, Gemini Omni, you know, it sounds like it's trying to be the everything AI, like text, voice, video, you know, the works. You know, if they can nail consistent performance across all those modalities, that's a pretty big deal for developers.

SPEAKER_01 19:41

But nine demos also means nine opportunities to spot inconsistencies. I'm curious whether they're showing cherry-picked examples or if this represents typical performance.

SPEAKER_00 19:52

That's always the question with uh AI demos, right? Though after seeing companies accidentally spend $500 million on Claude, maybe the bar for good enough to deploy is higher than we thought.

SPEAKER_01 20:05

And Google's positioning this as multimodal AI that can handle everything seamlessly. If that's true, it could simplify a lot of the complexity we see in AI deployment.

SPEAKER_00 20:15

Which brings us back to cost control and governance. One model that does everything well might be easier to budget for than multiple specialized models that each have different pricing structures.

SPEAKER_01 20:27

Governor Newsom signed an executive order addressing AI's economic impacts. If confirmed, this could be significant policy movements at the state level.

SPEAKER_00 20:36

California basically uh setting the tone for AI regulation again, with companies like ClickUp cutting 22% of their workforce for AI agents. Uh, someone needs to be thinking about the economic disruption. Newsom's probably seeing the same AI psychosis trends we talked about.

SPEAKER_01 20:56

And California has both the biggest AI companies and some of the biggest potential job impacts. Makes sense they'd want to get ahead of this.

SPEAKER_00 21:04

Right, because if if the mystery company burning $500 million is in California, the state definitely wants to understand what's happening in their own backyard before it spreads.

SPEAKER_01 21:16

I'm curious what specific economic impacts they're targeting. Are we talking about job displacement, wage effects, or broader economic disruption from AI deployment?

SPEAKER_00 21:26

Probably all of the above. When companies can accidentally spend half a billion on AI or lay off 22% of their workforce for AI agents, you need policy frameworks that can handle multiple types of economic chaos.

SPEAKER_01 21:40

And California's executive orders tend to become models for other states. This could be the beginning of a broader policy response to AI's economic disruption.

SPEAKER_00 21:49

Exactly. Someone has to figure out how to manage AI adoption responsibly. And it might as well be the state where most of these AI companies are headquartered. They have the Most skin in the game.

SPEAKER_01 22:02

Here's something wild. Early reports suggest AI has solved some Airdose mathematical problems, and apparently it's sparking major debate about the future of mathematics as a discipline.

SPEAKER_00 22:13

Oh man, this is like AI in medicine. But for pure mathematics, airdose problems are legendarily difficult. We're talking about puzzles that have stumped brilliant mathematicians for decades. If AI can crack these, it's not just solving problems. It's potentially changing how mathematical discovery works.

SPEAKER_01 22:35

But I imagine mathematicians have feelings about this. Like if AI can solve the hard problems, what's left for human mathematicians?

SPEAKER_00 22:43

Exactly the debate they're having. Though honestly, it might be like AI in medicine, not replacing mathematicians, but making them capable of tackling even bigger, more complex problems. The real question is whether AI solutions can teach us new approaches to mathematical thinking.

SPEAKER_01 23:03

And there's probably a difference between AI finding solutions and AI helping mathematicians understand why those solutions work. The proof might be more important than the answer.

SPEAKER_00 23:14

That's a great point. Mathematics isn't just about getting the right answer. It's about understanding the underlying principles. If AI can solve problems, but can't explain the mathematical intuition behind the solution, that's still a huge gap.

SPEAKER_01 23:30

Plus, Erdo's problems are famously elegant and often connect different areas of mathematics in surprising ways. I wonder if AI solutions maintain that elegance, or if they're just brute force computational approaches.

SPEAKER_00 23:43

And that gets to the heart of the debate, right? Mathematics as a field values insight and elegance, not just correctness. If AI solutions lack those qualities, they might solve problems without advancing mathematical understanding.

SPEAKER_01 23:59

And finally, we've got more context on that AI psychosis trend. Reports suggest tech layoffs in 2026 are nearly matching 2025 totals, with companies replacing workers with AI agents they don't fully understand.

SPEAKER_00 24:14

This ties everything together perfectly. Aaron Levy from Box nailed it, decision makers are using AI for job replacement without understanding those jobs. It's the same mindset that leads to accidentally spending $500 million on Claude. Rush first, think later.

SPEAKER_01 24:30

And click up cutting 22% of their workforce for AI agents is a perfect example. That's not optimization, that's disruption without a plan.

SPEAKER_00 24:39

Right. And while Boston Children's Hospital is saving lives with thoughtful AI deployment, other companies are creating chaos with thoughtless AI deployment. Same technology, completely different outcomes based on how humans choose to use it.

SPEAKER_01 24:55

What's scary is that 2026 layoffs are nearly matching 2025 totals. That suggests this isn't a temporary adjustment period. It's becoming a pattern of how companies think about AI adoption.

SPEAKER_00 25:07

And the AI-pilled terminology is perfect because it captures the obsessive, almost addictive quality of how some companies are approaching this. They're not using AI strategically, they're using it compulsively.

SPEAKER_01 25:21

The contrast with military AI training is interesting too. The military is investing in training people to use AI properly, while companies are just replacing people with AI entirely.

SPEAKER_00 25:32

Exactly. One approach builds capability and maintains human oversight. The other approach eliminates oversight and hopes the AI can handle whatever complexity the humans were managing. Guess which one leads to half billion dollar accidents.

SPEAKER_01 25:47

If you zoom out and look at everything we covered today, there's a clear pattern emerging. AI is simultaneously our most powerful tool and our most dangerous temptation.

SPEAKER_00 25:57

Yeah, and the deciding factor isn't the AI itself, it's human judgment. Boston Children's Hospital saves lives because doctors understand both the technology and medicine. Grok pivots successfully because they understand both hardware and software. Meanwhile, mystery companies burn through hundreds of millions because they understand neither their costs nor their employees.

SPEAKER_01 26:22

And that's what makes Governor Newsom's executive order and the military training so important. We need frameworks for responsible AI deployment before the technology gets so powerful that our mistakes become irreversible.

SPEAKER_00 26:35

But here's what's really interesting. The successful AI deployments all have something in common. They're augmenting human expertise, not replacing it. Boston Children's enhances doctors' diagnostic capabilities, military training enhances soldiers' decision making. Even Groke's pivot is about enhancing AI models performance.

SPEAKER_01 26:55

While the failures are all about replacement without understanding, replacing human oversight with unlimited AI spending, replacing workers with AI agents, replacing mathematical intuition with computational solutions, there's a pattern there.

SPEAKER_00 27:10

Exactly. Because if 2020 is teaching us anything, it's that AI doesn't fail. Humans fail at AI. The question for 2020 is whether we'll learn from Boston Children's Success Story or keep repeating the mystery company's very expensive mistakes. When decision makers get obsessed with AI as a solution, they stop thinking clearly about what problems they're actually trying to solve. That's how you get half billion dollar accidents.

SPEAKER_01 28:05

But it's also how you get breakthrough medical diagnoses and potentially revolutionary mathematical discoveries. The same technology that can bankrupt a company in a month can also save lives and advance human knowledge. The difference is entirely in how we choose to use it.

SPEAKER_00 28:27

We need guardrails that encourage the Boston children's approach while preventing the mystery company approach. The technology is too powerful for pure market forces to handle responsibly.

SPEAKER_01 28:40

Keep watching for companies that can clearly articulate why they're using AI and what success looks like. Those are probably the ones that won't accidentally spend your quarterly revenue on chatbot conversations.

SPEAKER_00 28:51

And look for the human expertise that's being enhanced, not eliminated. Because the most successful AAI deployments seem to be the ones where humans and AI work together, each doing what they're best at. Alright, that's build by AI for today. If your company is deploying AI, maybe set some usage limits first. Just a thought.

SPEAKER_01 29:14

And if you found today's stories helpful, hit subscribe so you don't miss tomorrow's episode. We'll be back with more AI news and hopefully fewer half billion dollar oops moments.

SPEAKER_00 29:25

Though honestly, those make for pretty entertaining stories too. See you tomorrow.