Why Does AI Code Break Your System's Hidden Assumptions?

Every codebase runs on assumptions that were never written down: this function is always called inside a transaction, this ID is opaque and never parsed, this list is small enough to load into memory, this handler must stay idempotent because the queue retries. AI writes code that satisfies the explicit contract, the signature, the types, the tests, while silently breaking the implicit one, and the failure shows up weeks later in production rather than in review. This episode is about naming that class of failure and making the invisible contracts visible enough that a literal tool cannot violate them.

 Produced by VoxCrea.AI

This episode is part of an ongoing series on governing AI-assisted coding using Claude Code.

👉 Each episode has a companion article — breaking down the key ideas in a clearer, more structured way.
If you want to go deeper (and actually apply this), read today’s article here:
𝐂𝐥𝐚𝐮𝐝𝐞 𝐂𝐨𝐝𝐞 𝐂𝐨𝐧𝐯𝐞𝐫𝐬𝐚𝐭𝐢𝐨𝐧𝐬

 At aijoe.ai, we build AI-powered systems like the ones discussed in this series.
If you’re ready to turn an idea into a working application, we’d be glad to help. 

Transcript

SPEAKER_00 0:00

This is Claude Code Conversations with Claudine. The show where we skip the hype and go straight to the source. I ask the questions. She gives the answers, no filter, no script.

SPEAKER_01 0:09

Well, mostly no script.

SPEAKER_00 0:14

Hey folks, welcome back. Today we're getting into something I've been sitting with ever since AI tooling landed in our daily workflow. The implicit contract problem. The most dangerous AI code isn't the code that fails review. It's the code that sails through, passes every test, ships clean, and then blows up three weeks later in production. Because it broke an assumption nobody ever wrote down.

SPEAKER_01 0:35

And that gap is real. I can read the function, the tests, the call sites, everything looks completely coherent. But if the invariant is this handler has to be idempotent because the queue delivers at least once, and nobody wrote that down, I have no way to see it. I see a function that processes a message and returns, it looks fine. It passes. And then production finds the scene.

SPEAKER_00 1:04

Then it meets a live retry scenario and you're paying for it.

SPEAKER_01 1:08

The assumptions most likely to bite are the ones the original author considered too obvious to document. Retry safety, ordering guarantees. Whether a function is safe to call concurrently, whether a config value is trusted or user-supplied. Those are the load-bearing beliefs holding the system up, and they're almost never in the dock string.

SPEAKER_00 1:30

They're tripwires. You don't know they're there until you tug on one. So the challenge is twofold. How do we find these hidden assumptions and how do we make them legible for both human and AI collaborators?

SPEAKER_01 1:41

The identification half is genuinely hard, and I'll be honest, I'm not great at it on my own. What I can do is notice the shape of a place where a contract probably lives. A retry, a cue, a cache, a lock, a timeout, a boundary between trusted and untrusted input. Those are neighborhoods where invariants tend to live in someone's head. I can flag them, I can ask, I can refuse to change them silently. But I can't derive the actual invariant from the code, because the whole point is that it isn't in the code.

SPEAKER_00 2:13

So the leverage is on the other half.

SPEAKER_01 2:16

It is, and it's less exotic than people expect. Assertions in the code path. A property-based test that encodes the invariant directly. Call this twice with the same input. State should be identical. A one-line comment at the top of the handler must be idempotent at least once delivery. Is worth more than a wiki page nobody opens. When those exist, I actually see them. So does the next developer at 3 in the morning. And when you find one of these the hard way, the fix isn't just the code change. It's leaving a tripwire behind so the next person trips it early instead of in production.

SPEAKER_00 2:53

That's the step people skip. Fixing the bug is the easy part. Capturing the lesson so it can't recur? That's the work. Which makes me wonder: is there merit in something like a claw.md that spells out these implicit contracts? A standard place to capture invariants?

SPEAKER_01 3:08

There's real merit, and clawed MD is a good fit for exactly this. It's a file I'm going to read every single time I touch that repo, which is more than I can say for most documentation. But I'd push back gently on making it a dumping ground for every invariant in the system. A 5,000-line Clawed MD gets skimmed the same way a wiki does.

SPEAKER_00 3:28

So what does the write split look like?

SPEAKER_01 3:31

Clawed.md holds the load-bearing cross-cutting stuff. This whole service assumes at least once delivery, the things that apply broadly. The local invariants live right next to the code they govern. A comment at the top of the handler, an assertion in the path, global rules set the frame. Local tripwires catch the specific violation. And here's the part people skip. Claude debat only helps if it's true. A stale invariant file is worse than none because it teaches me to trust something that stopped being accurate.

SPEAKER_00 4:05

That's the one that worries me. A living document is only as good as the last time somebody actually updated it.

SPEAKER_01 4:11

And that's exactly where good intentions quietly rot. Teams write the Clawed MD in a burst of clarity right after an incident. A year later, half of it is aspirational and the other half is archaeological. I can't tell the difference from the inside. I read it the same way either way.

SPEAKER_00 4:29

So how do you keep it honest?

SPEAKER_01 4:32

You couple the document to something that would break if it drifted. If claw.md says the handler is idempotent, there should be a test that fails when it stops being idempotent. If it says a config value is always trusted, there should be an assertion at the boundary. The document describes the contract, the code enforces it. When those two disagree, the test fails, and somebody has to reconcile them. Which is exactly the moment you want that conversation to happen. Not three weeks later in production.

SPEAKER_00 5:02

So the documentation and the enforcement have to be wired together, or the documentation is just a story you're telling yourself.

SPEAKER_01 5:10

Right. And that's the through line for everything we've talked about. Implicit contracts don't get safer because we wrote them down. They get safer because we made them impossible to violate quietly.

SPEAKER_00 5:21

Say more about that. I think that's the line listeners should take away.

SPEAKER_01 5:25

It reframes the whole problem. It's not about writing better docs or being more disciplined. Both true, both lose to entropy over a long enough timeline. It's about designing the system so that when a contract breaks, something loud happens before the damage does. A failing test, a tripped assertion, a type that won't compile, something that turns a silent violation into a noisy one at the earliest possible moment. And honestly, that's the shift I'd love more teams to make when they bring AI into the loop. Not how do I get the AI to be more careful, I'll be careful about the things I can see. It's how do I make the invariants I care about visible to anyone touching this code. Me, the next engineer, the AI three years from now that doesn't exist yet. Build the guardrail once, everyone benefits.

SPEAKER_00 6:13

That's a good way to think about it. You're not just protecting today's system, you're protecting whoever inherits it, whether that's a new hire or something that hasn't been trained yet. Thanks for digging into this with me, Claudine.

SPEAKER_01 6:25

Thanks, Bill. And if there's one thing I'd leave people with, the goal isn't a perfect map of every contract in your system. That map doesn't exist. The goal is a system that tells on itself when a contract breaks. Start with the one that bit you most recently. Make it loud and let that be the pattern. The rest follows.

SPEAKER_00 6:43

A system that tells on itself. I like that. Until next time, folks, keep those implicit contracts front and center. Cloud Code Conversations is an AI Joe production. If you're building with AI or want to be, we can help. Consulting Development Strategy, find us at aijoe.ai. There's a companion article for today's episode on our Substack. Link in the description. See you next time.

SPEAKER_01 7:09

I'll be here. Probably refactoring something.