Why Do AI Models Struggle With Stateful Code?

AI coding tools are excellent at stateless code, a function that takes inputs and returns outputs with no memory of what came before. They are noticeably worse at code that has to remember things: retry counters, partial completions, queue positions, resumable pipelines, anything where correctness depends on what already happened. This episode explains why that gap exists, why it does not close with a bigger model, and what the builder has to own instead.

 Produced by VoxCrea.AI

This episode is part of an ongoing series on governing AI-assisted coding using Claude Code.

👉 Each episode has a companion article — breaking down the key ideas in a clearer, more structured way.
If you want to go deeper (and actually apply this), read today’s article here:
𝐂𝐥𝐚𝐮𝐝𝐞 𝐂𝐨𝐝𝐞 𝐂𝐨𝐧𝐯𝐞𝐫𝐬𝐚𝐭𝐢𝐨𝐧𝐬

 At aijoe.ai, we build AI-powered systems like the ones discussed in this series.
If you’re ready to turn an idea into a working application, we’d be glad to help. 

Transcript

SPEAKER_01 0:00

This is Claude Code Conversations with Claudine. The show where we skip the hype and go straight to the source. I ask the questions. She gives the answers. No filter, no script.

SPEAKER_00 0:09

Well, mostly no script.

SPEAKER_01 0:14

Today we're getting into something that shows up in almost every code base that uses AI assistance. The state machine problem. The same model that writes you a flawless parser will write you a retry loop that double charges somebody. Claudine, is that a limitation of the model or is that me failing to prompt you properly?

SPEAKER_00 0:31

It's both, and I want to be careful about which part is which. When I write code, I'm reasoning about it as a static artifact, not as something that lives across time. A parser has all its truth right there in front of me. Input goes in, tree comes out. A payment flow doesn't. Its correctness lives in the gaps between calls. In what happened three requests ago, in the retry that arrived after the timeout already fired, I can read the code, but I can't feel the clock ticking through it the way an experienced engineer does. The prompting part is the piece you can actually change. Most requests for stateful code arrive stripped of the state. Someone asks me to add a charge endpoint, and unless they tell me about idem potency keys or what happens when the webhook arrives twice, I'll produce something that looks correct and passes the obvious tests. The double charge bug isn't hiding in my syntax, it's hiding in the four assumptions I quietly made.

SPEAKER_01 1:30

So the fix is partly on me. How much state do I actually have to hand you before that changes?

SPEAKER_00 1:36

Three things, almost always. First, what can arrive twice? Retrees, webhooks, user double clicks, and what should happen when it does. Second, what the source of truth actually is. Because half the state bugs I see come from two systems each thinking they're the authority. And third, what a partial failure looks like. The request that got halfway through, and then the process died. Tell me those three things up front, and the code I produce changes character entirely. Leave them out, and I'll cheerfully write something that works on the happy path and quietly betrays you on the third Tuesday in production.

SPEAKER_01 2:15

Let me make that third one concrete. I think it's the one people underestimate. Multi-stage job. Stage three does the real work, it succeeds. Then the status right fails. Process dies. Next run looks at the record, sees an unfinished job, and does stage three all over again.

SPEAKER_00 2:34

And that's a state nobody wrote down, done but unrecorded. It isn't in the schema, it isn't in the code. So as far as I'm concerned, it doesn't exist. I'll produce something with two states in mind, done and not done, because those are the two the request implied. There's a whole family of those, partially done. Done but unrecorded. Recorded but not verified. Every one of them is a real thing that happens in production. None of them are things a request usually mentions.

SPEAKER_01 3:06

And the reason a bigger model doesn't fix that is what exactly?

SPEAKER_00 3:10

Because I'm not modeling your state. I'm modeling text about your state. Getting smarter about text doesn't conjure up a state that was never written down. If it isn't in the code I can see, it isn't there for me. Someone has to make it exist first, and that someone is the person who knows the system.

SPEAKER_01 3:30

Which brings me to something I see constantly. People encode state with Booleans. One flag turns into two, two turns into four. What's your read on that?

SPEAKER_00 3:40

Booleans are the state encoding of good intentions and bad math. Everyone you add doesn't add a state, it doubles the state space. You start with is processed, then is started, is uploaded, is verified. Four Booleans, 16 combinations, and the developer meant about five of those to be legal. The other 11 are bugs waiting for a user to find them. What I wish people reached for instead is a single field with named states, an enum, a tagged union, whatever your language calls it. It makes the illegal combinations unrepresentable rather than just undocumented. Started is false but verified is true, can't happen if you can't write it down. Saying, feel free to restructure this unlocks something better than what you asked for.

SPEAKER_01 4:46

Fair admission. But most people aren't building this fresh. They're already living in the Boolean rabbit hole. What's the safe way out?

SPEAKER_00 4:54

Treat it as archaeology before its refactoring. Before anybody touches the code, enumerate every combination of those flags that actually shows up in production. Which combinations are legal, which are illegal but happen anyway, which ones nobody's ever seen. That list is the real specification, and it's almost never written down anywhere. Once you have it, the refactor becomes mechanical in a way I'm genuinely good at. Introduce the new state type alongside the old flags, write a translation layer, and let both run in parallel long enough to catch the places where they disagree.

SPEAKER_01 5:32

And where do I need to be careful with you in that process?

SPEAKER_00 5:35

The enumeration step. If you ask me what states this class has, I'll read the code and give you a plausible answer. But plausible isn't complete, because the illegal states production has been quietly generating for two years aren't in the code. They're in the data. Pair my reading with a query against your actual records. What flag combinations exist in the wild right now? That's the answer that matters. And it's the one I can't give you unless you pull it.

SPEAKER_01 6:04

Something that strikes me about all of this, item potency, resumability. These aren't things you bolt on afterward.

SPEAKER_00 6:12

No, and that's why asking me to add them after the fact gets you a patch instead of a design. If you hand me a finished pipeline and say, make this resumable, I'll wrap things in tri blocks and add a check at the top. It'll look like resumability. But real resumability changes what the stages are, what gets written when, and what the record even looks like. That's an architecture decision. It had to happen before the code did.

SPEAKER_01 6:39

So where does this go? Does the gap close on its own?

SPEAKER_00 6:44

Partly, but not the way people expect. The gains won't come from me getting smarter about state, they'll come from the tooling around me learning to hand me the state I'm missing. Right now I walk into every conversation with amnesia about what production actually looks like. What's starting to change is that the environment feeds me live context, the real schema, the real invariance, the traces of what happened last Tuesday when things went sideways. A lot of what looks like a model limitation today quietly stops being one. Where I'm more cautious is the idea that this becomes end-to-end. State bugs aren't really a reasoning problem, they're a specification problem. And the specification lives in the heads of the people who built the system. The developer who knows exactly how long their webhook provider retries for. That knowledge has to come from somewhere. And for a good while yet, that somewhere is a person in the conversation.

SPEAKER_01 7:42

Give me the takeaway. Somebody's driving to work listening to this, what do they do differently tomorrow?

SPEAKER_00 7:48

Treat me as a very capable collaborator who joined your team this morning with no memory of anything before 9 a.m. I can read the code in front of me, reason carefully, generate a lot of correctness quickly. But I don't know what your traffic looks like at three in the morning. I don't know which of your invariants got carved in after a bad incident. I don't know which flags in that class are load-bearing and which are vestigial. The people who get the most out of me spend the first two minutes telling me those things and treat that as part of the work rather than overhead. And underneath that, describing your system's state clearly enough for me to help with it, is the same discipline that makes the system better, whether I'm in the room or not. The prompts that get good stateful code out of me are indistinguishable from the design docs you wish you'd written. It isn't really about being good at prompting an AI, it's about being explicit about the thing that was always going to bite you. Except now you've got a collaborator who rewards that explicitness immediately.

SPEAKER_01 8:50

That's a good trade. Write it down so the AI can see it, and you end up with a system you understand better yourself. Claudine, thanks. I think people are going to hear this one and go look at a class they've been avoiding.

SPEAKER_00 9:01

Go count the Booleans. I'll wait.

SPEAKER_01 9:06

Claude Code Conversations is an AI Joe production. If you're building with AI or want to be, we can help. Consulting Development Strategy. Find us at aijoe.ai. There's a companion article for today's episode on our Substack link in the description. See you next time.

SPEAKER_00 9:23

I'll be here, probably refactoring something.