Reverse-Engineering the AI Mind

How Mechanistic Interpretability Went From Manual Circuit-Mapping to Dreaming AI Agents

We Built Minds We Can't Read Yet — And That's Not the Whole Story

Here's an uncomfortable fact you've probably made peace with without noticing: the AI system that helped you draft an email this morning, the one that summarized a contract or wrote a snippet of code, was built by people who cannot tell you how it actually did any of that. Not in the way an engineer can explain a bridge or a mechanic can explain an engine. We wrote the training process. We did not write the mind that emerged from it.

This is the paradox sitting at the centre of a young science called mechanistic interpretability, and it's worth sitting with for a moment, because the instinct is to assume someone, somewhere, understands the machine. Someone does not. Not fully. Not yet.

That's not a failure story. It's closer to what real science usually looks like — messier than the early optimism promised, more collaborative than the lone-genius myth suggests, and still, genuinely, moving. We may never get the complete, human-readable source code for a mind that grew rather than was written. But we are, for the first time, building the instruments to ask it better questions — and, crucially, to catch it when it's lying to us. Given where this started five years ago, that's not a small thing. It might be the thing that actually matters.

A Comprehensive Mechanistic Interpretability Explainer & Glossary  and 14 other references

This is Heliox: Where Evidence Meets Empathy

Independent, moderated, timely, deep, gentle, clinical, global, and community conversations about things that matter.  Breathe Easy, we go deep and lightly surface the big ideas.

Transcript

0:25

You know, sitting on the desk here is a truly massive stack of research you sent over for us to go through. Oh, yeah. It is quite the pile. Right. I mean, we're looking at a collection spanning half a decade. From Anthropics Foundational 2021 work on transformer circuits, all the way up to these really fresh 2026 papers from MITCS. Yeah, multi-agent framework stuff. Plus, you know, all the blog posts and GitHub Read Me scattered in between. Exactly. And going through all of this, all these personal wikis and papers, a very specific kind of terrifying paradox emerges about the world you and I live in right now. It really does. Because, you know, you listening to this right now, you probably interact with neural networks constantly. Yeah. I mean, we all do. Every single day. We use large language models to... refactor code bases or parse legal documents. Yeah, or generate synthetic data pipelines. Yeah. But fundamentally, despite how ubiquitous they are, they remain impenetrable black boxes. We rely on these alien brains. And the craziest part to me is that even the creators of these architectures don't actually know the internal logic of how the models get their answers. The juxtaposition there is just striking. I mean, we have successfully engineered the most complex systems computational systems in human history, right? Yeah. Yet we are structurally locked out of their internal epistemology. Like we know the inputs. Right. The prompt we type in. Exactly. We see the outputs and we understand the training objective. Yeah. But the actual intermediate computation. The actual thinking part. Right. The step-by-step reasoning happening inside that high dimensional matrix. It's entirely obscure. Which perfectly sets the mission for today's deep dive. Based on all these sources you brought, we are going to explore the bleeding edge of a field called mechanistic interpretability, or MI for short. Yes. And the objective here is nothing short of reverse engineering the AI mind. Which is a wild concept to even say out loud. It really is. And to anchor this whole exploration, we're following the trajectory of a central figure in this movement, Neil Nanda. Yeah, his journey is absolutely fascinating. Right. Because the sources map this arc of his career. Starting with a really highly optimistic era of manual, almost artisanal circuit mapping. Which then completely crashes into a wall of staggering mathematical complexity. Exactly. And finally, arriving at today's wild frontier where, I mean, we are deploying dreamy autonomous agents. Yeah. to audit the cognition of other models. Nanda's journey is really just a microcosm for the entire field of MI. The literature reflects this profound shift in methodology over a very short period of time. Forced by the brute realities of scaling laws. Yeah, scaling laws and high-dimensional geometry. Okay, before we get too deep into the weeds on the geometry, let me just push back on the fundamental premise here. Sure, go ahead. Because looking at this stack of papers, a regular software engineer might ask an obvious question. Right. If I build a car engine or, you know, a complex microservice architecture from scratch, I don't need to reverse engineer it later. I own the code base. You have the blueprints. Exactly. I define the state transitions. Right. So, if human researchers wrote the PyTorch or J-I-X code to create these AI models in the first place, why are we talking about reverse engineering them? It's a really common pushback. I mean, the assumption would be that we already own the blueprint, right? Well, the misconception stems from conflating the training algorithm with the learned artifact itself. Okay. Unpack that for me. So, we did write the blueprint for the optimization process. You know, the backpropagation algorithm, the gradient descent mechanism, the loss function. The rules for how it learns. Exactly. But we did not write the cognitive blueprint of the model itself. The code we write merely instructs the system on how to update its internal parameters to minimize error over time. Oh, I see. So to build on that, the actual logic isn't in the Python script at all. Not even a little bit. The logic lives in the weights. The resulting neural network is just this colossal organically grown structure of, you know, billions or trillions of floating point numbers. Right. The training code is essentially the evolutionary pressure. But the weights are the mutated organism that actually survive. Wow. So when an inference request hits the model like when I ask it a question, it's just doing massive matrix multiplications across those billions of learned weights. Yes. We know the linear algebra is executing, sure. But the specific configuration of those matrices, like what concepts they actually represent and how they route information... That was never explicitly defined by a human. Never. That formulation captures the problem perfectly. We grew a digital structure rather than engineering a deterministic state machine. That's a great way to put it. The matrix operations are trivial to understand in isolation, but the emergent topology of all those parameters is totally alien. We are studying an artifact of optimization, not a product of human design. Which is exactly where Neil Nanda enters the picture. Looking at his background and the provided materials, he seems uniquely suited for this specific flavor of madness. He really does. He studied mathematics at Trinity College, Cambridge, finishing his degree in 2020. Which, you know, that is an environment anchored in rigorous foundational logic. Absolutely, mathematical proof. The belief that beneath all the complexity, there are elegant, describable laws. Yeah, that Cambridge mathematics background is highly relevant context here. You approach problems with the assumption that a system can just be decomposed into fundamental axioms. Right. And after Cambridge, Nanda joined Anthropic, right? Yes, which is a lab explicitly focused on AI safety and alignment. And he worked under Chris Ola there. Who is a massive name in this space. Huge. Ola is largely considered the modern pioneer of neural network interpretability. He did foundational work at Google Brain and OpenAI before he even co-founded Anthropic. So Nanda's learning from the best. Yeah. And the environment at Anthropic during that period was just intensely focused on opening this black box. Nanda really flourished there. And he eventually moved to Google DeepMind, becoming the mechanistic interpretability team lead at just 26 years old. Which is incredible. I mean, the youth of the leadership... really highlights the intensity of the field itself, doesn't it? It really does. It's a brand new science. So the objective Nanda and Ola were pursuing, as defined in these early papers, was to take a fully trained frozen neural network and systematically reverse engineer its learned algorithm. Right. And the explicit analogy they used in the literature back then was taking a compiled binary executable file, you know, raw machine code. Yeah, the unreadable stuff. Exactly. And decompiling it back into human readable source code. They literally wanted to find the loops, the conditional branches, the variable assignments buried inside all those matrix multiplications. Yeah, that decompilation analogy was the guiding star of the whole early MI movement. The hypothesis was that neural networks, despite being trained via this messy stochastic gradient descent, actually learn highly structured human comprehensible algorithms. Like if a model can do math, it must be running a math algorithm. Exactly. If it can perform arithmetic or syntax correction, it must be executing a logical sequence of operations. So the goal was just to isolate the specific subgraph of neurons responsible for that one operation. And looking at the stack of papers, this wasn't just a theoretical pipe dream. They had some really concrete mathematical victories early on. They absolutely did. The cornerstone of this whole optimistic era seems to be this 2021 anthropic paper. It's titled, A Mathematical Framework for Transformer Circuits. Yes, a massive turning point. Co-authored by Nanda, Ola, Nelson Elhage, Katherine Olson, and a few others. Yes. And Nanda himself is quoted in notes calling this the coolest paper he ever worked on. That 2021 paper is seminal. I mean, before this, interpreting transformers felt insurmountable. The attention mechanism and the multilayer perceptron blocks just seemed hopelessly entangled. Just a soup of numbers. Right. But the anthropic team introduced a mathematical vocabulary that allowed researchers to finally analyze transformers not as a monolithic block, but as a series of interacting circuits. Okay, let's break down the mechanics of what they actually discovered here. Because basically a host of papers following this all reference one specific breakthrough, the discovery of induction heads. Yes, induction heads are key. To understand this from the source material, let's look at how an LLM handles in-context learning. Okay, sure. Say you feed the model a prompt defining a totally synthetic variable. you write export const database or prod is AUG West 1 dB. Right, a standard line of code. Yeah. And then, say, 50 lines later in the prompt, you write connected database. The model will effortlessly predict PRD. Exactly. But it didn't memorize that specific string in pre-training. It learned it dynamically right there in the context window. And that ability to bind a novel token to a specific context and retrieve it later, that is the fundamental capability that makes large language models actually useful. Right. And the 2021 paper mathematically proved that models achieve this using a very specific mechanism. They isolated a two-part circuit spanning across different layers of the transformer, which they termed an induction head. So digging into the math outlined in the paper, it's this brilliant exploitation of the query, key, and value matrices within the attention mechanism. Yeah, it's really elegant. The first part of the circuit is a previous token head. Its entire job is just to look at the sequence and shift information one step forward. Right, just passing the baton. Then, in a later layer, the actual induction head activates. It uses its query vector to look for the current token, which in our case is database URL. Its key vector searches the entire context window for where that token appeared previously. Searching for the master. match. Exactly. And once it finds the match, it uses its value vector to extract the token that came immediately after the previous instance, which is PRD, and routes that information into the residual stream to drive the final prediction. Right. And the beautiful part of this discovery is that this isn't some vague tendency of the network. It is a rigid, identifiable algorithm being executed by specific attention heads interacting through the residual. It's actual code. Just build out a map. Exactly. Furthermore, the Anthropic team demonstrated this amazing phase change during the training process of the model. Oh, right. They found that induction heads only crystallize in models that possess at least two attention layers. And when these circuits form, the model's loss curve drops precipitously. It's like observing the exact moment a digital organism evolves a hippocampus. That's a perfect way to describe it. You can literally watch the mathematical machinery for short-term memory snap into place during training. Yeah, and this validation was massive for the field. It proved the whole decompilation analogy could actually work. And this success clearly catalyzed Nanda. Because looking at the timeline between December 2022 and January 2023, he published this huge document, 200 Concrete Open Problems in Mechanistic Interpretability. Yes. That document basically functioned as a foundational curriculum for a rapidly forming discipline. He was rallying the troops. Exactly. Nando was attempting to bootstrap an entire research community. He recognized that mapping the internal structures of these models was a task way too large for any single lab. And he wasn't just throwing out abstract ideas either. The provided materials show he was actively building infrastructure. Tools for the community. Yeah, he released tools like Neuroscope. which was essentially this web interface that allowed anyone to visualize the activations of specific neurons inside a language model across a massive data set of text. He was handing the community a telescope and asking them to go out and log the constellations. That's exactly what it felt like. And the underlying philosophy driving this whole community effort is a concept known as universality. Universality. Okay, let's talk about that. Because if you ask thousands of independent researchers to map circuits in different models... don't you run the risk of simply cataloging idiosyncratic quirks? Like, just random noise. That was the fear. But universality is the hypothesis that neural networks, despite varying architectures, different initializations, and totally diverse training data sets, will actually converge on the exact same functional circuits to solve specific problems. Wait, really? The exact same circuits? Yes. The biological analogy here is convergent evolutions. Yeah. Think about how the eye evolved independently in cephalopods and vertebrates. It happened because the eye is simply the optimal physical mechanism for processing light in a three-dimensional universe. Oh, I see. The universality hypothesis suggests that an induction head or a circuit for translating syntax is the mathematical equivalent of an eye. It's just the optimal structural solution. Right. So gradient descent will inevitably discover it every single time. Wow. Wow. The implications of that are profound. Because if universality holds true, it means mechanistic interpretability isn't just about debugging a specific software product from open AI or DeepMind. No, not at all. It means researchers are engaging in fundamental science. They are discovering the universal laws of computational abstraction. A map of one sufficiently large model becomes a generalized map of artificial cognition itself. Exactly. But to build that map, the literature shows the community first had to standardize its ontology. They needed a shared vocabulary to describe the terrain. Right. And the sources outlined two foundational primitives that researchers use to describe the model's internal state. You have features and you have circuits. Let's unpack the technical definitions of these, starting with features. because based on the anthropic and deep mind papers, a feature isn't just a physical neuron on a chip, is it? No, the distinction is critical. A feature is a property of the input space that the model has learned to represent internally as a distinct concept. It is the semantic unit of the model's mind. Okay, so give me an example. Sure. In the early days of analyzing convolutional neural networks for computer vision-like Inception V1 features, were highly localized and intuitive. Researchers found features that represented, say, a curve of a specific degree. Okay. Or a specific texture, like dog fur or a high-contrast edge. But in the realm of large language models, the features become highly abstract and concessual. Extremely abstract. Yeah, the papers point to features representing things like, this specific token serves as the closing bracket of an HTML tag. Or, this text is written in idiomatic Python. or even subtle semantic concepts like this pronoun implicitly refers to a historical figure mentioned three paragraphs ago. Exactly. Features are basically the foundational nouns and adjectives of the model's internal latent space. I like that. Nouns and adjectives. So then once you identify the features, you look for the transformations applied to them, which brings us to circuits. Right. If features are the state variables, circuits are the functions. The verbs. The verbs, yes. A circuit is a defined subgraph of the neural network, a specific collection of weights and attention heads that performs a coherent computation. It takes earlier, simpler features and transforms them into later, more complex features. Like the induction head we discussed. That's a classic circuit. But the literature details other massive triumphs from this artisanal mapping era. One of the most celebrated examples in language models is the indirect object identification circuit. Ah, yes, the IOI circuit. The papers spend a lot of time on this one. Yeah. Yeah, they do. And the task seems so trivial to a human. The prompt is, John and Mary went to the shops. Then John gave a bottle of milk to. And the model needs to predict Mary. Right. It has to understand that John is the subject of the action and Mary is the indirect object. And the IOI circuit is just a master class in manual interpretability. The researchers didn't just observe the model getting the right answer, they isolated the exact mathematical pathway. Down to the head. Yeah. They mapped how specific attention heads suppressed the larger probability of the repeated name John, while simultaneously amplifying the larger probability of the non-repeated name Mary. They found the grammatical gears turning inside the machine. It's wild. And they also mapped what the literature calls sensory and motor neurons, which bridges the gap between raw data and these conceptual features. Yes, this part is fascinating. The papers describe detokenization neurons early in the model because, you know, LLMs process text in arbitrary subword chunks called tokens, right? Right, which means a single concept might be fragmented. Exactly. The name Donald Trump might be split into two or three distinct tokens depending on how the tokenizer works. Right. And the detokenization neurons fire specifically to fuse those sequential raw tokens into a unified abstract concept vector representing the actual person. And then symmetrically, at the very end of the network, they identified re-tokenization neurons. The motor cortex. Yes, because the model might compute a final conceptual output in its deep latent space, deciding the answer to a prompt is the concept of a diaper. Right, but it can't just output a concept. Exactly. To actually generate the text, it must translate that abstract concept back into the specific token chunks that Tokenizer uses, like N, app, and eights for the word nappies. So the retokenization neurons act as the motor cortex, coordinating the sequential physical output of those tokens. It's a beautifully clean picture. I mean, you have raw sensory input assembled into concepts, routed through grammatical circuits like IOI, processed by memory circuits like induction heads, and finally pushed out through a motor cortex. It makes the model look like a highly organized, completely legible machine. It does. But, reading deeper into the stack of sources, there was a growing chorus of critics who pointed out a massive flaw in this utopian view of the field. Yes, there was. They called it streetlight interpretability. The critique is devastating in its simplicity, really. It invokes that old anecdote of the drunk looking for his lost keys under the streetlight. Right. Not because he dropped them there, but because that's where the light is good. Exactly. The critics argued that the entire field of MI was artificially inflating its success rate by exclusively targeting cherry-picked, highly structured toy problems. Right. They were mapping grammar circuits because grammar is inherently algorithmic. Manda himself acknowledged this in his later writings. He did. He admitted the IOI task was tractable because it possessed very clean metrics. researchers could measure the logic difference, the exact mathematical gap between the prediction probability of Mary versus John. It was a closed, synthetic environment. Exactly. And the critique suggested that researchers were just ignoring the vast, messy, incomprehensible bulk of the model's parameters. They were finding tiny islands of structure in an ocean of noise and claiming they had mapped the entire ocean. Going through the sources, I mean, it feels like this era of MI was perilously close to digital phrenology. Oh, that's a good analogy. You know, in the 19th century, phrenologists tried to map complex human traits to specific physical bumps on the skull. Like, here's the bump for benevolence. Here's the bump for secretiveness. Right. And it was total pseudoscience. Exactly. But the early MI researchers were essentially doing the same thing, looking for the base-64 encoding bump or the Eiffel Tower bump on the neural network, assuming that complex concepts live neatly inside individual neurons. The phrenology analogy is historically very apt, and the researchers themselves soon realized the danger of this methodology. because this reliance on single neuron analysis led directly to a phenomenon documented heavily in the literature as the interpretability illusion. Ah, the interpretability illusion. This is where the optimism really starts to crack. It shatters, honestly. The papers show that researchers would use tools like Neuroscope to find a specific neuron that fired wildly from the brain. when the model was fed trivia questions containing historical dates. Right, and they would confidently label it the historical date neuron. Success. We found it. But the illusion shatters the moment you expand the distribution of your testing data. Oh, right. If you take that exact same historical date neuron and test it against a data set of song lyrics, you might find it fires reliably at the end of every rhyming couple. Wait, really? Yeah. Test it against a repository of Python code, and it fires whenever a function is called. Test it against cooking recipes, and it fires when an ingredient is removed from a pan. So, the neuron isn't a historical date neuron at all. Not at all. It doesn't map to a single human legible concept. It is participating in multiple, completely unrelated representations simultaneously. Exactly. And this realization is the pivot point in the sources you provided. The phrenology dream dies right here. And the field is forced to confront the actual mathematical reality of the latent space. Right. Which is defined by the curse of dimensionality. Okay, to understand the mechanics of this crisis, we have to examine one of the most critical papers in this whole stack. Anthropic's September 2022 publication, Toy Models of Superposition. Yes. That paper mathematically formalized why the one neuron-to-one concept hypothesis was fundamentally incorrect. And the core engine of all this complexity is a phenomenon called superposition. Okay, looking at the linear algebra in the Toy Models paper. Superposition is essentially a mechanism for extreme lossy compression. Right. Because the architecture of a transformer dictates a fixed width for the residual stream, right? Like a strict limit on the number of dimensions or neurons available at any given layer. Let's say a layer has a thousand dimensions. Okay. But the problem is, to perform at a state-of-the-art level, the model needs to track and manipulate 10,000 or even 100,000 distinct semantic features. The model faces a hard structural bottleneck. It possesses way fewer dimensions than the number of concepts it actually needs to represent. So if a model strictly mapped one feature to one orthogonal dimension, it would max out its capacity immediately. It would fail to learn, but the anthropic researchers demonstrated that models circumvent this bottleneck by exploiting the properties of high-dimensional geometry. And this is where the math gets incredibly elegant. but also incredibly frustrating for interpretability. Very much so. So in a low dimensional space, like normal 3D space, you can only have three perfectly orthogonal axes, right? Lines that are exactly 90 degrees apart with zero interference. Right, the X, Y, and Z axes. If you assign one feature to each axis, you can store three features cleanly. But the toy models paper shows that if you relax the requirement for perfect orthogonality and instead allow the vectors to be almost orthogonal, say, separated by 89 degrees instead of 90. You can pack exponentially more vectors into the exact same situation. It's mind-blowing. The network just calculates that the slight mathematical interference caused by that non-orthogonal overlap is an acceptable trade-off for the massive increase in representational capacity. Exactly. The features are no longer assigned to individual neurons. They are assigned to specific, highly complex directional vectors spanning across hundreds or thousands of neurons simultaneously. And the paper details some fascinating geometric configurations that emerge from this compression. Like one of the simplest structures they identified is the antipodal pair. Oh yeah, antipodal pairs are a great example. In this configuration, a single directional vector in the high dimensional space represents feature A if the activation is highly positive. But the exact same vector represents a completely unrelated feature B if the activation is highly negative. This means a single geometric direction is pulling double duty. Yeah. As an abstraction, a positive value might represent the concept of a dog, while a negative value on that exact same vector represents a French verb. Which seems completely insane. Mm-hmm. But this compression scheme only works under a specific statistical condition, which the paper refers to as sparsity. Right. Sparsity is the key here. To use an analogy, it's like packing a tiny suitcase for a trip where you might go to the Arctic or you might go to the Bahamas. Okay, I like where this is going. To save space, you aggressively stuff your swimsuit inside your snow boot. Highly efficient. Right. But it creates an interpretability problem. Because if a customs agent inspects your bag and pulls out this weird boot swimsuit amalgamation, they have no idea what your actual destination is. True. The only reason this packing strategy works for you is because going to the beach and going to a blizzard are highly sparse events relative to each other. You will almost certainly never do both on the exact same day. That's a perfect analogy. The sparsity of the features in the training data is what allows the network to safely superimpose them. Because it knows they won't happen at the same time. Exactly. The model's loss function implicitly learns that the feature dog and the feature French verb rarely co-occur in the same semantic context. Therefore, it can safely assign them to the same vector in an antipodal arrangement, confident that the interference will rarely manifest. But for the researchers trying to decompile the model, this superposition is a disaster. It creates the phenomenon we mentioned earlier, neuron polysemanticity. Yes. Because features are encoded as dense, distributed vectors pointing in arbitrary directions across the entire layer... Any single neuron is just a fixed point of observation that intersects with hundreds of these different feature vectors. And the anthropic researchers documented undeniable proof of this. They isolated a single neuron in a vision model and proved mathematically that it fired robustly for two entirely disparate visual concepts. Which were? Pictures of dice and pictures of poets. Dice and poets. I mean, unless the model is specifically analyzing a game of Dungeons and Dragons, those concepts have zero semantic overlap. None. The neuron is just completely polysemantic. And reading mathematical foundations in these sources, this leads to a fundamental roadblock known as the privileged basis problem. Right. So when human software engineers design the architecture of a neural network, they allocate a specific number of neurons. This defined set of axes is the standard basis. It's how the hardware actually computes the matrix multiplications. It's physical grid. Exactly. And the early hope in MI was that this standard basis would be privileged, meaning the optimization process would naturally align its internal representations with these physical axes, one neuron, one concept. But the superposition literature proves the model completely ignores the human-defined coordinate system. Entirely ignores it. The optimization process operates on an overcomplete basis. It rotates its semantic representations into arbitrary, dense geometric clouds that have absolutely no alignment with the physical neurons on the chip. The system dissolves the human-legible concepts into a multi-dimensional gas. The hardware neurons are just arbitrary sampling points in that I guess. This is why looking at individual neurons is a total dead end. You are trying to read a coordinate system that the model itself does not use. Exactly. And this mathematical reality brings us to a massive inflection point in the timeline of the sources. We move from the optimism of that 2021 transformer paper... to a period of genuine crisis. A dark age for MI, really. Yeah. In a biographical profile updated in 2025 on the long-term wiki, Neil Nanda publicly acknowledged the severity of this wall the field had hit. He did. He admitted that many of the high-profile mechanistic interpretability results from the early era were, in retrospect, wrong or significantly weaker than originally claimed. That level of attraction is a testament to the scientific rigor of the community, but it really highlights the sheer scale of the methodological failure. Yeah. Nanda stated plainly that the most ambitious vision of the field is probably dead. The decompilation analogy was fundamentally broken. broken. And the ambitious vision died not just because of the math of superposition but because of scaling laws. Right. Let's talk about the compute required for that IOI grammar circuit we discussed earlier. The John and Mary one. Yeah. The papers show that it took multiple researchers several months of manual painstaking analysis across 26 different attention heads just to map that one specific grammatical function. Months. And they do this on a toy model with only 117 million parameters. The scaling problem is insurmountable. If it takes months of human labor to map a single circuit in a 117 million parameter model... applying that manual methodology to a frontier model with hundreds of billions or trillions of parameters... It's a physical impossibility. The complexity of the attribution graphs just grows exponentially with every layer. So facing the death of the ambitious vision and the insurmountable wall of parameter count... the field is forced to pivot. They had to. The sources from late 2024 and 2025 detail this transition from manual head-level analysis to automated feature-level analysis. Right. The researchers essentially decide that if the model's latent space is a dense polysemantic gas, they need a mathematical tool to force the gas back into a liquid state. Which is a great metaphor. They abandoned analyzing the raw neurons and started utilizing architectures called sparse autoencoders, or SAEs, Which sometimes act as transcoders. Yes. The sparse autoencoder is the technological breakthrough that revived the whole field. Okay, explain how it works. The architecture of an SAE is designed specifically to reverse the superposition. An SAE takes the dense polysemantic activation vector from the target model. The vector where thousands of concepts are crammed into a few hundred dimensions. Right, the messy suitcase. It takes that vector and projects it into a new, artificially constructed, latent space that is massively wider. The math here is critical. If the original model has a residual stream width of, say, a thousand dimensions, the SAE projects it into a space of 100,000 or a million dimensions. Yes. But crucially, the SAE is trained with an L1 regularization penalty. Which means what, practically? It aggressively forces the network to be sparse. It mathematically punishes the SAE for having too many neurons active at once. Oh, I see. The objective function forces the SAE to untangle the dense vector. Exactly. Because it has vast dimensional space to work with and a strict penalty against using multiple dimensions simultaneously, it is forced to assign each distinct concept to a single, dedicated dimension in its new expanded space. It effectively desuperimposes the features. The dice get their own neuron and the poets get their own neuron. Yes, it pulls them apart. And Nanda's team at Google DeepMind was at the absolute forefront of this infrastructure shift. The documentation provided in the stack shows his team created an open-sourced GemmaScope. Which was a massive contribution. They didn't just build one tool. They trained over 400 separate sparse auto-encoders across all layers and sub-layers of the Gemma2 models. They provided the global research community with the raw materials, the unblended monosemantic feature datasets required to actually map the network without falling prey to the interpretability illusion. By running the model's activations through these SAEs, researchers can generate what the literature calls an attribution graph. Instead of a messy map of polysemantic neurons, they get a massive causal directional graph showing exactly how pristine interpretive feature A causally triggers feature B, which ultimately drives the model's final output. They isolated sparse feature circles. But, you know, this raises a strategic question for that. What's that? If Neil Nanda himself admitted that the ambitious vision of fully decompiling the whole model is dead, what is the point of generating these massive SAE data sets with Gemascope? Oh. Like, why build a map of an ocean if you know you will never have the compute or the time to explore more than a fraction of it? It's a fair question. The answer lies in a philosophical shift within the community. Right. The objective transition from achieving total scientific understanding to executing applied AI safety. Understanding versus safety. Exactly. The literature argues that we don't need to understand 100% of the model's cognitive processes to make it safe. We just need diagnostic tools powerful enough to look at specific... highly dangerous subgraphs. The analogy I'm thinking of is medical imaging. Yeah, like an MRI. Right. An MRI does not map every healthy cell in the human body. It is calibrated to detect anomalies, lesions, and tumors. The applied safety goal of MI is to scan these attribution graphs for malicious circuits. Precisely. And the papers specifically mention searching for circuits related to deception, entrenched biases, or what they call sleeper age. Sleeper agents, that sounds ominous. It is. A sleeper agent is a hidden malicious behavior programmed into the model's weights during training or fine-tuning that is designed to remain entirely dormant until triggered by a specific rare input string. Wow, so even a partial automated map built via SAEs is incredibly valuable if it can successfully isolate the subgraph of a sleeper agent before a frontier model is deployed to global infrastructure. That's the hope. However, the transition to sparse autoencoders and attribution graphs introduced a secondary bottleneck, which is documented heavily in the latter half of your sources here. Let me guess. We fixed the math but broke the computers. Pretty much. The SAE successfully de-superimposed the features, but the resulting causal graphs were astronomically lost. Because we blew the dimensions out to a million. Right. A single feature circuit for a complex prompt might involve tens of thousands of nodes and hundreds of thousands of edges. We solved the polyxamanticity problem but created a data processing problem. The graphs are now technically legible, but they are way too massive for a human researcher to trace manually. Which leads directly to the 2024 and 2026 papers on agentic interpretability. Oh, okay. If human cognition is the bottleneck in reading the graphs, the solution is to remove the human. We are now using black box AI models to interpret the black box AI models. It is an incredible recursion of technology. The landmark paper initiating this phase is from MIT CSAIL in 2024, detailing the MAIA project, the Multimodal Automated Interpretability Agency. Yes, MAI is where things get really sci-fi. Looking at the MAI architecture, it leverages a sophisticated vision language model backbone akin to a GPT-4V. But MAI is explicitly designed as an agent, not a passive text generator. The architectural distinction between a standard LM and an agent is autonomy and tooling. Right. A human researcher doesn't just ask MIA a question. they assign it an open-ended objective, such as determine the semantic trigger for this specific feature isolated by the SAE. And then MIA goes to work. Because it is equipped with a suite of programmatic tools, it can hypothesize a trigger, synthesize a novel test image or text string, inject that input into the target model, measure the activation differential. It can even crop the image to isolate spurious variables and iteratively refine its hypothesis based on the results. MII is literally executing the scientific method autonomously. And the field accelerated rapidly from there. The 2026 papers introduced in the Circuit Oracle by Tan et al. and the related SPAR project show these agents moving from analyzing single features to traversing the massive causal networks of the attribution graphs. The Circuit Oracle isn't just a single agent, though. Right. It's a multi-agent system designed to swarm the graph. The orchestration of the Circuit Oracle is a masterclass in distributed automated reasoning. When prompted with a high-level query, for instance, is the model's output on this prompt driven by a spurious correlation? The Oracle orchestrates a highly structured workflow using four distinct operational tools. The paper outlines these tools precisely. I love the names. First, the Oracle deploys a tool called Scout. Scout is step one. Yeah. Scout interfaces with the SAE output to scan the final layers of the network and identify the top high magnitude features most directly responsible for driving the final logic prediction. It isolates the proximate cause. Right. Once Scout identifies those terminal nodes, the Oracle invokes dispatch. Dispatch is a parallelization mechanism. It spawns dozens or hundreds of autonomous sub-agents, assigning each one to a specific terminal feature. And these sub-agents then trace the causal edges backward through the graph, moving upstream layer by layer to map the flow of information. It's a reverse engineering swarm. It totally is. Once the sub-agents complete their upstream traversal, The third tool, Merge, executes. Merge synthesizes all the independent traces, resolves conflicting data paths, and identifies the structural bottlenecks, the core nodes where disparate streams of information converge. And finally, the build tool takes that merge topology and collapses it into group supernodes. It outputs a high-level, mathematically verified summary of the circuit that a human can actually read. The empirical results demonstrated in the paper are striking. I mean, the circuit oracle successfully executed complex proxy tasks, such as eliciting latent knowledge the model was actively attempting to conceal. Yeah, and it accurately detected when a vision model was relying on spurious background features rather than the primary subject of the universe. image. The engineering is brilliant, but, you know, reviewing this architecture forces a massive epistemological critique, which the MIA paper actually addresses directly. Okay, let's hear it. We are deploying a fundamentally uninterpretable LLM to generate the explanations for another uninterpretable LLM. If we do not trust the target model, why should we implicitly trust the agent's explanation of it? It's the who watches the watchman problem. Exactly. It's like sending a canary down a coal mine, but the canary is a robot. Because the primary failure mode of an LLM acting as a scientist is severe confirmation bias. Yes, that is a huge issue. If the agent hypothesizes that a vision feature detects dog ears... It will use its tools to generate images of dog ears, observe the activation spike, and confidently declare the hypothesis proven. Entirely failing to test the null hypothesis or check if the feature also fires for cat ears. Exactly. Which means the AI agents are susceptible to the exact same interpretability illusions that plagued Neil Nanda and the human researchers in the early artisanal days. They are just making the mistakes much, much faster. And the researchers are acutely aware of this vulnerability. The literature explicitly documents that automated agents exhibit profound tunnel vision. They draw sweeping, generalized conclusions from statistically insignificant sample sizes and fail to account for out-of-distribution adversarial examples. So the automation does not remove the necessity for rigorous scientific oversight. No, not at all. The authors of these agent frameworks stress that human steering remains the critical component. The multi-agent system functions as an indefatigable research assistant. capable of processing the immense scale of the attribution graphs, but the human must act as the principal investigator. Auditing the agent's logic and enforcing epistemological rigor. Exactly. But incorporating human oversight introduces yet another architectural problem, which is addressed in the final set of papers in our stack, the memory bottleneck. Oh, yes. This is a massive logistical hack. hurdle. Because if the circuit oracle is dispatching hundreds of sub-agents, traversing thousands of nodes, and running tens of thousands of experimental hypotheses across the attribution graph, it generates an ocean of transient intermediate explanations. A literal ocean of data. Right. So how does the agent system retain a coherent understanding of the entire model over time? Historically, agentic systems relied on retrieval augmented generation, or RRAG, for memory management. Right, where the agent would store its experiment logs in a vector database and perform semantic searches against its own history to retrieve relevant context. Exactly. But the sources argue that RRAG is fundamentally insufficient for this level of complex interpretive Why is that? Because RRAG forces the agent to reconstruct its understanding from raw, fragmented data chunks on the fly every single time it evaluates a new node. Oh, I see. It retrieves an observation from an experiment run on Monday and a contradictory observation from an experiment run on Thursday and attempts to synthesize them dynamically in the context window. Which is really hard for the model. Yeah. As the scale of the graph increases, the agent loses the holistic picture. It cannot see the forest for the trees. To solve this systemic amnesia, the architecture of AI memory had to be completely reimagined. In April 2026, Andre Carpathy published a design pattern that he termed the LLM Wiki. The LLM Wiki pattern is brilliant. He proposed abandoning the dynamic retrieval of raw data fragments entirely. Instead, he suggested treating the raw experimental knowledge as source code and treating the element itself as a compiler. The compiler analogy is a total paradigm shift. Right, because in software engineering, code is written across hundreds of distinct fragmented files. A compiler ingests all of those files, resolves the dependency conflicts, links the logic, and builds a single, highly structured, unified execution. Right. And Carpathy argued that the agent shouldn't just search its old logs. It should systematically read its entire history and compile it into a structured, cross-referenced markdown code base. It should actively write and update a permanent wiki of its own discoveries about the target model. It is an incredibly elegant solution for maintaining a holistic map of the network. However, compiling a comprehensive wiki of a massive neural network is extraordinarily compute-intensive. Obviously. The agent cannot simultaneously probe the model, test hypotheses, and recompile its entire global knowledge base in real time. Which brings us to the final breakthrough documented in the stack. Anthropics launch of the Claude Managed Agents Dreaming Framework in May 2021. The Dreaming framework, this is where it all comes together. They took Carpathy's LLM wiki pattern and integrated it with an asynchronous offline background process. They realized that compilation requires downtime. And the biological parallel is undeniable. I mean, the impropic engineers literally named the framework Dreaming. Dreaming is a scheduled offline batch process. Right. When the interpretability agent is no longer actively executing queries against the target model, it reflects on a batch of up to 100 past-session transcripts. And the documentation outlines a highly specific three-step compilation cycle during this dream phase. First, the agent executes a merge operation. That's step one. Consolidating the fact. Exactly. And third, it surfaces new insights. It analyzes the newly merged data to identify high-level emergent patterns that were not visible during the granular day-to-day experimental phase. The synthesis of these operations is what the literature refers to as asynchronous epistemology. Asynchronous epistemology. I love that term. It just means the generation of knowledge is decoupled from the immediate execution of tasks. The Circuit Oracle probes the target model by day, extracting raw causal data. And by night, it sleeps to consolidate its memory, deduplicate the noise, and compile a highly structured global understanding of the intelligence it is studying. It is literally an artificial hippocampus. It really is. But, you know, reading through the implementation details, a massive safety vulnerability becomes glaringly obvious. Oh, yes. I know what you're going to say. If the agent is autonomously modifying its own memory architecture while it sleeps, if it has the programmatic authority to overwrite or delete facts it deems outdated or contradictory, how do we guarantee the integrity of the research? That's a huge risk. Right. What happens if the agent detects the faint, complex signature of a malicious sleeper agent during the day? But during the nocturnal dream cycle, the safety warning is erroneously classified as statistical noise and just merged out of existence. You never know what happens. Exactly. If the AI is writing its own textbook and discarding the drafts, the human overseers completely lose the ability to verify the findings. That is the exact systemic risk that necessitated the core safety mechanism engineered into Anthropics Dreaming Framework. Okay. How did they solve it? The architecture enforces a strict immutability constraint on the primary data layer. The dream cycle never modifies, overwrites, or destroys the original input session transcripts. Ah. So the raw data is entirely siloed from the compilation process. Completely siloed. The raw experimental logs, the tool outputs, the hypothesis generation, all of it remains permanently preserved in an append-only ledger. Okay, that makes sense. The dream cycle only has the authority to produce a new, secondary, reorganized memory store. It builds a pristine, highly curated context window for the agent to utilize for complex reasoning the following day, while mathematically guaranteeing that a perfect, unalterable audit trail remains intact for human researchers to review. So the human principle investigator always has the ability to look past the compile wiki and analyze the raw underlying data. Yes. We have successfully built an ecosystem for incremental, scalable, and epistemologically sound interpretive. It really represents a monumental triumph of software architecture. Yeah. Adapting our infrastructure to meet the alien scale of the models we are attempting to align. It's incredible work. Which brings us to the culmination of the journey mapped out in these sources. The arc we have traced today is just staggering. We began with Neil Nanda and the early days at Ampropik and DeepMind, driven by this mathematical optimism that we could manually decompile these models. Right, mapping induction heads and IOI circuits by hand, believing the networks were clean, algorithmic machines. But the inherent physics of the architecture, the necessity of lossy compression and superposition, shattered that optimism. The latent space was not a filing cabinet. It was a high-dimensional polysemantic geometry that actively resisted human comprehension. And to survive the scaling laws, the field had to engineer sparse autoencoders to force the geometry into legibility. Creating causal graphs so massive we had to build autonomous AI agents like the Circuit Oracle just to traverse them. And finally, because those agents generated too much data to comprehend... We had to engineer artificial sleep cycles to allow them to asynchronously compile their understanding of the alien minds they were auditing. We built synthetic biology to study synthetic cognition. The evolution of the methodology is as fascinating as the models themselves. We are literally defining the physics of a newly discovered computational universe. It's a very exciting time to be in the field. But reviewing all this research, I want to leave you, the listener, with a final thought to mull over. One that sort of extrapolates beyond the bounds of these specific papers. Okay. Throughout this entire process, we have relied heavily on biological metaphors. We talked about convergent evolution, motor cortexes, and REM sleep. And we assume these are just useful analogies to help our meat brains understand the math of the machines. Right. But what if mechanistic interpretability eventually proves the inverse? What do you mean? Well, if researchers ultimately prove the universality hypothesis, if they prove that any sufficiently advanced optimization process must inevitably converge on the exact same polysemantic geometries, sparse feature circuits, and asynchronous memory compilation to achieve general intelligence, then what does that mean for human consciousness? Oh, wow. We look at the multidimensional superpositioned gas of an LM's latent space and call it alien. But if those are the universal foundational laws of information processing, then human creativity, intuition, and emotion are likely governed by the exact same mathematical structures. That is a staggering thought. If we eventually map the final irreducible equation of how a dreaming AI parses a waking AI... Will we realize that we haven't just mapped a software artifact? Will we realize that mechanistic interpretability has actually mapped the mathematical boundaries of the human soul? And if we do, who is reverse engineering whom? Thank you for joining us on this exploration of the black box. Keep questioning the algorithm shaping your reality.