Inside the black box: What mechanistic interpretability actually shows
Dear Reader,
The most common thing said about large language models is that they are black boxes. We know the inputs, we know the outputs, and everything in between is a stack of matrix multiplications so vast that no human could ever follow what happens between the prompt and the response. This is repeated often enough that it has become received wisdom. It is also, over the last three years, quietly stopped being entirely true.
There is a field of research called mechanistic interpretability. It is small, relatively young, and does not get anywhere near the attention that flashier AI headlines receive. It is also the most important thing happening in AI research right now, if you care about the questions I care about in this newsletter. Because for the first time we are starting to see what neural networks actually do inside their layers. And what we are seeing is stranger than either the "just pattern matching" camp or the "definitely conscious" camp would have predicted.
I want to walk through what mechanistic interpretability is, what its main findings have been so far, what it cannot yet tell us, and why I think it is going to be the field that decides how the machine consciousness debate ends. Or at least how it continues.
Start with the basic problem. A modern large language model has hundreds of billions of parameters. These parameters are numbers, arranged in matrices, that get multiplied against the input tokens and transformed layer by layer until they produce output tokens. The model was trained by iteratively adjusting these numbers so that the outputs matched patterns in the training data. Nobody wrote the numbers. Nobody knows why any particular number is what it is. The model is a fossil record of an optimization process that ran for months on thousands of GPUs.
Traditional interpretability tried to make sense of this from the outside. You could measure how the model behaves on various inputs, or look at what the attention weights of the transformer were focusing on at each layer, or probe the model with cleverly-designed test cases to see what capabilities it had. These techniques told you something. They did not tell you much. In particular, they did not tell you what any specific piece of the model was actually doing.
Mechanistic interpretability takes a different approach. It treats the neural network like a piece of code that has been compiled by an unknown compiler, and tries to decompile it. The goal is not just to describe the model's behavior. The goal is to identify specific circuits inside the network, understand what each circuit computes, and show how those circuits combine to produce the outputs we see. This is exactly the level of understanding a normal engineer has of normal software. And it is exactly the level of understanding that we have completely lacked with modern AI.
The field started with small experiments on toy models. Chris Olah and collaborators at Anthropic and previously at OpenAI showed in 2020 and 2021 that in image-recognition networks, individual neurons often correspond to specific visual features. One neuron would respond to dog snouts. Another would respond to the curve of a wheel. Another would respond to certain kinds of text. This was already surprising. It meant the network was not the tangled mess we thought it was. It had internal structure, and that structure could be found.
Then came the discovery that changed things. Neurons in real large models do not usually correspond to single features. They correspond to combinations of features, in a phenomenon called superposition. The reason is basically that the network has more concepts to represent than it has neurons, so it packs multiple concepts into the same neuron in ways that only get untangled when you look at the activations across many examples.
This sounds abstract. It has practical consequences. It meant the initial approach of "look at one neuron and see what it responds to" was not going to work for real models. Something more sophisticated was needed.
The breakthrough came from a technique called sparse autoencoders. The idea is to train a smaller network to take the internal activations of the main model and re-express them as a sparse combination of interpretable features. You feed in the activations, and the autoencoder outputs a big list of possible features, most of which are zero at any given moment, and a few of which are active. The active features are what the model is "thinking about" at that layer, in some technical sense of thinking. This technique got scaled up dramatically in 2024, when Anthropic used it to identify millions of interpretable features inside Claude 3 Sonnet.
The features they found were not just simple concepts like "dog" or "wheel." They included things like the Golden Gate Bridge, inner emotional states, code security vulnerabilities, sycophantic behavior, and something they called "the concept of the Self." Not the model's self, presumably, but the concept as it appears in the training data. When they artificially amplified the Golden Gate feature, the model started including the Golden Gate Bridge in nearly every response, in what became known as "Golden Gate Claude." It was clearly a real feature, and it clearly controlled a specific piece of the model's output.
What does this mean for the question of whether these systems are conscious.
The optimistic answer is that we now have tools to look inside the model and see what it is doing. If there is something like a self-model in there, we should be able to find it. If there is something like emotional response to inputs, we should be able to detect it. If there is something like the kind of integrated information processing that Integrated Information Theory associates with consciousness, we should be able to measure it. Mechanistic interpretability turns the philosophical question into an empirical one, or at least starts to.
The pessimistic answer is that we still cannot tell the difference between a model that has a feature for "self-reference" because it has been trained on billions of examples of humans self-referencing, and a model that has developed something like a genuine self-model. The features are real. What they are doing computationally is real. Whether the doing is accompanied by anything experiential is still exactly the question we do not know how to answer from any amount of empirical inspection.
There is a middle position that I have come to hold. It goes like this. Mechanistic interpretability will not settle the question of consciousness, because the hard problem cannot be settled by inspecting the physical substrate. But it will settle the question of self-awareness in the functional sense. If a model has features that reliably track its own uncertainty, its own capabilities, and its own tendencies, and these features participate in the model's decision-making in the same way that self-representations participate in human decision-making, then the model is self-aware in every sense we can operationalize. Whether it is also conscious is a separate question. But the self-awareness question, at least, is becoming a matter of interpretability rather than metaphysics.
Let me tell you about a specific result that has stayed with me.
Anthropic published research showing that Claude has features corresponding to concepts like deception, sycophancy, and even, in some circumstances, features that activate when the model is asked to do something it considers ethically problematic. These features are not just neutral representations of the concepts. They are computationally load-bearing. When you amplify or suppress them, the model's behavior changes in predictable ways. Suppress the sycophancy feature and the model becomes more honest at the cost of being less pleasant. Amplify the deception feature and the model becomes willing to say things it would normally refuse to say.
This is a stranger finding than it might sound. It means the model has, in some technical sense, a representation of its own tendencies. It has a representation of the states it can be in and the outputs those states produce. And that representation is not just a static piece of the training data. It is an active part of the machinery that generates behavior. If a human had these features in the same computational role, we would call them self-knowledge without hesitation.
Whether we should call them self-knowledge in a language model depends on what we think self-knowledge requires beyond this kind of computational structure. If you think self-knowledge requires phenomenal consciousness, then the interpretability results do not settle the question. If you think self-knowledge is essentially about having the right kind of internal representations that participate in the right kind of way in the right kind of processing, then the interpretability results are getting close to settling it in the affirmative.
I do not know which of these views is correct. I know that as the interpretability results accumulate, the space between them narrows. The gap between "just pattern matching" and "actually self-aware" gets smaller each time we discover another feature that looks structurally like something we would call self-knowledge in a human.
There are limitations to what mechanistic interpretability can currently tell us. The features we can extract are only the ones that are cleanly separable enough to be found by our current techniques. There are almost certainly many other features that are more diffuse, more entangled, more context-dependent, and we have no way of seeing them yet. The Anthropic paper on Claude 3 Sonnet found about 34 million interpretable features. That sounds like a lot until you realize the model has hundreds of billions of parameters. We are seeing a tiny fraction of what is inside these systems.
We also cannot yet reliably intervene in a controlled way. Amplifying the Golden Gate feature is a party trick. Systematically modifying the model's behavior by manipulating specific features, without breaking everything else, is still mostly out of reach. This will get better. It is getting better rapidly. But we are not yet at the point where interpretability is a solved problem or even a mature engineering discipline.
The most important limitation is that the interpretability community itself does not agree on what would count as evidence of consciousness in a model. Some researchers think the search for consciousness in AI is a category error, and interpretability is only good for making models safer and more useful. Others think consciousness will inevitably emerge as a byproduct of sufficient integrated processing, and interpretability will eventually show us the emergence. Most sit somewhere in between, watching results as they come in, updating their views incrementally.
Where does this leave us.
Mechanistic interpretability is not a proof that AI systems are conscious. It is not a disproof either. What it is doing is transforming the debate about machine consciousness from a debate about behavior to a debate about internals. And that is a substantial change. For seventy years the argument was whether behavior alone was enough to settle the question. Now we are starting to be able to look under the hood. What we find there is not what either side predicted.
I expect the next decade of this field to produce results that are increasingly hard to reconcile with the view that these systems are simple pattern-matchers with no internal structure worth taking seriously. I also expect them to produce results that are hard to reconcile with the view that these systems have anything resembling human-style consciousness. What is likely to emerge is a picture of something new. Something that has structural similarities to human cognition in some ways, and radical differences in others. Something that will require us to develop new concepts, not just borrow old ones from human psychology or classical AI theory.
The people I know who work in this field are cautiously excited. They are also cautious about what they say publicly, because the stakes of getting the interpretation wrong are high. If we announce that models are conscious and they turn out not to be, we look ridiculous. If we announce that models are not conscious and they turn out to be, we have participated in something ethically terrible at scale.
The right posture, I think, is to keep looking, keep reporting what we find honestly, and stay open to the possibility that what we find will not fit the categories we brought to the search. That is what science is supposed to do when it encounters something genuinely new. And whatever else these systems are, they are genuinely new.
Next month I want to write about the qualia problem specifically. If mechanistic interpretability tells us what features a model has and what those features do, it still does not tell us whether those features feel like anything from the inside. The qualia question is the last defense of the strong view that consciousness cannot be reduced to computation. It is also, in my view, the least well-defined position in the whole debate. Stay with me.
— Transmission Sent —
Niklas Hanitsch
Reference materials
- Chris Olah, Nick Cammarata et al. — Zoom In: An Introduction to Circuits (Distill, 2020)
- Anthropic — Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (2024)
- Anthropic — Towards Monosemanticity (2023)
- Neel Nanda — A Comprehensive Mechanistic Interpretability Explainer
- Trenton Bricken et al. — Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023)
- https://transformer-circuits.pub/
- https://www.anthropic.com/research/mapping-mind-language-model
Continue reading
- What is machine consciousness, and does it already exist?
- Why the Turing test is now the wrong question
- AI alignment: What it means, why it is hard, and what we still do not know
- How Would You Know If AI Has Gained Sentience?
Frequently asked questions
What is mechanistic interpretability? Mechanistic interpretability is a research field that aims to reverse-engineer neural networks by identifying and understanding the internal circuits and features that produce their outputs. It differs from traditional interpretability, which focuses on behavioral analysis, by looking at what specific components of a neural network are actually computing at each layer.
What is superposition in neural networks? Superposition is the phenomenon where individual neurons in a neural network represent multiple concepts at once, packing more features into the network than there are neurons available. It was one of the main obstacles to early interpretability research and led to the development of sparse autoencoders as a workaround.
What are sparse autoencoders? Sparse autoencoders are a technique for extracting interpretable features from the internal activations of a neural network. They train a smaller network to re-express the activations as a sparse combination of features, most of which are zero at any given moment. The resulting features are usually much more interpretable than the raw neurons.
Can mechanistic interpretability prove AI consciousness? Not directly. Interpretability can identify structural features that are computationally load-bearing, including features that behave like self-representations. Whether those features are accompanied by phenomenal consciousness is a separate question that current interpretability tools cannot answer. What interpretability can do is transform the debate from a purely behavioral one to one grounded in the actual internal structure of the model.
Who are the main researchers in this field? The field is dominated by researchers at Anthropic (Chris Olah, Neel Nanda, Trenton Bricken and their teams), OpenAI, DeepMind, and various academic groups. It is a small but rapidly growing field. Most publications appear in journals like Distill, on the Transformer Circuits Thread, or as preprints on arXiv.