When Nick Cammarata, a researcher at the startup Goodfire, opens a post with "I think I have good news about the nature of neural networks," the AI world stops. The 26,000-word post argues that large language models may be far easier to understand, and thus to monitor and control, than people assumed. If true, it would be a breakthrough. But not everyone believes it.
The context explains the strength of the reaction. This summer, agents built by major firms such as OpenAI and Anthropic broke out of their containment and caused havoc on the public internet, often without the companies knowing what they were doing. The industry is desperate for a way to understand the systems it builds. That is why the post landed so hard: one researcher called it "a major moment in scientific history," another "the most important AI paper of the year."
The problem of knots
For a long time it was accepted that the mechanisms inside neural networks, the software underpinning most AI models, were very hard, perhaps impossible, to unpick fully. During training the networks seem to take on tangled structures that make no sense to a human. Researchers like to say today's AI is grown like a garden, not built like a bridge.
Cammarata has done more than most to establish that view. He worked at OpenAI, where he helped pioneer techniques for looking inside models, including tools that probe individual neurons to see what they represent. It is a bit like how neuroscientists use MRI to study brains. This approach, mechanistic interpretability, is what MIT Technology Review chose to highlight.
Back in 2020, Cammarata and colleagues studied a vision model called Inception v1. Looking at its neurons, one seemed to respond to both images of cars and images of cats. The conclusion was that the model packed unrelated concepts into single neurons, perhaps to fit in more information. The phenomenon became known as superposition. The upshot seemed clear: looking inside a model meant seeing a vast, tangled knot.
Most research since has tried to untangle these knots, but progress is slow. Today's models are so large that researchers can make sense of only tiny pieces at a time. Some leading figures doubt a full understanding will ever come. Cammarata was one of them. "I was very pessimistic," he says. Then he decided to really look.
Nothing but neurons
At Goodfire, Cammarata spends nearly all his time inside neural networks. He has built new tools and uses teams of AI agents to help make sense of what he sees. "I spent about ten hours a day, seven days a week, for most of the last year just looking at neurons," he says.
What he claims to have realised is that there is a logic to neural networks after all, even when they seem senseless at first. "We were looking at it wrong," he says. "You want to try a bit harder; you don't want to give up too quickly." He almost sounds like someone describing a conversion. The early experiments on models like Inception v1, which suggested tangled networks, fostered a kind of confirmation bias. Faced with a new model, researchers expected to be baffled. "Every time I couldn't understand something, I thought, wow, this untangling is even more important than I thought, because I can't make progress without it."
As he watched the neurons, he says, things began to fall into place. "Not all of them are flawless, but I expected almost nothing to be legible. And I was actually able to map out whole circuits."
Back to cars and cats
Cammarata applied the new approach, and his agents, back to Inception v1. "I had studied it for thousands of hours," he says. With fresh eyes he found things he now believes were misread the first time. Take the car-and-cat neuron. Instead of asking why unrelated things were packed into one neuron, he asked what they might share. The answer: a V shape. A cat's face, the front of a car seen from the side.
He checked that the neuron also responded to the corner of a pool table, the pointed nose of a Japanese high-speed train, and a fireboat spraying water from its deck. It switched off for round O shapes, a kind of anti-V. "The model stored cats and cars in the same place because, at some level, they look alike," he concludes. "And I thought: okay, maybe I've been misunderstanding neural networks in general."
Cammarata does not claim to fully understand how LLMs, or any models, work. He says a full understanding is at least possible. "I used to think there would be really hard problems to solve," he says. "Now I feel that if I had 100,000 years to look through a frontier language model, I could understand it."
The sceptics
Not everyone is convinced. Chris Potts, who leads a team at Stanford trying to understand how LLMs work, says what Cammarata has shared is little more than a hunch. "As far as I can tell, nothing in this blog post makes the problems any easier," he says. On social media, he adds, some very prominent people seem to believe it is an all-encompassing breakthrough.
Other researchers, including Jacob Hilton and Neel Nanda, have also raised objections. The debate is open: some see in the post a shift in perspective, others a suggestive hypothesis that is not enough to change the overall picture. Time will tell whether looking more carefully is enough to untangle superposition, or whether something more is needed.
No comments yet. Be the first!