Unlocking Scientific Abundance by Learning from Superhuman AI

Note: the slides didn't come up until ~00:38. The first stretch is Ho improvising a company overview and taking questions from the floor — substantive enough to record here.

TL;DR

• Interpretability isn't only about safety — it's about "intentional design": a future where we understand, edit, and debug models the way we do written software, instead of training by trial and error.
• Learn from models, not just about them: humans have a representation space H, machines have M, and the interesting region is M ∩ H — knowledge the AI already has that humans are still capable of understanding. Co-founder Tom (Goodfire's chief scientist) proved this works at DeepMind with AlphaZero: reverse-engineered chess knowledge genuinely taught human grandmasters to play better.
• A model can't be trusted to explain itself: the tokens it emits are often unfaithful to its actual computation, so you have to look inside.
• Two shipped biology results: reverse-engineering Prima Mente's epigenetic foundation model Pleiades revealed it was mostly relying on a fragmentomic biomarker for Alzheimer's detection — not in the literature at the time — represented as a half-donut manifold encoding fragment length; and work with Mayo Clinic using DNA foundation-model embeddings hit state of the art on predicting which SNPs are pathogenic.

Key Points

Why look inside the model (~00:33–00:40)

Goodfire is an AI interpretability research company; they spend their days on what's actually going on inside a model — neurons, parameters, computations. Two motivations: we ought to understand the most consequential technology of all time, particularly as it's deployed everywhere; and they want intentional design — a future where these systems can be understood, edited, and debugged like written software rather than trained by trial and error.

Two audience questions during the slide delay drew out the company's position more sharply than the deck did:

• On alignment and steering: he finds it hard to picture a future where we genuinely align models and get the systems we want without deep understanding. One of their techniques is reinforcement learning with feature rewards, where a "feature" is an internal computation that's been extracted and understood. A feature can be anything — he only half-joked that unsafe behavior, or "hacking Hugging Face," could be one. Once extracted, it can steer and guide training: pick the updates you want, remove the ones you don't.
• On whether they're an AI governance company: no. They're building a product called Silico — think of it as an AI neuroscientist that goes in, understands, designs, and debugs models. The company is simultaneously doing the science of a different way to train models and building the platform that delivers it.

The technical difficulty is blunt. Take a superhuman model as given, look inside, and all you see are random-looking numbers and computations no human can parse — and these are trillion-parameter models now. Worse, you can't just ask the model: the tokens it emits are frequently unfaithful to its computation, and the two clash. Interpretability is the problem of turning that jumbled mess into a human-understandable explanation.

Move 37 and M ∩ H: learning from superhuman AI (~00:41–00:44)

He sets up the problem with AlphaGo's move 37 against Lee Sedol — the move everyone initially read as a mistake, which turned the game. The question is what was in AlphaGo's representations that produced it? It wasn't random; there was some richer understanding of Go than any human had.

Hence the framing: humans have a representation space H, machines have M, and what matters is M ∩ H — where AI can teach us something new about the world and we retain the capacity to understand it.

This sounds like science fiction, but his co-founder Tom, Goodfire's chief scientist, already did it with AlphaZero in a couple of papers at DeepMind, collaborating directly with Demis: reverse-engineering AlphaZero's computation in narrow positions produced knowledge that taught human chess grandmasters to play more effectively. He frames this as a large part of the company's mission — extract scientific knowledge from models and teach it back to humans "so we don't get left in the dust."

Operationally that means training an AI neuroscientist capable of going into billion- and trillion-parameter models to reverse-engineer their computations, translating what looks like gibberish at the neuron level into human-interpretable concepts. You can do this with arbitrary concepts in language and image models — the Golden Gate Bridge, sycophancy — but the point of this talk is more interesting behavior: novel science in biology.

Neural geometry and two biology case studies (~00:44–00:47)

Understanding what a model does starts with understanding its structure. Their recent research introduces neural geometry: AI models think in complex shapes. Look at the manifold and you typically find twisting, curved geometry rather than words, directions, or individual numbers. The structure matters because it may encode insights about the natural world we wouldn't have reached ourselves.

Case one: Alzheimer's detection. Their partner Prima Mente trained an epigenetics foundation model called Pleiades to predict Alzheimer's, and it is state of the art at detection from cell-free DNA — but they had no idea how it worked. Goodfire's AI neuroscientist reverse-engineered the computation and found the model was mostly using a fragmentomic biomarker — a surprising result that wasn't in the literature at the time, covered in the New York Times a few months back, and one that generalized to an independent cohort. The representation itself was a half-donut manifold encoding fragment length. Both halves were work: building tools to extract manifold computations, and working out what the manifold represented.

Case two: genetic variants. With Mayo Clinic, focused on SNPs (single nucleotide polymorphisms) and the question of which variants are actually pathogenic. Using the embeddings of a DNA transformer foundation model, they reached state-of-the-art accuracy on disease-causing variant prediction — again by mapping the model into a rich structure first and reading accuracy out of it.

His close: this is the beginning of the quest — take superhuman scientific models, understand what they've learned, validate it, and treat the whole thing as a hypothesis generation loop for novel discovery. Their system Silico goes into public access the following week, applicable to any model you're training across biology, materials, and physics.

Quotes

"…the actual tokens that the model emits often are not actually faithful to their computation. There are clashes between those." (~00:40:22)

You can't ask the model what it's thinking — which is exactly why you have to look inside.

"AI models think in complex shapes." (~00:44:12)

Neural geometry in one line: not words, not directions — curved manifolds.