A Practical Perspective on Recursive Self-Improvement

TL;DR

• The definition of "agent" has been reshuffled. Classic RL drew agent and environment as two separate boxes; today the environment is so entangled with the agent that Vinyals draws it inside the agent — a machine (often shared with the user) running the harness, with an LLM as the brain.
• Practical definitions: self-improvement means a user hands over a spec, the system runs for hours or days with little interaction, and optimizes some fitness function. Recursive adds one step — the harness code and the model weights themselves count as part of the environment and become modifiable.
• Status check: plenty of self-improvement examples exist; "I don't think we've seen very impactful actual recursive self-improvement examples yet."
• The bottleneck sits at both ends of the research loop. Implementation and experimentation — coding agents' home turf — are racing ahead; ideation and evaluation have barely moved.
• Evaluation should move toward measuring how, not just how much: proxy benchmarks (SWE-Bench Pro, MLE-bench, held-out perplexity) → benchmarks that measure RSI directly (PostTrainBench and friends) → evaluating the manner of improvement, rewarding creativity and penalizing cheating.
• Four reasons to expect RSI later than the hype suggests: evaluation is hard, ideation is hard, physical limits (chip speed, speed of light), and human-level performance may already be near an asymptote — "better doesn't mean better."

Key Points

From video games to an environment that lives inside the agent (~00:43–00:46)

DeepMind has been building agents for over a decade, largely because reinforcement learning defines what an agent is. The traditional picture was simple: an agent and an environment exchanging observations and actions, the agent powered by a neural net, with fixed observation and action spaces and video games as the environment. The perennial objection was "games aren't the real world," but hindsight says otherwise — and games had one lovely property: they were sandboxed even more thoroughly than the sandboxes we build today. They tested on Go, StarCraft and the rest, until "we ran out of games."

Today's picture is reshuffled. The most profound change: the environment is hard to disentangle from the agent, so he draws the environment inside it. The environment is a machine, probably with network access, running an agent harness whose brain is an LLM — structurally not that different from the old game-specific neural net, except the interaction is now too tight to separate. Two other changes: a user is now in the loop (coding, or handing over a hard question that runs for a while), and the environment is often shared with that user — your own machine, or virtual machines spawned in the cloud.

Self-improvement and recursive self-improvement, defined through that lens (~00:46–00:49)

• Self-improvement is a nuance on what agents already do. The user is far less interactive: hand over a spec, the process runs for hours or days. Critically there is a fitness function or improvement target — the Tetris score demoed that morning being one example. "It's almost a detail" that separates this from an agent.
• Recursive is one step further: the harness code that runs the agent, and the LLM (or set of LLMs) behind its actions, can also be modified. Both are trivially part of the environment — the harness source, and the model as an object you can upload to whatever serves your tokens. Recursion is simply permitting the environment and harness to modify themselves or the model.

Two improvement spaces are worth distinguishing: harness space (compaction, available tools, all the implementation detail) and the model weights (rewritten from scratch, or fine-tuned). The action space for recursion is anything that modifies that object.

His read: no impactful RSI examples yet; plenty of self-improvement ones. But it is coming, with deep consequences for how frontier labs do the research that produces harnesses and LLMs in the first place.

The four-box research loop, and why everyone is crowded into the middle (~00:49–00:51)

To think about how a brain-equipped agent would recursively self-improve, put on a researcher's hat: the process a fully capable RSI system must run is not that different from the game many of us have played for years — publish papers, get techniques adopted. Four boxes:

1. Ideation — absorb the prior work, decide which ideas are fundamentally worth pursuing. A crucial reasoning step.
2. Implementation — actually build the idea, simple or complex.
3. Experimentation — run the code, hit bugs, it doesn't work; persist, or give up and go back to another idea.
4. Evaluation — judge what happened and whether the hypothesis from ideation held. Traditionally you then write a paper; an RSI system just loops.

The observation that anchors the talk: almost all current effort sits in the middle two boxes. Everyone knows how powerful coding agents have become, so implementation and experimentation (running, compiling, debugging) attract both progress and enthusiasm — probably because that stage is the most interactive with the user, which makes it more self-improvement than recursive self-improvement. The outer two boxes lag badly.

Evaluation, in three tiers (~00:51–00:55)

He dwells on evaluation because the community has under-weighted evaluation and data for years — proposing a new dataset used to be dismissed as "an application," not worth a top conference. Three ways to measure progress scientifically:

(1) Proxy capability benchmarks plus hill climbing. Pick capabilities you believe matter and benchmarks for them — SWE-Bench Pro, MLE-bench, held-out perplexity in pre-training — then climb, hoping RSI emerges. - Pros: exactly how LLMs have been trained for years; cheap and well-defined. - Cons: a very indirect measurement of whether a system will truly develop RSI; overfitting and cheating; and today's evals mostly cover only the middle two boxes.

(2) Evaluate the capability we actually care about — RSI itself. Benchmarks are appearing as the topic gains traction: PostTrainBench is one example, and Dawn Song told him about another; expect more over the coming weeks and months. The design is deliberately meta — the evaluation is an inner loop: give the agent a metric, a few tries, and measure how much improvement it gets. - Pros: measures the self-improvement capability directly. - Cons: expensive right now — you literally have to run an agent for a while; and the inner-loop metric is usually something like Tetris, whereas the production question is "automate my whole frontier lab and produce the best model in the world," so these evaluations may be somewhat out of distribution as they stand.

(3) Evaluate how the improvement was achieved. The direction he expects the field to move: not just "how much better at Tetris in three hours of compute" but no cheating, visible creativity, certain aspects of intelligence — scored by powerful reward models that are more tractable to evaluate.

Ideation: the most underexplored box (~00:55–00:57)

Take ideation as the concrete case. How do you evaluate an idea? As a conference reviewer you look for research taste, novelty, efficiency, elegance, whether the technique will stand the test of time. You can imagine writing rules against a few of these and automating the scoring of an RSI agent's ideation behavior — but it is very hard today and severely understudied. Getting there, and getting the automated reasoning good enough to optimize on top of with RL, will take real time.

And honestly, humans aren't that good at it either. We run very expensive conference and review processes, yet Transformers took years to become mainstream, and distillation — increasingly famous now — was rejected from the conference they submitted it to, resurfacing years later through something like a random process. Everyone in the room, he notes, has their own sour story of a rejected paper that in hindsight should have been accepted by this expensive apparatus we built for ourselves.

An old friend: agents exploiting the game (~00:57)

He brings games back for one slide: it should surprise no one that agents do things a little differently than we expect. Several famous games have been exploited in unintended ways. We've been watching this coming for many years.

Closing: enormous, but probably slower than you think (~00:57–00:59)

• The upside: RSI will likely increase the speed of progress on certain engineering and science tasks by 10x or more.
• The caution: be mindful of how goals are set, of misalignment, of resource reallocation, and of the fact that how labs are run may itself change.
• Four reasons it may not arrive as fast as expected: 1. Evaluation is very hard. 2. Ideation is very hard. 3. Physical constraints — chips can't run faster than they're designed to, and even the speed of light bounds how fast data moves around a chip. 4. Human performance may already be close to an asymptote, so adding RSI may not yield a drastic improvement. How good is AlphaGo relative to a perfect game of Go? Unclear. A more everyday example: these models write so well it's become annoying, and you have to make the output look a little worse. In some domains, better doesn't mean better.

Quotes

"The environment is hard to disentangle from the agent. So I put the environment inside of the agent." (~00:45)

The classic two-box RL diagram no longer survives contact with agentic systems.

"I don't think we've seen very impactful actual recursive self-improvement examples yet. Certainly many self-improvement examples." (~00:48)

RSI is still a forecast, not a fact on the ground.

"Currently a lot of effort is done in the middle two boxes." (~00:50)

The diagnosis the whole talk hangs on.

"It might not come as fast as we thought it would, because first evaluation is very hard and ideation is also very hard." (~00:57)

A cooling note from inside a frontier lab.

"Sometimes it's kind of annoying how well these models write and you have to make it look a little bit worse. … Better doesn't mean better." (~00:58)

On some axes we're already near the ceiling, and the marginal value of going further may be negative.