Robotics: Endgame

TL;DR

• "Two curves is all you need." The first is the neural scaling law — compute and data in, loss down, predictable enough to forecast the result before spending a dollar. The second landed this year: agentic scaling, where the x-axis is still training compute but the currency is tokens, and models self-improve via swarms of agents running their own experiments. The two stack: agentic scaling starts where foundation scaling ends.
• On the model side: take a pretrained video diffusion model and have it jointly decode future video and actions, yielding a World Action Model (WAM) — the robot dreams the next few seconds and acts on the dream, replanning and re-dreaming on the fly when perturbed. Test-time training (TTT) then stretches the native context 1000× to 8,000 timesteps (~5 minutes of "muscle memory"), unlocking post-deployment learning and in-context imitation of humans.
• On the data side: teleop VR rigs ("medieval torture devices") fundamentally don't scale. What's needed is the Tesla FSD property — data collection fading into the background. EgoScale puts 99.9% of training on human egocentric video: 20,000 hours of in-the-wild pretraining, 50 hours of mocap-glove fine-tuning, 4 hours of teleop (<0.1% of the mix), mapping pixels straight to 22-DoF humanlike dexterous hands — and yields a log-linear scaling law for dexterity.
• On the agentic side: simulators and control stacks are just code, so you can copy homework from LLMs — turn robotics into coding environments and let teams of coding agents run physical auto research on real hardware (auto-reset, auto-improve, auto-evaluate), climbing a task from 0% to 99%. ASPIRE goes further: it doesn't update weights at all, but grows a compounding skill library that transfers sim-to-real and across embodiments.

Key Points

Two curves, and why robotics is still alchemy (~02:21:49–02:23)

He opened at NeurIPS 2017, where Ali Rahimi took the stage and declared that machine learning had become alchemy. He was right: nothing was rigorous, everyone was vibing, things seemed to work but nobody could tell for sure, and the future felt fuzzy and fragile. Nine years later the LLM crowd is speedrunning AGI on the back of mythical beings literally named Mythos.

What ended the alchemy? Two curves.

1. The neural scaling law — more compute and data in, loss down, so predictable you can forecast the outcome before spending a single dollar. This curve is the guiding light for how trillions of dollars of investment should flow.
2. Agentic scaling, which only landed this year. The x-axis is still training compute, but measured in a new currency: tokens. This is auto-research, where models self-improve with more and more agents running experiments on their own — and capability is the final boss fight for LLMs.

The beautiful part is that these are not separate curves. Agentic scaling starts where foundation scaling ends, so what began as alchemy is now the most reliable curve extrapolating into the near future.

Then the confession: as a roboticist at the LLM party, he feels lonely and left out — because robotics is stuck in the age of alchemy, and won't get out until the field figures out what to scale and what happens at scale. The rest of the talk covers two things: the model and the data strategy.

Model: from AI video slop to World Action Models (~02:23–02:26)

The first pretraining paradigm for digital AI was next-word prediction. He expected the second to be something glorious; instead the name we gave it is AI video slop. He can watch fat cats playing banjo all day — peak internet — but nobody takes it seriously until we recognize that these video world models are learning to simulate physics internally.

Videos from Nvidia Cosmos 3 show the model picking up gravity, buoyancy, reflection, and collision purely from predicting the next blob of pixels at scale. An emergent property: it can solve visual puzzles, running simulation forward in pixel space. And his favorite: watch closely, and Cosmos finds that if you're not looking, geometry is optional.

How do you turn that into something useful for robotics? A new kind of policy model. The feed in the upper right corner is not a real camera — it's the robot dreaming what will happen over the next few seconds in video, then acting on top of that. The policy is robust: perturb it, knock things over, and it replans and re-dreams on the fly. Mechanically, you take a pretrained video diffusion model and have it jointly decode video and action into the future — a world action model (WAM).

These models only dream a few seconds ahead, and real tasks need longer memory. So they added a technique that extends the model's native context length 1000× to 8,000 timesteps — five minutes of muscle memory. The technique is TTT (test-time training): embed a tiny model inside the model, and at inference, as incoming signals flow in, take gradient steps on that tiny core, continually compressing the observed history into fast weights. In effect, the robot keeps learning after deployment. The demo is fully autonomous, single-shot: a robot assembling a car from scratch.

It also enables in-context learning from humans: a person demonstrates how to reconfigure a circuit board into a novel arrangement the robot has never seen, and conditioned on that long video prompt — a lot of tokens in context — the robot imitates faithfully. That opens a new scaling axis, context scaling: performance rises reliably out to 8,000 timesteps, three orders of magnitude beyond prior state of the art.

Data: EgoScale and a scaling law for dexterity (~02:26–02:29)

The last three years were the golden era of teleop VR headsets — complex rigs that look like medieval torture devices, absorbing enormous industry investment and no small amount of suffering, in service of something that fundamentally does not scale.

The contrast is Tesla: when you drive one, you're feeding the biggest physical data flywheel in the world, and the beautiful part is you don't even notice — the upload is an ambient process. Robot manipulation needs an FSD equivalent: data collection has to fade into the background and stay out of the loop, so we can organically capture the full glory of human dexterity across all walks of life.

Hence the all-in bet on human egocentric video with detailed annotations (fine-grained language, hand poses). EgoScale puts 99.9% of training on human egocentric video:

• 20,000 hours of high-quality in-the-wild video for pretraining
• fine-tuning on only 50 hours of mocap-glove data
• plus 4 hours of teleop — less than 0.1% of the training mix

The result is a fully autonomous policy mapping directly from pixels (the eyes' view) to 22-DoF humanlike dexterous hands, performing dexterous tasks like sorting poker cards and long-horizon tasks like manipulating a syringe from only a handful of demonstrations.

And one more scaling law: they found a neural scaling law for dexterity — a clean log-linear relationship between the volume of human video and optimal loss. The hope is to push it to 1 million, 10 million, and someday 100 million hours, and for the law to keep extrapolating.

That completes part one: x-axis compute, y-axis robot capability, showing how to scale robot foundation models end to end by gradient descent on sensory data.

Agentic: turning robots into coding environments (~02:29–02:33)

To carry the curve further, a simple idea: all the simulators and the robot control stack are, after all, just code. That's exciting because it means we can finally copy homework from LLMs — turn robotics into a set of coding environments that take motor actions as input and emit sensory signals, with decades of robotics libraries and perception stacks integrated as agentic tools.

He takes the audience to the lab at night: no humans inside, just motors humming and GPUs spinning — Night at the Museum, but the robots come alive. They talk to each other through code, tinker with the control stack, run experiments on real hardware, and go online to read papers. (They found their robots read a lot of Sergey's papers, so: "thank you, Sergey, for contributing to our agentic token budget.")

The setup gives a team of coding agents a robot fleet and one simple goal: solve a task as fast as possible subject to safety constraints, and make no mistakes. Every robot station tries a different research idea. Closing that loop needs three things: auto-reset of a physical scene, auto-improvement of the policy, and auto-evaluation. Get those right and the agents hill-climb a task from 0% all the way to 99% success. This is their first attempt at physical auto research. (His dream: the whole team takes a holiday and the boss doesn't notice, because they just read the agents' reports. His team's reply: be careful what you wish for.)

The reverse direction works too: instead of solving one task, let robots autonomously discover skills while attempting a huge variety of tasks in simulation and the real world. Recent work called ASPIRE enables a new kind of continual learning that does not update the model's weight matrices — instead it learns a skill library that compounds over time, capturing the know-how a robotics practitioner would accumulate. That unlocks yet another scaling law: x-axis training compute measured in tokens, y-axis the number of validated skills. ASPIRE's learned skills transfer sim-to-real and even across embodiments. So: compute = tokens = skills.

That completes the full curve: Total robot intelligence = foundation intelligence + agentic intelligence. The world action model sets the floor; physical auto research raises the ceiling. (Closing in Jensen Huang mode: "the more you buy, the more you save.")

Endgame: the physical Turing test and 2040 (~02:33–02:35:25)

To him, solving robotics means passing the physical Turing test across a wide range of tasks — you can't tell whether a human or a robot did it. Deceptively simple, but this is the next, if not the final, grand challenge for AI. (Deadpan, after a robot demo glitched on stage: judging by how the robot just went out, work is cut out for us.)

The last scaling law of the day is the calendar. It has been 14 years from AlexNet's first forward pass to the Opus, Mythos, and GPT-class models of today. Add another 14 and you land on 2040. His view: humans are terrible at feeling the scaling curve — looking backwards it seems flat, looking forward it is almost vertical.

Physical AGI will happen gradually and then suddenly. But first the field needs its guiding principles and scaling laws, so robot learning can move from alchemy to chemistry.

Quotes

"Two curves is all you need." (~02:22:20)

Echoing "Attention is all you need"; the skeleton of the whole talk.

"Agentic scaling starts where foundation scaling ends." (~02:23)

The two curves are a relay, not a substitution.

"Cosmos finds that if you're not looking, geometry is optional." (~02:24)

The sharpest line about what video world models actually learn: physics-when-observed.

"Total robot intelligence equals foundation intelligence plus agentic intelligence. World action model sets the floor and physical auto research raises the ceiling." (~02:33:50)

His designated "if you take away only one page from my talk" slide.

"Physical AGI will happen gradually and then suddenly." (~02:35:05)

"Turn the practice of robot learning from alchemy to chemistry." (~02:35:15)

The closing callback to Ali Rahimi.