Real-World Superintelligence

TL;DR

• The gap is about sandboxes, not intelligence. The time horizon of tasks a coding LLM can solve has doubled every year, yet the same models still fail at running a vending machine business. Code and math have rollout sandboxes that scale essentially for free; real-world rollouts are slow and expensive.
• Video is the most general modality: the most abundant source of real-world scenarios, it teaches physics, spans scientific observations at many temporal and spatial scales, and even captures the digital world's interfaces. So Runway's bet is that pixel/next-frame prediction is the right auxiliary task for learning world representations — and the bitter lesson applies to video models: more compute predictably buys better physics.
• Three more steps turn a video model into a usable simulator: (1) convert the bidirectional diffusion base into an autoregressive, causal model and distill it to real time, unlocking interactivity and counterfactual exploration; (2) move from static-environment navigation to multiple agents taking actions in the world; (3) specialize into GWM Robotics, an action-conditioned model that makes policy evaluation scalable and lays the groundwork for RL inside the world model.
• The easily-missed requirement: a world model has to simulate failure. Traditional video models carry a strong success bias (a video of a goal going in is easier to generate than one missing), but to evaluate a policy you need wrong actions to reliably fail.

Key Points

Why real-world progress lags (~02:48–02:49)

He opened with the graph everyone knows: the time horizon of software tasks a coding LLM can solve has doubled every year. Progress in the language domain, and especially with coding agents, has been incredible. Yet there's a contrast — that progress in math and code does not transfer as well to tasks that involve interacting with the unpredictable real world. We hear daily about some open conjecture being disproven, and meanwhile LLMs still fail at rudimentary tasks like running a vending machine business effectively.

The reason, he argues, comes down to how effectively we train those models. For coding and math we have the luxury of sandboxes we can run at effectively infinite scale, so rollouts scale trivially. For most real-world problems we care about, there is no simulator we can easily run at scale, and scaling rollouts in the real world is slow and expensive.

Why video (~02:49–02:52)

The natural response is world models. Most people know the concept from David Ha's paper in the 2010s, but the idea goes back to mid-20th-century cognitive science and early model-based RL: humans constantly predict and try things out in their heads, understanding the outcome of actions before taking them, and that's what enables planning. To do this with agents, you need a simulator of experience.

To bootstrap such a simulator, video is the most general modality available:

• it's the most abundant source of real-world scenarios;
• it can teach physics and essentially all the tasks humans care about;
• its generality goes beyond human-scale camera footage — video can represent scientific observations across many temporal and spatial scales;
• and it captures digital-world observations, which matters enormously if you want agents that work with the variety of interfaces we use every day.

So Runway's approach has been predicting the world by predicting the next frame — treating pixel and frame prediction as the right auxiliary task to train at massive scale, producing models with powerful world representations usable for many downstream tasks.

How good is the quality? He ran three real-versus-generated pairs with the audience. Then the reveal: in a user study with 1,000 participants, reliably fewer than 10% could tell them apart. If you live in video generation and stare long enough you can still call it, but it's getting harder. On quality, we've crossed the threshold where it's easy to fool human perception.

That came from a decade of increasing compute and data scale. The milestones: Gen-2, their first text-to-video model released a few years ago, through to Gen-4.5, Runway's latest — a massive improvement in simulating physics, producing dynamic motion, and generally feeling like plausible real-world video.

From creative quality to physical accuracy (~02:52–02:54)

How much of the creative-video progress translates into what physical AI needs? Quite a lot. Video generation can simulate a great many long-tail scenarios that are basically impossible or incredibly expensive to simulate otherwise.

And it's measurable. Good benchmarks now compare real to generated video across many categories of physics content — solid mechanics, fluid dynamics, thermodynamics, optics. Even on the stricter bar of whether the video is physically accurate rather than merely plausible, increasing compute scale reliably improves the physics of these models. In other words, the bitter lesson really applies to video models, and they've seen it hold predictably across every video model they've trained.

Making it a simulator: real-time, interactive, multi-agent (~02:54–02:57)

The next stage toward real-world usefulness is making the models real-time and interactive — critical for robotics whether you want to build a simulator or a policy model on top of a video model.

Their approach: take the foundation video model, which is a bidirectional diffusion model, make it autoregressive and causal, then run another distillation stage to make it real-time. That makes the model interactive and lets you explore counterfactuals easily — if I take this action versus that one, what happens? — with frames generated on the fly from the actions you take.

Two axes keep advancing: generality and flexibility of world models. He traced the arc: David Ha's original world model was trained on one particular racing game; then came narrow-domain world models for things like self-driving footage (generating roads and cars moving along them); then last year's Genie, high fidelity but mostly static-environment navigation. Their focus has been on making world models genuinely dynamic — not just navigating a static world, but taking actions in it, and ideally with multiple agents. The demo instructs two different agents simultaneously and simulates the outcome.

GWM Robotics: moving policy evaluation inside the world model (~02:57–03:01)

One of the released world model variants is robotics-specific: GWM Robotics, an action-conditioned video model that simulates the outcomes of actions for single-arm or bimanual robots.

• In many cases the generated rollouts are very difficult to tell apart from ground-truth teleop data.
• It simulates fine-grained interactions with cloth and other non-rigid objects.
• The immediate payoff is making evaluation far more scalable: take any policy model — a VLA or otherwise — predict the outcome of an action, and feed that back into the policy.
• And it matches the real world: take a policy like π0.5, roll out identical actions against ground-truth teleop data and inside the world model, and task success correlates well between simulation and reality. That massively speeds up how fast you can evaluate policy models.
• Looking forward, this is also a strong foundation for doing RL inside the world model, scaling interactions far faster than reality allows.

He closed on the point most likely to be overlooked: world models must be able to simulate failure. Traditional video models carry a bias toward success — it's much easier to generate a video of someone scoring a goal than missing one. But if you want to evaluate how well a policy works, you need the model to reliably fail when the policy performs the wrong actions.

The ultimate goal of world models is to increase the number of observations you can put into them. He believes learned simulators are a far more scalable path to building environments where agents can train — especially where hand-authoring a traditional simulator is very difficult, or performing the actions and gathering data in the real world is very expensive.

Quotes

"For coding and for math we have this luxury of having sandboxes that we can run at effectively infinite scale." (~02:49)

The bottleneck for real-world AI in one line: not model intelligence, but the cost of a rollout.

"Our approach at Runway has been predicting the world by predicting the next frame." (~02:50:30)

The thesis statement of the talk.

"The bitter lesson really applies to video models." (~02:53:30)

Physical accuracy improves predictably with compute, not with architectural priors.

"A lot of traditional video models have this bias towards success." (~03:00:20)

The most counterintuitive point: the very models that produce beautiful successful footage are the wrong ones for evaluating a policy.