World Models for Physical AI Simulation

TL;DR

• Opening news: she announced her departure from Nvidia the day before; she's now "a civilian at University of Toronto" and invited the room to speculate about her next steps. Nvidia let her keep most of the deck, so the work shown is Nvidia's.
• Not a talk about agentic AI — about world models, the technology she's found most exciting for six years and which is now "around the corner."
• Simulation is robotics' foundational infrastructure. Evaluating an LLM means sampling text; evaluating a robot means deploying the policy on hardware in closed loop. Real robots are bounded by physical time and require buying more robots; virtual robots are bounded by compute time, which keeps getting cheaper and scales across GPUs.
• Three generations of simulation: (1) handcrafted graphics — one to two months of artist time to build a digital twin of a single San Francisco street; (2) NeRF and Gaussian splatting from ~2020 — real capture straight to simulation environment, which solved the content problem and let Nvidia run two million autonomous-driving simulations per day; (3) generative world models — purely data-driven, able to hallucinate the long tail nobody ever recorded.
• Her definition of a world model is interactive: not text in, five seconds of video out, but a human or a robotic policy in the loop with every single frame generated live. That system is OmniDreams, and it runs in real time.
• The one-year curve: March 2025, five seconds of video per five minutes of GPU time; March 2026, real-time interactive; two months later at CVPR, running on an RTX 5090 — "literally a game engine you can have in a backpack."
• Her favorite result: dropping the production driving policy into the generated world and having it drive — the policy confuses the simulated world with the real one, which is exactly the quality bar you need.

Key Points

From ChatGPT to agentic AI to physical AI (~00:18–00:19)

Three eras: ChatGPT in 2022, when AI reached the masses for the first time ("my mom knew about ChatGPT"); today's agentic AI, which she can no longer imagine working without; and the next break point, physical AI, where intelligence moves to the edge and into robots across industries — perhaps a decade out, depending on which robot you're waiting for. Her group at Nvidia focused on one particular robot, the autonomous vehicle, which serves as the running example; general-purpose robotics is the hot upcoming field.

Generations of robot software (~00:20)

For a car: simple vision → hardcoded driving rules → machine learning at every stage (perception, prediction, planning) → recent end-to-end systems that map sensor data to a planned trajectory → and the emerging generation built on large pre-trained foundation models, which carry vast knowledge of how the world works, cover corner cases that are nearly impossible to collect in the real world, and can reason. The field genuinely believes foundation models are the game changer for that last mile of corner cases.

Why simulation is the infrastructure (~00:21–00:23)

The LLM development loop is cheap: train models, sample text, evaluate, sometimes with a human in the loop for conversation. The robotics loop is not: the robot's brain must be deployed on real hardware in closed loop and observed (she showed a Skild AI pick-and-place clip pulled from the internet). The alternative is a virtual robot in a cloud digital twin doing the same pick-and-place — which gives a genuinely useful signal about which policy is better, and scales training far more cheaply.

The argument that carries the talk: real-world development is bounded by physical time, and there's no running faster than physics, so scaling scenario coverage means buying more robots. Cloud development is bounded by compute time — and compute keeps getting faster than real time, and scales across GPUs instead of robots. "So it's a much, much, much more scalable pipeline." Her conclusion: you cannot develop robotics without really, really good simulation.

Three generations of simulation (~00:23–00:25)

1. Graphics: artists carved out 3D assets, wrote behavior models, wrote the ray-tracing equations. It worked, but testing a car on a specific San Francisco street cost one to two months of artist time to build that digital twin.
2. NeRF and Gaussian splatting (~2020): a complete game changer, because you could go from a real-world recording captured on a robot straight to a simulation environment — with the restriction that reconstruction only gives you what was actually observed. Pipeline: capture → surface reconstruction for collision checking → splats (little appearance blobs in space) telling you how to render new viewpoints. At Nvidia, this powered two million autonomous-driving simulations per day.
3. Generative world models: reconstruction is bounded by what you recorded, and robotics lives in the long tail — the stuff that's so hard to see and that you most need to test and train on. So it's all about imagining the hard cases. World models are purely data-driven AI, no human crafting content, so the only limit is data.

What generative world models buy you (~00:26)

• Hard scenarios on demand: rough weather, a fully generated collision ("no one got hurt in that video — it was a completely generated accident"), exactly the cases you want to test software against.
• Viewpoint changes ≈ embodiment changes: going from a low sports car to a truck, which matters enormously for testing across vehicles.
• Editing: trivial with generative models and much harder with reconstruction — a text prompt is enough to rewrite the scenario.

The specific architecture: interactive (~00:26–00:27)

"There's a specific architecture I have in mind when I talk about world models today, and it's interactive." Not text in, five seconds of video out — there is a user or a robotic policy in the loop.

Her first user was "my favorite human": as soon as the model was working, she walked into their office and had them try it with a real steering wheel. A human drives, and every single frame is generated by the world model. That model is OmniDreams, and the bonus is that it runs real time. "Does simulation need to run real time? Not necessarily — it just needs really high throughput. But real time is a great bonus."

How it's built (~00:27–00:29)

1. Base model: a pre-trained video generation model, typically diffusion — noisy frames denoised into a few seconds of video, with bidirectional attention, pre-trained on vast publicly available internet data.
2. Domain post-training: adapt to a specific robotic domain such as AV — and not just "stuff that happens on the road" but stuff as captured by a robot, which usually means many cameras rather than one.
3. Make it interactive: switch to a causal architecture that generates frame by frame with actions in the loop, rather than short video snippets at a time.
4. Make it efficient: distill the many-step diffusion model down to a few steps, then apply every trick in the book for a real-time implementation. That stack is FlashDreams, which has been released and can be plugged into different world-model architectures.

In the demo, physics is deliberately minimal — ground physics only, so the car stays on the ground and rides over speed bumps, but there's no collision checking beyond the road surface, so "you can actually go through a restaurant, no problem."

The one-year trajectory (~00:30)

March 2025: five-second clips at roughly five minutes of GPU time each — clearly impractical. March 2026: real-time interactive. June 2026 at CVPR, two months later: running on an RTX 5090, a consumer-grade GPU — "literally a game engine that you can have in a backpack." That slope is why she says the technology is around the corner.

Results, and the moment she cared about most (~00:30–00:32)

From a text prompt or a single first frame you get scenes far more complex than a graphics engine could produce — too many simultaneous phenomena — and that reconstruction techniques struggle to generate too. The model produces the multiple sensor streams a real robot carries. Editing the same environment changes weather and lighting, and the little green trajectory on screen is the robot's own policy driving through the generated world.

That was her favorite result: they plugged in the production policy and it drove. "It's confusing this simulated world with the real world, which basically means that your quality bar is at the level that you actually need." And on the long tail: "Even though Nvidia has never captured an elephant on the road, we can actually generate it." The closing demo showed a person driving on the left while on the right the robotic software turns the wheel, driving inside a three-camera world model — every frame generated, all in real time.

Quotes

"I actually no longer [work at] Nvidia. I just announced it yesterday. … Today I'm a civilian [at] University of Toronto and I'll let you speculate about my next steps." (~00:18)

The talk opened with news.

"In the real world we cannot run faster than the physical time. … On the other hand we're bounded by compute time — computers are getting better and better, we can actually run faster than real time, and we just scale across GPUs instead of robots." (~00:22)

The core argument: robotics scales only by leaving physical time behind.

"It's confusing this simulated world with the real world, which basically means that your quality bar is at the level that you actually need." (~00:31)

The most practical test of whether a simulator is good enough: the production policy can't tell.

"Even though Nvidia has never captured an elephant on the road, we can actually generate it." (~00:31)

The long-tail problem in one sentence.