Panel: Robotics & World Models

Setting (~03:08:47)

Moderator Guru Chahal (Partner, Lightspeed Venture Partners) had set the session's frame at the top (~02:09:26): Lightspeed has invested across the AI stack from Anthropic, Mistral, SSI, and Reflection through robotics companies like Dexterity and Skild, and the common view is that robotics is still in its pre-ChatGPT moment — a view he broadly shares, while arguing the inflection point may be closer than most people realize.

He deliberately skipped the opening "define a world model" question (it would eat the next twenty minutes) and refused the go-down-the-line format, instead pulling out points from the five talks where the panelists' stances looked like they might diverge.

Topic 1: Does generating video alone give a model an understanding of physics? (~03:10–03:12)

Anastasis Germanidis (the original claim): this is mainly an empirical conclusion from scaling the models and measuring how they perform on physics-related tasks. Their methodology: if you want to introduce inductive biases, introduce them in the data you train on, not in the architecture. So they very intentionally learn everything directly from pixels with no 3D priors, and it has worked well — past a certain scale you get good geometric consistency, which is also measurable. He sees no fundamental reason it won't keep scaling.

Jim Fan (agrees, then narrows it): despite Guru's ban, he offered a definition to ground the discussion — a world model is a model that predicts the next sensory state conditioned on actions. Under that definition, video is convenient, and video is one of the sensors: it's called an RGB camera. But a robot can have more than one RGB camera, and many other sensors besides. He argues today's world-model discussion is missing modalities: beyond video there is tactile, force, and inertial sensing, and all of it needs to go into these models for a genuinely holistic simulation of how dynamics evolve. "Video is convenient, video is easy to scale, it's abundant — but that's not the full story."

Sergey Levine (the strongest pushback): if you generate videos that people find compelling, it is very easy to end up with video that looks great to humans while not reflecting the model's ability to simulate the physical counterfactuals you actually care about. The crux: when you give a model a prompt, it doesn't have to predict what will happen to a real physical system if it does something. It only has to produce a result that, to you, looks like a good reflection of the prompt. Which means the model gets to choose how to set everything up so it looks good — and that choice is enormously important. Whereas if you need to plan and act in the real world, you must handle the actual situation you're in, the actual plan you're making, and whatever counterfactual effects follow.

His example is blunt: train a model on really good driving data, then ask what happens if you drive into a building, and it will probably produce the building morphing into a beautiful road with you driving happily down it. People have done this, and that is exactly what happens. So the answer is: yes, the model can handle physics; you just have to be very careful not to let it fool you. And as Anastasis said, it comes down to data — it's just that the data that produces really great video is not the data that produces counterfactuals.

Topic 2: Is the VLA era over — is it all world-action models now? (~03:12–03:16)

Jim Fan: it depends on the level of abstraction you work at. Splitting by system 1 / system 2:

• System 1 (fast, reactive motor skills): WAM is very promising. A lot of physical common sense — how do you grasp a cup — is, from first principles, very difficult to break down and describe frame by frame in language tokens.
• System 2 (reasoning, coding, tool calling): he argues Opus and the GPTs are actually VLAs — these models are agentic and they take actions. And more people use VLAs than WAMs today, because VLAs also serve other purposes, including the agentic scaling he covered in his talk: models that understand the world through perception, call tools, and help with auto-research.

He doesn't see one completely replacing the other. It's about the abstraction level you operate at.

Sergey Levine (thinks the question is subtly wrong): "It's the right question for you to ask us, but maybe the wrong question for a researcher to ask themselves" — because it puts the model before the data. The key to a really good foundation model is being able to productively ingest lots of very diverse data, and video is one great source of it. In fact, most VLAs that are actually good are trained on lots of video data — it just happens in a web-scale pretraining phase that trains the understanding backbone.

So the right question isn't VLA-or-video, it's how do you use the most data. On top of that sits inductive bias, and as the saying he quoted goes, "inductive bias is training data in disguise" — its effect may be large when you test at small scale, but it dissolves away once you scale up. What we should actually do is leverage as many data sources as possible, then be pragmatic about what outputs the model needs in order to leverage them.

Michael Spranger (two points from the application side):

1. Language isn't all that helpful for their tasks. For racing and table tennis, you can describe the skill but there's very little information in the description. Cognitive science's classic distinction between declarative and procedural knowledge applies: to ride a bike you have to actually do it and learn the physics.
2. Vision isn't always available. Vision is a great thing to condition on, but in some applications you can't. Ship a game to a customer with 20 agents racing and you cannot render from 20 different viewpoints — most of the PlayStation's compute goes to rendering for the player.

So these aren't yes/no questions: what domain do you care about, what are you solving, what's the downstream application? Large foundation models that capture visual and language knowledge are often extremely helpful, but you still need to distill the thing you actually care about in a given moment and ship it to a consumer.

Anastasis Germanidis (adding on): there's broad convergence toward omni-models that predict visual, audio, and text modalities alike. So, to Sergey's point, it comes down to what you specialize the model on during pretraining — is the data mix biased toward low-level motion planning or high-level reasoning? It's less about WAM-versus-VLA architecture and more about what you dedicate the model's capacity to — and that's the important factor.

Guru's summary: "It's not an either-or. It really is use-case dependent, and both have interesting applicability."

Topic 3: How much real-world RL versus the digital world? (~03:16–03:22)

Wei Zhan (separates by application):

• Autonomous driving: they care more about behavior and the visual modality, and both are relatively easy to scale from real-world data and inject into a synthetic world. Today, most leading players in productionized end-to-end ADAS rely mainly on real-world data, with synthetic data used in post-training — but the trend is clear that large-scale synthetic data will play an increasingly key role in making autonomy robust and safe.
• Robotics: at this point, for locomotion and whole-body tracking, RL-based methods are very useful and even dominant. But for complex tasks like dexterous manipulation, it's hard both to build a high-fidelity synthetic world capturing the key physics attributes and to define a good reward function for closed-loop training. So behavior cloning and imitation-based methods still dominate there.

Sergey Levine (splits supply side from demand side): the future will look very different from the present.

• Now: robotics is in the bootstrap stage — still figuring out what it would take for robot models to be good enough to be usable with end-to-end learning.
• Later: we won't be bootstrapping forever. Eventually there will be as many robots deployed as there are vehicles on the road today, and therefore very plentiful data from robots interacting with the world autonomously — at which point RL methods that can consume real-world data will be tremendously important.
• A second reason: as imitation learning methods keep improving, the challenge increasingly becomes bridging the last gap — how do you go from a 98% successful policy to 100%? There the domain gap really matters, because everything has to line up perfectly. So real-world RL will be needed there too.

But he timestamped it explicitly: "that's what's coming in the future; I don't think that's the state today" — calling Wei's account of the present an excellent summary.

Jim Fan (copy the LLM homework; three environment options): "Great scientists invent, greater scientists copy." He watches what the LLM people are doing, and they're not just creating data anymore — they're creating environments. It's about acquiring millions of coding environments and doing RL in them; that's how we got today's Mythos and GPT-class models. Not through imitation learning — eventually through reinforcement learning. Robotics needs to build environments too, and there are three options:

1. The real world. "Sergey and π have done exceptional work over there." Absolutely important — the physics is perfect because it is the real world.
2. Classical simulators (Isaac Sim and the like). The bottleneck isn't simulation speed — simulation runs very fast — it's the assets, environments, and tasks that used to require armies of artists. But real-to-sim recipes now look super promising: automatically scan the world and transport it into a classical simulator, so you're data-driven while still leaning on a fast classical physics solver.
3. World models as neural simulators. A very practical use of a video world model is as a data-programmed neural simulator: take actions as input, generate video and sensor states, use it as an environment. It runs very slowly, but it has a huge amount of diversity — as much as a world model can afford.

Ultimately robotics RL will run across all three options, using different types of compute under different constraints. He'd like to see a more holistic approach.

Topic 4 (rapid fire): Most amazing recent demo — not from your own company (~03:22–03:27)

Topic 5 (rapid fire): What year is robotics' "Claude Code moment"? (~03:27–03:35)

Guru's framing: LLMs had a ChatGPT moment, and then a Claude Code moment — the point where the first commercial use case genuinely takes off and rockets. He wanted a number, plus a word or two on the use case.

Michael's upper bound (an explicitly non-technical argument): "Can I give an upper bound? For some economies we're going to have to solve it in 10 years." Take large Japanese corporations in manufacturing, or any industry: 60% of the workforce is 50 or older, and in 10 years they will all be gone.

Guru's close: "So those are the bounds. Two to ten. Let's solve it by then."

Quotes

"If you train a model on really good driving data and then you ask what happens if I drive into a building, it'll probably produce the building morphing into a beautiful road and you're driving happily down the road. And people have actually done this and that's exactly what happens." (~03:11:40)

Sergey's sharpest counterexample to video-models-understand-physics.

"Yes, the model can handle physics. You have to be very careful not to let it fool you." (~03:12)

The conclusion of that exchange — not "don't use it," but "don't be fooled by it."

"Video is one of the sensors. It's called an RGB camera." (~03:11)

Jim Fan demoting video from "a representation of the world" back to "one sensor," and using it to surface tactile, force, and inertial modalities.

"Inductive bias is train data in disguise." (~03:13:30)

Sergey's borrowed line: architectural priors that look powerful at small scale dissolve once you scale.

"Great scientists invent, greater scientists copy." (~03:19:30)

Jim Fan's methodological credo, immediately followed by "the LLM folks aren't creating data anymore, they're creating environments."

"With robots the demo is never quite what meets the eye." (~03:29:50)

Sergey's moral from the Atlas backflip: the real challenge isn't in the demo reel.

"For some economies — and this is not a technical argument — we're going to have to solve it in 10 years." (~03:35:05)

Michael's hard upper bound, argued from Japanese manufacturing demographics rather than from capability curves.