Robot Foundation Models

TL;DR

• The real gap is the level of the prompt. Research benchmarks ask for atomic tasks ("put the corn in the pot"); what people actually want to say is "I've got guests coming tonight" — an entire job that requires grounding, planning, and generalization to new context.
• Test-time compute buys robots a second, less obvious benefit. Beyond solving harder problems by thinking longer, it decomposes unfamiliar problems into pieces that sit inside the training distribution ("I've never grasped this, but web-scale pretraining taught me to localize objects in images"). RL on top curates the decomposition so the sub-instructions are ones the robot can actually execute.
• Thinking need not be in language. Recipes suit language; rock climbing does not. Physical Intelligence's π0.7 folds a lightweight world model into its decision loop, reasoning in both language and images before grounding down to actions — enough to operate an air fryer it was deliberately never trained on, taught purely by spoken coaching.

Key Points

From atomic tasks to whole jobs (~02:12)

He opened with a joke: tired after work, he asked his favorite LLM to clean the house, fold the laundry, and make dinner. It obliged with a list of things he should go do himself. What he wants instead is an embodied foundation model sitting in a robot that can actually do it.

Robot foundation models are a young but fast-growing field. Physical Intelligence has been at it for over two years — a robot pulling espresso for 13 hours straight, assembling boxes in a factory — and strong results are coming out of academia and other companies too. Rather than parade results, he zoomed in on one question: given how good the results are, why aren't there robots around us following arbitrary prompts?

The answer is the level of the prompt. Research tasks look like "put the corn in the pot" (from the bridge dataset). Nice, but not what we want. We want "I've got guests coming this evening" — a prompt that specifies an entire job, not an atomic task — and one whose meaning depends on what the robot knows about you and your home. A great deal of grounding and problem-solving sits between that sentence and a motor command.

Test-time compute, and using RL to optimize the thoughts (~02:14–02:17)

Test-time compute in the LLM world is familiar: generate more tokens, think harder, solve harder problems. The less obvious benefit is that it breaks an unfamiliar problem into familiar pieces. The robot can reason: "You want me to pick up this object; I've never done that. But in web-scale pretraining I have localized objects in images — let me use that as a stepping stone."

His illustrative task is deliberately artificial: make the blue block the only thing on the plate, with a piece of (toy) sushi currently sitting on it. The correct response has to be semantically sensible, not just effective — you don't put sushi on the table, that's unsanitary, so it should go into an appropriate container. The reasoning trace handles that, and further recognizes that the sushi may be visually unfamiliar, so it chooses to emit a spatial coordinate rather than lean on the action output to absorb the visual distribution shift. At run time the thinking trace refreshes every few seconds based on what's happening.

Thinking models are most powerful when the thoughts themselves are optimized with RL. Here RL plays a dual role: it curates the task specification into something closer to the robot's training distribution. You propose several textual decompositions of a high-level prompt and keep the one the robot is most likely to succeed at.

The example: "put the hammer on the plate" — thoroughly out of distribution. An off-the-shelf VLM (Gemini, in this case) doesn't know what the robot can and can't do, so it simply says "go ahead, put the hammer on the plate," and the robot — which doesn't know what a hammer is or how it goes on a plate — fails. After RL, the model spoon-feeds it: "move a little to the left so you're right over the hammer," and only then "put it on the plate," at which point there is essentially one way to do it. RL bridges the distribution gap.

Multimodal thinking and π0.7 (~02:17–02:20)

Following a recipe suits step-by-step language. Rock climbing does not — you don't think "move my arm 37 centimeters left," you think spatially, in terms of what things will look like. So robot foundation models can think in other modalities. In earlier work, instructions were decomposed not into text but into an intermediate image: imagine what the scene would look like after progress toward the goal, treat that as the thought, and condition the action on it. This works because image generation can be pretrained at internet scale on language-image associations, giving the robot a way to imagine substeps.

That brings him to recent large-scale work at Physical Intelligence: π0.7, a large vision-language-action model that incorporates a lightweight world model into its decision-making, reasoning about the task in language and in images and then grounding all of it into actions.

The demo: Lucy teaches the robot to use an air fryer — an appliance they deliberately kept out of training, so it must generalize zero-shot. Told to air-fry the sweet potato, it succeeds, but slowly and with a lot of fumbling. A little extra supervision makes it far more reliable, and crucially the supervision is not in actions: Lucy talks the robot through the task step by step, producing additional image-plus-language data. The actions in that data are the robot's own, not ground truth. The data is used only to fine-tune the thinking process — teaching the robot how to think. Afterward you can watch the model's own generated language thoughts alongside the intermediate image thoughts, and the task runs fluently.

Takeaways (~02:20)

• Reasoning models let robots generalize better and follow more complex prompts. That matters long-term, because the prompts we ultimately want are like "you're a robot butler: clean the house at 6pm, get dinner ready, do my laundry on Saturdays" — one prompt, possibly weeks of robot operation.
• Thinking doesn't have to be in text. The physical world is complex; use the right modality for the job.
• You can even use multiple modalities simultaneously, at the right level of abstraction: language for following a recipe, imagined imagery for figuring out where your hand should go to push a button on a new appliance.
• Together this provides both a way to solve harder problems and a way to transfer knowledge more effectively from sources represented in other modalities.

Quotes

"Why is it that with all these really nice results, we aren't seeing robots that can follow arbitrary prompts all around us today?" (~02:12:30)

The framing question of the talk — not "what demo is next," but "why hasn't the demo become ordinary life."

"Break down an unfamiliar problem into more familiar steps." (~02:15)

What test-time compute is really worth to a robot: pulling out-of-distribution problems back into distribution.

"The thinking doesn't have to be in text. The physical world is complex and you should use the right modality for the job." (~02:20:30)

A direct rebuttal to language-only chain-of-thought for embodied systems.