Real-World Reasoning Agents

TL;DR

• Today's VLM visual encoders are stateless: each image is encoded independently, so two nearly identical images produce the same caption. Their Stateful Visual Encoder adds cross-image attention in the encoder's middle layers, tuned specifically to preserve small state changes that would otherwise be attenuated away.
• Touch is the underappreciated modality. Foundation models reason, plan, and act well but still don't react. T-Rex pairs a slow-fast multi-process architecture with tactile sensing to peel a card off a deck, squeeze toothpaste, identify mahjong tiles by feel, insert and turn a key, handle raw eggs, and screw in a light bulb without cracking it.
• A world model doesn't have to predict pixels. The World Motion Model operates over SE(3) pose trajectories: tokenize 6-DoF pose frames, stack them over time with one reference frame per object or rigid body part, train by denoising, and get future prediction, action-conditioned prediction, inpainting/motion planning, and retargeting.

Key Points

Seeing: Stateful Visual Encoders (~00:44–00:47)

He opened by crediting the previous talk — "Vincent's talk was the perfect precursor to mine, so I don't have to spend any time explaining why I'm interested in physical AI."

The problem is concrete: we have agents that can see, but they don't always see well. The visual encoders in today's VLMs and agentic VLMs are designed for internet vision tasks, and are not always well suited to physical AI tasks — or even to precise internet vision tasks.

His demonstration: two images run through the same encoder and LLM produce the same caption, and the audience themselves had to look back and forth to spot the difference. "And that's exactly what we don't allow most visual encoder architectures and VLMs to do."

Their fix is "so simple you'd be surprised it's novel": allow cross-image weights in the encoder's middle layers, tuned specifically to detect changes too small to survive in a normal representation. With that change encoder, the model can caption what changed, or tell whether a small box in a web interface has been checked — without a brute-force approach.

The paper compares several ways to build this, and reports significant improvements on longitudinal medical report generation, image-editing control, and geospatial change detection.

Feeling: T-Rex (~00:47–00:51)

"We want to manipulate things in the real world the way people can — a huge challenge of physical AI right now. Robots generally can't do these things. We want to be able to screw a light bulb in and have it turn on."

His diagnosis: current foundation models reason, plan, and act, but are still not good at reacting, and tactile remains an underappreciated modality. Real roboticists know this; a lot of "AI roboticists" assume you can scale your way out of it without understanding dynamic forces in the real world — "I'm not sure that's right."

Human intelligence is dual-process, so agentic systems should be dual- or triple-process: a slow-fast architecture. T-Rex, in collaboration with colleagues at NVIDIA, embodies all of this — it is both a dataset/data-collection effort and an architecture with multi-process control, tactile sensing, and reactivity.

What it learns is when to change force based on what it feels: adjusting for slip to catch something about to fall, or applying just enough pressure to peel one card off a deck. The platform is two Shadow Hands with head- and wrist-mounted cameras, with the tactile sensor readout visible in the lower-right tile of the demo video.

The demonstrated tasks were previously impossible without tactile reactive policies: squeezing toothpaste out of a tube ("we did not actually yet ask the robot to brush its teeth — maybe next time"), separating two cups out of a stack, identifying which of three mahjong tile types it holds purely from the feel of the indentations, inserting a key into a lock and twisting it, squeezing a pipette bulb to dispense the right amount of liquid, extracting a card from a deck, and screwing in a light bulb with enough pressure not to crack it. He noted the eggs shown earlier were real eggs, not hard-boiled.

He suggested the dataset may be the single most important artifact of the project.

Reasoning: World Motion Models (~00:51–00:54)

The framing is a contrast with pixel-space world models. We now have impressive models that predict every pixel of the future — wonderful — "but if I just want to adjust something on that wheel as it goes by, or tighten a bolt on the wheel like an F1 pit crew, I don't need to predict what the trees are doing. I don't even need to predict what much of the car is doing. I want to predict the affordance, or the geometry, or how that wheel is moving over time."

Hence the World Motion Model: a model of dynamic 3D world trajectories — more precisely a world model over SE(3) trajectories, i.e. poses. His claim is that much of the physical world can be abstracted as the motion of rigid frames evolving over time, and this is the first sufficiently general framework for predicting and completing over that representation (and eventually running MPC on it).

Mechanically: tokenize the 6-DoF pose frame, stack it over time, with multiple reference frames — one per object or per rigid portion of a body in the scene. Train by denoising sequences. At inference, predict the future given the past, predict the future conditioned on actions, inpaint or motion-plan, retarget, or solve for dynamics. The backbone is an efficient transformer.

Demos included text-driven generation of robot motion trajectories to solve a task, either directly or via motion predictive control, and human-object interaction on the OMOMO dataset — generating whole-body humanoid motion to perform a manipulation task on an object, where their model outperforms baselines, with similar results for hand-object interaction.

He closed by describing the talk as a teaser for three BAIR projects that see, feel, and reason in motion about the real world; World Motion Models "will be on arXiv soon."

Quotes

"We already have agents that can see, but they actually don't always see that well." (~00:44)

Seeing is not the same as seeing well.

"You probably didn't realize those two images were actually different… and that's exactly what we don't allow most visual encoder architectures to do." (~00:45)

Humans need to look back and forth to catch the difference; encoders are never given the chance.

"Current foundation models can reason, they can plan, they can act, but they're still not so good at reacting." (~00:48)

Reason / plan / act / react — the last is what tactile sensing supplies.

"I don't need to predict what the trees are doing." (~00:51)

The motivation for modeling motion instead of pixels, in one line.