On the Visual Capabilities of Multimodal Models

TL;DR

• A deliberately different angle. Everyone else was talking about coding agents and how to extend them; he wants to talk about adding a new capability — multimodality. "When you hear me now, you don't just see a text, you see me, you see everything. So we live in a multimodal world."
• The structural skew in today's multimodal models: a very strong LLM with a visual head found somewhere and attached, trained together in post-training. Analysis shows most of the reasoning in multimodal understanding happens in the LLM part, not the visual part — and in-context learning with too many images actually confuses these models, for the same reason.
• A two-year jump in spatial cognition. Two years ago frontier models were near chance on mental rotation, perspective taking, maze completion, and shortcut discovery. By January this year they were at the level of a three-year-old child; by February 2026 they could solve spatial tasks that take a human about 10 seconds. On perspective taking, Opus 4.6 and Gemini 3.1 are now near 100%.
• The mechanism: recent work with colleagues at Apple probed models that cannot generate images and found they internally reconstruct visual intermediate states — and explicitly supervising that imagination improves performance further.
• Two takeaways: asking the right question is what advances the science; and multimodal models should not focus on text alone — training the generate-the-visual half helps a lot.

Key Points

The old problem: compositionality and the bag of words (~02:46–02:48)

His observation about the field: once people start asking the right question, solutions start appearing within a year or two.

An early example is compositional reasoning. Older multimodal models captioning an image of, say, three zebras and two giraffes would jumble the numbers and the names together — treating everything as a bag of words, so compositionality broke. His group worked on fixing this: decomposing words into tree structures, producing denser and better-aligned captions, teaching the model to be structured. Things improved from there.

The structural skew: reasoning lives in the LLM (~02:48–02:50)

Two analyses of what actually happens inside the understanding workflow:

• Work with colleagues on the performance gap in entity knowledge, published at ACL a year ago, asking where in a multimodal model the reasoning occurs. The finding follows from the architecture: every modern multimodal model is a very strong LLM with a nice visual head attached — we invest heavily in training LLMs, then take some visual language model we found somewhere, attach it, and post-train them together. When you analyze multimodal understanding, most of the analysis happens in the LLM part, not the visual part.
• A second analysis of in-context learning. ICL is great in LLMs, but in VLMs there's a strange effect: give too long an in-context sequence with too many images and the models start to get confused — again because training relies mainly on the language side.

His question: maybe we need a shift.

Spatial cognition: from chance to near-perfect (~02:50–02:54)

The talk narrows to spatial cognition, starting from colleagues' work two years ago, "Does Spatial Cognition Emerge in Frontier Models?", which ran classic spatial-cognition tests on multimodal models:

• Mental rotation — given a shape, identify which candidate is the same shape rotated.
• Perspective taking — e.g. what is the angle between the bat and the dog, answered from a given viewpoint.
• Maze completion — solve the maze.
• Shortcut discovery — show the model a walkthrough of a place that doesn't take the shortest path, then ask it to take the shortest path.

Two years ago, all models were very bad — frontier models including GPT sat near chance, only slightly better.

Then, once the question was on the table:

• January this year, a study the captions render as "baby vision" ran a similar exploration and found frontier models roughly at the level of a three-year-old child. His aside: nice to grow to three years old in a year and a half, though usually we'd prefer to grow slower.
• Another work compared how long a human takes to solve each problem against what models can do, finding that by February 2026 frontier models solve spatial-cognition tasks that take humans about 10 seconds. Since mental rotation takes humans real time to work through, being capped at 10-second tasks is "not so great, but better than what we had before."
• Task by task: on perspective taking, the best models were near chance two years ago; now Opus 4.6 and Gemini 3.1 are almost at 100%. Maze completion is still not great, but just three months ago it leapt from near chance (his example: GPT-5.4), and shortcut discovery did the same.

Why: the models are imagining internally (~02:54–02:56)

Recent work with colleagues at Apple asked: do multimodal models imagine electric sheep? The answer is yes.

They studied a range of spatial-cognition tasks in an open-loop setting — show the model only the first frame, then ask what step it should take to solve the game. Critically, they looked at models that cannot generate images (they can receive images but not produce them), and probed the weights with an attached transformer.

The finding: the model is internally imagining how to solve the problem. Asked whether two shapes are the same shape under rotation, the model's internal weights reconstruct the visual content — the same thing we do when we mentally rotate two shapes. The same holds for assembling characters.

That may be exactly why the last two years improved so much: models started to imagine. So they went further and gave the model the ability to imagine and supervised it to do so — and performance improved.

Two closing points (~02:56)

Skipping the remaining results for time, he leaves two:

1. Asking the right question is very important to advancing science.
2. For multimodal models, don't focus on text only — focus on generating the visual part too. It helps the model improve a lot.

Quotes

"When you hear me now, you don't just see a text, you see me, you see everything. So we live in a multimodal world." (~02:46)

Why a room full of coding-agent talks still needs someone talking about vision.

"You have very strong LLM and a nice visual head attached to it … we found somewhere, we attached to it." (~02:48–02:49)

The bluntest and most memorable description of current multimodal architecture: vision is bolted on, not co-designed.

"Nice to grow to become three years old in one and a half year — but usually we want to grow slower." (~02:52)

His deadpan on the pace of spatial-cognition progress.

"Once you start asking the question, you start to see improvement throughout time." (~02:51)

The through-line of the talk: the question itself is the engine.