Computer-Use Models Will Agentify the Web, Not APIs

TL;DR

• The target is step three of a syllogism: (1) agents, not humans, will drive most action on the web ✓; (2) the web will be "agentified" ✓; (3) via APIs across a dozen-plus standards — MCP, web MCP, payment protocols ✗.
• The long tail will never have APIs: real restaurant menus are plain text (easy), PDFs (medium), or a gallery of pixelated, un-OCR'd page scans (hard). Roughly 200,000 small business owners in a single European country maintain their own sites. "The cavalry we're waiting for is not coming."
• "Just have a coding agent read the HTML" also fails: e-commerce stock state isn't in the HTML at all — it arrives as a JSON blob at page load, and separate code decides whether an option renders greyed out. A browser is a renderer; pixels are the source of truth.
• Navigator: Yutori's pixels-to-actions model takes a screenshot and emits human-like actions — but isn't limited to human methods, since it has an execute-JavaScript action that can fill an entire form at once.
• Computer use is not stuck: an academic benchmark from Ohio State is essentially saturated at ~97%, and specialized small models already beat frontier models on cost and speed.

Key Points

The argument being attacked (~00:29–00:30)

The popular online narrative goes in three steps: (1) AI agents rather than humans will be the primary drivers of action on the web — booking meetings and appointments, buying things, retrieving information. (2) Asked how, people say the web will be agentified, made agent-friendly. (3) Asked how that happens, the answer is APIs — your agents, my agents, and enterprise agents calling through fourteen different standards: MCP, web MCP, payment protocols, and so on.

His claim: the first two steps are right, the third is wrong. What will actually agentify the web — specifically its long tail — are computer-use agents that operate browsers like a human, by looking at the screen and pressing buttons.

Why the flight-booking demo is absurd (~00:30–00:31)

The canonical demo in this literature: a user asks their iPhone to find a flight, and behind the scenes a browser-use agent opens flights.google.com and clicks buttons like a human. He calls it ludicrously funny — that's a database with aggregator API services already available. Send a structured query, get a structured result. Why would you click buttons?

The generalized question exposes the difference: "I'm planning an evening gathering with friends — are there any gluten-free items on this restaurant's menu?" People imagine a future myrestaurant.com/menu endpoint you can curl and filter with a natural-language query.

What the long tail actually looks like (~00:32–00:33)

He showed three difficulty tiers of real restaurant websites: easy — readable text; medium — menus as PDFs; hard — a gallery of individually scanned menu pages stitched together, pixelated, not even OCR'd.

Behind those sites are real people: roughly 200,000 small business owners maintaining their own websites in a single European country. "What you are imagining is that tomorrow, or in the next five years, they will all be revolutionized so that there are gateways for your agents to pull." His verdict: it isn't coming. The web is extremely long-tailed — individual sites may be of limited value, but the tail cumulatively carries enormous value, and infrastructure changes slowly.

Why reading the HTML doesn't work either: the osmium cube (~00:33–00:35)

The standard rebuttal is: fine, we have coding agents, just read the HTML. He counters with an e-commerce page and a trivial question — is this 24mm osmium cube in stock?

Scroll down to the quantity dropdown and human eyes can see three of the options are sold out and one is available. Send an agent to read the HTML and the selector carries descriptors for the items but nothing about quantity. Behind the scenes, a query on page load returns a JSON object listing what the store actually has, and a different piece of code decides at render time whether each dropdown option shows greyed out or in color.

The framing that makes it click comes from game engineering: a browser really is a renderer — assets underneath, code on top, pixels out the other end. The web was built for human eyeballs; that is the source of truth; so machines will need to operate with vision. Yutori wrote this up as a blog post, "The Bitter Lesson for Web Agents": if you don't look at the pixels, you get stuck feature-engineering your way toward a long tail you'll never reach.

Live mishap: a popup covered half his slides. While waiting for it to be cleared, he ad-libbed: "APIs are not arriving, but my computer-use agent is arriving. It'll do a better job than this."

Navigator: pixels to actions (~00:37–00:38)

Yutori's model is called Navigator: input a screenshot of the browser, output a human-like action — clicking, typing, scrolling.

• Example task: "I have a promo code — does it work?" There is no API for that, and the store back office will never expose one. Navigator opens the browser, walks a mock checkout flow, applies the code, checks whether the price dropped, and returns a structured object: yes, it works, and the price went down 22%. That is your API of the web.
• Vision-based, but not human-limited: the model has an execute JavaScript action. On a form-filling task, it writes custom code and populates many fields simultaneously instead of clicking field by field.

Rebutting "computer use is stuck" and "it's slow and expensive" (~00:38–00:40)

There's a narrative that coding agents keep progressing while computer use has stalled. That's not what reality shows: benchmarks keep falling. He cited a benchmark run by an academic group at Ohio State, essentially saturated over the last couple of years — as of about a month before the talk, accuracies sit at 97%, so new benchmarks will have to be created.

On speed and cost: yes, if you use a frontier general model. Comparing against Opus 4.7 and GPT-5.5, every action takes around 10 seconds with GPT-5.5, and a full task can run to $230 on some dataset. Yutori's specialized model is significantly smaller, just as accurate, faster, and much cheaper.

Closing: we've had 30 years of web built for human consumption, and that is changing — but the how matters. The universal web API, where you can pull any website and describe any task in natural language, will be delivered by a sea of agents driving browsers in the background.

Quotes

"The cavalry that we're waiting for is not coming." (~00:33)

Waiting for long-tail websites to sprout agent-friendly APIs is waiting for reinforcements that will never arrive.

"A browser really is a renderer. … The web was built for human eyeballs. That is the source of truth. And so machines will need to operate with vision." (~00:35)

The thesis of the talk in one breath.

"APIs are not arriving, but my computer-use agent is arriving. It'll do a better job than this." (~00:36)

Improvised when a popup blocked his slides — and an accidental live demo of his own argument.