From Games to the Real World: How Reinforcement Learning Is Powering Performance and Fun

TL;DR

• One infrastructure, three stages: the 2022 Nature-cover Gran Turismo racing agent → integration into the game as the first Power Pack people pay directly for an AI experience → the very same agent factory producing a table tennis robot that beats professionals.
• Gran Turismo is a good benchmark because it demands three things at once: physical realism (racing lives at the edge of control, and sometimes past it), tactics (opponents counter your moves, so you must learn to probe and deceive), and sports etiquette (don't crash into people, but drive timidly and you get overtaken). One end-to-end RL system solves all three.
• His core argument: LLMs are great and world models are great, but don't forget RL — its ability to produce highly specialized superhuman policies in both simulation and the real world is what will, in combination with the rest, actually solve physical intelligence.

Key Points

Gran Turismo: three axes at once (~02:36–02:39)

The story starts in 2022 with the team's first Nature cover for this project. Gran Turismo is a 30-year-old PlayStation title led by Kazunori Yamauchi, whose real interest is recreating the sensation of racing — which is why it isn't an easy game: you have to genuinely hone your driving to enjoy it.

Step one was an RL agent that beats the best drivers. He played footage from an actual competition and challenged the room to say which of the two cars was the AI. It was the gray car; the white one was a Gran Turismo world champion. You can't tell, and that's by design — they deliberately built an AI that is competitive and humanlike.

Gran Turismo makes an unusually interesting benchmark because it combines three demands:

1. Physical realism. The game is built on a physics engine that faithfully reproduces vehicle dynamics, in some cases on real-world tracks. To race, you first have to master driving — and racing itself sits at the edge of control, sometimes deliberately over it.
2. Tactics. You're not alone on the track. Make a move on an opponent and they counter, so you have to learn to deceive and to probe.
3. Sports etiquette. Racing is a collaborative-and-competitive dance: you're not supposed to crash into other cars, yet you must drive aggressively or you'll simply be overtaken and lose.

They trained one end-to-end system for all three with reinforcement learning: the game is the environment, inputs range from visual to state representations similar to an autonomous car's, the agent acts in the game, and reward follows the obvious shape — fast and overtaking is good, crashing into cars is bad, failing to make progress on the track is bad. That alone suffices to build highly specialized, extremely capable AIs in Gran Turismo and other titles.

Infrastructure meant heavy collaboration with PlayStation: the game only runs on PlayStation, so they had to use PlayStation cloud infrastructure to reach the compute scale required. Once the research was done, training was fast: 15 minutes of driving gets you around the track, about 24 hours yields a really nice driver, and a few days produces a superhuman policy.

What superhuman looks like: on a time trial at Dragon Trail Seaside there's a corner they call the "chicane of death," and no human driver has taken it the way the agent does. A second clip shows the tactical layer: white cars human, colored cars AI. The AI deliberately trades away position to set up a better entry, then completes a double overtake on the exit. Those subtle trade-offs are entirely learned.

Not just fast, but fun — and then shipped (~02:39–02:42)

Superhuman isn't the goal. They care about building agents that are fun. He played an interview with a high-end human racer: the most interesting part for her was watching the AI's racing lines — corners where she went wide and cut back in, the AI took tight all the way around. She also learned about prioritization: into turn one she braked later than the AI, but the AI got a far better exit and beat her to the next corner. "I didn't notice that until I saw the AI, and I was like, oh, okay, cool, I should do that instead."

Because the substrate is RL, changing the objective changes the behavior: swap "go fast" for "burn as much tire as you can" and you get superhuman drifting agents.

And as part of Sony, they get to ship to millions. After the Nature paper the team iterated hard on integration — within six months the technology was in the game — culminating in something he's very proud of: the Power Pack, the first ever paid Power Pack for Gran Turismo, where people pay directly for an AI experience. Because the experience is so humanlike, races can be longer and more fun, and Polyphony Digital built a whole Power Pack around the new capability.

From racing to table tennis: the same factory (~02:42–02:47)

This year's result takes exactly the same technology to table tennis. Structurally table tennis rhymes with racing: real-world physical realism, tactics, and an opponent trying to beat you. What they're demonstrating is a factory for agents that solve highly specialized competitive tasks.

The difficulty: the ball crosses the table literally in the blink of an eye, and spin — not speed — is the main difference between professional and amateur play, at roughly 9,000 rotations per minute with speeds up to 100 km/h. You must master that physical skill while someone across the table is trying to beat you.

They train the policy in a custom table tennis simulator, then bring it onto the real robot without any changes and play professionals. The footage shows the robot beating Miu Hirano, a highly successful top player. One clip slows down at the crucial moment: the ball clips the net — something they did not foresee and could not train for. Within fractions of a second, this fully end-to-end policy (no programming involved) has to revise its plan at microsecond resolution.

They also have a flywheel now: when the robot still loses to a player because of an unforeseen tactic, they retrain the policy overnight on that match's data and the robot then beats that player.

The closing argument returns to the thesis: LLMs are great, world models are great, but don't forget reinforcement learning and its ability to build highly specialized policies that hit superhuman capacity in simulation and the real world. Solving physical intelligence will take the combination.

Quotes

"I'll challenge you to tell me which of those two cars is driven by an AI versus a human." (~02:37:30)

Gray car AI, white car world champion. The indistinguishability is the design target, not a side effect.

"Racing itself is really at the edge of control. It's sometimes going over this edge of control and pushing the limits." (~02:38:20)

Racing isn't driving steadily; it's operating deliberately at the boundary of control — which is exactly what makes it a physics benchmark.

"I think world models are great. We should not forget about reinforcement learning." (~02:46:45)

Delivered in the middle of a session themed on world models — a deliberate counter-current.