When Language Models Learned to Reason

TL;DR

• The entire talk runs on four-to-five-year-old internal slides. "There's this saying that there's nothing more stale than yesterday's newspaper" — and these are ugly, because they were never meant to be presented anywhere. He framed it as time travel: what it felt like to work on reasoning back then.
• The central picture: add a third axis — thinking time — to a two-dimensional wheel of problems. Pre-training only pushes the solved-problem circle outward from the center and never reaches the hardest problems; reasoning asks whether that cylinder can grow over time instead.
• The evidence that justified scaling it up was embarrassingly simple: stack prompt instructions telling the model to think step by step and show its work, watch the output get longer, and watch accuracy rise with output length. That curve started everything.

Key Points

Why old slides (~00:33)

These are the earliest internal presentations OpenAI had when it started work on reasoning — explicitly not a talk about the future. The slides are ugly because they were team-internal artifacts from a long time ago. Think of it as time travel to see what working on reasoning used to look like.

What reasoning is: the wheel that tilts into 3D (~00:34–00:36)

The framing at the time: picture a colorful wheel covering the space of all problems, where the angle is the subject (AI, math, physics, biology) and the radial distance is difficulty. As they pre-trained GPT-2, 3, and 4, the circle of solvable problems expanded outward from the middle, covering every subject a little and solving slightly harder problems as scale grew.

The nagging question was when the genuinely hard problems would fall. His example: Andrew Wiles thought about Fermat's Last Theorem for seven years — and crucially, he wasn't solving everything at once; seven years bought him one problem, and most hard problems have that shape. Something about "just pre-train models to be better at general knowledge" seemed unlikely to ever expand the circle to cover the hardest problems.

So: what do humans do? We think. We reason. Can we give models the same capability? The slide then does something fancy — tilt the wheel into 3D, push it to the side of the screen, and open up a third axis that didn't exist before: thinking time. The question becomes how to grow those cylinders outward given more time.

Two things about that era are worth remembering. First, "it might look a bit ridiculous today, but there was a time where you would give a model more time to think and it would give you the exact same answer" — the answer arrived after a few milliseconds and the model had no idea what to do with the extra time. Second, the practical ceiling: Wiles thought for seven years, and the models had a context of a few thousand tokens, maybe 8,000. What can you do with 8,000 tokens? At most a few seconds of reasoning. Even granting that they'd learn to reason, how was this ever supposed to happen?

The first evidence: what if we just ask them to think? (~00:37–00:40)

Another open question was what substrate reasoning would use. There are many ways to spend test-time compute — a recurrent network taking many internal steps, for instance. Text turned out to work really well.

He then showed his first project after joining OpenAI: a small exploration of what models could do when simply asked to think, despite never having been trained to. It was one of the pieces that gave the team confidence to scale things up.

Three examples, silly by today's standards but useful for measuring the distance traveled:

1. Potatoes. Ask GPT-3 about 29 bags of potatoes plus 17 more, and the model answers "46 kilos of potatoes" — plainly wrong. Add "let's think about it before getting the answer," and instead of an immediate answer you get some extra text first, and then, at least sometimes, the right answer — compared to never.
2. Instruments. "I have a clarinet, a piano, a dog. How many musical instruments do I have?" The model says one. Break it down step by step and the model reasons that a dog isn't an instrument, a clarinet is — but a piano isn't. Still one. Only after adding another instruction to break it down further does it get there.
3. Counting. A chair, an amp, an oven, and a long list of things: the model says 20, the answer is 14. Back then you could check the model's probability of outputting 14 — it was 2%, essentially never. Add "let's think about it, break it down step by step," and you get a long enumeration and a better answer.

Turning intuition into a curve (~00:39–00:40)

Two plots, the earliest quantitative evidence. The first: keep stacking prompt instructions — think step by step, break it down, "I really want you to do all the work and show all your work and make sure there are no mistakes" — and a model that could barely solve problems solves more and more.

The second (which he noted should have appeared alongside the first): the average length of generated solutions. Asked plainly to solve, the model produced almost nothing. As instructions accumulated — including the trick of forcing the output to start with "First," so the model wouldn't immediately blurt an answer but instead adopt the mindset that a long list was coming ("if there's a first, there must be a second and a third, so let's output a lot of text") — length grew. And as the amount of text the model output increased, its accuracy went up.

"Since then lots of other people worked on this, we did a big project, and then lots of other components we can't talk about yet. But — very humble beginnings, one of many."

Quotes

"There's this saying that there's nothing more stale than yesterday's newspaper. Well, these slides are five years old." (~00:33)

The opening line and the frame for the whole talk.

"There was a time where you would give a model more time to think and it would give you the exact same answer, because after a few milliseconds you would get the answer and it would not know what to do with the extra time." (~00:36)

One sentence that marks the dividing line — absurd today, and exactly where this started.

"Very humble beginnings — one of many. I thought it was interesting maybe for you guys to see this in the midst of these talks about the future." (~00:41)