Continuous Model Improvement

TL;DR

• Compute has three dimensions: pre-training (bigger models, more data, and now more exploration), post-training (the easy problems are solved, so compute goes to harder problems, long-horizon problems, and multi-agent handled in training rather than only at inference), and test-time compute for product serving, where a family of models has to interact.
• "Data is the king, everything else is the queen." What matters about synthetic data isn't quality but coverage — creating complementary data that didn't exist before; and the human data we generate daily becomes the seed for it.
• Architecture's second dimension is the one people skip: inference cost. How popular a model becomes is ultimately decided by how fast and how cheap it runs, so inference has to be defined at the very beginning. He'd bet on data over architecture for quality, and have architecture focus on cost and efficiency.
• Distillation is the general pattern of machine learning: learning from web data distills from humans, so does learning from product telemetry, so does human labeling. The next better model will be built from existing models, not from scratch. And counter-intuitively — a more powerful teacher is not necessarily a better teacher, while a weak model can sometimes teach well.
• Two things about the harness: first, optimize everything around the model before you try to optimize the model; second, much of what you build into the harness today will be absorbed into the model, so harnesses simplify — even as we keep defining more complicated scenarios.
• Continuous improvement means continuous deployment at scale. Don't optimize only for benchmarks; production usability and real interaction data are what actually make the next model better.
• The point he deliberately saved for last: human involvement is still essential — above all to ensure self-improvement aligns with human values rather than with what the model wants to do.

Key Points

Engine one: three dimensions of compute (~01:37–01:39)

The first driver is obviously compute — the first scaling engine; without it you can't do anything. Most improvement is essentially compute-driven: more compute, train longer, the model gets better automatically. But compute has a second-order effect: it unblocks a lot of experiments, and much of the innovation actually comes out of experimentation and fast iteration.

Three dimensions today:

1. Pre-training — large models, more data, and today also more exploration, whether coming from the model or from humans. The more you explore, the better the model. This is roughly what people call the pre-training scaling law.
2. Post-training — a lot of the simple problems are already solved, so compute now focuses on how to solve harder problems, how to solve longer (long-horizon) problems, and how to handle multi-agent problems in training, not just at inference. Call it the post-training scaling law.
3. Test-time compute — for product serving, centered on making a whole family of models interact together.

Engine two: data, and why coverage beats quality (~01:39–01:41)

"We always say data is the king, everything else is the queen. And data is oxygen." Data is the second engine that makes everything happen. Several parts:

• Pre-training data — most pre-training progress comes from data. Every time you clean the data better, or refresh it (web data gets better and broader in coverage every day), the model improves.
• Synthetic data is about coverage — "synthetic data is not just about the quality; the most important thing is the coverage," creating far more complementary data than before.
• Human data is the seed — he thinks data is the most important thing going forward, and all the data we create every day will be the seed for synthetic data. Synthetic data becomes the mainstream; the value of the data we already have lies in how it produces better synthetic data.
• Post-training data means RL environments — the key is making the training-time RL environment run exactly as the user actually uses the product, so the model becomes useful in real product scenarios. RL needs trial and error, so whatever appears at inference must be present in training for the model to learn it. Very similar to building a simulation in training so you can generate data at volume.

Engine three: architecture — don't forget inference cost (~01:41–01:42)

Model architecture is complicated and the literature is vast, but he summarizes it into three optimization dimensions:

1. Higher model quality;
2. Lower inference cost — "a lot of people just ignore the second one";
3. Supporting much longer context, more efficiently.

The goal is simple: how do you make it better and cheapest?

His position is explicit: data matters more for model quality than architecture does; architecture should pay more attention to inference cost and inference efficiency. Because how popular a model becomes is finally decided by how fast and how cheap you can run it — which he considers more important than anything else. So inference must be defined at the beginning. And tracking GPU advancement, designs should move toward more compute and less I/O.

Engine four: distillation is what machine learning has always been (~01:42–01:44)

He argues much of machine learning is just distillation, and the only question is whether you distill from humans or from a model:

• Learning from existing web data → distilling from humans;
• Learning from product telemetry and interaction data → distilling from humans;
• Asking humans to label data → distilling their knowledge.

So distillation was always ubiquitous. In industry it's far more efficient than training from scratch — you don't need to build anything from zero. He believes the next better model will be built from existing models rather than from scratch.

Distillation is also a very general pattern: build multiple teachers — it's easier for a student to learn from one specific teacher, and teachers can run in parallel — then distill into a single model.

But he stresses that distillation brings a lot of new research problems, not simple ones. The type space alone includes strong → strong, strong → weak, and even weak → strong distillation. And the most counter-intuitive finding: a more powerful teacher model doesn't mean it's a better teacher — very much like humans — and sometimes a weak model can be a very good teacher. With distillation, everything is possible.

Engine five: what RL is good at, and three pain points (~01:44–01:45)

RL works well today: it is very effective — he corrects himself mid-sentence, effective rather than efficient — at solving problems, verifiable or non-verifiable alike. It's particularly good at fixing product behavior: define a grader, add some data, and it's far more efficient than the alternatives. And it works directly against the real environment. He goes as far as saying RL plus distillation plus synthetic data may be enough to get us to self-improvement.

The challenges:

• It's very slow — he thinks real breakthroughs are needed to make it much faster;
• Updates are very incremental — run an RL chain and you'll see each update is tiny; sometimes a LoRA-scale update is already good enough;
• And on the other side it's data again — what drives RL is data, and data only becomes more important for RL training. He believes maybe 99.9% of the data will come from synthetic data in the next step.

The harness: optimize around the model first (~01:45–01:47)

Since the day had already covered harnesses at length he moves fast, but two claims stand out.

First: optimize the entire system. The harness is a very big part of that system — don't start by optimizing the model. Speaking directly to the builders in the room: before you think about optimizing the model, try to optimize everything around the model.

Second: a lot of what you build into the harness today will at some point be absorbed into the model, so harnesses get simpler and simpler — while at the same time we keep defining more complicated scenarios and building more complicated harnesses. That absorption keeps going, and there's nothing we can change about it.

One more point he flags as very important: don't care only about benchmarks; care about production usability. Real interaction data is extremely valuable for building a better model in ways benchmarks are not. Hence: continuous model improvement really requires continuous deployment at scale. When a lot of people use your model, you can understand what's going wrong, extract insight from interaction data, and see what other models do in the same scenarios and learn from them.

Closing: four calls for the next round (~01:46–01:47)

1. Scaling keeps working, and we don't know where the ceiling is.
2. Models can already optimize the harness, and will also be able to optimize and change another model — because models are better than humans at this. He thinks it's going to happen.
3. Optimize the whole system; synthetic data becomes the mainstream, especially for the next round of model training.
4. And equally important: human involvement is still essential. We still need humans to guide the model — above all to make sure this self-improvement aligns with human values, instead of doing something the model wants to do.

Quotes

"We always say data is the king, everything else is the queen. And also data is oxygen." (~01:39)

"Synthetic data is not just about the quality. The most important is the coverage." (~01:39)

The value of synthetic data is filling distribution gaps real data never covered, not just raising quality.

"How popular a model becomes is finally going to be decided by how fast you're able to run it and how cheap you're able to run it. So we need to define the inference at the beginning." (~01:41)

"A more powerful teacher model doesn't mean it's a better teacher. That's very similar as humans." (~01:43)

"Before we think about optimizing the model, try to optimize everything around the model." (~01:45)

The one line for builders in the room.

"Continuous model improvement really requires continuous deployment at scale." (~01:46)

"Human involvement is still very essential … especially to make sure this self-improvement aligns with human values, instead of doing something the model wants to do." (~01:47)

Deliberately saved for the final slide.