Continual Learning for Agents

TL;DR

• If you can't touch the weights, touch the harness. Continual learning is usually associated with model training, but most companies run closed-weights models. Replit's answer is to put their hands directly on the harness and the ecosystem around the agent.
• Evals are good but narrow; production traces are the gold mine. Two pillars: evaluation, plus continuous A/B testing and real-time trace analysis. The long-tail events in production are the golden ones — they show where users push the product's boundaries and where things you assumed worked don't.
• The pipeline: cluster → frontier models analyze anomalous traces → auto-generate PRs → humans adjudicate. In principle this was buildable long ago; in practice frontier models only got good enough at analyzing traces in roughly the last six months.

Key Points

The premise: you can't reach the weights, but you can reach the harness (~02:58–02:59)

Continual learning has been a holy grail for the research field for a long time: we don't want static AI systems, we want systems that learn from usage. But in practice a lot of companies use closed-weights models — so with no access to weights, how do you make your AI system actually evolve?

His answer: there's another route, and it's exactly what Replit has been doing for many months — putting your hands directly on the harness and the ecosystem around the agent you're running.

He gave evals a fair hearing first: as an industry we've relied on evaluation for a long time, and benchmarks are amazing — he genuinely appreciates the people who work on them. The output is crystal clear: run the eval, get a number, decide whether your harness change was progress or a regression. But evals are by definition narrow, capturing only a subset of the capabilities your agent is actually exposed to. A lot of signal goes missing.

Meanwhile, once you hit product-market fit, the sheer volume of usage your platform receives is itself a gold mine of data. That gold mine is usually associated immediately with model training, but far more can be done with it: analyzing traces teaches you what works, what doesn't, why users are annoyed by your agent, and much else.

Hence two pillars: one still leaning heavily on evaluation, the other continuously running A/B tests and analyzing traces in real time to learn from the data what's actually working and what to fix now.

Why the long tail is the point (~02:59–03:00)

His emphasis on continual learning comes down to an order-of-magnitude gap in signal: the more traffic you receive, the larger the gap between it and a fixed-size eval.

• They recently launched their own benchmark — an end-to-end vibe-coding evaluation on the order of a few tens of applications. It's fixed in size, so they already know exactly how their agent behaves on it.
• What the production system experiences daily is a stream of long-tail events they can't predict. "Those long-tail events are actually golden" — they tell you how users are pushing the boundaries of your product, and they're usually the behaviors that break what you intended to work.

A/B tests are not a silver bullet (~03:00–03:01)

He was blunt: anyone who's run A/B tests knows they often look like "no clear result." You manipulate the harness one way and some metrics improve while others drop.

• Maybe you're optimizing for cost and agent capability degrades.
• Maybe you're optimizing for speed and user sentiment shifts.

You basically never get an answer clean enough to ship on the spot.

The actual pipeline: cluster, analyze, auto-generate PRs (~03:01–03:02)

So the right approach is to take the whole production workload — but not by analyzing every trace individually. With millions per day, that would be prohibitively expensive and too slow.

1. Cluster first. He stressed this is very basic machine learning: find semantic relevance across traces. The vast majority get discarded as intended behavior, but every day a few clusters pop up highlighting tail behaviors they've never seen from the agent before.
2. Anomalous traces go into analysis. After clustering, the system runs each anomalous trace through their analytics stack (which of course includes frontier LLMs), works out what went wrong, and immediately generates a PR.
3. Humans adjudicate. He wanted to leave a picture of the world where the AI engineer's job is still extremely relevant — he doesn't think this gets automated in the next few months at least. Once you have that stream of PRs and apply them, you run A/B tests, and some come back inconclusive. At that point, as the person leading an AI team, your job is deciding which changes go to production, which wait, and which get dropped entirely. - In other words: the vast majority of harness changes are generated by the AI system, but the human decision layer remains.

An important timing note: in principle this pipeline was buildable long ago, but he only started talking about it a few months ago because frontier models became extremely good at analyzing traces "only in the last six months or so." His analogy: the revolution software engineers are experiencing, where much of the code is written by agents, is happening on the production-workload side too.

The worked example: VMs booting slower than the harness (~03:04–03:05)

• Replit spawns hundreds of thousands of virtual machines for users every single day, completely transparently.
• The long-tail bug: sometimes the VM took longer to fully boot than the agent harness took to be ready to go.
• Agents are very eager to debug problems, so on those occasions the agent started spinning its wheels trying to figure out why it couldn't execute code or run certain tools.
• Because agents are fundamentally nondeterministic, no two traces showed the same errors — each agent picked a different debugging strategy — but all of them had in common that the VM wasn't booting fast enough.
• The key point: they would never have spotted this by reading logs manually, and it would never have shown up on their Datadog dashboard because it was a long-tail error. After clustering, they saw it was happening often enough, and the system immediately generated a PR and fixed it on the spot.

Closing (~03:05)

"Stop thinking about evaluation as just the last check before shipping. It's not a boolean flag that tells you whether to ship your new PR. Think of it as an engine that helps you ship a better agent every single day."

Quotes

"When you don't have access to weights, how do you make your AI system actually evolve?" (~02:58)

The framing question: what continual learning looks like in a closed-weights world.

"Evals are by definition very narrow. They capture only a subset of capabilities." (~02:59)

Benchmarks are useful, but the signal is orders of magnitude below production.

"Those long-tail events are actually golden." (~03:00)

They simultaneously mark where users push the boundary and where you break.

"You never get a crystal clear answer that allows you to immediately ship that change in product." (~03:01)

What A/B tests actually look like: cost, speed, capability, and sentiment pulling against each other.

"We would have never spotted this just by analyzing the logs manually. It would have never shown in our Datadog dashboard, because it was a long-tail error." (~03:05)

The blind spot in conventional observability that trace clustering is meant to cover.

"Stop thinking about evaluation as just the last check before shipping … think of it as an engine that helps you ship a better agent every single day." (~03:05)

The talk's closing line.