Opportunities and Challenges for Long-Horizon Agents

TL;DR

• The talk starts from a very concrete observation: his own Codex sessions have a median of about 10 minutes and a mean of about 20. "Why is it not more?" is, he argues, one of the most worthwhile questions for AI researchers today.
• We are barely a year into the era of agents. In his view o3 was the first genuinely agentic model, released last April; models have improved vastly since, with far more still to come.
• RL solved the old "failure probability grows exponentially with trajectory length" problem. Scaling RL produced a recipe that lets models recover from their own mistakes and train on their own trajectories, so the more the model thinks, the better it performs — the test-time-compute and token-maxing era. Today's best frontier readouts land around 12 to 16 hours of continuous work.
• But throwing more money at it doesn't extend that indefinitely. Two hard barriers:
• The economics are quadratic. Sampling a length-n trajectory costs ~n (more with quadratic attention), while modern RL assigns a single reward to the whole trajectory with essentially no token-level credit assignment — so information per token is ~1/n and the learning signal decays quadratically in horizon. Linear would be fine; quadratic is brutal.
• Latency. Sampling a 12-hour trajectory takes 12 hours, so you get two gradient steps a day, ~14 a week, ~60 a month. That is not a fast training method, and nobody has time.
• Today's long-horizon behavior is mostly glued together by the harness (/goal, plan mode, sub-agents) — none of it native to the model, backpropagated through, or trained. "Whatever we can backpropagate through wins over what we cannot."
• Two ways out: value functions (if someone cracks them, long horizons are solved — but he's skeptical, since finding a good value function is often harder than solving the original problem) and continual learning (attacking from the other side: run the seven-year loop daily, but not with the same model each day).

Key Points

Starting from "my Codex sessions are only 10 minutes" (~01:10–01:12)

Long-horizon agents are clearly one of the frontiers of AI research right now, and his way in is unusually concrete: the median of his Codex sessions is around 10 minutes, the mean around 20. Why is it not more? Why aren't agents working for him for longer? He thinks that is a very worthwhile question for AI researchers to be asking today.

Start with the obvious: we already live in the era of agents, but we are very early in it. In his mind o3 was the first really agentic model, and it was released only last April — so we are just barely more than a year in. The models are early; they have improved vastly since o3, and there are much larger improvements still to be had. Products are improving, adoption is coming, and a lot is not figured out — which is the opportunity these times hand us, for the decades of agent proliferation ahead.

What makes an agent an agent? A fuzzy term, but his pieces: agents use tools, because an agent should be able to act on your behalf; agents are most likely async — you tell it something and it goes off and does the work; agents are usually goal-oriented, where you specify something high-level and the agent fills in the blanks; and agents work over long horizons of time, not back-and-forth every five minutes.

How RL dissolved the exponential-failure problem (~01:12–01:15)

Extend the time an agent works and you hit an obvious observation: the neural networks we train are probabilistic at every step and they make failures, so the chance that something goes wrong grows exponentially with trajectory length. That was a very big reason we couldn't have agents for many years — the networks got off the rails. It's the classical assumption from the RL literature, and the reason we need reinforcement learning at all.

The moment came when he and his team scaled RL to larger and larger amounts of compute and finally found a recipe that trained models to recover from their own mistakes and train on their own trajectories — and suddenly the models started succeeding at harder and harder tasks. Two things followed:

1. The obvious one: spend more compute, and capability improves proportionally on the algorithmic plot.
2. The important one, and one of the biggest shifts in machine learning in recent years: the more the model thinks — the more tokens it spends on a task — the better performance gets. RL-trained models not only avoided the exponentially-increasing failure trap and learned to recover from it, they could keep thinking productively for longer and longer. Essentially: the longer your trajectory, the better your results. We entered the era of test-time compute and token maxing.

Then comes the plot everyone thinking about agents knows: how long agents can work for us. Since RL scaling began it keeps reaching new heights, with the latest readouts around 12 hours, 16 hours for some of the best frontier models today.

Why "just keep spending" doesn't carry you (~01:15–01:18)

So maybe this is it — we have RL, it works, throw a lot of money at it and get agents that work for whatever length of time we want? Unfortunately there are problems. We are not done.

Barrier one: the economics are quadratic. - Sampling a trajectory of length n costs roughly proportional to n (slightly more with quadratic attention, but treat per-token cost as fixed for a second). - But how do today's RL algorithms work? They assign a single reward for that trajectory. There aren't really any credit-assignment methods in modern RL that attribute that reward to specific tokens. - So one reward covers a trajectory of n tokens, meaning information per token is roughly 1/n while the trajectory costs n — your learning signal decays quadratically in the length of your horizon. - If it were linear, that would be pretty okay: to train agents to work twice as long, spend twice as much money. That's a reasonable trade. Quadratic is brutal. The big labs today are very good at spending a lot of money training models, but the quadratic cost makes scaling considerably harder than it looks at first sight.

Barrier two: latency, which is even harder. For all the investment flowing into AI, no one has time. Imagine sampling a 12-hour agent trajectory: it takes 12 hours, then you take a gradient step to reinforce the successful trajectories, then you sample another 12-hour trajectory. Two full gradient steps in a day. Roughly 14 a week. Roughly 60 a month. That is not a very fast training method. Training for very long trajectories and long-horizon thinking is meaningfully harder than many realize.

Today's long horizons are harness-glued (~01:18–01:19)

Most of the progress in the field — and the reason we have agents that can think for a long time at all — usually comes from putting some kind of harness on top of the model. Agents are very good at short-horizon tasks, and the truth is that a lot of long-horizon work can be glued together very successfully from smaller things.

• /goal in Codex and in Claude Code is very powerful: fix a goal, the model runs for however long it can, revisits itself, and continues.
• There's also plan mode, sub-agents, and various other ways.

The catch: none of this is native to the model. It is not backpropagated through. It is not trained. These mechanisms merely exploit the model's ability to work on short-term goals and stack them into longer spans. You can be very successful this way — but the history of machine learning has always been that whatever we can backpropagate through wins over the things we cannot.

Auto-research is the ready-made counterexample. The story of almost every auto-research rollout is that you get a lot of progress pretty early, and then further along it stalls and stops. He hasn't yet seen any particularly successful auto-research rollout with today's models. Research is inherently a hard-serial, long-horizon task, and today's models can't easily glue short-horizon thinking into larger-scale research projects — auto-research is exactly where you see it.

How did humans master long horizons? (~01:19–01:21)

He inverts the question. Famously, Andrew Wiles spent seven years working on a single goal — proving Fermat's Last Theorem — and succeeded. How do we train models to do seven-year projects?

He asked ChatGPT when humans started developing long-term planning, and got: roughly 10,000 generations of humans since we could do any significant long-term planning, and roughly 120 billion people in that time. So maybe that's it — 10,000 gradient steps and 120 billion trajectories is what it takes to optimize long-horizon planning as well as humans do. And gradient descent is more sample-efficient than evolution, so perhaps we have even more room. Something to think about and revisit.

Two research directions (~01:21–01:23)

(1) Value functions. Definitely a direction that can help: if we got good value functions for AI research, it would be extremely helpful. But he is honest about the state of things: when doing RL on large language models we basically threw out value functions, and everyone saw it works just as well. Value functions are a very, very hard problem, and he holds a fair amount of skepticism about how much progress can be made — very often, finding a good value function is much harder than solving the problem you were trying to solve. But if someone figures out value functions, we're good: we can solve long-horizon problems as long as we'd like.

(2) Continual learning. This attacks the same problem from the other side: update the model through the loop. If you have a long, seven-year loop — "let's prove this theorem" — you repeat the same thing every day, but not with the same model: the model updates on new data, learns something each day, and improves. He thinks continual learning could get us through this very well. The question is how — it is an unsolved problem, and one worth solving.

Closing: worth working on (~01:22–01:23)

If you're looking for what to work on right here and now in this age of machine learning, long-horizon agents is absolutely one of the valid choices. When we get that technology we'll be able to put agents to work on hard research problems, and have them run companies — very long-horizon and very economically valuable tasks. And from the user's side: if I want to delegate to an agent, I don't want to be telling it what to do every 30 minutes; I want it acting on my behalf for long periods of time.

The research isn't done. There's much work to do, and the objective is agents that can keep going for us and keep getting better over time.

Quotes

"If I look at my Codex session, my median is around 10 minutes. If I look at the mean, it's around 20. Why is it not more?" (~01:10)

The whole talk grows out of one personal statistic.

"o3 was the first really agentic model, and that model was released in April last year. So we are just barely more than a year in the era of agents." (~01:11)

A dated starting line for the era.

"You get one reward for a trajectory of n tokens … which means your learning signal decays quadratically in the length of your trajectory." (~01:16)

The core technical barrier for long-horizon agents, in one sentence.

"You were able to do full two gradient steps in a day. … That means like 60 gradient steps a month. It is not a very fast training method." (~01:17)

Latency is harder to engineer around than cost.

"The history of machine learning has always been that whatever we can back propagate through wins with things that we cannot." (~01:18)

His long-run verdict on harness-glued long-horizon behavior.

"Very often finding a good value function is much harder than solving the problem we are trying to solve." (~01:21)

Why he stays skeptical of the value-function route.