Self Optimizing Agents

TL;DR

• The industry's framing has flipped: "token maxing is basically over — everybody's speaking about token efficiency." The real question is best real customer outcome per dollar invested.
• Customers all say the same thing: the agent's quality is fine, but running it on every PR, call, booking, or transaction would be prohibitively expensive.
• The configuration space has four layers, each capable of exposing a new Pareto frontier: model and retrieval strategy → scaling (vertical and horizontal) → model portfolios → execution strategy (chaining, escalation, early stopping).
• The portfolio layer is the counter-intuitive one: different models cover different regions of the space, so if you measure covariance and each model's contribution carefully, the sum of using all models beats using them separately — worth roughly 50% cost reduction at comparable quality in their experiments.
• Why it must be automated: the search space is too large to cover by hand, and it keeps moving — new models ship, prices change, and traffic and task distributions drift, so any hand-tuned configuration is already stale.

Key Points

The problem: economics becomes the binding constraint in production (~00:19–00:21)

Over the past few months he has watched AI systems move from experimentation into production and start deploying at scale, which introduces a new set of challenges — the most obvious being how to operate economically at the frontier.

What he hears from customers, repeatedly: we have this agent, we're satisfied with how it functions and its overall performance, but if we wanted to apply it to every PR, every call, every booking, or every transaction, it would be prohibitively expensive.

Hence the shift in focus. The question worth asking is how to get the best possible performance per token invested, or the best real customer outcome per dollar invested. His illustrating chart plots one company's token usage (Coinbase in the example) as a line against cost as bars broken down per model; the state enterprises want is for usage to keep climbing while the cost curve decouples from it.

The typical optimization flow: take an agent, try several configurations, experiment, discover the Pareto frontier of operating points, select an agent candidate, and evolve the system from there. Most teams have a target operating zone in cost and quality — getting there is easier said than done.

Four dimensions of the configuration space (~00:23–00:33)

He describes the space as a caricature of levers: the model itself and its weights, model selection in a world where models have very different capability and performance profiles, and then the harness — tool specifications, prompts, skills, code, and the surrounding scaffold. That's a lot of moving parts before you begin.

Dimension one: model × retrieval strategy. Using BrowseComp — a deep-research benchmark that is verifiable because the answer is contained in the given corpora — they mapped candidates across models and retrieval configurations (dense, sparse, and late-interaction retrieval). Empirically, the GPT-5 family with late-interaction retrieval and MiniMax formed the Pareto frontier; you'd choose between those two. He notes this is the basic step most teams already perform before going to production.

Dimension two: scaling and execution strategy. Vertically, you can vary thinking tokens, loop iterations, or fix-repair loops. Horizontally, you generate multiple candidates and select or merge — best-of-n rolls the same model out in parallel and picks the winner. On the same benchmark, MiniMax (the cheaper, lower-quality option) reaches the quality of the top single-shot candidate once horizontally scaled, and GPT-5 also improves on quality relative to cost. The point is that adding this dimension opens a whole extra layer of viable operating points.

Dimension three: model portfolios. Rather than generating candidates from one model, generate them from different models — potentially with adjusted prompts and tool definitions — and combine the outputs into the most optimal result. He frames this as the industry moving from token maxing to model maxing.

The underlying observation: different models cover different areas of the space, so being diligent about measuring covariance and each model's contribution buys a lot. "The sum of all using all models is greater than using them separately." Applying a portfolio strategy exposes a new Pareto frontier, and the mixing proportions can be learned per task rather than discovered manually. His headline number: about 50% less cost for roughly the same quality compared to the previous state-of-the-art setting. Swap cost for latency and the same principle holds.

Dimension four: execution strategy. How you chain the calls matters: escalation strategies, which model to prioritize, and stopping thresholds. Moving to a different benchmark for variety (heard as "three rebench" in the captions — see To Verify), he showed the impact on cost/latency trade-offs:

• Plain best-of-n: run all candidates, then pick the best.
• Cascading: start with the weakest model and work up, stopping once you're confident enough in the result — saves cost.
• Parallel with early stopping: doesn't save cost, but saves latency — optimizes for speed.

At the same quality level, you pick the execution strategy that matches whether you care more about speed or cost.

His closing example is a hierarchy analogy from legal work: many junior interns do the busy work, an associate synthesizes a thesis, and a partner makes the final call. The same shape applies to coding — have a weaker model generate many rollouts, a stronger model enrich those rollouts with more relevant information, and a very strong model (he said "a Fable level model") produce the final patch and decision. The result is better quality at roughly 3x cheaper than using a frontier model throughout.

Why this has to be automated (~00:33–00:35)

All of these strategies are learnable, but the search space is huge and manual tuning will miss many optimization opportunities. More importantly, the environment keeps moving: a new model arrives, pricing changes, traffic and task distribution shift over time. Manually searching this space isn't practical.

AI21's answer is an enterprise toolkit that takes an existing agent — whatever framework, runtime, or models it uses — and optimizes it in production, aligned to that customer's production traffic, so it can continuously evolve toward the best price-performance according to the customer's own preferences.

He closed with three properties automated optimization needs: efficient, observable (so people can see the trade-offs and select among them), and future-proof (easy to adjust when a new model lands or the distribution shifts). Their research on the topic is posted to the company blog every few weeks.

Quotes

"Token maxing is basically over now. Everybody's speaking about token efficiency." (~00:20:43)

The framing for the whole talk: efficiency, not scale, is now the frontier-deployment story.

"Moving from token maxing to model maxing." (~00:27:49)

Not squeezing a single model harder, but harvesting what several models are each good at.

"The sum of all using all models is greater than using them separately." (~00:28)

The justification for the portfolio layer — differentiated coverage is itself an exploitable resource.