Ready for General Agents? Let's Test It.

TL;DR

• The thesis: extend Sutton's bitter lesson from models to agents. NLP moved from summarization/extraction models to general LLMs; agents will move from today's domain-specific agents (finance, customer care) to general agents, where domain knowledge lives in the environment rather than inside the agent.
• The real bottleneck is standardization, not metrics: every benchmark expects a different agent, every agent expects different information from the environment, and every researcher writes bespoke glue for each agent × benchmark pair. That does not scale.
• The fix is a Unified Protocol mediation layer: agents keep their own protocol, benchmarks keep theirs, and the framework translates between them using three shared primitives — task, context, actions. Neither side is modified. The framework is Exgentic; it produces the Open (General) Agent Leaderboard.
• Headline findings: the model dominates quality more than the harness; unmodified general agents are competitive with the best domain-specific agent on each task; agents with near-identical scores can differ enormously in cost.
• Two warnings about open-weight models: they trail closed models on average and collapse on tasks they likely weren't trained for (e.g. AppWorld), and — more consequentially — they are highly harness-sensitive: Kimi's best harness beats its worst by 18%. "Open weights are cheap" also doesn't survive the cost-quality plot.

Key Points

From the bitter lesson to general agents (~00:13–00:16)

The talk draws on three papers (ICLR, ICML, and hopefully NeurIPS). Starting from Richard Sutton's bitter lesson — generality ultimately beats domain-specific engineering — she traces the same arc through language models: everyone began with task-specific models for summarization and extraction, and everything converged on big general models.

Agents today sit at the domain-specific stage. Her definitions matter for the rest of the talk: in a domain agent, the domain knowledge is encoded inside the agent, and only once it's encoded can the agent work with the environment. In a general agent, no domain-specific knowledge lives in the agent; it all sits on the environment side, and what you evaluate is the agent's ability to explore and adapt to new tasks. The payoff is an agent that works across environments without per-environment re-engineering — so evaluation shifts from performance on a single task to performance across many environments and tasks.

The upside: one general agent covers many use cases instead of a hundred domain-specific ones; improvements to memory or search are centralized and benefit every use case; you can still customize, but from a better and more robust starting point.

She also presents the opposing view fairly, pointing at the ICML position paper for both sides in detail: domain-specific agents give more control, more efficiency, and more predictability; more autonomy for a general agent introduces unforeseen risk; and some argue that it's the model that should be general, not the agent.

The bottleneck: three unstandardized interfaces (~00:17–00:20)

Claiming an agent is general requires showing it works across environments and tasks — which requires evaluation, and that turns out to be far from trivial. The obstacles are standardization at three interfaces: the agent interface (each benchmark expects the agent to behave differently), the environment interface (each agent expects different information), and the researcher interface (everyone implements their own code per agent–benchmark combination).

She notes the contrast with the preceding LLMArena talk: here the benchmarks are open-sourced by students and community efforts. Surveying existing frameworks, not one supports multi-protocol benchmarks, multi-protocol agents, agent plug-and-play, and generality across genuinely different tasks all at once.

Unified Protocol and Exgentic (~00:20–00:21)

Exgentic introduces a mediation layer they call the Unified Protocol. The agent keeps the protocol it ships with; the benchmark keeps the protocol it was developed with; the framework transforms between them. Nothing in the benchmark changes, so benchmarks behave as intended, and nothing in the agent is touched. The translation rests on three primitives shared across benchmarks and agents: task, context, and actions.

This makes a full Cartesian product possible — any agent harness × any model × any benchmark — published as the Open (General) Agent Leaderboard: agent column, model column, per-task columns, average success, plus cost, with a Pareto frontier so you can pick the configuration that matches your accuracy/price tolerance.

What the leaderboard showed (~00:21–00:26)

1. General agents adapt with no modification: on average, most of them handle every task type thrown at them.
2. The model matters most: quality is driven mainly by the model, though the agent harness has a visible effect.
3. General vs. domain-specific: taking the top domain-specific agent per task from that task's leaderboard and comparing against the top general agent, the general agents are quite competitive — with no re-engineering at all.
4. Similar scores, very different behavior: some agents fail fast and cheap, others fail after burning a lot of money. Both are wrong answers, but the cost profile matters.
5. Open-weight models are not generally reliable: behind closed models on average (Kimi and DeepSeek in the plot), and on tasks they likely weren't trained for they sink outright — her example was AppWorld, where they underperformed averaged across all five agent harnesses.
6. Open-weight models are far more harness-sensitive: Kimi with its best harness beats Kimi with its worst harness by 18%. The practical implication is that swapping harnesses is relatively safe for closed models but requires re-evaluation for open-weight ones. The reverse case also appeared: OpenAI's agent harness lifted results with Claude and Gemini but crashed to zero with open-weight models — you can't pair any model with any harness.
7. Open-weight models are not necessarily cheap: responding to a claim from another session that day, she pointed at the cost-quality plot — they are competitive, not cheap.

Traces and what's next (~00:26–00:27)

Every Exgentic run collects traces in OpenTelemetry format. They're open source on Hugging Face — more than 10K full traces available for study.

Next: Exgentic V2, a somewhat different approach informed by V1, built on Kubernetes and Docker primitives. Alongside it, they're working on evaluating agentic inference platforms and on AI-native system evaluation.

Quotes

"Generality ultimately wins over domain-specific approaches." (~00:13)

The bitter lesson, lifted from the model layer to the agent layer — the premise the whole talk rests on.

"We are not changing anything in the benchmark, so they work as they intended. And similarly, we are not touching the agents." (~00:20)

The design philosophy of the Unified Protocol: standardize by translating in the middle, not by making everyone adopt one interface.

"You cannot just change the harness and assume that you will get the same quality for the open weights model." (~00:25)

The 18% best-vs-worst harness gap is a warning for anyone planning to swap harnesses to cut costs.