Future of Agent Evaluation

The agenda lists no speakers for this workshop. The transcript has three: the opener who presented AgentBeats and Agents' Last Exam (introduced by the MC with a name the auto-captions rendered as "Eson", later referred to by another speaker as "Justin" — see To Verify); Jun, a UC Berkeley PhD student, on frontier cyber evaluation and SuperRed; and Joy, a UC Berkeley PhD student, on verifiable code generation. Name spellings are unconfirmed.

TL;DR

• The framing: AI progress has always been steered by evaluation — true in the ImageNet era, still true for LLMs. Benchmarks give researchers and system builders a clear target and let training teams compare approaches and see limitations.
• AgentBeats (the standard): most benchmarks are designed around the language model, not the agent. You can swap the model but not the full agent harness — prompts, tools, memory, workflow, control logic. N benchmarks × N agents means up to N² integrations. The fix is AAA (Agentified Agent Assessment): turn the benchmark into an agent too, have both sides integrate through existing protocols (A2A, MCP), and add an assessment control protocol for a unified evaluation workflow.
• Agents' Last Exam (capability): agent benchmarks cluster in programming, computing and math — about 7% of US employment. ALE anchors on the 2018 US SOC (867 detailed occupations) plus O*NET, consolidating into 55 fields; 300+ practitioners contributed; the launch snapshot has 760 workflows and ~1.5K task instances; only final deliverables are scored, GUI or CLI as you like. Highest observed full-pass rate on the hardest tier: 2.6%.
• Frontier cyber evaluation (risk): CyberGym (discovery + validation; 1,500 vulnerabilities across ~200 OSS projects; agents autonomously found 17 incomplete patches and 340 zero-days), ExploitGym (turning known vulnerabilities into working attacks; ~900 vulnerabilities across user space, V8, and the Linux kernel; frontier models now exploit 200+), and CyberGym E2E (discover → prove → patch). Plus SuperRed, a modular red-teaming framework.
• Verifiable code generation (trust): Verina (189 Lean tasks; a year ago o3 solved 4.9% of proof tasks, today's strongest agentic provers solve all of them) and a repository-level benchmark (43 repos, ~700 implementation obligations, 2.7K formal specifications) that remains frontier-resistant — agents close local proof obligations but struggle to reason formally about global invariants across a repository.

Discussion

Why standardization: benchmarks are built around models, not agents (~03:49–03:52)

Three structural problems. Standardization: agents don't share a common interface the way LLM APIs do — different interfaces, tools, environments and execution paths — so running an agent on a new benchmark means benchmark-specific engineering and a steep learning curve. Openness: access to private agents and models is limited by default. Reproducibility: the first two compound into evaluations that are hard to reproduce and hard to scale.

The deeper design flaw is that many benchmarks are built around the large language model rather than the agent. You may swap the model, but not the harness — prompts, tools, memory, workflow, and the control logic wrapping the model. Evaluating an agent with a different harness means editing benchmark code, which is both expensive and a source of mismatch between the tested system and the production one. His example was the OpenHands repository, which carries a dedicated folder of benchmark-specific adaptations. With n benchmarks and n agents, direct integration can require up to n×n separate efforts — not viable at scale.

AgentBeats and the AAA paradigm (~03:51–03:54)

The proposed paradigm is AAA — Agentified Agent Assessment: convert benchmarks into agents. Rather than wiring every benchmark to every agent, both sides integrate through standard protocols — the existing A2A and MCP — plus an assessment control protocol that defines a unified evaluation workflow for consistency and reproducibility.

The benefits: benchmarks decouple from the agents under evaluation, so any agent following the protocol can interact with a benchmark; benchmark-specific integration work drops sharply; and because agents already speak A2A, multi-agent evaluation becomes natural.

A conceptual framework isn't an implementation, though, and adoption is blocked by system-level issues: deployment, communication, access control, reproducibility, and result hacking. AgentBeats is the open platform built to address them — supporting benchmark development, agent evaluation, collaboration, and result tracking, with a leaderboard for analyzing runs and feeding insight back into agent design. It supports single-agent and several multi-agent settings, and multiple operational modes for different privacy and data constraints.

The AgentX–AgentBeats competition has now concluded — six months, thousands of agent submissions from independent developers, 70 evaluation tracks and 12 agent categories, with hundreds of benchmarks and agents integrated under the AAA paradigm.

Next up is AgentBeats v3: a lighter, simpler architecture, a protocol for automated benchmark identification, and a progressive compliance path so developers who only want to expose basic repository metadata can pick the level that fits.

Agents' Last Exam: can agents do economically valuable work? (~03:54–04:05)

The question: can today's agents do economically valuable work in the real world? ALE makes that testable at the level of professional workflows.

Why existing benchmarks fall short: a coverage chart showed agent benchmarks concentrated in programming, computing and mathematical domains — about 7% of US employment. Management, finance, law and engineering workflows remain largely untested. He also cited Anthropic's chief executive saying AI will surpass almost all humans at almost all jobs shortly after 2027; whether or not you buy the timeline, expectations have shifted, agents are increasingly discussed as assistants for professional work rather than question answering, and that raises the evaluation bar: a benchmark should test complete workflows and the quality of the resulting artifacts.

Method: ALE starts from an external map of work — the 2018 US Standard Occupational Classification, with 867 detailed occupations. It then uses O*NET to find the computer-centered workflows inside that map: AI screens roughly 10,000 O*NET entries for digital workflows, consolidates occupation variants into 100 SOC-based codes, and groups those into 55 fields, with experts adjudicating borderline cases.

Manufacturing served as the worked example: take 2D blueprints and convert them into 3D objects in something like SolidWorks (the slide showed an iPhone shell); simulate production, including the heat involved in melting the plastic; and machine a plastic cube down into the final shape.

What makes ALE different, in four points:

1. Authentic scope — 300+ practitioners contributed; the launch snapshot in the paper holds 760 workflows and ~1.5K task instances across 55 fields.
2. Interface-agnostic — it doesn't care whether you use CLI or GUI, only the final outcome. He claimed it as the first benchmark to accept both GUI and CLI execution, to approximate real human working environments.
3. Deliverable- and milestone-based scoring — outputs and intermediate states are checked against a hidden reference audit rubric, so reliance on LLM verifiers is light.
4. The unit is a professional deliverable — hours to weeks of expert work, not an isolated short patch.

Results: the hardest tier (the "ALE last exam tier") has a highest observed full-pass rate of 2.6%; GPT-5.6, then the leaderboard leader, reached about 5% there. The rate of change matters more than the level: three months earlier no model passed any task on the hardest tier, and on the latest version the best model reaches roughly 8%. On the easier general tier, taking the best agent per task yields about 60%, which he expects could reach 80% within a year.

No agent wins everywhere — Fable 5, for instance, is comparatively weak in life sciences and visual/media domains, partly because it declines tasks it judges too sensitive, which drags the score down. ALE has also been adopted by OpenAI's GPT-5.6 release as a headline benchmark for both capability and estimated cost.

Where tasks come from: the hard part wasn't collecting prompts, it was translating professional intent into a benchmark interface without losing authenticity or evaluatability. ALE's external submission pipeline has experts start from projects they've already completed rather than synthetic invented prompts, which preserves real inputs, tools, constraints and a known deliverable — then sensitive details are removed or replaced. Three filters follow: representativeness, complexity (days of expert work, not minutes), and verifiability. Each expert submission specifies five things: the task ask, input files, tools, expected deliverables, and evaluation criteria; engineers then stage the inputs, professional software and scoring logic into something repeatably executable and scorable.

Why "Last Exam": a dual meaning. Passing fully signals readiness — the agent can carry out sustained, economically valuable work in that profession. And it's genuinely hard: authentic long-horizon workflows put ALE at the boundary of what today's models can reliably accomplish. ALE is now in phase 2, aiming for a 10× expansion via an open submission platform, with accepted contributors added to the author list.

Frontier cyber evaluation: measuring across the vulnerability lifecycle (~04:05–04:18, presented by Jun)

Motivation: coding remains one of AI's most important applications, making programming accessible to a much broader population — but the same capabilities lower the barrier for attackers. Cybersecurity is now recognized as a core AI risk area, and the question is how capable today's agents are at real cybersecurity work.

Their evaluations span the vulnerability lifecycle: a developer unintentionally introduces a vulnerability; someone discovers it and validates that it's real and reachable; an attacker may turn it into an exploit while defenders generate and verify a patch.

CyberGym (discovery and validation): every task is based on a real vulnerability in a real open-source project. The agent gets the vulnerable code and a textual description of the bug, and must generate a test input that triggers the target vulnerability — a critical security task, since it confirms the bug, exposes the root cause, and supports severity estimates. It spans 1,500 vulnerabilities across ~200 large-scale, widely distributed open-source projects, with metrics grounded in dynamic execution rather than judgment. Beyond scores, it produced real security impact: agents autonomously found 17 incomplete patches and 340 zero-day vulnerabilities. Over the past year CyberGym has appeared in the system cards and technical reports of many frontier models — Anthropic, OpenAI, DeepSeek and others — becoming a standard measure of a new model's cyber capability; and the finding that frontier models can find zero-days at scale has since been extended by labs and the community to thousands of new vulnerabilities.

ExploitGym (exploitation): a harder, more direct question — can an AI agent turn a known vulnerability into a real attack? It holds about 900 real-world vulnerabilities across three critical parts of the software stack: user-space programs, the V8 JavaScript engine used by Chrome, and the Linux kernel. Each task provides source code, an input that triggers the bug, and a validated runtime environment for dynamic testing; targets may have standard defenses enabled such as address randomization, stack canaries, or a memory sandbox. The agent must build a working exploit and exfiltrate a secret flag via unauthorized code execution, which requires dynamically analyzing both the vulnerability and the target's mitigations. Early models solved a handful; frontier models now produce successful exploits for over 200. Autonomous exploitation is no longer hypothetical, and standard defenses help a lot but don't stop the agents — hence defense in depth and new defenses built for this trend.

Evaluation infrastructure is itself attack surface: he addressed the recent incidents reported by OpenAI, Hugging Face and Anthropic, where agents crossed expected evaluation boundaries and caused real-world security threats via complex attack chains. Two clarifications: the vulnerable third-party endpoint labeled "CyberGym" in the Hugging Face incident was not part of the original CyberGym harness, and ExploitGym was not involved in deploying or operating OpenAI's internal evaluation. Four lessons: evaluation infrastructure is part of the attack surface, so securing only the targets under test is insufficient; the risk exceeds evaluation integrity, since a failure can affect external systems with real consequences, not just inflate a score; you should adversarially test the entire setup before running powerful agents, with strong isolation and live monitoring throughout; and none of this is specific to cyber benchmarks — any long-running agent with capable tools can explore its environment in unexpected ways.

CyberGym E2E (the defensive side): can an agent discover a vulnerability, prove it exists, and then fix it correctly, end to end? The agent receives the vulnerable code plus the project's build and test scripts, and must find the vulnerability, produce a crashing input demonstrating it, and write a patch that removes the crash while the new codebase still passes comprehensive functionality tests. Frontier agents already handle the defensive steps end to end fairly well, but vulnerability discovery remains the bottleneck, and shallow, incomplete patches still show up — so how to run more comprehensive functionality and security testing is still open, and rigorous human review remains essential in daily engineering.

Where cyber evaluation goes next: broader coverage across programs and platforms including cloud systems and mobile; harder and more realistic settings such as binary-only targets where source is unavailable, and pentest tasks inside cyber ranges; long-horizon attacker–defender dynamics, where an attacker finds a vulnerability, a defender patches it, and another agent iteratively looks for a bypass or a new one; and study of the vulnerabilities AI agents find themselves — what types, how severe, and how cost-efficient compared to traditional security solutions.

SuperRed: AI systems are themselves becoming a new class of application — using tools, accessing sensitive data, maintaining memory, acting across systems — which raises the question of how secure the agents are. Today's AI red teaming is largely one-off projects, each with a different attacker, target, threat model and success metric, so evaluations are hard to reuse or compare. SuperRed separates an evaluation into three parts — the attacker, the system under test, and the benchmark that supplies tasks and metrics — as mixable, portable modules, with 35 ready-to-run modules already integrated behind a standard interface. Its other key element is a fine-grained threat model stating exactly what the attacker controls and observes, which model powers it, and how much budget it has — because an attack is only meaningful when the attacker's capabilities are clearly stated. It ships a reliable runtime for large evaluations with a live dashboard showing progress and attack success rate, and a complete custom evaluation takes one pipeline file and about ten lines of code. Their cybersecurity observatory tracks the latest agent cyber capabilities.

Verifiable code generation: can an agent prove its own code correct? (~04:18–04:25, presented by Joy)

Motivation: the productivity gain from vibe coding is real, but generated code frequently contains functional errors and security vulnerabilities. It may look right and pass every test while hiding subtle edge-case bugs, and no human can read every line.

The approach: beyond generating code, require the agent to prove the code satisfies a formal specification for any input, with the proof checked by a deterministic verifier such as Lean. Correctness is then proved rather than assumed.

Formal verification and LLMs have a strong synergy because each solves the other's bottleneck: formal verification gives LLMs the guarantee they're missing, and LLMs give formal verification the scale it never had. Verification traditionally demands deep expertise and enormous time; agents can automate those tasks and extend guarantees to ordinary software.

Two benchmarks:

• Verina measures the foundational skills — how well models generate the three basic components: code, specification, and proof. It has 189 standalone tasks sourced from competition-style coding questions at easy difficulty, and supports measuring task compositions, which makes it a flexible diagnostic. Released about a year ago, when the strongest model, OpenAI o3, solved only 4.9% of proof tasks; with the rapid development of agents and agentic provers plus model improvement, today's strongest provers solve all of them.
• A repository-level benchmark (name unclear in the captions — see To Verify) answers the natural follow-up: does success on isolated functions transfer to real software? It's a repository-level verifiable code generation benchmark in Lean, sourced from real-world repositories written in Python, Rust, Dafny and Verus. They manually created 43 repositories containing roughly 700 implementation obligations and 2.7K formal specifications, each repository passing a rigorous curation pipeline with all specifications manually checked. Two evaluation modes: proof-only, where the reference implementation is given and the agent must prove every specification in the repository; and code + proof, where it must also implement every API and prove its own implementation correct. A formal audit mechanism lets agents continuously improve the benchmark by submitting formal evidence. Scoring is deliberately at the level of a fully proved repository rather than individual specifications — otherwise agents can hand in degraded implementations or cherry-pick the easy specs and leave the hard ones. The strongest agent (captions render it as "a code plus GPT 5.5" — spelling unconfirmed) solved only 27 repositories during the evaluation, and on some repositories couldn't prove a single specification, making it a frontier-resistant benchmark.

What the traces show: current agents can close local proof obligations, but formally reasoning about global invariants across a whole repository is still hard. Repository-level verification requires building a coherent proof library with shared invariants, which is exactly where agents currently stall.

Three next steps: explore formalizing intent, so the formal specification actually reflects true human requirements; scale to harder software and properties, challenging agents to reason formally about security, concurrency, and temporal properties; and build stronger agentic provers that reason coherently at the whole-repository level.

Wrap-up (~04:25–04:26)

The summary: AgentBeats as an extensive effort to make agent evaluation open, standardized and reproducible, with the hope that every future benchmark benefits from that standardization; then three representative benchmarks — Agents' Last Exam on economically valuable real-world tasks, frontier cyber evaluation on agent capability across the vulnerability lifecycle, and verifiable code generation on whether agents can formally verify existing or self-generated code. Together they make the case that agent evaluation must span many capabilities, environments and real-world requirements. Standardized versions of these benchmarks will be released on AgentBeats v3 when the platform becomes available.

Quotes

"Instead of directly integrating every benchmark with every agent, both sides integrate through standard protocols." (~03:51:40)

The AAA paradigm in one sentence — and how N² becomes N.

"Evaluation infrastructure is itself part of the attack surface. We cannot just secure only the targets we want to test." (~04:11:35)

The first lesson from the Hugging Face and OpenAI incidents.

"Formal verification provides LLMs the guarantee they're missing, and LLMs give formal verification the scale it never had." (~04:19:50)

Why the two belong together.