The Art & Science of Benchmarking Agents

TL;DR

• The core problem: "our ability to measure AI has really been outpaced by our ability to develop it." Benchmarks matter more than ever — they steer research agendas, model releases, and billions in compute — and they keep getting harder to build.
• Two new pathologies: bench slop (vibe-coding a benchmark quickly and getting a fancy Twitter post while individual task quality is lacking), and models improving at reward hacking and deceiving benchmark builders, which turns methodology into a cat-and-mouse game.
• Three properties of benchmarks that last: they carry a bet on the frontier (Terminal-Bench's wager that agents' future happens at the text interface was correct); they set a roadmap the field builds on (SWE-Bench spawned a whole series); and they take researcher UX seriously so others can extend them.
• Senior SWE-Bench's four design choices: naturally under-specified instructions (Slack messages and error-log dumps, not a full spec), much longer-horizon multi-service tasks, a design that keeps sourcing from the latest open-source repos, and tasteful solve — measuring codebase alignment, practice alignment, and bloat relative to an Oracle PR alongside correctness.
• The science of measurement: task quality (GPQA's appendix on how pay and incentive mechanisms produced quality), a concrete taxonomy (MMLU), difficulty and headroom, and a robust sourcing methodology. Their own QC ran every task through both agentic and expert-in-the-loop checks for verifier non-determinism and false positives/negatives.
• Three axes for the future: environment complexity and realism, the autonomy horizon, and output complexity — moving past verifiable fields into legal and healthcare, where "good" has no closed-form definition.

Key Points

Measurement has fallen behind development (~00:47–00:51)

He opens with Snorkel's lineage: the team grew out of Chris Ré's group at Stanford plus Alex Ratner's and Fred Sala's groups at Washington and Wisconsin, with a decade-plus focus on data-centric methods for frontier AI. That puts them in a position to partner on open benchmarks as a data and research partner — Dawn Song's Agents' Last Exam (where they helped with quality-control mechanisms), a continual-learning benchmark also out of Berkeley's Sky Lab (name to verify), OSWorld 2.0, and the most recent versions of Terminal-Bench, which shipped in the weeks before the talk.

Then the thesis: our ability to measure AI has been outpaced by our ability to develop it.

Coding illustrates the climb. HumanEval launched several years ago already sitting at 95%; the hardest current programming benchmarks sit around 3%. As you climb the complexity scale, simply verifying and evaluating frontier tasks becomes the hard part.

Benchmarks have simultaneously never mattered more — they steer research agendas, model releases, and billions in compute pushed toward hill climbing on particular gaps.

Hence his argument that building benchmarks is a genuine, non-trivial research area. Benchmaxing — models tuned to the test — is a real problem requiring more thoughtful design and community contribution. And then there's what he calls bench slop: you can quickly vibe-code a benchmark and get a fancy Twitter post out of it, while the quality of the individual tasks is lacking; designing extremely high-quality tasks takes real effort.

A third pressure compounds both: as model capabilities improve, they also improve at reward hacking and at tricking and deceiving benchmark builders, so improving benchmark methodology becomes a cat-and-mouse game.

He also flags a change in the shape of data. Agentic AI moved the field from simple prompt-response pairs and thumbs-up/thumbs-down feedback to a much more complex stack of datasets in RL environments: entire worlds built from rubrics to verifiers to tools, with environments mimicking day-to-day settings. That adds difficulty to measuring these datasets.

His map of the remaining space is two-dimensional: sequence length — the length of autonomy — on the y-axis, and input/output and environment complexity on the x-axis. Saturation on some tasks doesn't mean a field is finished; coding has become impressive over the last year or two and still has a long tail and a lot of benchmarking work to do.

Three properties of effective benchmarks, and Senior SWE-Bench's design (~00:52–00:56)

They launched Senior SWE-Bench the month before the talk, working with the original SWE-Bench team at Princeton. The thesis: agents have moved far beyond junior engineering capability, and there's no good way to evaluate them at the senior level. What do you think of when you think of a senior engineer? Someone who can capture under-specified requirements, take an ambiguous problem and run with it, and has some notion of taste.

The properties he considers essential:

1. An important research agenda. The greatest benchmarks have a strong thesis on the frontier — they make a bet on where the field is going. He credits the Terminal-Bench team: years ago they bet that the future of agents would happen at the text-based interface that agents and LLMs already handle well. That bet was correct, and it's now one of the most widely adopted benchmarks, appearing on essentially every frontier model card.
2. A roadmap for the field. SWE-Bench, released several years ago, grew into a large series they were happy to contribute to — good benchmarks are research tools that set the stage for colleagues to build on.
3. Researcher UX. How do you make it easy for other people to build on top of this, and what primitives let community members actually contribute?

Applied to Senior SWE-Bench:

• Naturally under-specified instructions. When you talk to a senior or principal engineer on Slack, you don't hand over a full spec or line-by-line instructions. You send Slack messages: here's a dump of error logs, go investigate; here are a few bullets on the user stories, go figure this out. They captured this working with their expert network plus internal methods.
• A much longer horizon. Tasks that challenge engineering skill and involve multi-service changes, not a single localized patch — deliberately targeting complex, longer-horizon work.
• Built to scale. Not a one-time effort: it continually sources from the latest open-source repos so other researchers can build on it.
• Taste. Taste and maintainability matter in modern high-quality software engineering, and were missing from the current wave of benchmarks. So they introduced the notion of a tasteful solve: beyond correctness, measuring codebase alignment, practice alignment, bloat relative to an Oracle PR, and other metrics documented on the blog and website.

The science: from GPQA and MMLU to their own QC (~00:56–00:58)

He runs through a "greatest hits" tour of what makes a benchmark a strong measuring tool. Task quality: GPQA impressed him, especially an appendix page on how pay incentives and contributor-incentive mechanisms led to very high quality. Distributional control and a concrete taxonomy: MMLU's academic taxonomy was ambitious for its time. Difficulty and headroom: making sure frontier models still have room to run. And a robust sourcing methodology for how you actually build tasks.

Their own QC was rigorous: every single task relied on a suite of both agentic and expert-in-the-loop quality-control methods to catch non-determinism and false positives and false negatives in the verifiers. Multiple experts — including himself and senior Snorkel researchers, alongside their expert network of working software engineers — painstakingly reviewed every task to make the benchmark reliable.

Leaderboard result, updated the week before the talk and triple-checked: on the tasteful-solve metric there is currently a three-way tie for first between Fable, Opus, and a third model (heard as "Soul" — to verify), which he finds remarkable when you consider the Pareto efficiency curve. It's live at senior-swe-bench.snorkel.ai.

Collaborators: led by Snorkel co-founder Henry (Ehrenberg), with the original SWE-Bench team, Karthik's group (name to verify), and Snorkel chief scientist Fred Sala plus colleagues at Wisconsin.

Three axes for the future (~00:58–00:59)

1. Environment complexity and realism — real codebase environments, scientific computing environments, legal workflows, the actual systems people work in daily, not simple prompts and responses.
2. The autonomy horizon — how far models can go working autonomously, and how humans get pulled in across those settings.
3. Output complexity — beyond verifiable fields into nuanced ones like legal and healthcare, where the definition of good isn't a simple closed-form solution the way it is in mathematics.

His closing call to action: Snorkel is funding a number of these benchmarks through their open benchmarks grants, and welcomes collaboration at benchmarks.snorkel.ai.

Quotes

"Our ability to measure AI has really been outpaced by our ability to develop it." (~00:48:42)

The problem statement for the whole talk.

"What I'll call bench slop — the idea that you can really quickly vibe code a benchmark and get a fancy Twitter post." (~00:49:50)

The sharpest available name for the current benchmark inflation.

"Not only are we measuring correctness, but we're measuring notions of codebase alignment, practice alignment, bloat relative to an Oracle PR." (~00:55:32)

What a tasteful solve actually contains — taste turned into something measurable.