From Training to Evaluation: Open Recipes for Building Agentic AI at Scale AI

TL;DR

• RL's next frontier is the non-verifiable domain. Math and code have ground truth, so correctness is checkable; the science and open-ended tasks frontier labs actually care about have no verifier. Scale AI's answer is to turn grading criteria (rubrics) into verification signals, producing a ranking-based reward model built from rubrics, and using it to push GRPO forward with measurable gains on open-ended tasks.
• Online rubrics are the natural extension. The policy keeps improving during training, so the rubric should too: compare a reference policy against the current policy, find what the current reward is missing, generate better rubrics, and fold them back into the reward model. Wang notes this lines up closely with the recursive self-improvement research trend.
• SWE-bench-style benchmarks are too clean. A real engineer doesn't read an issue and emit a patch — they write unit tests, go back and forth through the repo, and ask and answer their own questions to work out the next step. Hence SWE-Atlas, which he says frontier labs have widely adopted.
• The CLI vs MCP debate may be the wrong question. Their finding: once a model is capable enough (recent Claude 4.8, GPT-5.5/5.6 tier) and the backend is the same, the tool interface stops mattering — the model learns to use whatever is best.
• Computer-use benchmarks are expensive, so synthesize. Building on existing high-quality benchmarks, they generate longer-horizon, harder, more realistic tasks — knocking the OSWorld state of the art from above 80% down to roughly 30%, evidence that high-quality realistic datasets can be constructed synthetically.
• The closing frame is a training ↔ evaluation flywheel: post-training insights build better RL environments; evaluation failure modes improve post-training algorithms.

Key Points

Positioning and Scale AI background (~02:12–02:13)

Wang is an assistant professor at UC Santa Cruz and a research advisor at Scale AI, working closely with Emily Xue (the previous speaker). With only 10 minutes, he covered post-training briefly and spent most of the talk on evaluation and the synergy between the two.

Scale AI's mission is to build reliable AI systems for the world's most important decisions. Founded in 2016, now large, with offices in the US and other countries.

Post-training: from RLHF to rubric-based RL (~02:13–02:15)

Scale's current post-training research thrust is moving from RLHF to rubric-based RL — a direction that came out of providing data services to frontier labs, where it emerged as genuinely important.

Three layers of challenge: how to design an effective reward, how to use that reward with RL, and how to make the reward itself better during RL training. Results have appeared at ICLR and ICML this year, with NeurIPS submissions pending.

Concretely, they pushed the boundary of GRPO using a rubric-based reward paradigm — improving performance, and extracting recipes for spinning the data ↔ RL flywheel.

Why rubric-based rewards are the key move (~02:15–02:17)

This was the point he most wanted to land.

RL's traditional battleground has been verifiable domains — coding, math. There you know whether an answer is true or false; you have ground-truth verification.

But most of what matters in the real world, and to frontier labs, is open-ended and has no verifier — science, or personal tasks of all kinds. So a way had to be found to make RL work in those domains.

Their approach: set up a paradigm that turns evaluation criteria into verification. The reward model built this way has one crucial difference — it isn't just distinguishing a better response from a worse one; it ranks, based on rubrics — which is what lets it scale to real-world scenarios.

The pipeline: start simple, gradually build a very high-quality rubric, then use it in the RL phase — with improvements observed on open-ended tasks.

Online rubrics: rewards that evolve with the policy (~02:17–02:18)

A natural extension of the first work. The observation is that the policy keeps changing and improving during training, so a fixed rubric progressively falls behind.

The method: hold a reference policy alongside the current policy, identify what the current reward is failing to capture, generate a better rubric from that gap (an online rubric), and integrate it back into the reward model — yielding a better-trained model that's more robust on downstream tasks.

He flagged that this online-rubric evolution aligns closely with the recent self-improvement / recursive self-improvement line of work.

Evaluation: SWE-Atlas and what real engineering looks like (~02:18–02:20)

Evaluation is Scale's main focus: turn insights from training and customer conversations into worldwide leaderboards for frontier models.

The problem with SWE agent benchmarks: SWE-bench and SWE-bench Pro give you a text description of a problem from GitHub and ask the agent to produce a patch that fixes the bug in the original repo.

His objection is that the real world is far messier. Engineers don't just read the PR and the issue and emit a patch — you have to write unit tests, and sometimes go back and forth through the repo, asking questions and answering them before you know what to do next.

So they released SWE-Atlas, which he says is now widely adopted by frontier labs.

Professional reasoning and AI for science (~02:20–02:21)

• HLE (Humanity's Last Exam) as the representative professional-reasoning benchmark, with model families still hill-climbing on it.
• A drug discovery bench: building a genuine healthcare environment where experts generate real drug-discovery tasks that run from the initial concept all the way to delivering the drug. A preprint was released recently.

The tool interface debate: CLI vs MCP (~02:21)

A conclusion that clearly resonated in the room. Their high-level finding on which tool interface is better:

Once the model becomes capable enough — the recent Claude 4.8, GPT-5.5 or 5.6 tier — the interface stops mattering. The model learns to use the best available tool interface on its own. The caveat: you have to give it the same backend. In other words, arguing CLI versus MCP may be the wrong question.

Computer-use agents: making benchmarks harder by synthesis (~02:21–02:22)

Benchmarking computer-use agents is hard because it's very expensive to run — multimodal models, plus a long tail of verification details.

Their answer: take existing high-quality benchmarks as a base and synthetically generate longer-horizon, more challenging, more realistic tasks from them.

The effect is dramatic: on OSWorld, where a new Opus version's state of the art sits comfortably above 80%, the generated tasks drove it down to about 30%. Evidence for his claim that realistic datasets can be constructed synthetically while staying high quality.

Closing: the training–evaluation flywheel (~02:22–02:23)

His last technical slide was about synergy:

• From post-training insights → build better RL environments
• From evaluation failure modes → improve post-training algorithms

"Hopefully this can run forever" — with both better evals and better training results falling out of the loop.

Wrap-up and workshop plug (~02:23)

Scale works with frontier labs, governments, and enterprises, and is active in robotics and physical AI.

He closed by promoting the third edition of the Agents in the Wild workshop, accepted at NeurIPS, with a submission deadline at the end of that month. He is one of the core organizers; the panelist lineup includes Dawn Song plus two speakers whose surnames the captions garble (heard as "Joshua" and "Jav").

Quotes

"In many real world cases where Frontier Labs really care about … it's not a verifiable domain. So we need to find out a way to make it work for those open-ended domains." (~02:15)

The motivation for the entire post-training half: RL has to leave the comfort of math and code.

"Once a model becomes really capable … the model will learn to use the best tool interface possible. So that really doesn't matter — unless you give the same backend." (~02:21)

A cooling agent for the CLI-vs-MCP argument: at sufficient capability, the interface isn't the bottleneck; the backend is.

"We can pretty much bring down the performance of the original OSWorld … from something definitely beyond 80% to something like 30%." (~02:22)

Synthetically generated long-horizon tasks strip computer-use agents back down to size.