Building Reliable Agents: An Evals-First Approach

TL;DR

• The framing: in enterprise and regulated industries, an agent that's wrong 5% of the time is a liability — a wrong number on a customer invoice, a misstatement in a compliance filing, a bad write to a production database thousands of systems depend on. She cites the MIT study referenced that morning: 95% of AI pilots never reach production.
• What you evaluate changes: not a model and its output text, but a system unfolding over time in a real environment — model, prompts, tools, orchestration, and environment (production APIs, knowledge bases, files, business workflows). Good eval design separates tunable components from given ones.
• A four-layer framework: L0 business outcomes (North Star KPIs), L1 task success (black box; the operational center you iterate on), L2 component evals (open the box, look at sub-tasks), L3 diagnostics (the "why" behind L1/L2 failures). Common failure mode: optimizing L2/L3 without moving L1.
• Design principle: anything that can be deterministic shouldn't go through the agent. In the oil-well casing case the agent never computes a number or drives a workflow, so numerical fidelity is 100% by construction and the eval problem shrinks to answer quality and document completeness.
• Evals are living assets and a moat: you can swap the model at any time; the eval suite is what customers actually invest in, and it's what lets them ride the wave of foundation-model improvements.

Key Points

Why "mostly right" fails in the enterprise (~00:28–00:30)

Scale's mission is to build reliable AI systems for some of the world's most important decisions, and the tension is structural: inherently stochastic technology, deterministic business outcomes.

The credentials behind the approach: 90% of frontier labs use Scale's data engine, and Scale builds AI applications for governments — including the US Department of Defense and the Defense Innovation Unit — and for enterprises in regulated industries like health care, finance, and telco. Unlike consumer and prosumer agents used for day-to-day productivity, reliability is the whole game here. Their work with the Mayo Clinic on bringing agents into patient care is a case where, in her words, the goal is literally saving lives.

She cites the MIT study mentioned in that morning's talk — 95% of AI pilots never reach production — and makes the enterprise version of the point concrete: an agent that's wrong 5% of the time produces a wrong number on a customer invoice, a misstatement in a compliance filing, or a bad write to a production database with thousands of downstream dependents.

Every production agent at Scale passes eight engineering gates, covering data privacy, correct access-control frameworks with authZ and authN in place, and more. This talk covers one of them: rigorous evals.

You are evaluating a system (~00:30–00:31)

Relative to model benchmarking, enterprise agent evaluation shifts because an agent is a system capability. It unfolds over time and interacts with a real environment, so what's under test is the end-to-end system: model, prompts, tools, orchestration, and environment — where in an enterprise the environment means production APIs, knowledge bases, files, and the business workflows already present at that company. The useful distinction, and one good eval design preserves, is which components can be tuned and which are given.

The four-layer framework (~00:31–00:33)

The framework keeps business alignment, black-box evaluation, and debuggability separate but causally linked:

• L0 — business outcomes: the North Star KPIs and the value the agent actually drives — resolution rate, cost per ticket, analyst time saved.
• L1 — task success: treat the agent as a black box and ask whether it completed the task. Was the refund issued for the correct amount? Is the case record in the correct final state? This is the operational center of agent evaluation and the layer teams iterate against.
• L2 — component evals: open the box and look at sub-tasks — extracting the relevant facts, choosing the right fields, producing the right record changes.
• L3 — diagnostics: the "why" behind failing L1 and L2 metrics; what you need for debugging and optimization.

These form a chain, and the common failure mode she calls out is teams spending most of their time optimizing L3 diagnostics and L2 components without confirming L1 is improving too.

Case study 1: oil well casing design (~00:33–00:36)

Scale is helping a top oil company design wells better, including the casing — the steel pipe that goes around the well. Designing it is normally a multi-week loop in which an engineer runs hundreds of simulations and consults standards buried in PDFs and libraries.

The agent design has two deliberately separated halves. First, a deterministic core: the engineer stays in charge, runs the simulations with trusted physics-based simulators, results are checked deterministically against company standards, and the engineer makes the final call on approving safe well designs. Second, a casing design agent that follows along, extracts the right context from hundreds of simulations, answers the engineer's questions with claim-level citations grounded in the agent's interaction, and finally drafts the well design document — the Basis of Design (BOD).

The load-bearing architectural choice: the agent never computes any number or drives any workflow, so numerical fidelity is 100% by design. That collapses the eval question down to how well the agent answers the engineer's questions and generates complete, correct BODs.

Mapped onto the framework: L0 is the number of annual BODs signed off by subject-matter experts; L1 is whether the agent's answers and documents are complete, correct, and review-ready; L2 covers retrieval and context management; L3 covers terminology, grounding fidelity, and recall.

They build the evals with SMEs and calibrate the graders. Because SMEs are bottlenecked on time, they synthetically expand SME templates and have the SMEs validate the additional data. Hill-climbing on L1 correctness took terminology precision from 83% to 100% and claims precision from 68% to 90%. That earned the confidence of the customer's well design engineers, and the agent is now in a limited production rollout, collecting additional production traces, scoring them, evolving the eval sets, and improving from there.

This leads to her strongest claim: evals are living assets, and the most durable asset there is. You can always swap out the model or redesign the agent and test against the same suite. That suite is the IP moat customers invest in, and it's what lets them ride on top of the wave of foundation model improvements.

Case study 2: financial due diligence (~00:36–00:37)

With a top-four professional services and accounting firm, Scale built a financial due diligence agent that takes analysts from four to six weeks down to two days to reach the same human-validated insights. Same framework — L0 defined as total time to human-validated insights, evals built, L1 correctness hill-climbed, and a production rollout where every run teaches them something. She adds a practical note: having sufficient eval data is often one of the biggest bottlenecks, and limited production rollouts are a good opportunity to collect more and evolve the suite.

Process and closing principles (~00:37–00:39)

Every agent built at Scale gets an eval design review, run with a forward deployed engineering team that partners with enterprises and government agencies. A performance maturity matrix serves as the eval scorecard for each agent, and nothing ships until the scorecard is all green.

Closing principles: you're evaluating systems, not just models; use hybrid graders — deterministic wherever possible, model-based where necessary, human review for calibration and high-stakes use cases; treat evals as living assets that grow over time; and "evals first is how mostly right becomes production-ready."

Quotes

"You have technology that is inherently stochastic, and you are trying to deliver deterministic business outcomes using this." (~00:28)

The central tension of enterprise AI in one sentence, and the reason the whole framework exists.

"The agent never computes any number or drives any workflow. So, by design, numerical fidelity is 100%." (~00:35)

The best reliability wins are architectural — make the error impossible, then evaluate what's left.

"Evals are living assets … they are the most durable asset." (~00:36)

Models get swapped out; the eval suite doesn't.

"Evals first is how mostly right becomes production-ready." (~00:39)