Evals: The Engine for Agent Improvement

TL;DR

• The problem isn't that teams don't want evals — it's friction. Every team wanted high-quality production agents, and every team said "let me just ship the agent and I'll think about eval later" — a perfectly rational call when you're trying to prove product-market fit. The cost is getting caught in a relentless loop of retrofitting evals and reconstructing why the agent broke. The platform team's job was to remove those friction points one by one.
• Four interventions: (1) tracing on from day one in every environment; (2) automatically derived starter-kit evaluators pushed straight into Slack; (3) CLI experiences and skills that move eval ownership out of engineering and into product teams; (4) changing the narrative — the one thing no tool could fix.
• The best evidence was a failure. Voice ride-booking had a 95%+ offline eval, but the production eval showed turns per session running way above average. The cause: a rider booking to SFO while someone in the background said "hey, I want pizza" — and the agent took it as input and started rerouting them to the nearest pizza place.
• The destination is turning eval from a score into an engine: traces → failures automatically categorized → proposed updates to the agent and the evaluators → teams accept or reject. A score tells you about something you already know; an engine continuously moves the agent toward what the customer cares about.

Key Points

Three shows of hands (~03:01)

Agrawal closed out the Compass Stage. He warmed up the room with three questions:

1. Who took an Uber today? (a few hands)
2. More importantly: who has actually written an eval for an agent in production? — "let's see who the real builders are."
3. And the last one, which you should be proud of: did those evals tell you something that changed the way you shipped your product?

That third question is the dividing line: treating eval as a checkbox versus an eval that actually changes your product direction. Getting every agent team to the second state is what Uber spent over a year on.

Context: the Uber agent platform (~03:02)

Uber ships agents across its ecosystem — externally (booking a ride by voice) and internally. Powering all of it at enterprise scale is the Uber agent platform, which he manages, with a range of components so agent teams don't have to worry about infrastructure. This talk is about the eval piece.

The diagnosis: a rational decision and an inevitable loop (~03:02–03:03)

The pattern he saw across every team: everyone wanted high-quality production agents, and everyone did the same thing — "let me just ship the agent and I'll think about eval later."

He's fair about it: it was a very rational decision. They were trying to prove product-market fit; they wanted to see whether the thing even worked.

But it caught them in a relentless loop of retrofitting evals later and trying to figure out why that agent broke.

The platform team's conclusion was that there were true friction points stopping teams from reaching that state earlier — so make it frictionless.

Four interventions to make eval the default (~03:03–03:05)

1. Tracing from day one, in every environment.

(He noted this had come up across the day's talks.) The moment first development starts, tracing is on in every single environment. That gives teams the foundation for everything else: build an agent, vibe with it, send it out, and all the information about how it's been changing over time is already there rather than something to retrofit.

2. Starter-kit evaluators, delivered in Slack.

Once teams had data, the next blocker was that they didn't know how to start developing an eval — not the best way to write one, and not where.

So the platform used the context of how the agent was built plus its documentation to work out the best starter-kit evaluators, and sent those insights directly to the builders in Slack.

His illustration: "I don't know what an LLM judge is, but I do know what it means to have a tool contradiction, and make a change because of that."

3. CLI experiences to get eval out of engineering's hands.

He drew a sharp distinction: evals for agents are very different from QA tests — they're not purely engineering.

That gap had to be bridged so the teams closest to the customer — product and customer teams — could understand them. CLI experiences really democratized that: by building skills that manage the eval, product teams were able to own the entire process.

4. Changing the narrative — the one they couldn't tool their way out of.

The last intervention was, in his words, something they couldn't build a tool away for: changing what people believe an eval is for.

They shifted teams from asking "is your eval 90% plus?" to:

• Do you actually trust your eval?
• What have you changed about your roadmap because your evals told you that?
• How quickly have you been able to update your datasets? Is your dataset five months old and not really up to date with your product?

The result: shipping faster, and catching issues much earlier in the product lifecycle.

The pizza case (~03:05–03:06)

Uber is launching rider voice booking soon, and their evals surfaced this:

• The offline eval was 95%+ — looks great.
• But the production eval showed the number of turns per session running way above average.

Digging in, they found a customer trying to book a ride to SFO while somebody in the background said "hey, I want pizza." The agent took that as an input and started rerouting them to the nearest pizza place.

The lesson: the agent needed to understand true intent, and to have no-ops for when it shouldn't be listening at all.

He stressed that this was only possible because they had both evals and the product teams in the loop.

The endgame: from eval-as-default to eval-as-engine (~03:06)

Uber's arc: these interventions made eval the default, and now they're moving to the next stage — eval as an engine:

traces → failures automatically categorized → proposed as updates to the agent and to the evaluators → teams accept or reject → an improvement feedback loop.

His closing contrast:

On one side, an eval metric — a score for something you already know. On the other, an engine that helps you continuously improve agents, tuned toward what the customer cares about.

"That's what makes each of your trips magical at Uber."

Quotes

"That's the difference between having eval as a checkbox versus an eval actually changing your product direction." (~03:01)

The line his three opening questions were designed to draw.

"They said that, hey, let me just ship the agent and I'll think about eval. And it was a very rational decision." (~03:02)

He doesn't blame the teams — he treats it as a friction problem for the platform to solve.

"We changed teams' narratives from 'hey, is your eval 90% plus?' to 'do you actually trust your eval?'" (~03:04)

The one intervention no tool could deliver.

"Is your data set five months old and not really up to date with your product?" (~03:05)

Eval datasets rot, and a rotten dataset hands you a reassuring but meaningless score.