Spec-Driven Agents: Hierarchical Specs, Tooling, and Trajectory-Based Evaluation

TL;DR

• Trajectories beat final answers: most of the failure modes they found — repeated reads, backtracking, dead-end searches, inconsistency — are invisible in the final result and only surface in the trajectory. Inefficient trajectories don't just waste tokens; through context exhaustion they actively hurt accuracy.
• More tools is not better: split tools into setup tools (always called at startup) and discretionary tools (called by the model as needed). Moving a subset to startup forces an "onboarding" pass, shrinks the decision set the model faces mid-problem, and measurably tightens trajectory consistency.
• Tool overload is real: their full tool set was a superset of minimal with convenience tools pre-baked, and it was slower with marginally worse quality. What matters is tool orthogonality / separability — if the agent can't tell whether to call B or C, that's a problem.
• Restructuring knowledge into a diagnostic playbook (a router mapping symptom → problem type → handbooks) beat reading the raw guides across the board, with a 43% reduction in total tokens and a 48% reduction in total error rate, plus better instruction following.
• One line worth keeping: "the only thing worse than a system that doesn't work is something that just works occasionally."

Key Points

The problem: a diagnostic and remediation agent for the Redis Query Engine (~00:40–00:41)

They built an AI diagnostic and remediation agent for diagnosing issues with the Redis Query Engine, primarily to help the customer support team. Redis ships a rich tool suite and extensive documentation, but in practice many complex queries were escalating all the way to the development team — not sustainable. The agent exists to unblock a good fraction of those and shorten customers' time to solution.

What makes this hard: unlike SQL, an LLM's parametric knowledge of Redis queries is fairly limited and often wrong. Many of these queries are also context-dependent — the agent must work out what's actually missing from a query. So they built an agent with skills and tools running a clarify → diagnose → confirm loop to make it more reliable.

First lesson: the final result isn't enough (~00:41–00:43)

Three observations. Evaluating the final result alone is insufficient — how you get there matters a lot. Correctness matters, but so does consistency: "the only thing worse than a system that doesn't work is something that just works occasionally." And inefficient Redis queries are expensive — the lesson learned the hard way was that they're also more error-prone downstream, because of length and context exhaustion.

A more capable frontier model resolves some of this, but what actually moved the needle was restructuring the knowledge and how the agent accesses information — making sure the agent has enough to know how to identify information, how to access it, and when to use it.

Four failure modes (~00:42–00:44)

Three concern the final result, one concerns the trajectory:

• Correctness — the result is simply wrong: wrong paths, misread metrics. These are hallucinations.
• Completeness — it diagnosed only part of the issue and left the rest unresolved. One of the more common failures they saw.
• Usefulness, in two flavors: giving too much or too generic information, so it isn't actionable and the user is overwhelmed; or giving advice that violates semantic intent. His example: telling a user who asked "why is my query slow?" to reduce the number of search terms is technically correct but doesn't serve the purpose at all.
• Efficiency — repeated reads, backtracking, dead-end searches, wasting tokens and, as noted, hurting accuracy too.

And most of these only resolve by looking at the trajectory; they don't show up in the final result.

Intervention 1: tool architecture (~00:44)

Split tools into setup tools, always called at startup, and discretionary tools, called by the model based on the problem it's solving. His analogy: starting a new role, you go through onboarding where they teach you what's happening, who's doing what, who to talk to. Setup tools are that onboarding — enough context for the agent to proceed correctly, use the discretionary tools correctly, and know when to use them.

Intervention 2: the diagnostic playbook (~00:44–00:45)

Rather than feeding raw documents, they reorganized the knowledge base into a diagnostic playbook: a router plus a set of handbooks. The router maps a symptom to a problem type and routes to one or more handbooks.

The observation underneath it: users don't arrive with a symptom, they arrive with a concern. Mapping concerns to symptoms is the agent's job; making that easy for the agent is the team's job — which is exactly what the playbook is for.

Results (~00:45–00:48)

Both interventions were studied separately, measuring end-to-end latency, number of tool calls, and result quality.

On tool architecture: the baseline used an automatically generated tool set and performed poorly — slow, with debatable quality. Minimal and full were both informed by their domain knowledge of how the Redis Query Engine works, with full a superset of minimal: where minimal gives you get_shard_info and expects you to post-process to find the slowest shard, full also ships get_slowest_shard so you just call it. Full did not perform better — it was slower with marginally worse quality. Two conclusions: tool overload is very much a real thing, and tool orthogonality (or separability) matters — at any point, if the agent can't determine whether it should call B or C, that's a problem.

The setup tools configuration is just minimal with a subset moved to startup time, so the agent chooses among a smaller set while solving. Setup tools and minimal looked similar on aggregate metrics — until they drilled into the trajectories. Without setup tools, trajectories were far more inconsistent, with many more unique sequences. Not every unique sequence is an issue, but some are. The takeaway: reducing the decisions the LLM has to make at any point reduces variance and improves consistency, and forcing an onboarding context via setup tools achieves that.

On knowledge organization, metrics split into answer quality (answer grounding, specificity) and trajectory quality (first-pass success, dead-end rate). Playbooks beat raw guides across the board, with a 43% reduction in total tokens and a 48% reduction in total error rate; the last row of the table also showed better instruction following. His framing: making it easier for the agent to access information reduced cost and reduced forgetfulness, which is what improved reliability.

Open work and takeaways (~00:48–00:50)

Not everything works as they'd like. He flags recursive self-improvement — a recurring theme at the summit — as a prime candidate for pushing performance further, though they already see a reduction in the common failure modes.

Three takeaways: measure the trajectory, not just the final answer, because many of the issues never surface in the final result; structure knowledge so it's easy for an agent to identify and access; and audit your tooling to separate mandatory baseline context from discretionary or exploratory context. If you can get correctness, completeness, and usefulness in the result plus efficient trajectories, you're far more likely to have a production-reliable agent.

Quotes

"The only thing worse than a system that doesn't work is something that just works occasionally." (~00:41)

Consistency isn't a side effect of correctness — in production it's a first-class metric.

"Telling a user who asked why my query is slow to reduce the number of search terms is technically correct, but it doesn't actually serve the purpose at all." (~00:43)

A failure mode that final-answer grading almost never catches.

"Most of these things can only be resolved by looking at the trajectory. They don't show up in the final result." (~00:44)

The thesis of the talk.