Preferences > Benchmarks: Model Routing for How Teams Actually Build

TL;DR

• Cost is the number-one pain in agentic AI right now — and it arrived fast. Raha cited Uber burning a full year's agentic AI budget in four months and Walmart capping AI spend. At AWS he watched cloud cost optimization take years to become a real discipline (FinOps); this time the same shift happened in one or two years.
• The root cause isn't price, it's fit. Not every request needs a frontier model: classification and labeling run fine on small models, writing/summarization/translation on small-to-medium, code generation on medium; only scientific reasoning, medical, and security analysis genuinely need frontier depth.
• A third reason is risk. Betting the whole application on a single model means a single outage or quality regression takes the app down with it.
• Routing should encode preferences, not benchmark scores. Your cost ceiling, latency tolerance, prompts, and tools are all dynamic — and no public leaderboard knows any of them. Encode them in the router and rerank live, instead of blindly shipping every request to whatever tops a benchmark.
• The architecture is a server-side Plano proxy plus a 30B purpose-built routing model, both open source with no vendor lock-in. Because it is trained for routing and nothing else, it beats frontier models (GPT-5 series among them) on the routing task — at under 200ms added latency, zero client code changes, and no cost.
• Using an LLM as your router means paying a double tax: roughly 600ms extra latency and another round of token spend. A purpose-built system is the right abstraction.

Key Points

Speaker and DigitalOcean's position in the stack (~01:43)

Raha is a Fellow Engineer at DigitalOcean, joined about a year ago, working on data and AI. Before that he spent a long stretch at AWS building foundational services like OpenSearch and several ML services from the ground up.

DigitalOcean positions itself as a cloud-native, AI-native full-stack cloud: compute, storage, and networking at the bottom, data and learning above that, managed agents on top. The inference router sits in the middle, at the inference layer — a position that means it affects everything above it.

Since the inference service launched they've onboarded 70+ models, including the recently leaderboard-topping Kimi K3 and Claude Opus 5. His framing was blunt: "those are amazing models, I love them, but there is one problem, which is cost."

Three problems: cost, fit, risk (~01:44–01:45)

1. Cost — routing every task to a frontier model produces enormous bills. Two industry data points: Uber exhausted an entire year's agentic AI budget within four months; Walmart is capping AI spend. "Every company is feeling that."
2. Fit (the top reason) — you simply don't need frontier capability for every request; a small or medium model serves many tasks perfectly well.
3. Risk (the operational one) — depending on one model puts your whole application at the mercy of that model's availability and quality.

His task-to-tier mapping:

The inference router: between client and server (~01:47–01:48)

The router lives server-side, between the client and the models. You don't hardcode which model serves which task, and you don't build the routing system yourself — you declare which class of task should draw on which models, and the router dispatches intelligently using live signals: current latency, model availability, and cost.

Results in the field: a Canadian legal-AI startup (subscription legal advice combining AI with human-in-the-loop) is seeing a 40–50% reduction in inference cost. Raha said reductions of up to 80% are achievable in practice, without losing model capability or output quality.

"This is the new FinOps" (~01:48–01:49)

He drew the comparison to his AWS years: cloud cost optimization took a long time to mature into a discipline, and the big push came during COVID. This time there's no waiting — it's happening as we speak, with a massive shift inside one or two years. That's why he thinks the router is becoming the most important part of the stack.

The intuition: what should a router actually decide on? (~01:49–01:50)

Beyond the prompt and the tools, two things must enter the decision: the cost you're willing to spend and the latency your application can tolerate. The crucial observation is that these are dynamic, not static — and no public leaderboard or benchmark tells you anything about them.

So they have to be encoded into the router and applied in real time, rather than blindly routing to whatever some benchmark declares best for a task. That's the whole thesis behind "Preferences > Benchmarks."

Architecture: Plano plus a 30B purpose-built routing model (~01:50–01:51)

• Plano is the proxy in the middle. Requests hit Plano, Plano consults the routing model, then reranks the task's candidate model set against your cost and latency preferences — fully dynamically.
• The routing model is 30B parameters, purpose-built for routing only. Because it's trained for exactly this task, it beats frontier models — GPT-5 series included — on the routing benchmark.
• Both Plano and the routing model are open source: no vendor lock-in, self-hostable, contributions welcome.
• Plano supports observability from multiple systems. Here it plugs into DigitalOcean's inference system, but you can plug in your own and feed it live metrics to drive reranking.

Why not the alternatives — static routing and LLM routing (~01:51–01:52)

• Static routing: buttons on your homepage where "summarization" always goes to one model and "translation" to another. Too static, and every new model or task category means changing the client.
• Using another LLM to decide the route: you're paying a double tax — roughly 600ms more latency, plus paying frontier model prices just to make a routing decision.
• Their approach: purpose-built, server-side, under 200ms, no additional cost. Server-side placement also means adding a model or a task class is a configuration change, never a client change.

Operating it, and plugging in your own eval (~01:53–01:54)

• Routing rules are declared in natural language.
• Presets ship on top: using public benchmarks plus task-specific benchmarks, they pre-select a candidate model set per task class so you can start somewhere sensible and adjust from there.
• Plug in your own eval. The platform includes an evaluation framework, and his emphasis was on whose data it runs against: "the dataset that is needed for your company, for your task — not someone else's public leaderboard." Verify the router behaves on your data; if not, adjust and loop.

Cache-aware routing (~01:54–01:55)

Rushed for time but flagged as a must-consider: in agentic systems that discover, analyze, and write code in a loop, the same prompts recur on every iteration. Switch models mid-session and you lose the warm cache the previous model had built. Routing therefore has to be cache-aware. This is what they're building now.

Demo and quick facts (~01:55–01:57)

The live demo didn't survive the venue Wi-Fi (the flow: create router → name it → add a task, e.g. the summarization preset → save → launch in the playground in one click and compare against another model). He relayed his pre-talk test result from memory: the router came out roughly 20× cheaper than Opus 5 and finished the task about 67% faster.

Quick facts:

• Purpose-built routing model, 30B
• Adds ≤200ms latency, server-side
• Zero application code changes
• Completely free — none of the LLM-as-router double tax on latency and cost
• Managed on DigitalOcean or self-hosted

Roadmap: personalization (~01:57–01:58)

The part he's most excited about, having built a personalization service before: the more you use the router, the better it gets. Turn on logs and traces, and the feedback flows into a continuous loop checked against your evals, so the router keeps improving on its own. His closing line: you build nothing on your side, and the router still improves.

Quotes

"You do not need a frontier model to serve every single of the request." (~01:45)

The premise of the whole talk: the problem isn't that frontier models are expensive, it's that they're overkill.

"No public leaderboard or public benchmark tell you. So you need to encode those in the router." (~01:49)

Your cost ceiling and latency budget belong to you, not to a leaderboard — so routing logic has to be preference-driven.

"You are paying double the tax … you are increasing the latency by around 600 milliseconds or so. In addition, you are also paying more." (~01:52)

The price of using an LLM as your router, on both axes.

"Without doing anything, you're not building anything on your side. Without doing anything, the router is improving." (~01:57)

Once the routing layer has logs, traces, and an eval loop, improvement becomes the default rather than a project.