Building a Token-Efficient OpenClaw Agentic System

Format: a 10-minute AMD deck (originally prepared by an AMD VP who couldn't attend, delivered by Mahdi Ghodsi), followed by roughly 40 minutes of hands-on workshop led by Mahdi Ghodsi and Satya Devineni with TAs on the floor. Organized by theme below.

TL;DR

• The unit of account changed. Chat is a bounded task: prompt in, response out, done. Agentic work is unbounded — plan, call a tool, inspect the result, dislike it, call again — so the same job might take seconds or hours. The metric to report is cost and time per successful run, not tokens per second.
• This breaks three things: budgeting (two users of the same service can differ 10–100× in cost — their own team burned a full year's compute budget in a few weeks), the optimization target (from flops to a memory-bound whole-system problem), and the scorecard itself.
• It all comes back to caching: every turn repeats work the last turn already did. Three fronts at once — model level (69 of Kimi K3's 93 layers use KDA linear attention with a fixed-size recurrent state instead of a growing KV cache), GPU serving stack (tiered HBM → CPU DRAM → SSD → pooled memory, taking P99 TTFT from 17.3s to 5.4s), and CPU (cached tokenization: 1.7s → 0.09ms).
• The counterintuitive lesson: cache hit rate went down from 88.5% to 87.5% while TTFT improved 3× — because hit rate isn't the user experience, finished tasks are.
• The workshop's through-line: use OpenClaw to build a mixture of models (not experts), routing requests to local or cloud models for reasons of cost, quality, and privacy — through four escalating routing techniques, ending in Lemonade Router's semantic routing.

Themes

Theme 1: From tokens to outcomes — why the unit of account changed (~01:27–01:31)

Something that sounds obvious and turns out to be expensive: AI chat is a bounded task — the user sends a prompt, gets a response, reads it, done. Agentic AI isn't. The same query goes in, but the agent plans, maybe calls a tool, looks at the result, maybe doesn't like it and calls again, maybe spins up a whole graph of agents. The same work could be seconds or could be hours.

For anyone planning capacity or a team budget, this is a structural change. Previously you bought seats and assigned users to models or services. Now one user might take seconds and another might take hours, and the compute those two need differs by orders of magnitude.

Three things break:

1. Budgeting. Two people on the same service can differ 10× or 100× in cost. He suspected that's why his VP put the slide in: they budgeted a full year of compute for the team at the start of the year and blew through it in a few weeks — and it's happening across the industry.
2. The optimization target. Everyone used to obsess over flops, tokens per second, and whether the GPU was fast enough. Now multi-turn agent runs grow context so fast that it's a whole-system, largely memory-bound problem.
3. The scorecard. Tokens per second doesn't tell you much when what matters is the finished task, not one response.

Theme 2: One principle across three fronts — caching (~01:31–01:38)

The principle in a sentence: every turn repeats work from the last turn. By turn 12 you're still recomputing all 11 previous turns just to handle what's new in turn 12. The obvious answer is not to redo it. People have been doing this for years, but agentic work changes its weight. They attack it on three fronts — models, the GPU stack, and the CPU — with examples from their day-zero Kimi K3 collaboration with Moonshot AI.

• Model level: the KV cache grows with every response, reaching hundreds of gigabytes for large models — impossible to keep in HBM, the GPU's fastest memory. Kimi K3's approach: 69 of its 93 layers use KDA (Kimi Delta Attention), linear attention with recurrent layers, so those layers compute over a fixed token budget. On AMD MI355 (one node, eight GPUs, 288 GB each) their experiment used only ~15.5 GB.
• GPU serving stack: keeping every request's cache in HBM makes the data center economics untenable. They redesigned tiered caching around the new model: HBM → CPU DRAM → SSD → pooled memory. The key observation is that agentic runs have a lot of idle time — the agent fires a task and waits on a tool call or CPU work — so when eviction happens you want the GPU still busy. With tiering, the prefix is ready when the model is ready to run. Result: P99 time-to-first-token from 17.3s to 5.4s, roughly 3.2×.
• CPU side: agentic workloads make CPUs far more relevant — a lot of compute happens there, and tool calls and execution can be made much more efficient. His example: tokenization can be cached too, giving a flat 0.09 ms when you only compute what's new versus 1.7 s to recompute everything.

The counterintuitive slide: plenty of people report cache hit rate, and in their example it went down, from 88.5% to 87.5%. At first glance that's worse — but TTFT improved 3×, because 88.5% doesn't mean much if you still have to compute the first token. Conclusion: report cost and time per successful run, not tokens per second and not hit rate.

Theme 3: Mixture of models, and the three reasons for it (~01:41–01:47)

The workshop's through-line is moving routing from inside the model to between models. He set up the background: most developers hear "routing" and think MoE — the notable early paper from Geoffrey Hinton's team in 1991, revisited in 2017 on a far larger model where sparse gating and better expert-selection gating emerged, still state of the art today. But the same routing idea works with models, rather than experts, as the endpoints — the mixture of models they'd be building.

Three reasons:

1. Cost — "not every token deserves that best frontier model." Finding a file or running a simple command doesn't need to go to Fable 5.
2. Quality — escalate when it's needed, and use specialized models for tasks they handle well.
3. Privacy and security — he has plenty of personal files on his machine and doesn't want his social security number showing up in someone's training data, so it stays local.

He was explicit that none of this is AMD's idea. FrugalGPT (2023) used a judge model on the small model's response, keeping it if satisfactory and escalating to a cloud model otherwise, claiming 98% total cost savings with results comparable to GPT-4 at the time. RouteLLM (from Berkeley) moved the decision up front — a router picks the model before inference — showing over 2× cost savings. Then semantic routing: embed the incoming request and classify the resulting vector against your rules using cosine similarity.

Hardware and environment: the local model runs on an AMD Ryzen AI "Halo" box (Ryzen AI Max+ 395, purchasable as a small-form-factor machine), with CPU, iGPU and NPU sharing 128 GB of unified memory — enough to serve models up to roughly 200B parameters at lower precision. Escalation goes to Fireworks AI, served on AMD MI355 GPUs. Local inference uses AMD's own Lemonade (desktop app, download a model and go, semantic routing built in); one of Lemonade's founders, Krishna, was in the room.

OpenClaw's role: the agentic runtime for the day. He described it as having gone hugely popular earlier in the year and being probably the most popular open-source software right now, with its founder also at the conference. The workshop uses only its agent-creation mechanics: each agent gets its own workspace containing .md config files (SOUL.md, AGENTS.md) and openclaw.json for models and permissions. Three agents were pre-built: local-brain (local model), cloud-brain (cloud Kimi), and smart-router.

Theme 4: Four levels of routing (~01:47–02:06)

1. Manual /model switching. Anyone who's used a coding agent knows it; simplest possible.
2. A dedicated agent per model. An attendee asked why not just use one big agent. The answer: rules, permissions, and context can be set per agent — e.g. "the cloud agent must never touch my personal files." Demo: after disabling local-file access for cloud-brain and restarting the gateway, asking it to read a financial CSV returned "I don't have access to the local file system"; switching to local-brain and issuing the same command printed the report fine.
3. Automatic escalation routing. Edit the smart-router agent's SOUL.md so every response must begin with exactly one of two lines — CLOUD or LOCAL — plus classification rules (write code → cloud; read a file → local). Two demos: "what does a four-year vesting schedule with a one-year cliff mean?" classified LOCAL and answered locally; "write a Python function to calculate loan interest payments" classified CLOUD and routed to the Fireworks model. Under the hood, escalation spawns a sub-agent carrying the cloud model, passes it the context, and waits for it to return.
4. Lemonade Router's semantic routing (Satya Devineni's section). Architecture: prompt → OpenClaw → Lemonade Router running locally → sensitive questions to a small local model, complex questions to cloud Kimi K2.6, complex-and-sensitive to a larger local model.

Theme 5: Semantic routing in practice — three agents, three rule types (~02:06–02:19)

The scenario: you're running a startup — onboarding and offboarding, benefits, finance, legal. Each agent is scoped by OpenClaw skills (a SKILL.md in the skills/ folder) that point at data sources, state output expectations, and specify what the agent is and isn't authorized to do.

• HR admin agent — LLM-as-router. employees.csv holds PII: SSNs, emails, phone numbers. Onboarding an employee through the cloud model leaks all of it. Switching to user.hr-admin-router applies a one-line routing prompt — "if you see any personally identifiable information send it to a local model; for anything else, send it to Fireworks" — and the same onboarding request completes locally with zero data leaving the machine. The Lemonade dashboard at localhost:1305 shows the local model loading, plus TTFT, TPS, and other metrics.
• Benefits agent — keyword and rule routing. RAG over benefits-handbook.md. Rules: keywords like 401k or vesting and a prompt under 400 characters → local; comparison, analysis, or benchmarking questions, or prompts over 800 characters → cloud Kimi K2.6. Two demos hit the two rules, with the user never switching models.
• Finance agent — semantic similarity plus an LLM classifier. Data analysis over CSVs. An embedding model computes cosine similarity (scored 0–1) between the input prompt and predefined concepts: "what is our current burn rate" maps to simple metric lookup with a threshold above 0.6 → local. Multi-source analysis maps to deep modeling, where either of two rules triggers escalation — an LLM classifier labeling the question complex above 0.5, or a semantic score above 0.72 → cloud.

Challenge and wrap-up: roughly 10 minutes at the end for attendees to solve a task using the fewest tokens possible — local, cloud, or Lemonade Router, their choice — scored on a leaderboard with AMD swag as prizes. Virtual attendees could request free compute credits through the AMD Developer Program form and receive a personal link by email to follow the same workshop later.

Quotes

"So the winner is whoever finishes the most tasks per dollar." (~01:38:29)

The agentic-era scorecard in one line.

"…not every token deserves that best frontier model out there. If for example you're looking for a file or looking for simple command, you don't need to send that to your Fable 5." (~01:44:37)

The entire case for mixture of models, compressed.