How Agentic AI Is Rewriting the Rules of AI Infrastructure

TL;DR

• Today's inference stacks are chatbot-era artifacts: one model at a time, optimized for static execution paths, each request assumed independent, hardware essentially homogeneous. Agentic workloads break every one of those assumptions.
• Two inevitable trends: dynamism and heterogeneity. Dynamism means many models, tool calls, fluctuating demand, interdependent tasks, and workflow-level optimization; heterogeneity means CPUs plus a growing set of hardware specialized for different parts of the task. And agentic workflows involve 10–100× more computation than non-agentic ones — which makes the middle software layer more decisive, not less.
• Trillions of choices plus rapidly evolving components means you will pick wrong. End-to-end SLOs on one side, infrastructure objectives on the other, and a cross-dependent choice space in between. His conclusion: optimizing parts of the stack will not work; only full-stack optimization does — and since "you can't optimize what you cannot measure," benchmarking and simulation tooling (Chakra, ASTRA-sim) are prerequisites rather than by-products.

Key Points

Tokenomics and why the software stack became the bottleneck (~00:48:03–00:49:30)

Krishna opened by acknowledging the clock: "We're at 1:45, I know it's officially the end of this session — my superpower as a professor is I can rush through any number of slides in 5 minutes."

His frame is the era of tokenomics (~00:48:35): we feed AI workflows into AI systems and tokens come out, so the metrics that matter have become tokens per second, tokens per watt, tokens per dollar. In an ecosystem with a workflow on one end and hardware on the other, the software stack in the middle becomes crucial to token efficiency.

The problem is that most inference serving stacks in production today were built for the chatbot era (~00:49:02), and share a consistent profile:

• typically one model at a time;
• optimized for static execution paths;
• each request assumed independent;
• heavy kernel-level optimization;
• primarily homogeneous hardware — GPUs, for the most part.

What agents change: two inevitable trends (~00:49:30–00:50:30)

1. Dynamism (~00:49:33): many models, tool calls, dynamically fluctuating demand, interdependent tasks, and a lot of workflow-level optimization opportunity.
2. Heterogeneity (~00:49:49): he credited earlier speakers for covering this well — there is already heavy use of CPUs for anything that isn't an LLM, plus a range of hardware specialized and optimized for different parts of the task.

Then the amplifier (~00:50:11): citing a Google talk from the previous day, agentic workflows involve 10–100× more computation than non-agentic ones. Which makes the software stack in the middle matter even more.

Trillions of choices, and "you can't optimize what you cannot measure" (~00:50:30–00:52:30)

Demystify the picture and you find, all the way down from workload to model layer to software to hardware, enormous numbers of choices and optimization opportunities at every level. Abstracted, one end is end-to-end SLOs — whatever the agent application cares about — and the other is infrastructure-level objectives for running data centers efficiently.

What you actually face is a cross-dependent space, trillions of choices, diverse SLOs, and rapidly evolving components (~00:51:00). His read is blunt: under those conditions going wrong is close to inevitable, the chances of going wrong are very high, and the cost of going wrong is higher still.

So how do you extract token efficiency from that ecosystem? He reached for the standard line: "You can't optimize what you cannot measure" (~00:51:22). Over many years he has worked with academic and industry partners to understand the end-to-end AI systems stack from hardware to software to workloads, and worked closely with benchmarking and standardization bodies to define APIs so that benchmarks and tools interoperate — which has driven wide adoption. Two results he called out:

• Chakra (~00:51:51), an initiative started with MLCommons and released a few weeks earlier at MLSys: a benchmarking methodology for distributed AI platforms.
• ASTRA-sim (~00:52:02): a simulation platform for studying distributed AI networks.

Across everyone he has worked with — hyperscalers, hardware vendors, OEMs, test vendors — the same finding keeps surfacing: the only way to get efficiency is optimizing across the stack; optimizing just parts of it will not work.

Closing: InfraVana (~00:52:30–00:53:01)

That leads to what he is personally most excited about: they have just started InfraVana, building an automated, agent-aware and hardware-aware full-stack optimizer. He reports already seeing massive speedups over state-of-the-art inference frameworks, with more to come once they emerge from stealth.

He closed by inviting questions at the conference or by email, and officially called the session to a close.

Quotes

"You can't optimize what you cannot measure." (~00:51:22)

The premise behind all of his benchmarking and simulation work, and why Chakra and ASTRA-sim are prerequisites rather than side projects.

"It's kind of almost inevitable that you would go wrong when you pick something, and the chances of going wrong are actually very high and the cost of going wrong is even higher." (~00:51:05)

In a trillion-choice space, picking correctly stops being a matter of experience and becomes a matter of systematic search and measurement.

"The only way to get efficiency is optimizing across the stack — optimizing just parts of the stack will not work." (~00:52:15)

The consistent finding from hyperscalers through OEMs, and the thesis InfraVana is built on.