vLLM: Building Open and Efficient Inference for Agents

TL;DR

• The API didn't change; everything under it did. vllm serve still hands you an OpenAI- (and Anthropic-) compatible endpoint that any agent framework speaks out of the box. Agents rewrote the engine, not the interface.
• Three axes of pressure: bigger models (trillion-parameter agent models like Kimi K3 and DeepSeek V4 force serious model parallelism), longer context (hundreds of turns, up to a million tokens, with recomputation strictly forbidden), and unbounded token demand growing faster than GPU supply.
• Parallelism has no universal winner. vLLM supports seven kinds, including a decode context parallelism that doesn't exist in training — but the right choice depends on model architecture × cluster setup × workload shape. In one real case a TP × PP × EP configuration across 16 GPUs beat the obvious single-host 8-way TP baseline on both TTFT and per-GPU throughput.
• Hybrid models turn memory partitioning into a dynamic problem. Full attention's KV cache grows linearly with context; linear attention (e.g. Kimi Delta Attention) keeps a fixed-size state per sequence. vLLM replaces static partitioning with one shared pool, per-attention-type allocators, and automatic rebalancing.
• Token economics have flipped: once models are smart enough, a token is worth far more than it costs to generate, so demand is effectively unbounded and there aren't enough GPUs on earth — hence a plugin architecture spanning 10+ hardware backends.

Key Points

Where vLLM stands, and the two ways to use it (~03:45–03:47)

Kwon introduces himself as a co-creator of vLLM and co-founder of Inferact, the startup built around it. vLLM is an open-source LLM inference engine whose goal is to make LLM inference "efficient and effortless." It started roughly three years ago at UC Berkeley during his PhD and has grown fast ever since — around 88k GitHub stars as of shortly before the talk. He stresses that it is a highly collaborative project across academia and industry, with major contributors including Red Hat, NVIDIA, AMD, Google, and Moonshot alongside Inferact, and that it is widely deployed for production inference.

Two entry points. Offline batch inference goes through the Python LLM class: pass a Hugging Face model name, call generate, done — model loading, optimization, scheduling, memory management, and GPU utilization are all handled underneath. Online serving is a single vllm serve command yielding an OpenAI-compatible endpoint; Anthropic APIs are supported too, so any agent framework that speaks either works out of the box.

Which sets up the hinge of the talk: that API looks much as it did a few years ago — agents barely changed it. What agents changed enormously is everything underneath.

The three pressures agents apply (~03:47–03:49)

First, large models. Frontier agent models — Kimi K3, DeepSeek V4 — are around a trillion parameters, forcing everyone to get serious about model parallelism. He frames this as opportunity: it opens many creative ways to shard a model, each with different trade-offs.

Second, long context. Agent sessions run for hundreds of turns, up to a million tokens, and context only grows over a session. Managing the KV cache of prior turns is therefore critical, and the engine must never recompute tokens from a previous turn — otherwise you get enormous duplicated computation and wasted compute.

Third, enormous token demand. As agents get smarter and more capable there are near-infinite places to deploy them, so demand is effectively unbounded — and growing faster than GPU supply. vLLM's response is to make efficient inference work on many more hardware backends so that all available compute can be turned into tokens.

Axis 1: no universal winner in parallelism (~03:49–03:53)

vLLM today supports seven main kinds of parallelism: tensor, pipeline, data, expert, sequence, and two flavors of context parallelism — one of which, decode context parallelism, is parallelism around the KV cache and has no analogue in training. As a general-purpose engine, vLLM implements all of them efficiently, and mixtures of them.

That, he says, is not enough. These parallelisms must be selected and tuned for the target model architecture, the target cluster setup, and the target workload shape. There is no universal winner.

The worked example: prefill for a DeepSeek model on B200 GPUs in a disaggregated serving setup, where this pool of GPUs only does prefill. The most straightforward deployment is single-host 8-way tensor parallelism — standard and simple. The winning configuration is nothing like it: a combination of tensor, pipeline, and expert parallelism (plus sequence parallelism) spread across 16 GPUs per model replica, delivering substantially lower TTFT and substantially higher per-GPU throughput than the 8-way TP baseline.

Why: pipeline parallelism enables parallelism across chunks within a long prefill; sequence parallelism enables more communication–computation overlap; and expert parallelism yields better GEMM shapes than 8-way tensor parallelism. The takeaway is that in the agentic era you need the right insight and performance model to configure this correctly — and vLLM's contribution is the common substrate on which you can play with all of these parallelisms.

Axis 2: KV cache, from static partitioning to dynamic rebalancing (~03:53–03:58)

A defining property of modern LLMs is that they are hybrid: global attention layers interleaved with cheaper mechanisms such as sliding-window or linear attention (Kimi Delta Attention being his example). That interleaving is what makes million-token context feasible at all, since global attention alone consumes too much memory at that scale.

Elegant in theory, genuinely hard in practice, because layers of different attention types have completely different memory behavior. Full attention's KV cache grows linearly with context; a linear-attention layer like KDA keeps a fixed-size state per sequence regardless of actual context length. So how should GPU memory be carved between them?

The straightforward answer is static partitioning — reserve x% for full attention, y% for the rest. It works, but the optimal split depends on batch size and context length, both of which move continuously during inference.

vLLM's answer is dynamic partitioning: one shared GPU memory pool, with each attention type getting its own allocator drawing from it. The full-attention allocator requests from the pool by token count; the KDA allocator, being linear attention, allocates one large block for the entire sequence; and logic between them shares the same memory space dynamically. The engine rebalances automatically so no GPU memory is wasted, and the user never has to reason about the split.

Managing KV cache within GPU memory isn't sufficient, though. Agent sessions are long-lived and intermittent: generate some tokens, then wait — for tool calls, sometimes for a human — then continue. During those waits the KV cache has to live somewhere. vLLM's abstraction is the KV connector, which stores idle KV cache in external memory (CPU memory or disk) and brings it back on demand. He notes real design effort went into making that abstraction work with third-party libraries like Mooncake and with mechanisms like prefill/decode disaggregation, where KV movement is itself dynamic and complex — cache has to move from prefill instances to decode instances, and from prefill instances to a distributed KV storage pool. The purpose of all that infrastructure is one guarantee: in a many-turn agent session, as long as storage allows, never recompute tokens from previous turns.

Axis 3: hardware — turning all the compute on earth into tokens (~03:58–04:00)

Kwon argues the economics of tokens are flipping: as models get more intelligent, the value of a token far exceeds the cost of generating it. Demand explodes accordingly, and honestly there are not enough GPUs in the world to serve it. So the question becomes: can we use all available compute on earth to generate tokens efficiently?

vLLM's answer is 10+ hardware backends today. NVIDIA GPUs are the major focus, but Google TPUs, AMD GPUs, and various other industry chips are supported through a plugin structure that shares vLLM's core while customizing hardware-specific parts. This makes particular sense in inference, he argues, because the API layer is already standardized — you keep using the same OpenAI or Anthropic API while different silicon generates the tokens underneath.

He skips the detail slides for time, leaving the TL;DR: bringing up new hardware and making it efficiently support new models is getting easier because of coding agents, but it still requires retaking the entire inference stack from the ground up, and the team is actively working on it.

Summary: the agentic era stresses inference along three axes, and vLLM answers each — large models with workload-aware model parallelism, long context with a dynamic and hierarchical KV cache, exploding token demand with diverse hardware backends. Everything shown is open source.

Quotes

"The API itself is pretty much similar to a few years ago … What has changed a lot for agents is everything underneath it." (~03:47)

The framing device for the whole talk: the agent revolution happened below the API.

"There's no universal winner." (~03:50)

Parallelism has no default answer, only an answer for a given model × cluster × workload.

"As models get smarter, the value of a token far exceeds the cost of generating it … and honestly there are not enough GPUs in the world to serve it." (~03:58)

Multi-backend support is not a compatibility nicety; it is a supply-side necessity.