Building Frontier Inference and Training Infra for Agent: A Case Study of SGLang and Miles

TL;DR

• RadixArk's mission is to make frontier-level AI infrastructure open and accessible, built entirely in the open on two pillars: the SGLang inference engine and Miles, an enterprise RL post-training framework forked from and co-evolving with slime.
• Three inference challenges: high cache reuse (the cost lever for agentic workloads), deploying and sharding trillion-parameter models, and million-token context as the new norm.
• Three answers: unified hybrid radix cache plus HiCache (moving KV cache from HBM down through DRAM to external storage); 5D parallelism plus DCP/PCP with PD and EPD disaggregated deployment; and chunked pipeline parallelism, sequence splitting, and sparse attention. On GLM-5.2 they went from best-in-class at day zero to a further 2.2× improvement, reaching roughly 500 tokens/s per user.
• Three training challenges: far more diverse environments (some requiring lab experiments to produce a reward signal), train–inference mismatch (which silently turns your RL off-policy), and throughput, which determines how many experiments you can run.
• Eliminating mismatch takes three layers: deterministic kernels in the inference engine, token-in-token-out alignment of multi-turn agentic chat templates, and algorithmic fixes (rollout routing replay, truncated masked importance sampling).

Key Points

RadixArk, SGLang, and Miles (~04:01–04:04)

Zhu introduces himself as co-founder and CTO of RadixArk and a Berkeley PhD. RadixArk's mission is to make frontier-level AI infrastructure open and accessible to everyone, and it builds entirely in open source: SGLang for inference, already adopted as a production inference engine widely; and more recently Miles, an RL framework gradually being picked up as frontier-level post-training infrastructure.

The SGLang ecosystem has three parts: the core engine for language and vision-language model inference; diffusion, a newer project for image and video generation (and world-model inference); and omni, the most recent, targeting ASR and TTS on the audio side. The whole thing is optimized for agentic workloads, with fast day-zero model support and broad hardware coverage — NVIDIA, AMD, TPU, Trainium, Intel, and others.

Miles is the enterprise-grade RL framework, forked from and co-evolving with slime, adding features for the latest NVIDIA and AMD hardware. slime was built by Zhipu AI together with the community for training their GLM model series, so it has been battle-tested on production training at very large scale. Miles likewise offers day-zero support for all open models — Kimi K3, Thinking Machines' Inkling, NVIDIA's Nemotron, and so on — so the day a model ships you can train and customize with Miles and serve with SGLang. Architecturally, Miles natively uses SGLang as the rollout stage and NVIDIA Megatron as the training stage for best performance.

On adoption: hardware companies, hyperscalers, enterprises, AI labs (especially newer ones), developer tools, and neoclouds are running SGLang and Miles as part of their infrastructure. Ecosystem collaborations include close work with Google Cloud on throughput for their internal inference stack, and Cloudflare, IBM, xAI, and Meta all using SGLang as a backend for their inference engines.

His framing for the rest of the talk: 2026 is the year of agentic infrastructure. On inference, demand is exploding and the challenges are genuinely unique. On training, every detail matters — it's still early, but everything has to be done right if you want to train the model without hurting its generalization capability.

Inference: three challenges, three sets of answers (~04:04–04:09)

High cache reuse comes first. He points at DeepSeek V4's very low cache-hit pricing: such pricing presumes you can actually achieve high reuse, and ensuring a high cache hit rate directly determines total serving cost. SGLang brings two things here — a unified hybrid radix cache, designed so prefix caching runs smoothly across all the recent hybrid models and new architectures, and HiCache, which lets you move KV cache down from HBM to DRAM and even to external storage, scheduling it efficiently across memory tiers. On Qwen3-Coder specifically, cache rate, TTFT, and throughput all improved substantially.

Large model scaling comes second, at one trillion, three trillion, and beyond. The standard toolkit is 5D parallelism — data, tensor, context, pipeline, expert — plus newer strategies like DCP and PCP. Deployment strategies are evolving too: teams start colocated, then move toward PD (prefill/decode) disaggregation at larger scale and with more imbalanced prefill/decode profiles; for vision-language models there is also EPD, which disaggregates the encoder as well. On the runtime side, speculative decoding runs from EAGLE to MTP to D-Flash and DeepSeek's recent DSpark, and scheduling includes an overlap scheduler plus their recent spec v2 for better native speculative-decoding speedup at inference.

Long context is third, particularly scaling to a million-token window. Chunked pipeline parallelism stops sending the whole prefill sequence at once, chunking long prompts and processing them in parallel. A second, orthogonal dimension of chunking splits sequences across GPUs. And sparse attention optimization (HiSparse) processes a full KV with a hot buffer, so sparse attention occupies less memory and yields much higher throughput.

Combining all of this with tuning, his concrete result: on GLM-5.2 the day-zero performance was already best-in-class, and over time they achieved more than a further 2.2× improvement, reaching up to about 500 tokens/s per user.

Summarizing the inference half, he lists what SGLang introduced first at large production scale: RadixAttention, speculative decoding, PD disaggregation, sparse attention, native Miles integration, and more recently D-Flash, DSpark, and HiSparse.

Training: three challenges in agentic RL (~04:09–04:13)

Environment diversity has grown sharply. It is not just more tool calls and longer context — some environments are very hard to execute, and some may even require lab experiments to produce a reward signal. RadixArk's response has been close collaboration with most environment suppliers so Miles natively supports the major open-source environments, among them Prime Intellect's verifiers, Daytona, and NVIDIA NeMo Gym.

Train–inference mismatch is a very popular topic in frontier labs right now and one of Miles's biggest focuses. If there is a large mismatch between your training and inference engines, it will naturally and silently turn your RL more off-policy and hurt the entire training run. Three layers of fix:

1. Deterministic kernels in the inference engine — a prerequisite for exact token-level agreement between inference and training engines.
2. Chat template fixing — token-in-token-out, so multi-turn agentic chat templates stay aligned and tokens from prior turns match exactly, with no drift introduced by re-tokenization.
3. Algorithmic fixes — rollout routing replay and truncated masked importance sampling, both aimed at stable training at large RL scale.

Throughput is the third: higher throughput means faster iteration across more experiments and better GPU utilization. This one is mostly engineering discipline — they provide day-zero support for most models and verify each with their own in-house training runs, checking that KL stays controlled and reward actually goes up, so users can adopt the framework without that concern.

Two recent investments close the talk. Fully asynchronous RL now has sophisticated, mature support: overlapping rollout with training time and disaggregating the two to make the best use of GPUs, compared against the synchronous case. And low-precision training: rollout in lower precision with some form of quantized training on the backend. They natively support 8-bit and 4-bit rollout, and recently worked with a hardware partner to support NVFP4-native rollout in the RL stage with no performance loss.

Quotes

"2026 is actually the year of agentic infrastructure." (~04:04)

His unifying header for both the inference and training challenges.

"If you have very large train–inference mismatch between different engines, then that will naturally and silently turn your RL to be more off-policy and hurt your entire training run." (~04:10)

The nastiest failure mode in agentic RL: it never raises an error, it just quietly makes your training worse.