Talk Session 2: AI Systems
Disaggregated LLM Serving with Shared Memory KV Cache at Rack-Scale
Jongryool Kim — Senior Director / Head of AI System Infra., SK hynix
Put a physically disaggregated memory pool in the middle of the rack so multiple nodes see the *same* address space, and the KV cache no longer has to be shuttled between nodes over RDMA — prefill writes once, decode reads directly, the old KV cache stays in the pool for the next request's reuse, and GPU HBM is freed immediately while PCIe contention disappears.
TL;DR
- Two modes, and they're betting on the second. The physically disaggregated pool supports (1) memory pooling, where each node dynamically allocates extra memory but regions stay isolated between nodes, and (2) sharing mode, where multiple nodes see the same memory address space and can access the same data. Sharing is what SK hynix cares about.
- Faster than RDMA. The usual upgrade path for inter-node data movement is TCP/IP → RDMA, but pooled-memory-based data sharing is faster than RDMA. And once the KV cache lands in the pool, GPU HBM is released immediately — even if the prefill or decode side hits OOM, prefill can keep going.
- Three compounding benefits: faster inter-node movement, immediate HBM release, and elimination of PCIe bandwidth contention on both the GPU and network side when storing and reusing KV cache. Applied to agentic AI serving — which must reuse large amounts of KV cache — they measured improvement over a Mooncake-based KV cache, with the caveat that these are very early numbers.
Key Points
The architecture: what "physically disaggregated" means here (~00:36:59–00:38:50)
Jongryool Kim is a senior director at SK hynix, and his subject is a new memory form factor: physically disaggregated pooled memory — a memory box that sits apart from the servers themselves.
In the diagram: several server nodes plus a separate memory-pool box that multiple servers use simultaneously, in one of two modes:
- Memory pooling — each node dynamically allocates additional memory, but each node's region is isolated from the others.
- Sharing mode — multiple nodes see the same memory address space, so every node can access the same data.
"We are very interested in this sharing feature," he said plainly — and that's the technical pivot for the whole talk.
They integrated the pool into an LLM serving system quickly and demonstrated the deployment at several events last year. The demo has four servers running LLM serving components, with a Niagara box in the middle of the rack (~00:38:45) as the actual physical pooled memory.
How the KV cache flows, and why it gets faster (~00:38:50–00:40:50)
The demo scenario uses the pool to transfer the KV cache:
- Run prefill first; once prefill completes, the KV cache must be delivered to the decode side. Two memory operations — a store and a load — deliver it to the decode node.
- The important part: the old KV cache stays in the memory pool, so the next request can reuse it directly with no additional store operation at all.
The result is a performance improvement over full recomputation, and over NIXL and in-server DRAM-based KV cache storage.
But he was more interested in why the benefit exists, and gave three reasons:
- Faster inter-node data movement. People generally use RDMA rather than TCP/IP, and RDMA is fast — but pooled-memory-based data sharing is faster than RDMA (~00:39:59), which is where the serving-system speedup comes from.
- GPU HBM can be released immediately (~00:40:11) once the KV cache is offloaded to the pool. The practical consequence: even if the decode side or the prefill side runs out of memory, prefill can continue, because that KV cache is already uploaded to the pool.
- Contention disappears. Storing and reusing the KV cache creates heavy contention on PCIe bandwidth on both the GPU side and the network side; routing the transfer through pooled-memory data sharing removes that additional contention.
Applying it to agentic AI serving, and what's next (~00:40:50–00:41:48)
Agentic AI serving is characterized by having to reuse a large amount of KV cache, so they applied the pooled-memory environment to an agentic AI serving system and measured improvement against a Mooncake KV cache (~00:41:11). He volunteered the caveat himself: these are very initial performance numbers.
Looking ahead (~00:41:20): by the end of this year, working with roughly 25 collaboration parties, they are preparing five more system PoCs, including multimodal serving and runtime memory management, aiming to build a real system with pooled memory for HPC-class national systems.
His closing line: "Let's see how the pooled memory can change the AI system."
Quotes
"This pooled memory based data sharing is faster than RDMA." (~00:39:59)
The core claim: not an alternative way to move data, but a faster one than today's fastest.
"We can release the GPU HBM immediately by offloading the key cache to the pool memory." (~00:40:11)
The KV cache stops being an HBM tax and becomes a shared asset sitting in the pool.
"Let's see how the pooled memory can change the AI system." (~00:41:45)
The talk's closing line, and a fair summary of where the approach stands: direction clear, evidence still accumulating.
提到的專案與資源 / Projects & Resources
| 名稱 Name | 說明 | Description | 備註 Notes |
|---|---|---|---|
| Niagara (SK hynix) | 放在機櫃中央的實體池化記憶體裝置,支援 pooling 與 sharing 兩種模式 | SK hynix's physical pooled-memory box placed mid-rack, supporting both pooling and sharing modes | 已查證:SK hynix 的 CXL 分離式記憶體原型(多埠、支援 memory pooling / sharing);演講未明說 CXL |
| Mooncake | 對照組:以 KV cache 為中心的分離式服務架構與其快取層 | Comparison baseline: KVCache-centric disaggregated serving architecture and its cache layer | 逐字稿 "moonake" |
| NIXL | 對照組之一,推論資料傳輸函式庫 | One of the comparison baselines; an inference data-transfer library | 逐字稿 "nxl";名稱待確認 / to verify |
| TraCT | 與本講題幾乎同名的論文:Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale | Paper with nearly the same title: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale | arXiv 2512.18194;演講中未提及名稱,關聯性待確認 / not named on stage, relationship to verify |
| RDMA | 目前節點間資料搬移的主流做法,作為效能對照基準 | The prevailing approach to inter-node data movement, used as the performance reference point |
逐字稿勘誤 / Transcript Corrections
| 字幕原文 Heard as | 應為 Should be |
|---|---|
| Jang R Kim / Junior Kim | Jongryool Kim |
| skinex / SKH Highix | SK hynix |
| Rexcale | rack-scale |
| disegregated / disagregated | disaggregated |
| prepare / prefare | prefill |
| keeper cache / cuber cache / key cache | KV cache |
| GPU HPM | GPU HBM |
| realm serving / random serving | LLM serving |
| moonake | Mooncake |
| nxl | NIXL(待確認 / to verify) |
| niagara | Niagara |
| five more system PC | five more system PoCs |
| multimord serving | multimodal serving |
| relate the GPU HPM | release the GPU HBM |
待確認 / To Verify
- 逐字稿的 "nxl and nm cache is in server DM based cable cash stoing" 判讀為「NIXL 與 in-server DRAM based KV cache storing」,但 "nm cache" 也可能是另一個專案名,需看投影片。/ The comparison baselines heard as "nxl and nm cache" — read as NIXL and in-server DRAM-based KV cache storage, but "nm cache" could be another project name.
- 講者自述「I also working for the SRC as a cell」——SRC 疑為 Semiconductor Research Corporation,職稱不明。/ He said he also works for "the SRC" in an unclear role; possibly Semiconductor Research Corporation.
- 最後提到的國家級系統:逐字稿為 "for the HPC US-based national system",究竟是美國或韓國的國家級系統待確認。/ The national-scale system mentioned at the end — whether US or Korean is unclear from the transcript.
- 對比 Mooncake 的效能提升幅度未在逐字稿中出現具體數字。/ No concrete numbers for the improvement over Mooncake appear in the transcript.
- Niagara 是 CXL 裝置,但演講逐字稿中未出現 "CXL" 一詞;此資訊來自外部查證。/ Niagara is a CXL device, but "CXL" never appears in the transcript; that detail comes from external verification.