Topology as an End-to-End Contract: P/D Disaggregated Inference in llm-d on GKE
Production lessons from RDMA-backed Prefill/Decode serving across H200, B200, GB200, and GB300 on GKE
The Contract, and What Upholding It Is Worth
This post is the technical summary of a four-platform bring-up story, and the story converges on one systems lesson: topology is not only a transport optimization; it is an end-to-end contract spanning allocator, kernel, NIC, transport, engine, and router. Concretely, the contract imposes four obligations: topology must be discovered by the platform, preserved through resource allocation, verified at runtime, and consumed by the router — the allocation, verification, and routing sections that follow are one production stack's evidence for each.
Disaggregating LLM serving into separate Prefill and Decode (P/D) phases changes what kind of system you are running: the KV state that decode needs often has to leave the accelerator domain that produced it. The failures we hit while bringing up llm-d on GKE shared one shape: topology information the hardware knew was dropped somewhere between the allocator, the runtime, and the router, and the loss surfaced far from its origin. The bring-up itself was a first: llm-d's P/D guide had no GKE RDMA recipe before this work introduced one on A3 Ultra, then carried the same allocation-and-transport contract through A4, A4X, and A4X Max (allocation).
Three results from our own environment frame what upholding the contract is worth:
- Restoring the verified RDMA (Remote Direct Memory Access) path cut mean KV-transfer time by more than 93% versus the silent-fallback path it replaced.
- On a separate H200 benchmark cluster (evaluation), 436 of 5,400 requests on an intentional TCP baseline (UCX's TCP path over the datacenter network, DCN) timed out, while the RDMA configuration on the same cluster completed all 5,400.
- Mean time-to-first-token (TTFT) descended a ladder on one cluster: 2.15 s under unconstrained placement, 1.157 s with deterministic DRA allocation and the RDMA channel verified end-to-end, and 0.33 s with topology-scored P/D pairing on top (an experimental router prototype — routing) — a further 3.5x mean improvement. (All means; percentiles and provenance in evaluation.)
An earlier llm-d post, Networking for Distributed Inference in llm-d, covered the transport stack itself — NIXL (NVIDIA Inference Xfer Library), UCX (Unified Communication X), and backend performance; this post covers how topology survives from resource allocation all the way into routing.
