2 posts tagged with "UCX"

Production lessons from RDMA-backed Prefill/Decode serving across H200, B200, GB200, and GB300 on GKE

The Contract, and What Upholding It Is Worth​

This post is the technical summary of a four-platform bring-up story, and the story converges on one systems lesson: topology is not only a transport optimization; it is an end-to-end contract spanning allocator, kernel, NIC, transport, engine, and router. Concretely, the contract imposes four obligations: topology must be discovered by the platform, preserved through resource allocation, verified at runtime, and consumed by the router — the allocation, verification, and routing sections that follow are one production stack's evidence for each.

Disaggregating LLM serving into separate Prefill and Decode (P/D) phases changes what kind of system you are running: the KV state that decode needs often has to leave the accelerator domain that produced it. The failures we hit while bringing up llm-d on GKE shared one shape: topology information the hardware knew was dropped somewhere between the allocator, the runtime, and the router, and the loss surfaced far from its origin. The bring-up itself was a first: llm-d's P/D guide had no GKE RDMA recipe before this work introduced one on A3 Ultra, then carried the same allocation-and-transport contract through A4, A4X, and A4X Max (allocation).

Three results from our own environment frame what upholding the contract is worth:

• Restoring the verified RDMA (Remote Direct Memory Access) path cut mean KV-transfer time by more than 93% versus the silent-fallback path it replaced.
• On a separate H200 benchmark cluster (evaluation), 436 of 5,400 requests on an intentional TCP baseline (UCX's TCP path over the datacenter network, DCN) timed out, while the RDMA configuration on the same cluster completed all 5,400.
• Mean time-to-first-token (TTFT) descended a ladder on one cluster: 2.15 s under unconstrained placement, 1.157 s with deterministic DRA allocation and the RDMA channel verified end-to-end, and 0.33 s with topology-scored P/D pairing on top (an experimental router prototype — routing) — a further 3.5x mean improvement. (All means; percentiles and provenance in evaluation.)

An earlier llm-d post, Networking for Distributed Inference in llm-d, covered the transport stack itself — NIXL (NVIDIA Inference Xfer Library), UCX (Unified Communication X), and backend performance; this post covers how topology survives from resource allocation all the way into routing.

Networking: The Critical Path in P/D Disaggregation​

llm-d's prefill-decode disaggregation unlocks significant efficiency gains by separating compute-heavy prefill from memory-bandwidth-heavy decode onto dedicated GPU pools. But it introduces a hard dependency on the network: the KV Cache must be transferred from prefill to decode before the first token can be generated. This transfer time lands directly on the Time to First Token (TTFT) — making networking a first-order concern for end-to-end inference latency.

This post dives into llm-d's networking stack — how it works today and how it's evolving in collaboration with NVIDIA.