Topology as an End-to-End Contract: P/D Disaggregated Inference in llm-d on GKE

Production lessons from RDMA-backed Prefill/Decode serving across H200, B200, GB200, and GB300 on GKE

The Contract, and What Upholding It Is Worth

This post is the technical summary of a four-platform bring-up story, and the story converges on one systems lesson: topology is not only a transport optimization; it is an end-to-end contract spanning allocator, kernel, NIC, transport, engine, and router. Concretely, the contract imposes four obligations: topology must be discovered by the platform, preserved through resource allocation, verified at runtime, and consumed by the router — the allocation, verification, and routing sections that follow are one production stack's evidence for each.

Disaggregating LLM serving into separate Prefill and Decode (P/D) phases changes what kind of system you are running: the KV state that decode needs often has to leave the accelerator domain that produced it. The failures we hit while bringing up llm-d on GKE shared one shape: topology information the hardware knew was dropped somewhere between the allocator, the runtime, and the router, and the loss surfaced far from its origin. The bring-up itself was a first: llm-d's P/D guide had no GKE RDMA recipe before this work introduced one on A3 Ultra, then carried the same allocation-and-transport contract through A4, A4X, and A4X Max (allocation).

Three results from our own environment frame what upholding the contract is worth:

• Restoring the verified RDMA (Remote Direct Memory Access) path cut mean KV-transfer time by more than 93% versus the silent-fallback path it replaced.
• On a separate H200 benchmark cluster (evaluation), 436 of 5,400 requests on an intentional TCP baseline (UCX's TCP path over the datacenter network, DCN) timed out, while the RDMA configuration on the same cluster completed all 5,400.
• Mean time-to-first-token (TTFT) descended a ladder on one cluster: 2.15 s under unconstrained placement, 1.157 s with deterministic DRA allocation and the RDMA channel verified end-to-end, and 0.33 s with topology-scored P/D pairing on top (an experimental router prototype — routing) — a further 3.5x mean improvement. (All means; percentiles and provenance in evaluation.)

An earlier llm-d post, Networking for Distributed Inference in llm-d, covered the transport stack itself — NIXL (NVIDIA Inference Xfer Library), UCX (Unified Communication X), and backend performance; this post covers how topology survives from resource allocation all the way into routing.

Why P/D Makes Topology Serving State

In a node-local monolithic deployment, weights, KV cache, and every intermediate byte live inside one accelerator domain. Disaggregation breaks that assumption: prefill and decode scale and fail independently, and the KV state decode requires often crosses worker and accelerator boundaries. The moment it does, placement, path, and policy converge on one end-to-end latency — which GPU and NIC each worker holds, which rail (the network plane behind its GPU's NIC) its bytes traverse, and which prefill/decode pair the router selects are no longer independent knobs but one composed path.

Figure 1: Same hardware, three KV-cache transfer paths (the layout is illustrative — it shows the routing paths, not production node shapes). The labeled GPU boxes are GPUs — orange P pods prefill, green D pods decode; the small chips at each node's edge are their rail-local NICs (one labeled), each on its own rail; a decode pod spanning multiple GPUs owns each of their NICs — which is why cross-rail pairing exists at all. Rail-aligned (A) runs straight down one lane, prefill NIC to a decode NIC on the same rail; cross-rail (B) changes lanes through the switching fabric to reach a decode NIC on another rail; the same-node hairpin (C) — prefill and decode sharing one node — still exits to the fabric and comes back to its own decode NIC. Each panel draws a single rank-to-rank transfer leg; in the TP=1 prefill / TP=4 decode setup evaluated later (TP: tensor parallelism), one request spans several such legs. How far up the switching tiers each path actually climbs depends on placement — node distance, rack boundaries — not on the path type alone.

Co-location does not buy NVLink. It is tempting to assume co-located prefill and decode pods get NVLink KV transfers via CUDA IPC, as in a single-process engine. In practice, separate pods do not automatically receive a usable CUDA IPC/NVLink path; enabling one requires additional device-domain and runtime configuration. The boundary is a property of Kubernetes pod isolation: crossing it takes deliberate configuration — either hostIPC, which genuinely relaxes pod isolation, or a ComputeDomain backed by IMEX, NVIDIA's inter-node memory-export service — the community's supported, isolation-preserving direction for cross-pod NVLink, as NVIDIA's own guide to multi-node NVLink on Kubernetes describes.

What we measured when we bridged it. On GB200 we tested cross-pod KV transfer over the NVLink fabric, bridged through exactly that ComputeDomain mechanism. Functionally it was correct: transfers completed reliably. But the cross-pod software path we tested (UCX carrying the transfer over its CUDA transports, across the ComputeDomain) delivered roughly one-sixth to one-tenth of the throughput of the RoCE (RDMA over Converged Ethernet) path. Two hypotheses fit: the youth of the cross-pod software path itself, and contention — bulk KV streams sharing links with the µs-scale tensor-parallel collectives that gate every decode step. The public record keeps the contention hypothesis live: NVIDIA documents access and fault isolation for NVLink — link partitions, IMEX domains, ComputeDomains — but no per-flow traffic classes or bandwidth-isolation controls of the kind its InfiniBand and Ethernet lines treat as first-class, and contention-based NVLink side-channel research (arXiv:2404.03877; NVBleed, arXiv:2503.17847) is consistent with that gap. RoCE at least exposes the knobs: on GKE, collective data and control traffic ride dedicated traffic classes (NCCL_IB_TC=52, NCCL_IB_FIFO_TC=84) while bulk KV rides the default class. Until controlled mixed-load microbenchmarks separate the two hypotheses, the production call is conservative: bulk KV stays on the path whose traffic-class controls we can configure and whose delivered behavior we have verified. Our early manifests set those markings explicitly; the GKE platform's default NCCL stack now sets them, so the shipped recipes no longer need to — we removed the explicit settings in llm-d#2058 once the default landed. Other environments may not share that default.

The hairpin. With bulk KV on RDMA, even same-node neighbors take a long path: prefill GPU → PCIe → rail-local NIC → up into the switching fabric — the rail's top-of-rack (ToR) switch, and depending on the destination rail, the spine — and back down to the decode GPU's NIC. The additional traversal is single-digit µs — small at request scale, material at RDMA scale, where it recurs inside a high-fan-out transfer pipeline. The path also depends on physical facts — rail identity, PCIe roots, platform hairpin support — that a topology-blind scheduler never sees.

Where this applies. The co-location and hairpin concerns arise only where prefill and decode pods can share a node — partial-node deployments such as gpt-oss-120b, which we serve with single-GPU prefill and four-GPU decode workers on eight-GPU H200 hosts. Full-node deployments occupy every GPU on the host — all eight of an H200 node, for example — so the hairpin cannot arise. Everything else in this post binds regardless of model shape: rail alignment, transport verification, and topology-aware routing govern the cross-node path every KV transfer takes.

One Contract, Four Platforms

Every platform on this stack answers to the same rule: resource allocation has to be topology-deterministic, and the placement it produces has to stay visible all the way up. What differs, platform to platform, is how much of that they can express today.

The hardware makes the invariant concrete. GKE A3 Ultra pairs each H200 GPU with a dedicated ConnectX-7 NIC on its own PCIe path — eight 400 Gbps RoCE rails per node — and the default Kubernetes scheduler knows nothing about rails: early on, generic resource requests packed pods arbitrarily, and transport broke before a single KV block moved (UCX connection timeouts, RDMA_CM_EVENT_ADDR_ERROR during address resolution). What those early failures mapped, the hard way, was the constraint set of the full stack, not of the silicon: on the VPC platforms — A3 Ultra, A4, and A4X — both same-host hairpin and cross-rail RDMA are supported (probe-verified), yet a packet is only as routable as every layer beneath it. Running the address-resolution errors to ground landed on exactly such a layer: the workload's environment was missing its per-rail route and neighbor tables. The shared GKE base sets UCX_IB_ROCE_REACHABILITY_MODE=all for this case — instructing UCX to assume RoCE endpoints are reachable rather than run its own route check; the setting does not create reachability, the injected routing environment supplies the actual path — and the finding fed the fix: the routing environment has since been addressed on the platform side. Reachability established first, cost optimized above it; where a path is not available in the tested platform state, that too is per-platform data — the bare-metal hairpin case below.

The contract binds on four platforms — A3 Ultra and A4 use the same allocation pattern in this recipe, so what follows treats them together — with hardware details per Google's accelerator-optimized machine and GPU networking documentation. Two mechanisms recur below: DRA — Kubernetes Dynamic Resource Allocation, the API through which a claim requests devices and constrains their physical attributes — and CEL selectors, per-device Common Expression Language filters inside those claims:

Platform (GPU)	NVLink domain	RoCE NICs (per node)	Allocation today	Same-host hairpin
A3 Ultra (8× H200)	8 GPUs, node-local	8× CX-7 @ 400 Gbps, one per GPU	GPU + NIC in one DRA claim, pcieRoot-matched	Works (probe-verified)
A4 (8× B200)	8 GPUs, node-local	8× CX-7 @ 400 Gbps, one per GPU	Same as A3 Ultra (shared base overlay)	Works (probe-verified)
A4X (GB200 NVL72)	72 GPUs, rack-scale	4× CX-7 @ 400 Gbps (4 MRDMA VFs); 4-way rail topology	NICs are DRA-allocated (mrdma DeviceClass); GPUs come from the device plugin in the tested recipe — CEL selectors cannot lock a PCIe root there, so in-claim pcieRoot alignment applies once GPUs, too, are DRA-allocated	Works (probe-verified)
A4X Max (GB300 NVL72)	72 GPUs, rack-scale	4× dual-port CX-8 @ 800 Gbps (8 MRDMA PFs, two per card); 8-way rail topology	The same allocation split as A4X, plus a platform rule: each CX-8's two PFs bind as a pair (firstAvailable chain of CEL prefix matches) to bring the link up	Not available in the tested platform state (below)

The anatomy of one constraint. Same-host hairpin RDMA (send on one NIC, receive on another in the same host) is the constraint whose roots run deepest — and the clearest illustration of paths as per-platform data. Today the virtualized CX-7 platforms offer it (probe-verified); A4X Max, whose GB300 nodes run bare-metal, does not in the tested platform state — a difference that internal GKE analysis traced below Kubernetes: in the tested bare-metal path, local RDMA address resolution produced frames with identical source and destination MAC addresses, which the physical ToR dropped — kernel behavior meeting switch behavior, nothing Kubernetes can see. Constraints of this kind come with a platform's newness and its layering, and they evolve — like the routing environment above, only much deeper in the stack. Our part is the same on every platform: treat the constraint set as topology data — hairpin availability should surface as a per-platform attribute, never an assumed default — and align the deployment within it so the physical link delivers what it was built for. The expression is mechanical: rail-aligned allocation makes each endpoint deterministic; where a platform offers no hairpin today, anti-affinity keeps prefill and decode on separate nodes; where it does, the roles can co-reside — as the evaluation deployment does.

Within each platform's supported paths, we probed the placement penalty itself: roughly 2–3 µs per additional switch layer. That sounds small at request scale; it is not small at RDMA scale. In this workload a request-level KV handoff fans out across more than a thousand registered-memory entries and several rank-to-rank legs, with the engine grouping the selected descriptors into asynchronous NIXL transfers. The work is partially overlapped, so the cost is no literal per-packet sum — but finite queues, outstanding-operation windows, completion boundaries, and concurrent traffic leave repeated latency-sensitive points where path cost can affect transfer completion and queue-drain time, and transfer latency lands directly in TTFT. The evaluation measures the resulting end-to-end effect. Strict rail alignment also buys wire-up determinism: traffic stays on the path the network policy was engineered for.

None of this is unique to GKE. Some constraints reflect platform capability at a point in time. Every fabric carries its own constraint set — which paths exist, which need routing machinery behind them, which a security posture forbids outright; uniform any-to-any reachability, support, and cost cannot be assumed in a real cluster. That is why topology awareness earns its keep: the constraint set is per-platform data, and a deployment shape has to be built within it, not against it.

On A3 Ultra and A4, expressing the invariant needs nothing extra: GPU and NIC are both DRA-allocated, so we ask for the pair in a single ResourceClaimTemplate, co-constrained with matchAttribute: "resource.kubernetes.io/pcieRoot" — both devices guaranteed on the same physical PCIe root. We introduced that paired-claim pattern in llm-d#1821 — the guide's first GKE P/D recipe — and it has since been refactored into the shared gke/base that the platform overlays now build on (llm-d#2062); A4 inherited it through the shared eight-rail design. The pattern extends beyond the inference engine: llm-d's SGLang P/D guide adopted the same claim-and-constrain shape (llm-d#2059).

A4X is where that same claim has to wait: in the tested recipe, only the NIC side of the pairing lives in DRA. The RDMA NICs (and the NVLink compute domain) are DRA-managed — each worker's claim requests a dedicated NIC through the mrdma.google.com DeviceClass (llm-d#2043) — but GPUs still come from resources.limits through the legacy device plugin, whose placement hints reach NUMA granularity at best. With one side outside DRA, the pcieRoot co-constraint has to wait; once GPUs join the NICs under DRA, the same one-claim constraint above applies unchanged. (Managed DRANet, whose DeviceClass these claims use, already handles node-level, connectivity-aware scheduling — the finer, in-claim co-constraint is what's still out of reach.)

A4X Max inherits that same wait, then adds a requirement of its own. Each physical CX-8 exposes two physical functions (PFs), surfaced to pods as gpu0ipvlan0 and gpu0ipvlan1 — both mapping to physical gpu0; the pair must be bound together for the RDMA link to come up. We authored a dedicated overlay to bind it (llm-d#2068): a firstAvailable fallback chain, each alternative a CEL prefix match against an interface group (nic-group-0 through nic-group-3), so whichever alternative wins, both PFs land on the same physical device.

A word on versions: firstAvailable arrived with DRAPrioritizedList as an alpha feature in Kubernetes v1.33; DRA's structured-parameters redesign had landed earlier, in v1.31. Measurement provenance is described in the evaluation.

Figure 2: One contract, four platforms; A3 Ultra and A4 use the same allocation pattern in this recipe and share the first column. The gray bar is the invariant itself — GPU and NIC anchored to the same PCIe root; solid anchors are alignment guarantees held today, dashed anchors wait on GPU DRA; the green bar inside the CX-8 box is the PF pair binding — a device-level connectivity guarantee, not PCIe-root alignment. A3 Ultra/A4 co-constrain GPU and NIC on pcieRoot in one claim; on A4X the tested recipe DRA-allocates each worker's NIC explicitly, GPUs remain on the device plugin, and the alignment constraint waits; on A4X Max each CX-8's PF pair must additionally be bound together for the link to come up at all. Same invariant, three different expressive limits.

A KV-cache transfer does not care that every device was "correctly" allocated; it cares whether the bytes cross one PCIe switch or an entire network. Topology preservation must be a contract the platform upholds, not a patch reapplied per platform — the closing section picks up the upstream path to making it one.

Verify the Transport You Actually Have

The earlier llm-d networking post argued that distributed inference should verify its network paths rather than trust configuration. Here is the practice that principle became in our deployments — and the design reality that motivates it. UCX, the transport layer beneath NIXL, treats transport selection as a preference list: if the preferred RDMA path cannot be established — a misconfiguration, a failed link establishment — it quietly moves to the next transport that still works, TCP included. That is a deliberate design for reachability, and on general-purpose infrastructure it is the right default. A fabric engineered for GPUDirect RDMA inverts the trade: there, the same fallback silently gives up the performance the fabric was built for — nothing errors, and only UCX's own debug logs record the choice.

Silence is what makes this expensive. A fallback engaged under P/D load surfaces far from its origin: transfers stay slow but alive, so connections hold, health checks pass, and requests keep arriving, while resources held by in-flight transfers accumulate until decode workers are OOM-killed — the damage starts as latency and ends as OOM, two subsystems away from its cause. We met exactly this during bring-up: before RDMA was fully established on A3 Ultra (H200), transfers fell back onto the still-admissible TCP path — averaging well over 100 ms with peaks stretching into seconds instead of the fabric latency GPUDirect RDMA was supposed to deliver, and nothing errored. Nor is configuration the only trigger: on A4X, we traced a later fallback to the process loading the workload image's bundled verbs userspace ahead of the host's own libraries — a cause no manifest review could reveal and one that required process-level inspection to unwind. Binding the process to the host-matched verbs stack through LD_PRELOAD and the host-driver mounts shipped in llm-d#2043 repaired that failure; it did not make the next one loud — that is the allowlist's separate job below.

The practice. Decide the transport set instead of inheriting the preference list: an explicit UCX_TLS allowlist — rc_mlx5,rc,cuda_copy,cuda_ipc,sm,shm,self, which we contributed to the recipes in llm-d PR #2043 — removes TCP entirely and restricts wire transports to the reliable-connection (RC) family, plus the CUDA and shared-memory paths needed for local staging (UD remains admissible for UCX's connection bootstrap via the rc alias, but never carries KV data). Pinning buys two things at once. Early failure: a path that cannot be established fails at connection setup, loudly, before load ever arrives. And a stable link: with the transport family pinned — and UCX logs showing rc_mlx5 as the selected data path in our runs — the wire you engineered — the policy routing, the traffic classes — is the wire you get, run after run. The allowlist ships in the A4X-family recipe overlays; our benchmark deployments set the same value explicitly.

Result. With the link failure fixed and the allowlist in place, UCX logs and port counters confirmed that KV traffic rode the RDMA path throughout the run. The engine's application-level telemetry reported transfers averaging ~48 MB completing at ~6.6 ms mean (P90 ~8 ms), down from the 100-ms-plus fallback mean — a more-than-93% reduction. These are per-transfer application metrics — not per-packet measurements, and not the duration of an entire request-level handoff, which spans more than a thousand registered-memory entries across its rank-to-rank legs. The multi-second peaks disappeared outright, because the pathway that produced them no longer exists, and transfer backlogs now clear promptly under sustained load.

When development calls for a closer look, the transport's own evidence goes deeper than any dashboard — each check takes minutes on a live pod:

# 1. Transport actually selected: run workers with
#      UCX_LOG_LEVEL=info UCX_PROTO_INFO=y

# 2. Bytes on the wire: during a transfer, RDMA port counters climb
#    while eth0 carries only modest pod control-plane chatter —
#    KV-scale volume there means a fallback path is live.
watch -n1 'cat /sys/class/infiniband/*/ports/*/counters/port_xmit_data; \
           cat /sys/class/net/eth0/statistics/tx_bytes'

# 3. Read the trail generally: at every stage — bring-up, allocation,
#    connection setup — check worker logs and UCX's own routing
#    output; the transport it names is the transport you have.

Together, the checks read the chain both upstream and downstream of a transfer. The llm-d community is productizing this habit as preflight checks that gate startup on verified network paths.

Metrics close the loop. A serving stack is never static: drivers update, kernels roll, topologies change, new neighbors arrive on the fabric — and a regression can originate at any layer long after bring-up verified everything. That is a monitoring job, not a log-reading job: standing, real-time telemetry on the RDMA NICs, with alerting on it, catches what the startup gate and the bring-up checks cannot. Three properties — learned across these bring-ups — make that telemetry genuinely useful. Per-NIC granularity, keyed to the hardware device (mlx5_0, or its PCI address): a rail-aligned architecture pairs each GPU with its own NIC, so per-device counters are what confirm traffic is riding the intended rail and what reveal imbalance across interfaces — an aggregate bucket can do neither. Congestion and error counters alongside throughput: throughput shows traffic is flowing, while Congestion Notification Packets (CNPs) and RoCE retry/drop counts are often the clearest window into fabric contention when heavy workloads share rails. And resolution: RDMA works at microsecond scale, so minute-level aggregation smooths away short-lived bursts — fleet dashboards can stay coarse, but a node-local, high-frequency endpoint pays for itself during active debugging. These are also the counters we monitored during earlier debugging.

Configuration intent is not runtime evidence; the only transport you have is the one your telemetry proves you are using.

Topology Must Reach the Router

Rail-aligned allocation and a verified RDMA path set the floor; the router is the lever above them. Pairing choice — which prefill worker feeds which decode worker — decides which of the fabric's paths each transfer rides, a higher-level degree of control that the evaluation measures at a further 3.5x in mean TTFT. So topology has to reach the router — and the policy matters as much as the mechanism: what follows is the policy we converged on, and the engineering reality behind it.

Two rules define the serving policy we converged on:

• Feasibility is a hard filter. Reachability comes first: paths the platform does not offer — a hairpin unavailable in the tested platform state, an unbound NIC pair — are excluded outright, exactly as the constraint set dictates.
• Cost is a soft score. Among feasible paths, prefer the pairing with the lowest expected transfer cost — locality is the leading signal, not the only one — and spill over to a more distant worker when the preferred domain saturates, rather than queue behind a "perfect" path.

llm-d's router composes filters and scorers into per-workload scheduling profiles, moving from a many-signal weighted blend toward profiles built around a dominant signal with load guardrails. Dominant topology weighting worked here — deterministic pairing is exactly what our benchmark ran, and it performed with the preferred-locality pools holding headroom throughout (evaluation). The question is what generalizes: a fleet does not promise aligned headroom everywhere — pools saturate, workloads share clusters — and a hard constraint turns every capacity dip into queuing behind a "perfect" path. Soft scoring fits that engineering reality, because the topology score composes with the router's load- and cache-aware signals: the nearest pair is preferred, and selection degrades gracefully to a more distant one when local capacity runs out. In our experimental router build, the topology score joined that composition as one more scorer. Locality is an optimization; capacity is an invariant. And how a topology signal composes with the shipped scheduling profiles — when locality should dominate and when it must yield — is itself a promising research question.

The mechanism keeps concerns strictly separate: the router never talks to cloud-provider hardware APIs. Something outside it — a DRA plugin, a webhook, an operator — discovers the physical layout and attaches locality metadata to each pod; the router only consumes that metadata during its two-stage, decode-first selection flow, scoring each prefill candidate's proximity to the already-selected decode pod. For the benchmark the topology score carried a dominant, effectively hard weight: a prefill pod outside the target decode pod's locality domain was not selected while capacity remained in the preferred locality domain — the cleanest way to isolate the effect; at this load the preferred-locality pools had headroom throughout. Two boundaries keep the claim precise. Each TP=1 prefill worker owns exactly one rail-pinned GPU-and-NIC endpoint (a per-rail DRA claim), so choosing the pair fixes the sending lane of every transfer leg; the TP=4 decode pod it feeds is multi-homed across four rails, and which decode rank receives each KV shard — hence where each leg lands — is the transport's business, not the router's (rank-level selection is the open problem this section closes with). What the 0.33 s arm measures is therefore deterministic pod-level topology pairing over individually rail-pinned endpoints — and that is the finding: pod-level determinism alone, with no rank-level control, moved the mean and the tail this far. Deterministic weighting was the right experimental policy, and it performed; for production the design keeps locality as a soft score and the encoding pluggable — fleet-scale flexibility is the deciding factor: topology enters as one score among load and cache signals, and the design expects each provider to supply an injection plugin that models its own fabric's geometry.

One point of status keeps the numbers honest: what we benchmarked is a prototype, not a shipped feature. In our experiment, a dot-separated label (llmd.ai/topology-cidr, e.g. us-central1.spine1.leaf4.host12; despite the name, a hierarchical locality path, not an IP CIDR) ran macro to micro — populated manually — and a longest-prefix-match (LPM) scorer, patched into the router for the experiment, computed proximity as common-prefix depth. Adopting the approach in llm-d proper requires two pieces: a routing plugin that scores on topology, and a network plugin that injects the topology labels. Our design deliberately keeps label injection outside the router, because how a network encodes topology — rail fabrics, NVLink domains, TPU ICI slices — differs by provider, and each provider should ship its own injection plugin. The prototype scores pod-level locality — host, leaf — not rank-level rail selection; what it establishes is the size of the opportunity, and the evaluation reports the numbers in full. Upstream, the llm-d community is landing its own, structurally different topology model in llm-d-router. A topology-extractor datalayer plugin has merged, stamping each endpoint with hostname, rack, zone, and region drawn from standard Kubernetes labels (#1678). A topology-affinity filter (threshold-based and configurable, failing open when topology data is unavailable — the right default for a performance signal; reachability, per the rules above, is the part that must not fail open) and a topology-affinity scorer, which consume those attributes during P/D selection, have merged as well (#2299). The label-injection side remains open — the provider-shipped injection seam our design anticipates. The encodings differ — a hierarchical path scored by prefix depth versus structured locality attributes — but the upstream plugins implement the same locality-composition direction our prototype converged on — topology filtered or scored as a performance preference — while provider-specific reachability constraints (a hairpin the platform does not offer, an unbound PF pair) still need separate enforcement through allocation, placement, or network policy.

The results, reported in full in the evaluation, land as a three-step ladder on one cluster: 2.15 s mean TTFT under unconstrained placement, 1.157 s with deterministic allocation and the verified channel — the NIXL/UCX/rc_mlx5 verification work — and 0.33 s with topology-scored pairing on top, a further 3.5x in mean TTFT with throughput held within ~1–2% of the offered 45 QPS. Percentiles, provenance, and attribution live there.

Figure 3: The topology-disabled versus topology-enabled serving stack on the same cluster — a bundled comparison: unconstrained scheduling ran at 2.15 s mean TTFT (P90 3.92 s); deterministic allocation, the verified channel, and topology-scored pairing (the experimental prototype above) together brought it to 0.33 s mean (P90 0.41 s), with the channel-verified middle tier (1.157 s) separating the two steps in Figure 4. Arrows show pairing choice, not the wire path — same-node transfers still exit to the fabric (Figure 1).

One open problem stands out, and it is conditional. Today the scorer chooses among pods; for wide expert-parallel, full-node deployments the consequential decision is one level finer — which decode rank (which specific GPU within the selected worker) receives a given KV transfer, since rank choice determines the NIC, rail, and PCIe path. If future experiments bear out its value, extending topology scoring to decode-rank selection is the natural next step, so that scheduling-time placement, request-time routing, and rank-level transfer targeting consume the same locality metadata.

Evaluation and Ablations

This section supplies the evidence from one cluster: a controlled transport-path ablation — a deliberately configured DCN/TCP path against the verified RDMA path — and a routing comparison measured on the same cluster and workload, with llm-d's published baseline alongside as a reference point.

Setup. The benchmark served gpt-oss-120b on 16× H200 GPUs across two GKE A3 Ultra (a3-ultragpu-8g) nodes — eight prefill replicas (TP=1), two decode replicas (TP=4) — driven by inference-perf through the llm-d-benchmark harness. The workload is the guide's benchmark, unmodified: 45 QPS constant for 120 seconds — 5,400 requests with fixed 5,000-token inputs and 250-token outputs — after which the harness drains outstanding requests against a 300-second client timeout. The direct three-arm ladder reports one 120-second window per arm, and its 1.157 s-to-0.33 s comparison comes from a fixed deployment; for the topology-scored arm we additionally report the 25-run repetition results below. Serving image: vllm/vllm-openai:v0.23.0; exact cluster and component revisions are recorded internally, the guide supports reproduction of the non-prototype setup (not bit-for-bit reconstruction of these runs), and the topology-scored arm is not yet publicly reproducible. One placement note (see co-location above): at this replica count each 8-GPU node hosts four prefill workers and one decode worker — co-residence is a consequence of replica counts, not something the recipe enforces.

Transport ablation: DCN/TCP versus RDMA on the same cluster. To put a number on what the verified path is worth, we ran the identical workload over a deliberately configured DCN/TCP path — KV transfers on the cluster's standard datacenter network instead of the RDMA rails, the same UCX/DCN transport family a silent fallback lands on (verification). The TCP run collapsed under transfer-queue saturation: 436 of 5,400 requests timed out, and mean request latency among completed requests reached 223.06 s — requests were completing long after the 120-second load window; the system was queuing, not serving. This is UCX's TCP path, not TCP generally. The RDMA configuration completed all 5,400 requests with zero timeouts — this is the verified channel that serves at the ladder's 1.157 s mean-TTFT tier (the routing comparison below). The verified path hit peak wire utilization of 359.35 Gbps — roughly 90% of the ConnectX-7 400 Gbps line rate — with UCX logs and port counters confirming RDMA throughout.

Routing comparison: control versus topology-scored pairing. All three rungs of the ladder were measured on this cluster: the unconstrained control (2.15 s mean TTFT, P90 3.92 s); the channel-verified tier — deterministic DRA allocation with the verified RDMA channel, no topology-aware routing — at 1.157 s mean (P90 1.587 s); and topology-scored pairing at 0.33 s mean (P90 0.41 s). The channel-verified tier is also where llm-d's published P/D baseline now sits on the same 16× H200 GKE A3 Ultra class with the same 45 QPS harness — a baseline our DRA and transport updates helped move down from its earlier published level; the older figure predates this generation of environment changes.

Repetition and stability. The topology-scored result is not one lucky window: we re-ran it 25 times with the same deployment configuration — same manifests, same scripts, same topology shape — deliberately spread across different hosts and regions, reducing the likelihood that the observed level was specific to any one host or regional fabric. It held steady: mean TTFT averaged 0.33 s across runs (range 0.22–0.39 s, standard deviation 0.04 s), per-run P90 averaged 0.45 s (standard deviation 0.05 s), and the reported run's mean TTFT rounds to the across-run mean. These 25 runs are operational validation of the topology-scored arm alone — distinct from the ladder's direct one-window-per-arm comparison.

The ladder reads as two steps. The first step — 2.15 s to 1.157 s — comes from the bundled allocation-and-transport change: the control's unconstrained allocation let pairings and network paths vary draw by draw — reachable, but riding longer and more variable routes than the rail-aligned lanes the fabric is engineered around (allocation); the router configuration was identical across these two rungs. The second step — 1.157 s to 0.33 s in mean TTFT, a further 3.5x, with P90 falling from 1.587 s to 0.41 s — comes with topology-scored pairing, in its deterministic, hard-weight experimental form, on the same deployment. At RDMA scale a 2–3 µs per-layer difference is not negligible: the request-level handoff spans more than a thousand registered-memory entries and several rank-to-rank legs, the 45-QPS workload keeps transfers overlapping under load, and finite outstanding-operation depth, completion boundaries, and queue interaction keep path latency from being fully hidden. With deployment, workload, and verified channel fixed, the comparison attributes the end-to-end improvement to topology-scored pairing at the system level.

Figure 4: Left: the DCN/TCP configuration could not sustain the offered load; verified RDMA could. Right: the TTFT ladder measured on this cluster — unconstrained control, channel-verified, topology-scored. All means; percentiles and the published baseline references above.

Limitations. The direct three-arm ladder uses one reported 120-second window per arm — short and high-pressure, chosen to stress transfer queues; only the topology-scored arm carries the additional 25-run repetition results above. All routing arms ran on this cluster and platform; the published guide figures serve as reference points, not experimental arms. The testbed is two nodes, so the topology decision space is minimal: these numbers demonstrate the mechanism, not its ceiling — validating the scorer on multi-leaf clusters where topology genuinely varies is planned follow-up, along with a rail-aligned point-latency control row to join the micro- and macro-level evidence. Finally, whether disaggregation itself wins over aggregated serving for a given workload is out of scope: this post takes P/D as given and optimizes the network path underneath it.

Reproduction context. The public reference for this workload is the benchmark report in the llm-d P/D disaggregation guide. The reproducible recipes — resource claim templates, transport configuration, and benchmark methodology — live in the guide. The topology-scored arm is an internal prototype measurement, not yet a public recipe. The measurement establishes that topology is a materially useful routing signal under this workload; composing it as a soft score is the engineering judgment for fleet-scale deployment, where aligned headroom cannot be assumed; saturation-level validation is follow-up, and upstream llm-d-router has independently converged on the same soft-preference direction.

Design Principles and What Comes Next

The layer where a failure appears is not necessarily the layer where it originates. A silent transport fallback — any break dropping KV transfers onto a still-admissible TCP path — surfaced first as latency and, at its terminal state, as decode-side memory exhaustion, with every intermediate layer reporting healthy (verification). Missing topology at allocation time surfaced as connection failures before a single KV block moved (allocation). Topology-blind pairing surfaced as tail TTFT, far from any component a network audit would inspect (routing). In each case the symptom arrived layers away from its cause: each component looked locally valid while the composed path violated the end-to-end invariant. That is why topology has to be verified as a composed path rather than audited one layer at a time, and why we state these lessons as a contract between layers rather than a checklist inside any one of them.

Four obligations keep that contract intact.

Discover — the platform must expose real physical topology, not just device inventories. PCIe root, NUMA node, rail and NIC identity are the attributes every per-platform expression in the allocation section existed to recover, and the ones our claim templates consume.

Preserve — resource abstractions must carry physical attributes forward, not launder them into logical identities. The four platforms of the allocation section supply the invariant to test against any allocation API: can the scheduler co-constrain two devices on a shared physical attribute?

Verify — runtime evidence, not configuration intent, is the source of truth for the transport path. The verification checklist exists because a manifest can declare GPUDirect RDMA while every transfer rides eth0; only runtime signals reveal the breach.

Consume — the serving layer must ingest topology metadata and act on it. Graceful degradation is not a fifth principle; it is the policy rule that governs this one, and the one the routing section shows is easiest to get wrong: path feasibility and capacity are hard constraints; among feasible paths, locality is a soft preference — score cost softly, and spill over before you queue.

One note of proportion belongs at the end of this list: topology awareness is an extremely complex engineering problem, and the network is only one axis of it. The same serving decision that scores proximity must also weigh upstream and downstream load-bearing capacity, KV-cache state and the reuse it enables, per-node pressure, and the load on the NICs and the network itself. What the contract does is organize the topology axis — discovered, preserved, verified, and consumable — so that a production placement policy can compose it with all the others instead of rediscovering it under each one. The spill-over rule under Consume is the first instance of that composition, not an exception to the contract.

Encouragingly, upstream Kubernetes has standardized the vocabulary Discover needs: KEP-4381 (Kubernetes Enhancement Proposal) establishes resource.kubernetes.io/pcieRoot as a cross-driver device attribute — designed for exactly the matchAttribute co-constraint the allocation section uses. The vocabulary is already crossing providers: Amazon EKS's EFA DRA driver documents the same matchAttribute: "resource.kubernetes.io/pcieRoot" constraint for aligning EFA interfaces with NVIDIA GPUs, while provider-specific attributes extend the mechanism to Neuron connectivity groups. The device names differ; the contract does not. The concrete ask is precise: device drivers, GPU and NIC alike, should publish it — the NIC side of which we have raised upstream in the DRA network-driver tracker (kubernetes-sigs/dranet#261); the GPU side requires the GPU driver to publish the attribute through DRA. Once both sides of a pairing carry that attribute, the platform-specific CEL fallback chains of the allocation section can collapse into a generic device claim co-constrained by matchAttribute — the form A3 Ultra and A4 already use. The hardware is moving as well: rack-scale NVLink fabrics, reaching Kubernetes workloads through ComputeDomains and IMEX, will reopen the NVLink-versus-RDMA calculus for KV transfer as the cross-pod software path matures. We read our GB200 measurement as a verdict on today's delivered path, not on tomorrow's.

The deepest lesson of the four-platform bring-up this post summarizes: per-platform assumptions rot as hardware evolves; topology should be discovered and consumed as data rather than baked into deployment logic. A stack that treats topology as data absorbs the next platform by learning new attributes; one that bakes it in starts over. From H200 to GB300, what changed was the per-platform expression of the allocation section; what carried over was the contract.

The reproducible recipes we contributed behind the allocation and transport results — DRA templates, transport configuration, verification checks, and platform overlays for GKE A3 Ultra/A4, A4X, and A4X Max — live in the llm-d P/D disaggregation guide. The guide ecosystem carries analogous per-provider network recipes — OpenShift RoCE, AWS EFA, and OCI among them; the four obligations are the portable part, and ours are one provider's expression of them. The topology-scored routing (above) is not among them yet — llm-d-router's own topology work is the upstream path to running it from a release. Try the recipes, measure what topology is worth on your own clusters, and join the discussion in the llm-d community.