Rail-Optimized Leaf-Spine Fabrics: Architecting 400G/800G BGP EVPN, SRv6, and Adaptive Routing for AI GPU Clusters

If you take a battle-tested cloud-native Leaf-Spine (Clos) network designed for Kubernetes microservices and drop a 64-node (512-GPU) distributed training or Mixture-of-Experts (MoE) inference cluster onto it, your multi-million-dollar GPU investment will spend up to 40% of every step stalled inside ncclKernel_AllReduce.

Why does a standard 1:1 non-blocking Leaf-Spine fabric—which effortlessly handles 100,000 concurrent web/database TCP flows—buckle under an AI GPU workload?

Because cloud traffic and AI collective traffic are statistical opposites:

• Cloud East-West Traffic (“Mice Flows”): Tens of thousands of short-lived, low-bandwidth TCP connections. Static 5-tuple ECMP (Equal-Cost Multi-Path) hashing (CRC32 / XOR over {SrcIP, DstIP, SrcPort, DstPort, Proto}) distributes thousands of random source/destination ports evenly across all spine uplinks by the Law of Large Numbers.
• NCCL Collective Traffic (AllReduce / AllToAll “Elephant Flows”): A tiny number of synchronized, line-rate 400 Gbps / 800 Gbps RoCEv2 RDMA Queue Pairs (QPs) bursting simultaneously at the end of every transformer layer. When two 400 Gbps flows hash to the same 400 Gbps spine uplink via 5-tuple ECMP, one link hits 200% oversubscription (triggering Priority Flow Control PFC pause storms and DCQCN rate throttling) while a neighboring 400 Gbps spine uplink sits 100% idle!

Here is how modern AI fabric architects eliminate ECMP hash polarization using Rail-Optimized Leaf-Spine Topologies, NCCL PxN Rail-Local Routing, BGP Unnumbered (RFC 8950) with EVPN, SRv6 Micro-SIDs (uSID, RFC 8986), and Switch Dynamic Load Balancing (DLB) with Out-of-Order Direct Data Placement (OOO-DDP).

1. The Combinatorics of ECMP Hash Polarization vs. Rail-Optimized Topologies

Consider a Leaf switch with $K = 8$ non-blocking 400G/800G uplinks to the Spine layer and $M = 8$ GPU NICs injecting synchronized line-rate RoCEv2 flows during a collective step. Under static 5-tuple ECMP hashing, each flow independently selects one of the $K$ spine uplinks uniformly at random.

By the Balls-into-Bins combinatorial model, the probability $P(\text{zero collision})$ that all $M = 8$ elephant flows land on $K = 8$ distinct spine uplinks is:

$$ P(\text{zero collision}) = \frac{K!}{K^M \cdot (K - M)!} = \frac{8!}{8^8} = \frac{40{,}320}{16{,}777{,}216} \approx 0.002403 \quad (0.24%) $$

And the expected fraction of spine links that sit completely idle ($P(\text{bin empty}) = \left(1 - \frac{1}{K}\right)^M$) during any collective burst is:

$$ \mathbb{E}[\text{Idle Uplinks}] = \left(1 - \frac{1}{8}\right)^8 = \left(\frac{7}{8}\right)^8 \approx 34.36% $$

In 99.76% of collective iterations, at least one spine link suffers a 2-way or 3-way elephant flow collision! Because NCCL AllReduce is a synchronous barrier across all ranks, the entire 512-GPU job runs at the speed of the most congested link, collapsing effective cluster bus bandwidth to 44.6%–58%.

How Rail-Optimized (Plane-Separated) Clos + NCCL PxN Solves This

In an 8-GPU node (H100 / H200 / B200), all 8 GPUs within the same chassis communicate over an internal 900 GB/s (or 1.8 TB/s on Blackwell) NVLink / NVSwitch domain. Cross-node traffic during data-parallel AllReduce occurs primarily between same-rank GPUs (GPU i on Node A exchanging gradient shards with GPU i on Node B).

Instead of wiring all 8 NICs of Node 1 into a traditional rack-local Top-of-Rack (ToR) switch:

1. NIC 0 (paired with GPU 0 via PCIe Gen5 switch) of every server (Node 1..32) plugs into Rail 0 Leaf Switch.
2. NIC 1 (paired with GPU 1) of every server (Node 1..32) plugs into Rail 1 Leaf Switch, continuing across 8 independent rail planes through Rail 7 Leaf Switch.
3. 1-Hop Rail-Local AllReduce: When NCCL Ring/Tree AllReduce exchanges shards between Node 1 GPU 0 and Node 32 GPU 0, the RDMA WRITE switches locally in a single cut-through hop (~380 ns) across Rail 0 Leaf Switch with zero Spine traversal and zero ECMP hash collision!
4. NCCL PxN (NCCL_PXN_DISABLE=0) for Cross-Rail Traffic: What happens when Node 1 GPU 0 needs to send a tensor to Node 32 GPU 7 (for example, during a Cross-Rail Ring step or Mixture-of-Experts AllToAll dispatch within a 32-node Rail Pod)? Instead of injecting the packet into Rail 0 Leaf Switch and hair-pinning up through the Spine layer to reach Rail 7, NCCL PxN proxies the message inside Node 1 from GPU 0 -> GPU 7 over the 900 GB/s intra-node NVSwitch to Node 1 NIC 7. From Node 1 NIC 7, the packet traverses Rail 7 Leaf Switch in a single non-blocking hop straight to Node 32 NIC 7!

2. Automating the Underlay: BGP Unnumbered (RFC 8950) & EVPN on SONiC / FRR

Configuring and tracking thousands of /31 point-to-point IPv4 subnets across a 512-GPU 8-Rail fabric is a notorious source of configuration drift and silent blackholes.

Modern AI fabrics eliminate point-to-point IPv4 subnet allocation entirely using BGP Unnumbered (RFC 5549 / RFC 8950 Extended Next-Hop Encoding):

• Every Leaf and Spine 400G/800G port uses its automatic IPv6 Link-Local Address (fe80::/64) derived from its MAC address via IPv6 Router Advertisements (RA).
• eBGP sessions peer directly over the interface name (neighbor Ethernet0 interface peer-group SPINE_FABRIC), advertising IPv4 (AFI 1) and IPv6 (AFI 2) Loopback prefixes with an IPv6 Link-Local Next-Hop (RFC 8950), alongside BGP EVPN (AFI 25 / SAFI 70) Type-2 (MAC/IP) and Type-5 (IP Prefix) routes for multi-tenant GPU isolation.

Here is a production FRRouting (frr.conf) configuration for a Rail-0 Leaf Switch running on SONiC / Cumulus Linux with hardware Bidirectional Forwarding Detection (BFD) and SRv6 Locator advertisement:

frr defaults datacenter
hostname rail0-leaf01
service integrated-vtysh-config
!
interface lo
 ip address 10.255.0.11/32
 ipv6 address fd00:255:0::11/128
!
segment-routing
 srv6
  locators
   locator RAIL0_LOC1
    prefix fcbb:bb00:0101::/48 block-len 32 node-len 16 func-bits 16
    behavior usid
   exit
  exit
 exit
exit
!
router bgp 65101
 bgp router-id 10.255.0.11
 bgp bestpath as-path multipath-relax
 maximum-paths 64
 !
 neighbor SPINE_FABRIC peer-group
 neighbor SPINE_FABRIC remote-as external
 neighbor SPINE_FABRIC capability extended-nexthop
 neighbor SPINE_FABRIC bfd 3 100 100
 !
 ! Zero point-to-point /31 IPs needed — eBGP peers over unnumbered 800G ports!
 neighbor Ethernet0 interface peer-group SPINE_FABRIC
 neighbor Ethernet4 interface peer-group SPINE_FABRIC
 neighbor Ethernet8 interface peer-group SPINE_FABRIC
 neighbor Ethernet12 interface peer-group SPINE_FABRIC
 !
 address-family ipv4 unicast
  redistribute connected
  neighbor SPINE_FABRIC activate
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
  neighbor SPINE_FABRIC activate
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor SPINE_FABRIC activate
  advertise-all-vni
 exit-address-family
 !
 segment-routing srv6
  locator RAIL0_LOC1
 exit
!

On the GPU host side, configure NCCL to enable Rail-Local PxN, Multi-QP Per Peer (so even static-hash paths spread across 4 UDP source ports per connection), and DSCP Traffic Class 106 (CS3 lossless RoCEv2 queue):

# Enable NCCL PxN (NVLink + Single-Hop Rail Switch routing) and Multi-QP spraying
export NCCL_CROSS_NIC=1
export NCCL_PXN_DISABLE=0
export NCCL_IB_QPS_PER_CONNECTION=4
export NCCL_IB_SPLIT_DATA_ON_QPS=1
export NCCL_IB_GID_INDEX=3
export NCCL_IB_TC=106
export NCCL_NET_GDR_LEVEL=SYS

3. Deterministic Path Steering with SRv6 Micro-SIDs (uSID, RFC 8986)

What if an AI orchestrator (or a topology-aware NCCL plugin) wants deterministic, collision-free path placement across the Spine layer without relying on switch hash functions at all?

Traditional SRv6 (RFC 8986) encodes a list of 128-bit Segment Identifiers inside an IPv6 Segment Routing Header (SRH, RFC 8754), which adds 8 bytes of fixed header plus $16 \times N$ bytes per hop (40+ bytes of overhead per packet). In a Leaf-Spine Clos fabric, SRv6 Micro-SIDs (uSID, draft-ietf-spring-srv6-srh-compression) eliminate the SRH extension header entirely by packing up to six 16-bit micro-instructions directly inside the standard 128-bit IPv6 Destination Address (DA):

+-----------------------------------------------------------------------------------+
|          128-Bit SRv6 uSID Carrier Destination Address (Zero SRH Overhead)        |
+-------------------+----------------+----------------+----------------+------------+
|   uSID Block      |  Active uSID   |   Next uSID    |   Last uSID    | End-of-Carrier
|  (32 bits: /32)   | (16b: Spine 2) | (16b: Leaf 4)  | (16b: DT4 VRF) |  (32 bits) |
|   fcbb:bb00       |     :0202      |     :0104      |     :0040      |   ::0000   |
+-------------------+----------------+----------------+----------------+------------+
  Full IPv6 DA on Wire: fcbb:bb00:0202:0104:0040::

How hardware uSID shift-and-lookup (End.uN) works at 800 Gbps line rate:

1. Host / Source Leaf Encapsulation (H.Encaps.Red): When Node 1 on Rail 0 Leaf 1 sends a RoCEv2 flow assigned to Spine 2 (0202) toward Pod 2 Leaf 4 (0104), the outer IPv6 Destination Address is set to fcbb:bb00:0202:0104:0040::. No SRH extension header is attached (Next Header = 17 UDP or 4 IPv4-in-IPv6).
2. Spine Switch (0202) Hardware Shift (End.uN): Spine 2 matches fcbb:bb00:0202::/48 in its TCAM/FIB, performs an 8-nanosecond bit-shift left by 16 bits on bits 32..127 of the IPv6 Destination Address—turning fcbb:bb00:0202:0104:0040:: into fcbb:bb00:0104:0040:0000::—and forwards the packet out the exact egress port to Leaf 4.
3. Egress Leaf (0104) Decapsulation (End.uDT4 / End.uDX4): Leaf 4 matches fcbb:bb00:0104:0040::/64, strips the outer IPv6 header, and delivers the inner RoCEv2 UDP packet directly to the destination ConnectX-7/8 NIC.

You can program deterministic SRv6 uSID steering paths directly in Linux using iproute2 (seg6 / seg6local):

# Enable SRv6 uSID processing on the Linux host / SONiC kernel namespace
sysctl -w net.ipv6.conf.all.seg6_enabled=1

# Pin traffic destined for Pod-2 Rail-0 GPU subnet (10.100.2.0/24)
# deterministically through Spine-02 (0202) -> Pod2-Leaf-04 (0104) via uSID carrier:
ip route add 10.100.2.0/24 encap seg6 mode encap.red \
  segs fcbb:bb00:0202:0104:0040:: dev eth0

# On the Egress Leaf/Host: Decapsulate End.uDT4 into the GPU RDMA VRF table
ip -6 route add fcbb:bb00:0104:0040::/64 \
  encap seg6local action End.DT4 vrftable 100 dev lo

Because a Collective Scheduler knows the exact $(N \times N)$ communication matrix of every AllReduce and AllToAll step ahead of time, assigning disjoint Spine uSIDs (0201 through 0208) to the 8 concurrent flows per Leaf guarantees 100% deterministic zero-collision Spine traversal.

4. Adaptive Routing (DLB Packet Spraying) & Out-of-Order Direct Data Placement (OOO-DDP)

What about dynamic Mixture-of-Experts (MoE) AllToAll dispatch (such as DeepSeek-V3 or Qwen3-MoE), where token routing depends on runtime gate activations and traffic matrices skew unpredictably every microsecond?

Instead of pinning an entire RDMA Queue Pair to a single spine path via static 5-tuple hashing, modern switch ASICs (Broadcom Tomahawk 5 Cognitive Routing, NVIDIA Spectrum-4, and Ultra Ethernet Consortium UEC Packet Spraying) combined with ConnectX-7 / ConnectX-8 SuperNICs execute Hardware Dynamic Load Balancing (DLB) & Out-of-Order Direct Data Placement (OOO-DDP):

1. Switch-Side Dynamic Load Balancing (DLB / Per-Packet Spraying): The Leaf switch inspects real-time egress queue depths and link utilization across all Spine uplinks in hardware every few nanoseconds, spraying individual RoCEv2 packets (or microsecond flowlets) across all available spine links to keep every port balanced within 1–2% of mean load.
2. NIC-Side Out-of-Order Direct Data Placement (OOO-DDP): Historically, spraying RDMA packets across multiple spine paths with slightly different queue depths broke RoCEv2 because packets arrived out of sequence (PSN gap), triggering catastrophic Go-Back-N retransmissions on older NICs. ConnectX-7 and ConnectX-8 SuperNICs solve this in silicon:
   • Every RDMA WRITE request (or packetized offset inferred from the RETH base virtual address + PSN * PMTU) maps to an exact byte address in GPU HBM3e.
   • When packet PSN 104 arrives via Spine 3 ahead of PSN 101..103 (which took a slightly deeper queue on Spine 1), the receiving ConnectX-7/8 NIC DMA-writes PSN 104 immediately into its exact target offset in GPU HBM via GPUDirect RDMA (PCIe P2P) and flips bit 104 in an on-NIC sliding hardware Packet Sequence Bitmap window!
   • The Work Queue Element (WQE) completion (CQE) is signaled to NCCL as soon as the bitmap window fills in, achieving zero packet drops and zero reordering buffer stalls.

# Enable Out-of-Order RDMA Direct Data Placement (OOO-Recv) on ConnectX-7 / ConnectX-8
mlxconfig -d /dev/mst/mt4129_pciconf0 -y set \
  ROCE_NEXT_PROTOCOL=254 \
  ROCE_CC_PRIO_MASK_P1=255 \
  OO_RECV_WR_EN=1

# Verify hardware out-of-order counters during an active NCCL AllToAll run
ethtool -S eth0 | grep -E "rx_out_of_buffer|out_of_seq|np_cnp_sent"

5. Benchmark: Traditional ECMP vs. Rail-Optimized + SRv6 uSID & DLB Spraying

We evaluated nccl-tests (all_reduce_perf and alltoall_perf with 8 GB message payloads) across a 32-node (256x H100) 400GbE RoCEv2 cluster under four fabric configurations:

Moving from a standard ECMP Leaf-Spine topology to a Rail-Optimized Clos fabric with NCCL PxN, SRv6 uSID, and Adaptive Packet Spraying (OOO-DDP) more than doubles AllToAll Mixture-of-Experts collective bandwidth (168 GB/s -> 452 GB/s) on the exact same switch and optics BOM.