PIVOT AI LAB
PROJECT PIVOT AI INFRASTRUCTURE ARTICLE 01
FIELD NOTES / AI NETWORKING / ARTICLE 01

From Nexus to AI Fabrics: What Actually Changes When Ethernet Meets GPU Networking

An enterprise network engineer’s field notes on SONiC, RoCE, 100GbE, and the operating demands of distributed GPU infrastructure.

September–October 2026 lab evidenceEthernet • RoCEv2 • SONiC • RDMAEditorial preview / no publication
2 × 3Virtual spine / leaf design
100 GbEPhysical NIC and switch port speed
+136,874Correlated ECN / CE test deltas
41.5×Observed congestion / baseline latency

I've spent much of my career in enterprise networking, where keeping a service available often comes down to understanding exactly which device is making a forwarding decision, how that decision changes under failure, and whether the application is actually recovering.

One production environment included 52 VXLAN/EVPN networks across the United States, built with an IS-IS underlay, iBGP EVPN, route-reflector spines, and border leaves. That experience remains directly useful when building GPU networks. BGP, ECMP, MTU, failure detection, and packet-path analysis do not disappear when the endpoints become accelerators.

What changes is the workload and the visibility required to operate it. Distributed training can cause synchronized exchanges among many endpoints. A fabric can retain healthy routing adjacencies while congestion slows communication enough to waste expensive compute cycles. RDMA introduces queue pairs, retries, congestion signals, and host-adapter behavior into a network engineer's troubleshooting path.

I built Project Pivot to test those differences rather than assume that knowing VXLAN and routing protocols would be sufficient. The work moved through a virtual SONiC Clos, Kubernetes and RDMA simulation, and physical 100GbE RoCE tests on a Dell switch and NVIDIA adapter. The results are more useful when the successful measurements and the test boundaries are presented together.

01 / Familiar protocols, different operating conditions

A conventional enterprise fabric may run an IGP beneath MP-BGP EVPN and VXLAN. My earlier production deployment used IS-IS as its underlay and iBGP EVPN with spine route reflectors. Those were deliberate choices for that design, not universal requirements for EVPN.

For Project Pivot I chose a two-spine, three-leaf SONiC Clos with eBGP in the underlay, ECMP, and BFD. Linux workload endpoints attached to the leaves. An EVPN/VXLAN overlay then provided a second set of validation targets: VLAN 100 to VNI 10100, Type-2 MAC/IP routes, and Type-3 inclusive-multicast routes.

Comparison of enterprise Cisco EVPN and Project Pivot virtual SONiC Clos
Figure 01. Two architectures, different design choices. The left describes prior production experience; the right is the virtual topology exercised in Project Pivot. Neither drawing claims the physical Dell Z9100 ran SONiC.

I injected link-down failures, silent packet loss while an interface remained up, and VTEP isolation. I inspected BFD, BGP, and EVPN state to determine whether traffic retained a usable path. A separate traffic experiment demonstrated the limitation of reading forwarding health from protocol state alone: the control plane could look stable while workload latency climbed sharply.

The most important transferable habit is to verify the data plane, not stop at an Established neighbor or an UP interface. AI fabrics increase the cost of getting that distinction wrong.

02 / SONiC exposes the forwarding software stack

Engineers who have spent years on Cisco Nexus and NX-OS know its operational model: commands, feature configuration, platform counters, forwarding state, and vendor-integrated hardware support.

SONiC offers a more explicitly componentized view. FRR, database services, Switch State Service, and syncd operate as separate processes or containers. Redis databases pass desired and observed state among components; orchagent processes state before syncd and SAI participate in forwarding-programming operations.

SONiC route programming pipeline
Figure 02. Simplified SONiC route programming path, from FRR through Redis and orchestration to the SAI-backed forwarding target. Real implementations contain additional agents and platform-specific behavior.

That architecture changes troubleshooting. A learned BGP route may exist in FRR while another stage of propagation or hardware programming is defective. The virtual environment let me test routing and switching logic, overlay behavior, BFD, and failure response. That software image could not validate physical ASIC queues, lossless buffers, or actual switch silicon.

This led to two separate, intentionally non-interchangeable labs: virtual SONiC for control-plane and overlay work, and a physical Dell Z9100-ON running SmartFabric OS10 for RoCE and congestion measurements.

03 / The physical 100GbE experiment

The hardware phase used a Dell PowerEdge R430 with Proxmox VE 9.1.9, one dual-port NVIDIA ConnectX-5 100GbE adapter, and a Dell Z9100-ON running OS10 10.5.0.6C1. Each physical NIC port connected to a separate 100GbE switch port via QSFP28 DAC. Two Linux network namespaces, gpu-a and gpu-b, provided endpoint contexts with IP addresses 10.100.0.1 and 10.100.0.2 on VLAN 100.

Both namespaces resided on one physical host and one physical adapter. Their names describe workload roles, not the existence of actual GPUs. The design exercised two physical NIC ports and a switch path; it was not a multi-server GPU cluster.

Physical 100GbE RDMA testbed with two namespaces on one host and dual-port ConnectX-5
Figure 03. The physical hardware path. GPU-A and GPU-B are Linux namespaces on the same R430, not separate GPU servers.

PCIe, driver and firmware inspection preceded bandwidth testing. Host MTU was 9000; switch MTU was 9216. In the recorded test scope, RDMA WRITE achieved approximately 76.8 Gb/s, READ 67.3 Gb/s, and SEND 47.7 Gb/s. One RDMA WRITE run recorded approximately 9,158 MiB/s, which is about 76.8 Gb/s. These are host-to-host-namespace RDMA measurements across physical ports, not GPU-to-GPU or NCCL results.

A separate fault/recovery exercise brought the receiving port down and back up. The original reliable-connected (RC) workload failed. A fresh QP pair subsequently completed RDMA WRITE at 76.75 Gb/s. That proves that a new connection could be established after link restoration; it does not prove that the original QP or application recovered automatically.

This is the sort of distinction that matters in production incident reviews. Link-up, protocol restoration, QP recovery, and application recovery are four different observations.

04 / Congestion becomes a first-class network behavior

RoCEv2 carries RDMA communication over UDP/IP. When the network starts to congest, the behavior of queues, priorities, notification packets, and adapter rate control can determine the fate of the workload.

In Project Pivot, DSCP 26 mapped to traffic class 3 and egress queue 3 on the physical Z9100 configuration. Priority-3 PFC was enabled for the tested direction. The switch also had ECN-capable WRED and controlled queue shaping for congestion experiments.

RoCE end-to-end congestion path and distinction between observed ECN CNP and unproven DCQCN sender rate response
Figure 04. Observed classification, ECN marking and CNP feedback are shown as verified. The adapter's independent DCQCN rate-reduction and recovery response was not demonstrated.

PFC and ECN serve different roles. Priority Flow Control can pause traffic on a selected priority when buffer conditions demand it. ECN can mark capable IP packets under congestion; for the tested RoCEv2 feedback mechanism, marked traffic at the receiver can lead to Congestion Notification Packets (CNPs) back to the sender. DCQCN is the sender-side rate-control behavior associated with that feedback, and must be measured independently if it is to be claimed as validated.

I verified physical priority-3 PFC signaling in the tested direction: the switch transmit counter and GPU-A NIC receive indication each increased by 3,016 in the synchronized comparison. That supports real PFC frame delivery across the tested link. It does not establish bidirectional pause behavior or production-quality losslessness.

The strongest ECN evidence came from an October 12 controlled experiment. The switch's ECN-marked count and the receiver's CE-marked RoCE count both increased by 136,874 over the comparison interval. The receiver's CNP-transmitted count rose by 72,268 and the sender's CNP-handled count by 72,548. The equal marking deltas strongly support switch-to-receiver correlation, although no packet-by-packet identity was established and the two CNP counter changes must not be treated as an exact packet balance.

The test demonstrated ECN/CNP feedback. I did not independently demonstrate DCQCN sender-rate reduction and recovery. The fixed queue shaper and PFC activity prevented a defensible attribution of throughput changes solely to DCQCN. After documenting the limitation and reviewing the platform behavior, that objective was explicitly deferred rather than extending the test indefinitely.

05 / Healthy links, expensive latency

Congestion has a cost even when packet loss is not the headline result. In a controlled physical RDMA SEND latency comparison using 64-byte operations, the recorded averages were 1.53 µs unloaded, 63.57 µs congested, and 1.69 µs after recovery. The congested measurement was approximately 41.5 times the unloaded average.

Measured RDMA latency under unloaded, congested and recovered conditions
Figure 05. Measured 64-byte RDMA SEND average latency on the physical testbed. These are bounded lab measurements, not an ASIC-only latency characterization.

The physical shaper experiment told a related story. With queue shaping applied, RDMA throughput was approximately 5.82 Gb/s; removing the shaper raised the observed rate to 48.08 Gb/s; restoring it brought the result to approximately 6.00 Gb/s. ECN/CNP and PFC signals occurred in the congested experiments. The measurements describe the effect of the test setup and do not isolate a single cause for all rate changes.

An earlier virtual-incast experiment showed the same operational concern through a different mechanism: two senders were limited to approximately 39.54 Mbit/s aggregate and latency rose to roughly 90–112 ms, even though the routing view remained healthy. That was a virtual traffic-shaping experiment, separate from the physical 100GbE observations above.

Production GPU workloads are sensitive to the slowest participants in synchronized communication. Congestion and retry behavior are therefore worth correlating with collective operation timing and accelerator utilization, where those metrics are available. Project Pivot did not run GPU collectives, so that final layer remains a design consideration, not a measured result.

06 / Spectrum-X and InfiniBand widen the engineering vocabulary

NVIDIA Spectrum-X treats Ethernet AI networking as a coordinated platform involving the switch, host adapter, congestion behavior, and path selection. NVIDIA describes adaptive routing and other fabric behavior that goes beyond simple per-flow static ECMP. The design choices are different from a generic enterprise VXLAN deployment and must be evaluated with the actual platform, workload, and software stack.

InfiniBand brings its own operational vocabulary: subnet managers, partitions, LIDs, paths, queue pairs, and transport behavior. Many network engineering instincts transfer, while management and troubleshooting are different enough to justify dedicated practice.

Project Pivot includes conceptual study of InfiniBand and physical Ethernet/RoCE tests. It does not claim a physical InfiniBand fabric or Spectrum-X deployment. The most useful comparison is what each architecture expects the operator to measure, control, and recover.

For scale context, Microsoft has published a Fairwater design intended to interconnect hundreds of thousands of NVIDIA GB200/GB300 GPUs as part of a wider AI superfactory approach. Microsoft's long-running SONiC work is a separate example of hyperscale networking software. Those two public examples should not be read as a claim that this lab replicated Fairwater or that every Fairwater network element uses SONiC.

07 / From a crisis bridge to the AI fabric

My Tier-3 escalation routine started with a time window and facts about the affected communication: source and destination, ports, expected path, routing changes, interface and firewall counters, QoS, and packet captures. When necessary, I used captures from both endpoints, aligned timestamps, and analyzed the packets to distinguish a network problem from an application problem.

The same method works here, with more layers to correlate.

Operational evidence layers from workload to GPU and Kubernetes down through RDMA and fabric
Figure 06. Cross-layer incident correlation. The GPU layer is illustrated as a production operating model; Project Pivot's GPU monitoring was simulated using KIND/Mokka, while the physical RoCE tests used a non-GPU R430.

A GPU-related incident might require comparing scheduler placement, pod readiness, the GPU device plugin and DCGM, host and NIC counters, RDMA queue-pair state, the RoCE priority mapping, switch buffer and ECN counters, and routing convergence. A single green dashboard tile cannot make those layers agree.

In another Project Pivot workstream, a KIND/Mokka mock-GPU environment exercised the NVIDIA GPU Operator, device-plugin allocation and DCGM-exporter telemetry with Prometheus. That proved operational integration against simulated GPU devices, not physical GPU health or model-training performance.

The useful engineering habit is to establish the exact point where the observed evidence stops. If a fabric recovers but a failed QP remains unusable, report both. If ECN and CNP counters move but independent sender rate adjustment was not recorded, report that boundary. Incident documentation gets better when observations and hypotheses are kept separate.

08 / The practical shift

The work changed how I evaluate a GPU backend network. I still begin with topology, addressing, MTU, routing, and deterministic recovery. I now expect the design review to extend through adapter firmware, RDMA behavior, DSCP-to-queue mappings, PFC/ECN telemetry, memory headroom, workload placement, and cross-layer incident records.

The biggest lessons came from measuring failure and performance in layers:

  1. The virtual SONiC Clos established routing, EVPN/VXLAN and failover behavior. It did not establish ASIC congestion performance.
  2. The physical OS10/ConnectX-5 setup established a real 100GbE/RoCE path with useful bandwidth, PFC and ECN/CNP evidence. It did not establish GPU training or complete lossless-fabric behavior.
  3. Cross-layer telemetry and fault experiments showed why operational health must be evaluated from the workload downward, with each claim tied to the measurement that supports it.

That's the bridge from enterprise networking to AI fabric engineering: deeper transport behavior, tighter coupling between compute and network, and a higher standard for proving that recovery and performance really occurred.

Evidence notes and provenance

This article draws on Project Pivot's recorded lab reports, particularly:

The article's diagrams are explanatory illustrations. They do not masquerade as packet captures, actual vendor switch screenshots, or original hardware telemetry exports. A final publication evidence audit should reconcile them with the archived reports and primary console logs.

Primary technical references

Field notes prepared for Pivot AI Lab, October 2026.