When the Interface Is Up but the Network Is Broken

A network interface can report UP while forwarding nothing useful. I have dealt with that distinction in production incident bridges, where a healthy port LED and an established routing session can make the first ten minutes of investigation misleading. The answer usually comes from collecting several observations during the same failure window: the path the traffic took, what the forwarding plane actually did, how the control plane reacted, and whether the application recovered.

I used Project Pivot's virtual SONiC fabric to put that situation under test. A two-spine, three-leaf Clos gave me redundant eBGP paths. BFD provided neighbor reachability detection; Linux endpoints supplied repeatable flows. I deliberately broke forwarding while leaving the interface logically up, then compared that experiment with conventional hard link shutdowns. The results produced a useful operational lesson: link status, protocol state, installed routes, and delivered traffic are different observations. None can stand in for the others.

Comparison of an obvious link-down condition with a silent forwarding failure where Ethernet0 stays up — Figure 01. Two failure modes in the virtual SONiC lab. A hard-down interface provides an obvious local signal. In the silent-loss experiment, the Linux interface still reported UP/LOWER_UP while an egress netem rule discarded packets. This is a virtual packet-loss injection, not an assertion about a physical ASIC fault.

01 / The failure surface: five virtual switches and two usable paths

The lab uses Containerlab, SONiC virtual-switch containers, and FRRouting. Two spines connect to three leaves. Linux endpoints named gpu1, gpu2, and gpu3 attach to leaf1, leaf2, and leaf3 respectively. Those names identify simulated workload hosts, not physical GPUs; this test did not run NCCL or RoCE.

Leaf1 connects to spine1 on Ethernet0 and spine2 on Ethernet4. With both adjacencies healthy, a representative remote prefix such as 10.255.0.12/32 has two equal-cost next hops through 10.0.0.0 and 10.0.0.2. The routed compute endpoints use separate /31 subnets, and the EVPN/VXLAN overlay relies on reachable leaf VTEP loopbacks. A failure on one uplink can therefore affect routed workloads and the transport that carries encapsulated overlay frames.

Five-node Clos showing leaf1 uplinks Ethernet0 and Ethernet4 with an impairment on the spine1 path — Figure 02. Simplified path model for the failure exercises. The red branch represents the injected impairment on leaf1 Ethernet0 toward spine1. The remaining route through Ethernet4 and spine2 is a potential forwarding alternative once the failed next hop is withdrawn. ECMP path selection varies by flow; the drawing is not a packet capture.

The distinction between two eligible routes and two operating paths matters. BGP may correctly install both next hops while a forwarding impairment prevents one from delivering packets. Even a clean failover requires the appropriate failure detector to notice the problem and the routing and forwarding software to stop selecting that path.

02 / A hard shutdown establishes the comparison case

I first tested the more straightforward condition: take down the uplink actually carrying the workload. The Day 5 harness identified the path selected by the flow, disabled the corresponding leaf interface, and retained the state changes alongside live ICMP and TCP observations.

The gpu1 → gpu2 test was using spine2, so I disabled leaf1 Ethernet4. BFD to 10.0.0.2 went down, the spine2 next hop became inactive, and traceroute shifted to spine1. The harness recorded 1.896 seconds of recovery, a 2.079-second gap between successful replies, nine consecutive lost pings, and two intervals with no reported TCP throughput. The existing TCP session survived.

For gpu1 → gpu3, the active path ran through spine1. Shutting down leaf1 Ethernet0 moved the forwarding path to spine2. The recorded recovery was 2.056 seconds, the reply gap 2.070 seconds, with nine lost pings and two zero-throughput intervals. That TCP session also survived. In both tests, BFD returned to up after restoration and the validators confirmed both ECMP paths were available again.

Active path removed	Measured workload recovery	Successful-reply gap	TCP session
spine2 / leaf1 Ethernet4	1.896 s	2.079 s	Survived
spine1 / leaf1 Ethernet0	2.056 s	2.070 s	Survived

These are workload-level measurements for hard interface shutdowns. They are not direct measurements of BFD detection time and cannot be reused as recovery timings for the later silent-blackhole test.

03 / Inject a failure that does not drop carrier

A silent forwarding impairment is more interesting because the interface does not advertise the problem. In the Day 8 experiment, I applied Linux traffic control's netem loss 100% to leaf1 Ethernet0. The relevant command targeted the virtual SONiC container's interface and discarded traffic through that egress queueing discipline. A recorded ip link check still showed UP,LOWER_UP; the interface had not been administratively shut down.

The test harness ran a 500 Mbit/s UDP iperf3 flow between 10.10.1.1:40000 and 10.10.2.1:5201. It collected BFD state, the route to the remote VTEP loopback, and spine-facing interface counters before, during, and after the impairment. The deliberate failure was on the spine1-facing uplink, with spine2 still reachable on leaf1 Ethernet4.

Timeline separating interface state, BFD detection, BGP next-hop changes, and workload throughput — Figure 03. Sequence of evidence to collect during a silent forwarding incident. The sequence is conceptual; the diagram does not assign a measured millisecond timestamp to a BFD transition or to workload recovery in the Day 8 experiment.

The point of the experiment was not to prove that every UDP packet was delivered during the transition. UDP lacks TCP's retransmission behavior, and the requested send rate alone cannot establish receiver throughput. The captured result instead showed the failure-detection and alternate-route mechanism operating while interface state remained apparently normal. Separate traffic and counter observations were needed to assess delivery.

A port monitor that reports only carrier transitions would miss the initial symptom. A BGP neighbor monitor may see the consequence, but without BFD, next-hop, and endpoint data it still cannot prove why traffic was lost or which path recovered.

04 / Follow the signal from BFD through the forwarding table

The FRRouting BFD session observations in this lab showed 300 ms transmit/receive intervals and a detection multiplier of three. Under the documented negotiation assumptions, that produces a nominal detection interval on the order of 900 ms when expected BFD packets stop arriving. That is a timer calculation, not an incident stopwatch. Scheduling, negotiated intervals, polling, control-plane events, and the surrounding stack can affect what an operator actually observes.

BFD has one job here: identify that the peer's bidirectional forwarding path is no longer healthy according to its control-packet exchange. FRRouting can use that state change to notify BGP. BGP then removes or invalidates the affected route and next hop; the forwarding table must reflect the new eligible set. Only after that can a flow previously mapped to the damaged branch be expected to use the surviving path.

Detailed separation of BFD timer, BGP withdrawal, kernel route update, and application recovery — Figure 04. Operational dependency chain. Only the nominal BFD timer comes from the displayed 300 ms × 3 settings; no other stage is assigned a fabricated duration. FRRouting documents how BFD integrates with BGP neighbor state.

This also creates a useful troubleshooting branch. If BFD remains up while packets are lost, I investigate whether the failure lies outside the path exercised by BFD, whether traffic classification differs, whether only a specific flow is affected, or whether the detector is misconfigured. If BFD goes down and BGP removes a route but traffic still fails, I turn to the surviving next hop, forwarding table, underlay packet path, and workload endpoints. BFD-down is evidence of detection, not proof of application restoration.

05 / The overlay introduces another place for state to disagree

Project Pivot added an EVPN/VXLAN Layer-2 overlay using VLAN 100 mapped to VNI 10100. The three leaf VTEPs were 10.255.0.11, .12, and .13. The spines exchanged EVPN control-plane information; the leaves encapsulated the actual frames. Type-3 IMET routes established VNI membership, and Type-2 MAC/IP routes identified remote endpoint locations.

That layering creates more investigation questions when a transit path breaks. Does the remote VTEP loopback still have a surviving underlay route? Does EVPN retain the expected Type-2 and Type-3 information? Has the forwarding database changed? Are VXLAN packets still arriving at UDP destination port 4789? What do the overlay endpoints see?

In Day 7 I tested hard failures, silent forwarding impairment, and VTEP isolation with a dedicated test-overlay-failures.sh harness. The repository preserved baseline, fault-state, and restored-state snapshots of BFD, BGP, route, VNI, FDB, and EVPN observations. The earlier Day 6 packet capture had already shown actual VXLAN encapsulation; Day 7 was about assessing what happens to those dependencies under failure.

State comparison matrix across physical interface, BFD/BGP, EVPN/VTEP, and endpoint forwarding for three fault types — Figure 05. Diagnostic matrix, not a scorecard of universally passing tests. A hard-down event, an egress packet blackhole, and VTEP isolation can produce different combinations of interface, control-plane, overlay, and workload symptoms.

I would not close a VXLAN incident simply because the EVPN neighbors returned to Established. An overlay host can still be unreachable because of its local access port, MAC/FDB programming, incorrect VNI mapping, transport path, or something beyond the VTEP. The control plane narrows the search. It does not replace a packet test.

06 / Show the failure in operational telemetry

By Day 9 the fabric had telemetry covering device state, interfaces, BFD, BGP, and EVPN, with Prometheus, Grafana, and Alertmanager used to examine and alert on changes. The later cross-layer operator view combined fabric health with Kubernetes status. During a separate Day 10 silent-forwarding exercise, the fault was limited to a SONiC uplink while Kubernetes node and workload readiness remained stable.

That stable Kubernetes state was useful: it demonstrated that the exercise did not produce an observable readiness failure in the monitored cluster. It did not prove the absence of short traffic disruption, loss in every flow, or correct delivery for all possible applications. Likewise, a healthy Grafana panel that counts established BGP sessions does not prove a specific data-plane path delivered packets during the sampling interval.

I prefer an incident view that aligns four clocks: the injected or reported fault time; BFD and BGP transitions; interface or packet-path evidence; and the application's observed gap. Samples need timestamps, and the interpretation must respect scrape intervals. If a fault lasts a second and a dashboard samples every 15 seconds, missing the dip in one panel says more about the sampling strategy than about the network.

Incident observability map showing interface status, BFD/BGP, forwarding and workload evidence feeding one timeline — Figure 06. Operator-oriented correlation model derived from the recorded lab workflows. It shows which sources should be compared; it is not a claim that a single automated system already correlates every signal or establishes application SLO compliance.

07 / Recovery means more than finding a surviving next hop

When the fault is removed, I want to see the fabric return to a known baseline rather than merely resume a successful ping. The acceptance check includes the original BFD peers up, both eBGP next hops eligible where expected, EVPN route and VTEP state consistent, restored interface counters, and endpoint validation across the relevant routes. The Day 5 test closed with both fabric and workload validators passing after link restoration; Day 6's rebuild test also confirmed an independently reproducible overlay baseline.

A restored routing entry can coexist with an application that needs a new TCP connection, RDMA queue pair, or higher-layer retry. The virtual SONiC experiments used ICMP, TCP, and UDP flows, so their outcomes should not be presented as validation of RoCE queue-pair or NCCL collective recovery. Those questions belong to separate Project Pivot RDMA and GPU-operating experiments.

The record should also say what is still unknown: whether the exact affected flow took the impaired branch before withdrawal, how a different hash seed would behave, whether a failure has equal effects on all traffic classes, and whether virtual scheduling translates to ASIC-speed forwarding. Keeping those boundaries visible makes the results defensible.

08 / What this changed in my incident workflow

For a Tier-3 production escalation, I have always started with the failure window, source and destination, the expected path, counters, and packet evidence. Project Pivot added a repeatable fault-injection discipline to that process. I can now reproduce an interface that stays up while forwarding fails, observe the specific detector that notices, trace the routing decision that changes, and retain separate evidence of what the workload did.

There is a practical order to the investigation: confirm the affected flow and time, check the selected next hop, compare carrier with BFD state, inspect BGP and FIB, test surviving forwarding, verify EVPN/VTEP state if relevant, and finally confirm workload restoration and full redundancy. A single show interface output is the beginning of that investigation, not its conclusion.

The operational takeaway: An interface-up alarm is not an availability guarantee. The useful evidence is the chain connecting failure detection to routing changes and then to recovered delivery. That chain is what an engineer needs on a live incident bridge, especially when the interface itself provides no obvious fault signal.

Evidence trail

The original Project Pivot repository at /home/bryan/pivot/fabric/sonic-clos/ on fabric-lab-1 holds the underlying reports and scripts:

docs/day5-workload-failover.md and scripts/test-failover.sh — exact hard-down measurements, surviving TCP sessions, and post-test validation.
docs/day6-evpn-vxlan.md — VTEP and VNI mapping, route types, observed encapsulation, and overlay rebuild.
docs/day7-evpn-vxlan-failure-testing.md, scripts/test-overlay-failures.sh, and evidence/day7-overlay-failures/ — overlay fault-injection runs and state snapshots.
scripts/test-workload-convergence.sh and retained Day 8/9 terminal captures — 500 Mbit/s UDP flow, leaf1 Ethernet0 silent-loss injection, BFD, routes, and counter samples.
docs/day10-cross-layer-observability.md — supporting observability and cross-layer readiness measurements.

All graphics in this article are editorial diagrams reconstructed from the engineering records. They are not fabricated screenshots, physical ASIC traces, or end-to-end application recovery measurements beyond those explicitly stated.