On an incident bridge, the question I hear most often is whether the network has recovered. It sounds simple until somebody answers it with a BGP neighbor state, somebody else answers it with a ping, and the application team reports that users still cannot complete a transaction. Each person may be looking at a valid signal. None of those signals is a complete answer on its own.
In my Network SRE labs, I deliberately broke routing paths, observed recovery, and tested the tools intended to explain the failures. The useful discoveries were not limited to whether traffic could take a redundant path. I found a static route that prevented an intended BGP route from being installed, saw a short interface flap leave a useful syslog trail, and learned that an incident collector can appear to run while saving empty device files.
These were separate Cisco CML exercises, not experiments on a customer network. The six-router BGP lab supplied the strongest fault-and-recovery evidence. The earlier OSPF automation project established a configuration-control foundation, while the later NX-OS VXLAN EVPN fabric supplied an additional example of a control plane that could look healthy while a tenant forwarding dependency was incomplete. I am using those observations to define a disciplined way to test reliability, including where the original records stop short of a full, timed acceptance test.
01 / Decide what a failure test is supposed to prove
Before changing an interface, I want to know what a successful result would look like. A redundant link only matters if it preserves a required forwarding service. A protocol adjacency may return to Established while the routing table still selects the wrong next hop. An endpoint ping may work through one path while a different traffic class or destination remains affected.
I separate a reliability test into five questions:
| Question | Evidence required | False conclusion to avoid |
|---|---|---|
| Did the fault occur? | Injected action, device events, interface state and timestamps | Assuming a test command actually changed the intended device |
| Did the control plane respond? | BGP/OSPF state and relevant learned routes | Treating a recovered neighbor as proof of forwarding |
| What did the router install? | RIB and forwarding/CEF state for the affected prefix | Confusing a BGP-table path with an active route |
| Did service continue or recover? | Source-to-destination probes with a defined endpoint and path | Treating a single successful ping as complete service assurance |
| Can we explain the event afterward? | Logs, monitoring history, collector status, preserved snapshots | Calling an empty incident folder a completed investigation |
This is a test contract, not an assertion that my early labs captured every one of those dimensions on every run. In the original exercises, the coverage improved as I discovered how easily a green signal can coexist with a missing dependency.
A bounded test also needs a rollback plan. If the intended change is to shut down one link, the operator must confirm which interface is targeted, what other path should remain, which probe represents the service, and what condition will stop the test. In a shared or production environment, the fault scope and authorization matter just as much as the CLI syntax.
02 / Establish the baseline before touching the fault
The BGP environment consisted of ISP1, R1, R2, R3, R4, and ISP2. The alternative transit paths passed through R2 and R3 between R1 and R4. The important baseline was not simply that every session reported an expected state. It was that traffic between the selected endpoints used the intended installed path and that the other route could become usable if the preferred path disappeared.
I would preserve the device and probe state before an exercise: the route selected by BGP, the installed IP route, the forwarding entry, the next-hop reachability, and endpoint results. I would then associate that snapshot with the test identifier, source and destination, devices involved, and synchronized event times.
For Cisco IOS-style devices in this lab, representative read-only investigation commands include the following. They illustrate the evidence I want; this is not a verbatim archived command transcript and syntax may vary by release.
show ip bgp 172.16.1.0
show ip route 172.16.1.0
show ip cef 172.16.1.0
show ip bgp summary
show logging
The first three commands matter together. The BGP table describes candidate paths inside that protocol. The RIB shows the route selected for IP forwarding, after competition among routing sources. CEF provides the forwarding-plane view on supported Cisco platforms. The endpoint test answers whether the intended traffic actually reached its destination.
I keep four observation windows: a known baseline, the fault interval, the routing and forwarding transition, and the restored state. A post-test screenshot of every neighbor back up is not a substitute for knowing what happened during the interval. It is especially weak when the event lasts only a few seconds.
03 / The BGP path failed the test before the link did
The most revealing April 21 finding came from R4. The router had a static route for 172.16.1.0/24 pointing toward 10.20.0.13. The prefix also existed in BGP, but a routing-information-base failure indicated that the intended BGP path was not the route installed for forwarding. A static route with the default administrative distance can take precedence over an eBGP path to the same prefix. Cisco's documentation describes precisely this class of RIB failure.
That discovery changed the failover investigation. The network could exchange BGP information while the routing table made a different forwarding decision. If I had tested only whether the BGP session was Established, I would have missed the condition that interfered with the intended route selection.
After the interfering static route was removed during the lab correction, R4 installed the BGP route through R3, using next hop 10.20.0.17. The follow-up RIB, CEF, and ping checks succeeded. Subsequent end-to-end testing from the ISP1 side toward ISP2 succeeded, and the project exercised a failure of the R1-to-R3 leg that moved traffic to the R2 backup path.
I regard that sequence as two different findings. First, the baseline had a route-selection defect: a BGP route was learned but was not the forwarding choice. Second, after correction, a redundant route could carry the documented failover test. They cannot be collapsed into one claim that the network had perfect redundancy all along.
This is why I test the alternate path before a planned fault. Otherwise, the first part of a fault-injection exercise may be spent debugging a broken baseline, and the result will tell me more about initial configuration hygiene than recovery behavior. Sometimes that is the most valuable discovery, but it needs to be labeled correctly.
04 / What the recorded BGP events establish
On April 22, the retained BGP incident template preserves an R1 event sequence rather than the previously illustrated R4 ten-second interval. R1 syslog reports neighbor 10.20.0.10 resetting at 18:42:00 UTC, Ethernet0/2 becoming administratively down at 18:42:02, and its line protocol going down at 18:42:03. These are device-reported event times; the inspected excerpt does not establish link-restoration time, root cause, or duration of user traffic loss.
A BGP-reset event tells me that the session changed state. The R1 syslog adds device and interface context, but the order of these retained messages does not by itself establish which condition initiated the other. I would review both ends of the link, routing changes and service probes before assigning cause or impact.
Short events put monitoring claims into perspective. A scrape interval, alert evaluation interval and notification grouping policy influence when a monitoring system observes and reports trouble. The preserved R1 syslog does not establish alert detection latency without the corresponding Prometheus samples and Alertmanager timestamps. Packet-loss duration likewise needs time-aligned traffic measurements.
The engineering difference is between event duration, control-plane recovery, forwarding recovery, and time to detection. They are separate measurements. When the supporting timestamps are missing, I leave the measurement unreported rather than produce a plausible-looking number.
05 / Failure injection can break the observer
The next discovery involved the automation host rather than the network path. The BGP incident workflow used Prometheus, Grafana, Alertmanager, syslog, an Alertmanager-to-Python webhook, and Ansible on automation-1. Its purpose was to preserve enough state to reconstruct a failure after the device had already recovered.
Early test runs created incident directories and stored alert or Prometheus material, but some router CLI files were empty. Interactive SSH authentication blocked unattended collection. From the outside, the workflow looked partly successful: the incident folder existed. Inside it, the evidence needed to explain routing behavior had not arrived.
The collection path was corrected. In the April 23 test, the recorded Ansible run collected device data from all six routers: ISP1, R1, R2, R3, R4, and ISP2. The recap recorded zero failed hosts. The associated incident bundle included Prometheus JSON and six device syslog sources.
I now treat collection completeness as its own reliability signal. A collector must identify every target as captured, empty, failed, timed out, or not applicable. It needs a nonzero overall failure result when mandatory data is absent, plus an artifact manifest showing device identity, collection time, command or probe, and exit status. A successful alert webhook invocation is not proof that every downstream collector succeeded.
The failure domain includes the instruments. If the management path fails at the same time as the production link, or a credentials change breaks automation, the absence of device output can be an observation about the evidence system, not evidence that the network was healthy. The system should report those conditions distinctly.
06 / An alarm may outlive the fault
The later BGP exercises showed a different time problem. By May 5, the workflow classified an InterfaceErrorsIncreasing event for R2 Ethernet0/0 as INVESTIGATE and selected an investigation path. A May 6 report preserved 189 output errors and 189 collisions while the associated BGP neighbor checks were up. Those counters justified interface-level investigation; they were not a demonstrated explanation of application impact or an instruction to change the routing protocol.
In another live auto-investigation, the collector processed an InterfaceOperDown alert for R2 Ethernet0/0 but found the interface up/up at collection time. The workflow classified the current state as RESOLVED. A separate test-mode input with an intentionally down/down interface returned OPEN.
This distinction is essential for a fault experiment. There is an event-time observation, which describes what triggered the alarm, and a collection-time observation, which describes what the engineer or automation found later. If the interface has recovered, a live command should not erase the historical event. If a synthetic fixture is down, the record must identify it as test input rather than an independent observed outage.
During a real outage bridge, I would want both records visible. The alert tells us why collection started; the live state tells us what is true now. The difference between them may mean the fault recovered, the alert was delayed, the sampling was coarse, or the collector queried the wrong target. The test must be able to distinguish those possibilities.
07 / A separate EVPN lab exposed a similar false pass
The Cisco NX-OS VXLAN EVPN CML fabric added another example from a different environment. It had two spines, three leaves, and three test endpoints. OSPF supplied underlay transport; MP-BGP EVPN signaled overlay reachability; NVE interfaces and VNIs provided the tunneling relationships.
During bring-up, tenant L3 VNI 99999 remained down until the relevant Vlan999 routing SVI was corrected with ip forward. A healthy OSPF neighbor did not prove that the routed VXLAN tenant was ready. A BGP session alone could not verify L3 VNI operational state. A complete acceptance sequence also needed route visibility and host reachability checks after the change.
That was a configuration/functional failure in a separate fabric exercise. I am not claiming it was part of the April BGP shutdown, nor that a fully scripted EVPN fault-injection campaign ran with measured convergence results. It illustrates why the assertions in a future fault plan should follow the dependency chain: underlay, overlay signaling, NVE/VNI state, route and endpoint forwarding.
The retained GitLab Job #190 from the VXLAN project reported all three leaves passing OSPF FULL, NVE up, and required VNI up assertions, followed by successful five-device Ansible collection. That is evidence of those particular checks. The excerpt does not establish that end-to-end forwarding was retested during every fault, so I would keep endpoint probes as a separate gate.
08 / Turn individual experiments into a fault library
I do not want a set of heroic lab memories that only one engineer can repeat. For a reliability program, each exercise should have a stable identifier, an explicit hypothesis, an authorized fault, a known service probe, observability checks, stop conditions, and a reproducible recovery procedure. The source record should keep observed outcomes separate from tests proposed for a later cycle.
| Scenario | Status in these labs | Reliable conclusion | Next acceptance criterion |
|---|---|---|---|
| R1–R3 leg failure with R2 alternate path | Documented exercise | Backup path used after routing correction | Capture per-probe loss and time-aligned RIB/FIB transition |
| R4 Eth0/1 shutdown and BGP reset | Documented event | R1 BGP reset and interface-down messages preserved; recovery interval unverified | Correlate both sides, probes, monitoring and precise timestamps |
| R4 static-versus-BGP route conflict | Documented finding/correction | Learned BGP route had not been installed; correction restored intended path | Include RIB/FIB baseline assertions before other faults |
| BGP incident collector with empty CLI files | Documented failure/fix | Initial evidence was incomplete; later six-router collection worked | Fail the incident workflow on missing mandatory artifacts |
| NX-OS L3 VNI not operational | Separate documented configuration fault | Underlay state alone could not establish tenant readiness | Validate EVPN, NVE/VNI and endpoint path after correction |
| Sustained brownout, packet corruption, link flaps under load | Proposed future fault tests | No results claimed here | Define workload, thresholds, limits, and exact evidence before execution |
The table is deliberately uneven. It would be easy to format all six rows as completed fault-injection tests. The retained evidence supports different levels of confidence, and the article should preserve that distinction.
For every future test, I would define the actual traffic symptom in advance. For example, if a route is supposed to fail over to the alternate transit path, success should require the expected prefix to remain installed, the next hop to change appropriately, and the representative service probe to recover within a previously agreed limit. The limit must come from the application's tolerance and the organization's service objectives. It cannot be retroactively invented from an old green ping result.
09 / Production guardrails: smaller faults, stronger evidence
There is no sensible reason to run an uncontrolled chaos exercise on a busy network just because a dashboard now exists. I would begin with an isolated lab or a tightly scoped maintenance test: one fault, one known alternate path, one affected service, and one rollback owner. The operators watching the test need the same event and observation clocks, plus a direct way to halt the injection.
A production-ready design needs change authorization, blast-radius analysis, maintenance or approved test windows, management-plane independence where practical, reserved rollback access, and an explicit prohibition on escalating scope automatically. Loss of the collector, unexpected effects on unrelated services, or failure to restore the baseline should stop the exercise. Telemetry collection uses read-only accounts; configuration changes require separately authorized privileges. Logs, packet captures, and test artifacts must be protected and sanitized before publication.
I would also keep the reliability targets grounded. Detection time is measured from the fault or first user-impact event to a recorded detection signal. Forwarding recovery time is measured using probes with known timing and sufficient frequency, not the interval between two screenshots. Error-budget impact requires an actual service-level indicator and defined objective, not a percentage pulled from a BGP uptime chart. These are criteria for a future instrumented test, not statistics measured by the historical exercises.
There is a danger at the opposite extreme: designing so many conditions that the test becomes impossible to run. Start with the single failure that would hurt users most, prove the alternate path and evidence pipeline work, and add complexity only when the previous test is reproducible. An engineer who cannot explain a one-link shutdown has no business trusting a multi-fault automation framework.
10 / What I would take into a network operations team
These labs changed the questions I ask during recovery. On R4, the issue was not whether BGP had learned the destination prefix; it was which routing source had actually won installation. The separate April 22 R1 syslog sequence supplied BGP and interface event timestamps without proving a traffic-impact duration. When incident folders came back with empty router snapshots, I had to repair the evidence path before trusting the workflow. In the EVPN project, a correct-looking underlay still left a tenant L3 VNI down.
Taken together, those findings describe the kind of reliability work I want to scale: a fault with a bounded scope, a clearly stated forwarding hypothesis, independent observations at each layer, and an incident record that distinguishes what happened from what we were able to prove. GitLab and automation can make the procedure repeatable. Prometheus and syslog can help preserve the timeline. They do not remove the engineer's responsibility to verify what each tool measured.
The strongest result of failure injection is not proving that one redundant link works. It is discovering assumptions before they become an outage: a static route that hides a backup path, a missing L3 VNI dependency, a monitor that samples too slowly to establish timing, or a collector that quietly saves empty output. Once those are visible, they can become explicit acceptance tests instead of surprises during an escalation.
Source record, technical references, and publication verification
Lab basis. The BGP findings come from the separate bgp-sre-lab Cisco CML project and its April–May 2026 records: April 21 R4 RIB troubleshooting and failover checks; April 22 R1 syslog messages preserved in incidents/incident-evidence-template.txt; April 23 six-router collection; and later interface investigation. The previous R4 18:18 interval was not independently verified during the publication audit. The separate VXLAN finding comes from the Cisco NX-OS fabric records, including Vlan999 / L3 VNI 99999 and GitLab pipeline #124 / Job #190. These were independent lab environments. Diagrams are editorial reconstructions.
Vendor reference. Cisco, Understand BGP RIB-Failure and BGP Suppress-Inactive Command, explains why an otherwise valid BGP path may not be installed when a route with a better administrative distance already occupies the RIB.
Verification before publication. Reconcile the summarized timestamps, next hops, interface identifiers and test statuses against the original bgp-sre-lab source and logs, and verify the separate VXLAN VNI and GitLab records. Do not describe a proposed test matrix as completed, infer packet-loss counts or convergence times, or present reconstructed figures as screenshots. Retain this article as an editorial draft until those checks and the visual review are approved.