The Incident Cleared.
What Did We Fix?

On an incident bridge, getting traffic moving again is the immediate priority. I have spent a large part of my career doing exactly that: establishing the time of failure, identifying the source and destination, following the routing and firewall path, and collecting enough device and packet evidence to determine where the failure occurred. Once the service recovers, there is a temptation to close the bridge, file the ticket, and get back to the queue.

The next incident often begins with a similar symptom. An unexpected route wins, an interface flaps, a tenant VNI stays down, or a collection script produces a plausible-looking report with no useful router output. A good incident record explains what happened. An engineering follow-up changes the conditions that allowed it to happen or improves the probability that the next failure will be detected and isolated correctly.

My independent Network SRE labs produced several examples across three separate Cisco CML environments: an OSPF configuration-control lab, a six-router BGP incident-response lab, and a two-spine, three-leaf NX-OS VXLAN EVPN fabric. They also exposed faults in the automation used to check those networks. The tests and logs documented the immediate findings; they did not constitute one deployed, production-wide corrective-action program.

For this article, I am taking those findings through a structured follow-up review. I will distinguish the corrections already demonstrated in the lab from the additional safeguards I would require before depending on the same workflows during enterprise changes. That distinction is important to anyone reviewing the engineering record.

01 / Give recovery, diagnosis, and prevention different exit criteria

When a BGP neighbor returns to Established, the control plane has recovered one relationship. It does not establish whether the correct route is installed, whether the original traffic path is restored, or whether the condition that created the outage remains. I use three different questions at the end of an incident:

1. Is the affected service operating again? Confirm representative source-to-destination traffic, relevant interfaces, installed forwarding state, and any alternate path expected to carry the flow.
2. Do we understand what caused the observed failure? Correlate configuration, device logs, control-plane changes, route selection, and packet-level data when available. Record uncertainty rather than making up a customer-impact window.
3. Which change will reduce recurrence or diagnostic delay? Define a concrete action, an owner role, a validation test, and proof that the action was completed.

Those are different closure criteria. A NOC can reasonably conclude the immediate incident response while a deeper corrective action remains open. That is normal operations, provided the remaining risk is visible, assigned, and tracked. I would not leave an undefined “monitor it” task as the only follow-up for a repeatable failure mode.

The lab findings divide into network state failures and observer failures. The R4 route conflict was a forwarding-path problem. The missing ip forward under an L3 VNI was a fabric configuration problem. Empty SSH capture files were an evidence-collection problem. A one-sided drift comparison was a validation-coverage problem. Each needs a different regression test, and each can undermine a successful-looking summary in a different way.

02 / Reconstruct the event without inventing its impact

The retained April 22 BGP incident template provides an R1 sequence: neighbor 10.20.0.10 reset at 18:42:00 UTC, Ethernet0/2 administrative-down at 18:42:02, and line protocol down at 18:42:03. The previously described R4 ten-second event sequence was not corroborated by the inspected historical artifacts. The preserved R1 messages support device-event correlation, but not a proven causal order, restoration timestamp or precise service interruption.

That distinction has governed many of my real incident reviews. When I have an interface-state change and a BGP reset in nearby device records, I have a hypothesis to test, not a measured outage. Without time-aligned reachability samples or sender/receiver packet captures, I have no defensible basis for claiming precise service interruption. A neighbor can reset while an alternate path carries traffic, and service probes can fail while routing adjacencies look healthy.

My incident notes need the following fields even when some remain UNKNOWN: original event time and clock source; first observed symptom; affected device/interface and neighbor; expected and actual forwarding path; user-facing service test and sample window; action taken; verification time; and evidence locations. Event time and observation time must remain separate. A recovery snapshot taken later cannot retroactively prove what happened during a transient event.

For a brief flap, the follow-up may be better telemetry granularity or a triggered capture rather than another routing configuration change. That is an engineering action with a measurable purpose: capture enough data during the next event to distinguish control-plane interruption from actual service loss. The BGP lab's syslog trail establishes the detection opportunity. A continuously correlated probe-and-packet timeline was not demonstrated by that particular event record.

03 / The R4 static route: learned is not installed

An earlier BGP failover exercise on April 21 showed a more concrete network defect. R4 had a static route for 172.16.1.0/24 pointing to 10.20.0.13. A BGP alternative existed, but the static entry interfered with the intended route selection. The project notes described a BGP RIB-failure condition. Seeing a prefix in the BGP table therefore did not mean BGP supplied the installed forwarding route.

After the interfering static route was removed, the lab observed the BGP route installed toward R3, next hop 10.20.0.17, and follow-up CEF and ping checks succeeded. The broader failover work tested traffic between the ISP-side endpoints and exercised the R1–R3 leg failing over toward R2. Those results support a functional alternate-path test after correction; they do not supply a measured convergence-time distribution or an application SLO.

The immediate correction was removal of the conflicting static route. The next engineering action would be to prevent the same class of route from silently winning again. In a change gate, I would validate the intended prefix in three places: the relevant BGP path, the installed routing information base (RIB), and the forwarding information base (FIB/CEF). Then I would test the actual endpoint flow, including the planned failure of its primary path.

This is also where change ownership matters. A static route might exist for a maintenance bypass or a temporary recovery. Automation should report the unexpected route and require an authorized decision, not simply delete any route that diverges from a template. In this lab, we knew the path selection was wrong and removed the conflicting entry. In a live enterprise, I would check the change record and operational intent before touching the route.

04 / Write the failover test so it cannot pass on the wrong path

The six-router BGP topology offered two middle paths: R1–R3–R4 and R1–R2–R4, bracketed by ISP1 and ISP2. That makes it useful for a focused regression test. The test definition should identify the exact prefix and source/destination, the expected primary route, the allowable backup route, and what constitutes a failure before anyone shuts down a link.

I would divide acceptance into baseline, fault, and restoration. At baseline, record BGP best path, actual installed route, CEF next hop, and endpoint reachability. During the controlled R1–R3 outage, preserve the routing change and verify the backup R2 path carries a representative flow. After restoration, check the intended path selection and repeat the endpoint test. Where the policy intentionally favors a non-default path, document that expectation explicitly.

A successful ping alone leaves too much room for interpretation. It might traverse the backup path during baseline because the primary route never installed. That was the exact family of error exposed by R4's static route. A regression test should report the selected path before declaring success, and fail when the observed forwarding behavior violates policy even if the destination responds.

For production changes I would scope the fault injection tightly, confirm the secondary route is already viable, announce the planned outage window, and have a stop condition and restoration procedure ready. Controlled failure testing is most useful when its blast radius is deliberately constrained. None of this requires claiming a faster failover time than the saved CML evidence demonstrates.

05 / Carry the same discipline into the EVPN fabric

The NX-OS fabric produced a different but related fault. Its L3 VNI 99999 remained down while much of the underlay and overlay configuration had been assembled. The tenant routing SVI, Vlan999, needed ip forward. After that configuration was corrected, the L3 VNI could reach its expected operational state.

This is a good example of why a corrective-action test should target the condition that failed. OSPF neighbors in FULL state are necessary for underlay reachability, but those checks would not detect a tenant L3 VNI left down. Similarly, an established EVPN session does not establish correct tenant forwarding. I would add a VNI-state assertion for the relevant tenant, confirm VRF/VNI association and the routed SVI, and exercise a representative inter-VLAN endpoint flow.

The test scope should follow the actual lab plan: VLAN 10 → L2 VNI 10100, VLAN 20 → L2 VNI 10200, and L3 VNI 99999. The earlier lab used servers on 10.10.10.101, 10.10.10.102, and 10.10.20.103 to exercise same-subnet and inter-subnet paths. These are retained private lab values, not production customer addressing.

The retained GitLab pipeline #124 / Job #190, commit 31d7f689, explicitly reported OSPF neighbors FULL, NVE interface up, and required VNIs up for LEAF1, LEAF2, and LEAF3. Ansible collection covered five NX-OS devices. Its success is valuable evidence for those printed checks. The preserved job excerpt does not separately establish a full end-to-end endpoint ping gate. An expanded regression suite would have to add that gate and preserve its own result before I would claim it ran automatically.

06 / Correct the tool when the tool is part of the incident

Some failures occurred in the evidence pipeline. In the separate BGP lab, the early webhook-triggered collector created incident folders and retained monitoring information, but the intended router CLI snapshots were empty. Interactive SSH authentication had prevented unattended collection. Later, after that collection path was fixed, the April 23 Ansible exercise reached all six routers and reported zero failed hosts. That was a demonstrated correction to a specific failure mode.

I would turn this finding into a recurring collection-quality test. The known target list should be independent of the files that happened to appear. Each required command should produce a fresh artifact tied to an incident ID, device identity, collection time, and exit status. An empty mandatory capture must report an incomplete bundle, even if the surrounding webhook handler returned HTTP 200.

The BGP parser exercises revealed a stale output file and a Python NameError during development. Both are reminders to test the investigation tools with known-bad fixtures: missing output, previous-run output, parse exceptions, partial router inventories, and a healthy live interface after a historical down alert. These are follow-up test recommendations, not a claim that a comprehensive fixture suite was implemented in the historical repository.

A separate October 13 review of the VXLAN drift checker found that it caught baseline files missing from current collection, but failed to symmetrically flag a new current-only file. That should become a test fixture with a known expected failure. Comparing the union of baseline and current inventories is the remedy I would implement, but I will not describe that fix as already merged or revalidated.

07 / Turn each finding into a small, verifiable action

Post-incident action items fail when they read like advice. “Improve monitoring” is not an executable engineering task. “Add a required five-device collection inventory assertion and fail the job when any expected configuration is absent” is. I would keep each action narrowly scoped, give it a responsible role, and define what artifact proves completion.

That status column is intentional. An issue can be understood and still remain an open engineering risk. In a production action register I would add a named owner, priority, due date, change or issue reference, verification reviewer, and closure evidence. I would not fabricate those assignments or dates for old CML experiments whose repository issue trackers were not re-read for this editorial draft.

08 / Decide whether a follow-up is a fix, a guardrail, or a visibility improvement

These categories matter because they affect change risk. Removing R4's interfering static route was a direct configuration correction. Requiring a RIB/FIB check before maintenance is a preventive guardrail. Improving the event-time capture of BGP and service probes is an observability action. A new incident runbook may reduce mean time to diagnose without reducing the frequency of the underlying physical fault.

I would prioritize the follow-ups based on service criticality, likelihood of recurrence, detectability, and blast radius. A widespread baseline-promotion defect in network automation can deserve more urgent treatment than a cosmetic dashboard problem, because it can make an incorrect reference state authoritative. Equally, missing evidence during a fleeting failure may deserve attention when it repeatedly prevents a reliable root-cause determination.

The reviewed VXLAN GitLab pipeline defined validate → deploy → baseline → post_validate, with manual controls on the deployment and baseline stages. A proposed safer lifecycle would retain an approved pre-change state, execute a scoped deployment, require verified network and service postchecks, and only then promote a new baseline under an authorized review. That sequence was recommended after source review; it was not demonstrated as a replacement pipeline.

Security controls travel with the action. A read-only collector should not share privileged configuration-changing credentials with a deployment job. GitLab protected environments can restrict who deploys, and deployment-approval capabilities depend on edition and configuration. A manual job button should not be mistaken for proof that a change has been reviewed and authorized. Webhook input and incident evidence also require authentication, validation, controlled access, and retention policies.

09 / Keep the runbook short enough to use during a real bridge

The value of a runbook is measured during the incident, when several engineers are staring at different screens and the application owner wants a status. I would organize a BGP/VXLAN runbook around the evidence that changes the next decision, with exact platform-qualified commands maintained in its repository rather than an enormous copied command list.

For the BGP path, the order is: confirm incident time and source/destination; capture current neighbor state; compare learned routes to the installed RIB; inspect CEF for the specific destination; test a representative flow; compare expected primary and alternate path; preserve logs before clearing counters or resetting sessions. For the VXLAN path, begin with underlay/VTEP reachability, move to EVPN route exchange and NVE/VNI state, and then verify MAC/ARP/gateway information and actual endpoint communication. The correct investigation branch depends on the earliest failed dependency.

For the evidence collector, the runbook should state its own health checks: required targets, credential/transport health, per-command capture status, parser errors, and the identity of each output artifact. A missing file is a collection problem to report; it must not quietly become an empty “healthy” response. Incidents with incomplete evidence need an explicit handoff to the operator, not a fake root cause.

When I ran Tier-3 telecom bridges, I often had to establish whether the network or application owned the failure. I worked from incident timing and source/destination details through path tracing, interface and firewall counters, QoS behavior, sender/receiver captures, and Wireshark analysis. A good runbook supports that process. It does not replace the engineer's judgment or encourage random configuration changes during diagnosis.

10 / Close the engineering ticket with proof

I would accept closure of a corrective-action ticket only when its specified behavior is demonstrated. If the task says “detect a new current-only configuration,” show a fixture that was previously missed and now causes the required failure. If the task says “validate alternate BGP forwarding,” show the expected path before the fault, the selected backup while the fault is present, and restoration with real source-to-destination traffic. If the task says “complete six-router collection,” show the expected inventory and six fresh, nonempty required captures.

The proof should be linked to the code or configuration revision that introduced the action, the test job or incident record, and the approving engineer. For production systems, verify within the permitted change window, preserve a rollback route, and monitor for regression after restoration. Where the data is incomplete, mark the result unverified and retain the outstanding risk. A software test can pass while a physical path remains wrong; a healthy service probe can pass while a baseline checker is incomplete. Both dimensions need independent acceptance.

Google's SRE postmortem guidance places emphasis on concrete, owned, tracked actions rather than vague promises to be more careful. The current NIST SP 800-61 Rev. 3 addresses cybersecurity incident response within broader risk management; I use it here as a supporting model for disciplined incident improvement, not as a claim that this routing lab underwent a NIST compliance assessment. GitLab's documented permission and approval controls are similarly reference material for the proposed change lifecycle, not evidence that the historical lab employed all of them.

The strongest outcome from these Network SRE experiments was not that every fault disappeared. The work made specific engineering disagreements visible: the BGP route was learned but not installed as intended; the interface recovered before a future engineer could have reached the console; the L3 VNI depended on a routed SVI detail; an incident folder existed with no CLI evidence; and a successful pipeline did not prove every possible drift condition was checked. Each finding identifies a testable next improvement.

That is the operating approach I would take into a larger environment: stabilize the service, preserve the timeline, explain the path, give the remaining defect an owner, turn the defect into a regression test, and close the action only against actual evidence. The incident can be over while the reliability work remains active. The engineering record should make that obvious.