The Network Is Up.
Who Owns the Next Incident?

A network can pass its build checklist and still be difficult to operate. The interfaces are up, the routing neighbors are established, and the deployment report is green. Then the engineer who built it signs off, and someone else receives the first call at 2:00 AM. They need to know which route should win, which service path matters, what the alarms can actually prove, and whether the automation collecting evidence is healthy enough to trust.

I have spent much of my career on the receiving end of that handoff. In enterprise and telecom Tier-3 operations, a useful escalation starts with incident timing and a source/destination pair. From there I work through the path, routing and firewall decisions, interface counters, QoS, and packet evidence when required. I do not want to spend the first half of the incident reconstructing the design from old configuration files or discovering that a “successful” capture contains nothing from the affected router.

My independent Network SRE work developed across three distinct Cisco Modeling Labs (CML) environments: a four-router OSPF configuration-control project, a six-router BGP incident-response project, and a two-spine, three-leaf Cisco NX-OS VXLAN EVPN fabric with GitLab-based validation. Those labs established useful pieces of an operating practice. They did not become one deployed production network, and I did not perform a formal production handover. Article 10 takes the findings already recorded in those projects and assembles the readiness review I would require before transferring an equivalent service to an operations team.

01 / Treat acceptance as an operator exercise

A diagram and an approved configuration are necessary handoff material. They are not a demonstration that the receiving team can operate the network. My acceptance question is deliberately practical: Can an engineer who did not build this environment identify the failing dependency and make a safe next decision using the information provided?

That question has five parts. The operator must know the intended service, the current observed state, the signals that indicate deviation, the permitted intervention, and the evidence required for closure. If one part is missing, the handoff is incomplete. An empty packet-capture location is not a substitute for a path test. A list of devices is not a service dependency map. A green CI result is not a guarantee that every expected device was queried.

The difference resembles a production readiness review (PRR). Google's SRE literature describes PRRs as checks of production setup, operating expectations, and service ownership before SRE assumes responsibility. I am borrowing that review discipline, not claiming our CML labs underwent Google's formal review process. In a network organization, the receiving NOC, escalation engineers, automation owners, and change authority must understand their respective duties rather than assume the previous engineering team will be awake to explain everything.

The output I want is a compact, versioned operational acceptance record: topology and dependency scope; known-good paths; alarm and probe coverage; exact runbook links; management-access test; change and rollback boundaries; named owner roles; and open risks. The record should identify its last verification date and revision. Without those fields, a handoff can quietly age into fiction.

02 / Start with the known-good path, not the inventory spreadsheet

Each lab tested a different question. The OSPF project was about repeatable configuration and neighbor verification. The BGP project made alternate routes and incident signals visible. The VXLAN fabric added an underlay, EVPN control plane, VXLAN tunnel state, tenant gateways, and endpoint reachability. An operator should not mentally collapse them into one topology just because all three used GitLab or automation-1.

For the six-router BGP lab, the important diagram runs from ISP1 → R1 → R3 → R4 → ISP2, with R1 → R2 → R4 available as the alternative internal path. The acceptance sheet must identify the destination prefix, preferred next hop, backup next hop, and the source and destination used in the traffic test. If the intended path is R3, reaching the destination over R2 while R3 is healthy is not the same as passing the routing-policy check.

The April 21 exercise on R4 proved why this matters. R4 had a static route to 172.16.1.0/24 via 10.20.0.13. A BGP alternative existed, but the static route interfered with intended route installation. After removing the conflicting static entry, the BGP route installed through R3, next hop 10.20.0.17; CEF and ping checks succeeded. A handoff that simply stated “BGP neighbor established” would not have detected the original path error.

For the NX-OS EVPN fabric, the necessary record includes SPINE1/SPINE2, LEAF1/LEAF2/LEAF3, the VTEP roles, and the tenant mapping: VLAN 10 / L2 VNI 10100, VLAN 20 / L2 VNI 10200, and L3 VNI 99999. The lab's test endpoints were 10.10.10.101, 10.10.10.102, and 10.10.20.103. Those values make the troubleshooting path testable rather than abstract. They are lab addresses, not a customer's production addressing plan.

The dependency map must also record management access, the telemetry collection path, and the location of the authoritative baseline. In a live environment, it would include circuit and application owners, routing-policy intent, approved maintenance windows, and escalation contacts. No such real owner names or production circuit identifiers are invented here.

03 / Tell the next engineer what each green signal fails to prove

An operator does not need twenty dashboards. They need the right distinctions. OSPF FULL says the underlay neighbor relationship formed. BGP Established says the session is up. NVE peer Up and VNI Up describe specific overlay conditions. None of those statements alone proves that a tenant's application flow crosses the correct gateway and reaches the intended endpoint.

The VXLAN lab exposed the gap when L3 VNI 99999 remained down until Vlan999 had the required ip forward configuration. That was a specific configuration and operational-state fault, not a general collapse of the underlay. An incident playbook should branch at the earliest failing dependency: loopback reachability first, then EVPN route distribution, NVE/VNI state, MAC and ARP learning, the VRF/gateway, and finally representative endpoint traffic. Cisco's NX-OS verification guidance supports treating overlay control-plane and forwarding views as distinct checks; the exact CLI syntax depends on software version and feature set.

In the BGP project, the same principle runs from learned BGP path to installed RIB route, then CEF forwarding next hop, then source-to-destination test. An unexpected static route can preserve reachability while violating intended path policy. That is an operational defect even when the probe is green. A dashboard can show an established neighbor and still conceal an invalid release decision.

For a receiving team, I would therefore define three classes of signal: availability (does the representative service flow work?), path correctness (is it traversing the authorized forwarding decision?), and observation quality (is the collector able to prove either claim?). This is an operator-focused model, not three historical SLO dashboards already implemented across all labs.

04 / Give alarms a route into an evidence bundle

The BGP project supplied an actual monitoring-to-collection pattern. On automation-1, Prometheus, Alertmanager, syslog, Python, Bash, and Ansible helped turn an alert into a device-level incident record. An Alertmanager webhook triggered the Python-side investigation workflow. That is not the same as a device configuration-change webhook launching GitLab CI; the latter was discussed as a future extension and was not established as an executed end-to-end flow.

The retained April 22 BGP incident template supplies a useful handoff example: R1 reported neighbor 10.20.0.10 resetting at 18:42:00 UTC and Ethernet0/2 administrative-down and line-protocol events at 18:42:02–18:42:03. The formerly described R4 ten-second interval was not independently corroborated. An incoming operator should receive the device-event timestamps alongside the unresolved recovery status and a request for forwarding/service verification, rather than a claimed outage duration.

The early incident collector gave the other side of the lesson. It created incident directories but initially left some intended router CLI outputs empty because interactive SSH authentication interrupted unattended collection. Following collection-path repairs, the April 23 test collected from ISP1, R1, R2, R3, R4, and ISP2, with zero failed hosts reported by Ansible. That verified progress. It did not create an all-purpose guarantee that every later collection would succeed.

I would make the collection bundle self-describing: incident ID, alert event time, collector observation time, expected target list, per-device output status, command/parse errors, and references to raw CLI and telemetry. A result should say CAPTURED, EMPTY, TIMED_OUT, or FAILED at artifact level. “Directory created” is not a collection success criterion. And when the interface has recovered before inspection, the operator needs both the past event and current observed state, not a forced binary conclusion that the alert was false.

05 / Separate the people who detect, diagnose, approve, and change

Handoffs fail when responsibility is implied. In a larger enterprise, I would give the NOC responsibility for initial impact assessment, alert correlation, customer communications, and escalation against a runbook. The network escalation engineer owns protocol/path diagnosis and determines the safe technical intervention. The automation/platform owner is accountable when the collector or validator is broken. The change authority approves intended production changes and risk exceptions under the organization's change policy. The service owner supplies the business-impact view and validates meaningful service recovery.

Those are roles, not invented individuals attached to the old labs. A small team may have one engineer wearing several hats. Even then, the ticket should identify which decision was made in which capacity. The person who detects configuration drift should not automatically promote the drifted configuration into the approved baseline, and a webhook receiving untrusted network events should never acquire the authority to rewrite routers.

An initial NOC handoff might read: “R1 neighbor 10.20.0.10 reset at 18:42:00 UTC; Ethernet0/2 administrative and line-protocol down events at 18:42:02–18:42:03; live neighbor state and service path impact not yet verified; follow-up source/destination check assigned to escalation.” That statement gives the next engineer a bounded working picture. “BGP down, network outage” is faster to type but often less true.

The escalation handoff must retain time context and the exact path. If service is reachable on a backup route, the incident priority may still be elevated because the design has lost redundancy. If traffic is failing while adjacencies are green, the case should move immediately to installed routes, CEF/forwarding, gateways, and endpoint evidence. A runbook that insists on retrying the same control-plane command five times adds delay without insight.

06 / Make changes reversible before making them routine

A receiving operator must know what changes are permitted and how to stop. In the reviewed VXLAN GitLab workflow, the recorded stage order was validate → deploy → baseline → post_validate, with deployment and baseline steps manual. The October 13 review identified a sequencing concern: baseline promotion occurred ahead of post-validation, with rsync --delete able to replace stored reference files. The pipeline was real, but that sequence was not the stronger production control I would adopt.

The proposed order is: record and preserve the known-good baseline; verify the pre-change service and device inventory; check approval and maintenance scope; apply only the authorized change; validate protocol state, installed forwarding and representative traffic; review evidence completeness; then approve promotion of the new baseline. If required checks cannot be performed, that is an UNKNOWN, not an automatic PASS. For rollback, record the exact restored version and separately verify service, rather than assuming that an undo command guarantees recovery.

This is also a security boundary. GitLab documents different behaviors for optional and blocking manual jobs; a manual button is not itself change approval. Protected environments and deployment approvals can impose additional controls where supported by the organization's GitLab edition and configuration. The original shell-executor Runner on automation-1 also required trust in the host and pipeline writers; GitLab's security guidance describes meaningful risk in shell execution. I would isolate read-only collectors from privileged deployment credentials, protect runner scope, and retain audit evidence for all manual decisions.

07 / Let the incoming engineer run a short drill

I would not transfer a new fabric to on-call rotation based solely on the builder presenting screenshots. The receiving engineer should walk through a safe, time-boxed tabletop or isolated lab drill, using an incident they did not design. Three historical findings offer good scenarios, each with a distinct expected response.

Scenario A: R4 takes the wrong path. The operator receives a routing complaint while BGP peering is healthy. They must compare BGP route availability with the installed RIB and CEF decision for 172.16.1.0/24, identify the unexpected static next hop, and check the intended traffic path. Success is the correct diagnosis and documented change recommendation, not automatically deleting the route.

Scenario B: The EVPN control plane looks healthy, but tenant routing is down. The operator works through OSPF/VTEP transport, EVPN session visibility, NVE peers and required VNIs, and identifies the fault domain around Vlan999 / L3 VNI 99999. Success is choosing the right diagnostic branch and defining a valid tenant forwarding test. This uses a documented historical problem; it is not a claim a fresh drill was run for this article.

Scenario C: The alarm fired but the evidence bundle is incomplete. The operator receives an Alertmanager event with six expected router targets and one missing CLI snapshot. They must distinguish evidence failure from network health, preserve valid partial artifacts, and escalate collection reliability separately. Success is reporting incomplete evidence explicitly, not treating five available devices as a six-device success.

Before attempting any live fault injection, I would establish an isolated lab or a formally approved maintenance envelope, define rollback and stop conditions, and explicitly agree on acceptable service risk. For a real production ownership transfer, the tabletop can be the first gate; controlled maintenance testing comes later. The accepting engineer must be able to say “I cannot prove service recovered yet” when the data is insufficient.

08 / Measure whether the handoff works, not how many pages it has

A long runbook can still leave the operator stranded. The useful measures ask whether the team can carry out a decision. I would track time to a defensible fault-domain statement, percentage of incidents with complete required evidence, percentage of changes with explicit post-change service probes, runbook execution failures, and time to identify the accountable escalation role.

These are proposed operational indicators. The Cisco CML projects did not collect a validated 30-day handoff SLO series or prove a specified reduction in mean time to recovery. A numerical target would need a baseline, a defined observation window, and consistent handling of skipped/unknown results. For example, the denominator for capture completeness should be the complete required target list, not merely the devices that returned files. Otherwise the metric rewards a broken inventory.

I would also make the runbook's age visible. Every service dependency changes: a new VNI, a routing-policy update, a moved management address, a collector credential rotation, a new application flow. Changes should update the operational record and trigger a small regression check. That does not require an elaborate autonomous system. It requires a durable ownership rule that is reviewed when implementation changes.

The evidence retention policy matters as much as the alert. Store the incident ID, device and command provenance, review decision, and artifact lifecycle in protected storage. Strip tokens, passwords and unnecessary identifying details before publishing examples. Keep private operational details out of public screenshots. A public engineering article and an internal production runbook have different disclosure boundaries.

09 / Be explicit about what the existing labs proved

A readiness review has to distinguish implemented, observed, proposed, and unverified. The distinction is especially important here because these labs were educational systems. I can show specific test outcomes and design choices; I cannot turn them into a fabricated production acceptance sign-off.

A formal handoff record would add owner sign-offs, exception approvals, recovery references, test-job links and dates. None of that is present here as a newly executed event. Where evidence is missing, I would leave the finding open and identify what test is required next instead of writing “accepted” because the diagram looks finished.

10 / What I would hand the operator on day one

I would keep the final handoff small enough to open during an incident: a one-page topology with the intended traffic path; the versioned source of approved configuration intent; current access and device inventory; protocol and service verification commands qualified by platform; alert meanings and their blind spots; the exact capture/report location; escalation roles; maintenance/rollback boundaries; and a list of known exceptions with owners and review dates. Behind that front page sit the configuration repository, raw evidence, diagrams, post-change results and full troubleshooting references.

The first receiving engineer should be able to answer a few questions without calling the project builder. Which service is at risk? Which device and forwarding decision matters? Which test proves it? Which collection failures limit the conclusion? Who can authorize a change? How do we restore and verify the prior state? Those are the decisions that determine whether the response is controlled or improvised.

For the original BGP project, a complete handoff would have led an operator directly from the R4 prefix complaint to the learned-route/RIB/CEF discrepancy. For the VXLAN lab, it would have prevented OSPF adjacency health from masking the down L3 VNI. For the webhook collector, it would have flagged empty CLI artifacts as a failure in the observer rather than proof that the routers were healthy. Those are concrete examples of how the record changes the next action; none relies on an invented outage metric.

This is the final standard I would apply when promoting a network from a successful build to an operable service: the person taking responsibility can reproduce the health checks, locate the limits of the evidence, follow an approved recovery decision, and hand the next shift a record that remains useful when the alarm has cleared. The network may already be forwarding. Operational readiness is what allows the next engineer to keep it that way.