From Manual Operations
to Network SRE.

01 / Why I started building this series

After more than two decades working across telecommunications and enterprise networks, I wanted to examine how much of the routine work around changes and incidents could be made repeatable without losing the engineering judgment behind it.

In Tier-3 escalations, I have worked through the same practical sequence countless times: establish the incident timeline, identify the source and destination, trace the actual forwarding path, inspect protocol and interface state, and compare packet captures when the evidence warrants it. The challenge is getting those facts assembled quickly and consistently, especially when the problem is intermittent or a device recovers before anyone connects to it.

Cisco Modeling Labs gave me a controlled place to test an operating approach. I could intentionally introduce faults, collect device state, compare results against a known baseline, and check whether automation surfaced a real network problem instead of merely reporting that a script completed.

I organized the work into three successive lab environments. Each had its own topology and purpose. The shared thread was a disciplined way to establish expected state, detect change, preserve evidence, and validate recovery.

02 / How the work progressed

The sequence mattered. I began with familiar routing and configuration-management problems, expanded into BGP fault detection and incident collection, and then brought automated checks into a more complex VXLAN EVPN fabric.

03 / OSPF: Build a reliable baseline first

The first topology used four Cisco routers in CML: an edge router, R1, and three internal routers, R2 through R4. I used automation-1 to run Ansible and work with GitLab and the local GitLab Runner.

I collected running configurations and routing state, saved device-specific snapshots, and compared subsequent collections against approved baselines. The work also checked OSPF adjacency health and expected default-route behavior. Those checks gave me a repeatable record of what the network looked like before and after a change.

One practical lesson came from configuration drift. A raw text comparison included changing fields that contributed little operational value. I adjusted the comparison to filter volatile content while retaining changes that could alter forwarding or routing behavior. That made the result more useful for an engineer reviewing an unexpected difference.

Management reachability also had to be correct. Ansible could not collect meaningful evidence until SSH, return routing, and NAT worked as expected. That experience reinforced the need to validate the management path before interpreting automation failures as device failures.

04 / BGP: Capture evidence while the event is happening

I then built a BGP environment to test redundant forwarding paths, fault detection, and incident response. During failover exercises, routing details, including static routes that affected the expected path, had to be corrected before the test behavior matched the design.

The monitoring stack added Prometheus, Telegraf, Blackbox Exporter, Grafana, Alertmanager, and centralized syslog. I used it to observe routing and reachability conditions and to route alerts into an automated evidence-collection workflow.

When an alert fired, Alertmanager could send a webhook to a Python listener. The listener initiated collection of available Prometheus data, syslog, and router CLI output through Ansible. The lab records document incident capture during failure and recovery events, along with the evidence files produced by the collector.

This was the most direct connection to my operations background. The alert told me where to start. The collected state helped reconstruct what happened without relying solely on what the network looked like after recovery. It also gave me a clearer record to investigate when a session was down but another path continued to forward traffic.

05 / VXLAN EVPN: Validate the complete fabric state

The final environment raised the level of complexity. I built a Cisco NX-OS spine-leaf fabric in CML with two spines, three leaves, and test endpoints. OSPF provided underlay reachability; MP-BGP EVPN carried overlay reachability; VXLAN connected the virtualized Layer 2 and Layer 3 segments.

The project used Layer 2 VNIs 10100 and 10200 , Layer 3 VNI 99999 , and anycast gateway configuration. I tested underlay adjacencies, BGP EVPN state, NVE peers and VNIs, and connectivity between lab endpoints.

One troubleshooting example captures why a configured feature cannot automatically be treated as operational. The Layer 3 VNI remained down until the relevant SVI configuration included interface Vlan999 with ip forward . Early endpoint probes could also be affected while ARP and EVPN entries populated. Those observations informed the validation checks.

The fabric made a useful test case for automation because passing an OSPF check alone could not establish that overlay signaling, VNI state, and endpoint forwarding were all healthy.

06 / Make the checks executable with GitLab and pyATS

Across the projects, I moved collection and validation tasks into repeatable workflows. The VXLAN repository brought those ideas together with pyATS, Ansible, Bash, Python, and GitLab CI/CD. Git tracked the intended automation and configuration artifacts; the runner executed jobs against the live lab.

The pyATS validation ran checks against LEAF1, LEAF2, and LEAF3 concurrently. The implementation also needed to preserve which device failed and return a failing process status so GitLab could report an unsuccessful test correctly.

A retained execution provides concrete evidence. GitLab pipeline #124, job #190 , associated with merge-request commit 31d7f689 , invoked ./scripts/run_validation.sh on automation-1 . The job reported OSPF neighbors FULL, NVE interfaces up, and required VNIs up on all three leaves. Ansible collected running configurations and operational state from the five fabric devices. The drift test reported no differences, and the job completed successfully.

=== Running pyATS VXLAN validation ===
LEAF1: OK - OSPF neighbors are FULL
LEAF1: OK - NVE interface is up
LEAF1: OK - All required VNIs are Up
LEAF2: OK - OSPF neighbors are FULL
LEAF2: OK - NVE interface is up
LEAF2: OK - All required VNIs are Up
LEAF3: OK - OSPF neighbors are FULL
LEAF3: OK - NVE interface is up
LEAF3: OK - All required VNIs are Up
LEAF1: PASS
LEAF2: PASS
LEAF3: PASS
Overall Result: PASS
=== Running drift detection ===
No drift detected.
Overall Result: PASS - No drift detected
Job succeeded

Job #190 confirms the OSPF, NVE, VNI, configuration-collection, and drift checks shown in its log. The broader EVPN and endpoint-reachability tests will be examined in the dedicated fabric article.

The reviewed GitLab workflow defined validate , deploy , baseline , and post_validate stages, with manual controls on deployment and baseline handling. Repository changes could initiate CI execution. That is a different trigger from Alertmanager sending a network incident webhook; each had its own implemented purpose.

A later source review also identified baseline-promotion ordering and file-inventory drift checks worth hardening. Those findings deserve technical treatment in the CI/CD article, alongside the successful tests, rather than a general warning that obscures the functioning workflows.

07 / What I took away from the completed labs

By the end of this sequence, I had exercised three practical capabilities: repeatable collection and comparison of network state; alert-driven preservation of incident evidence; and GitLab-executed validation of a Cisco VXLAN EVPN lab.

The projects also exposed failures that would have been easy to overlook with a green status alone. Management reachability could break the collection path. Static routing could defeat expected failover. A VNI could be configured but down. A comparison script could miss a class of file changes. Each required inspecting actual network or automation behavior and refining the check.

I used AI-assisted scripting for portions of the Python automation. I defined the engineering objectives, assembled the environments, integrated and tested the components, interpreted the output, and troubleshot the network and automation failures. That division of work is part of the technical record.

These labs established a foundation for applying reliability engineering practices to networks I already knew how to operate. They also produced the code, failure cases, logs, and diagrams that the rest of this series can examine in detail.

08 / Where the technical series goes next

This introduction establishes the scope. Each follow-up article will focus on implementation details, failure scenarios, and test evidence for its part of the lab series. The links below currently return to the relevant overview sections and will point to the full articles as they are published.

09 / Lab repositories and supporting evidence

I maintained the OSPF, BGP, and VXLAN EVPN projects in separate GitLab repositories, preserving configurations, automation workflows, test results, and supporting documentation from each phase. GitLab pipeline #124, job #190, provides a concrete example of the automated VXLAN fabric checks discussed in this article. The original console excerpt is available for review. The articles that follow examine the implementations and test results in greater technical detail.