01 / The lab and the questions it needed to answer
The first topology was a routed chain: R1 at the edge, followed by internal routers R2, R3, and R4. R1 connected through the CML external connector, while automation-1 provided the GitLab instance, local GitLab Runner, Ansible inventory, and playbooks. The transit addressing was revised as the lab developed to avoid overlap with the home management network.
I needed to establish four things for each change: whether the routers were reachable from automation, whether OSPF adjacencies and learned routes matched the expected design, whether current configurations differed from the reference, and whether that difference was important enough to interrupt the workflow.
Those are separate checks. A configuration can match a saved baseline while a neighbor is down. A router can also be healthy but temporarily unreachable from the automation host because the management path is broken. Treating both cases as a successful configuration comparison would hide the very problems I wanted the workflow to detect.
02 / The first troubleshooting problem was the management path
Before the automated collection was dependable, I had to straighten out the network path between automation-1 and the routers. The CML lab initially had an addressing overlap with the external home network. I revised the internal transit addressing and worked through the edge routing, SSH access, NAT behavior, and return routes.
An important checkpoint came when the 192.168.1.0/24 management-side network was advertised into OSPF. R2 learned a path through R1 and could reach automation-1 at 192.168.1.41. That verified bidirectional reachability at the point where it was needed for automation. It did not, by itself, prove every SSH and playbook operation was correct, so I validated those separately.
This was a useful reminder from years of production troubleshooting: when a collection task times out, start with the actual route and session path. Check the source, destination, return path, access controls, and listening service before debugging the automation code. A playbook cannot compensate for a missing route.
Management reachability was a prerequisite for the experiment, not a side issue. If the collector cannot reach a device, the correct outcome is collection incomplete. It should never be silently translated into no drift detected.
03 / Building the baseline around real device state
The ospf-sre-lab repository gave the work a consistent home. The retained April repository inventory identified three key Ansible playbooks: playbooks/backup_configs.yml, playbooks/collect_state.yml, and playbooks/validate_ospf.yml. The project also held the Ansible inventory, baseline and backup directories, drift comparison scripts, and a GitLab CI definition.
The tasks served different purposes. Configuration backups recorded what each device was running. Operational state collection captured a view of routing and interfaces around the test. OSPF validation evaluated neighbor health and expected default-route behavior. Configuration comparison looked for differences between a new snapshot and the reference files.
I kept the checks separated because their failure meanings are different. If OSPF loses a neighbor, I need protocol and interface evidence. If the router is unreachable, I need to investigate the management path. If its running configuration changes, I need a focused diff and a record of the approved baseline used for comparison.
GitLab provided a repeatable execution point for the configuration drift comparison. The retained CI configuration invokes the comparison script; it does not invoke the separate Ansible OSPF health-validation playbook. It did not turn Git into a live copy of the network. Repository files represented a reviewed reference; Ansible still had to obtain a fresh observation from the routers.
04 / The drift test: one meaningful change and one false alarm
I deliberately changed a description on R3 to test whether the comparison would detect a genuine difference. The test record includes the marker PHASE1_TEST_DRIFT. The preserved Git history confirms that change, and the comparison script is designed to return a nonzero status when normalized configurations differ. I could not independently verify the original failed GitLab job execution from the retained repository evidence. This mattered because the job needed to communicate failure, rather than merely print a warning and exit successfully.
At the same time, R1 was showing formatting-only differences, including extra IOS configuration separator lines (!). A plain text diff gave those differences the same visual importance as the deliberate R3 change. In one review, the comparison also surfaced volatile configuration metadata and certificate-related noise. Those lines could obscure the actual engineering finding.
I refined the normalization so repeated separator noise stopped creating false positives. The important constraint was preserving the configuration content that could affect routing or device behavior. A normalization rule that simply discards every unfamiliar line would be dangerous. The filtered output needed to remain faithful to the running configuration, even while becoming easier to review.
The final Phase 1 repository checkpoint, bb0df1d on April 17, 2026, was titled “Finalize drift normalization and clean Phase 1 baseline.” That is the documented stopping point for this OSPF phase. It captures the practical result: an intentional drift finding, a comparison problem exposed by normal device output, and a revised clean reference after normalization work.
05 / Making failures visible to GitLab
A pipeline status is only as meaningful as the checks feeding it. The GitLab drift job invokes scripts/diff_configs.sh. That script compares normalized intended and collected configuration files, sets DRIFT_FOUND=1 on a difference, and exits with that status. The failure-signaling logic is verified in the source code; the historical CI job result remains unverified.
But I also ran into a more basic reliability issue. When unreachable devices were included in state collection, stalled Ansible connection attempts could consume significant resources on automation-1. The recorded troubleshooting included serial execution with ansible-playbook -f 1 and recommendations for reachability prechecks, shorter connection timeouts, and bounded retries. Those safeguards are sensible for a production design; the available OSPF checkpoint does not prove that every proposed guard was incorporated into the final repository.
This distinguishes an automation error from a network finding. An SSH timeout, a malformed inventory, a failed OSPF assertion, and a configuration mismatch should not all end up as an undifferentiated red job. At minimum, the report should identify the device, attempted task, failure class, and whether sufficient evidence was collected to make a drift judgment.
For a stronger operational workflow, I would enforce these outcomes:
| Observation | Appropriate status | Engineering response |
|---|---|---|
| All required devices collected, expected OSPF state verified, no unexpected diff | Pass | Retain timestamped evidence |
| OSPF neighbor or expected route missing | Validation failure | Review adjacencies, interfaces, LSDB and RIB |
| Unexpected R3 configuration difference | Drift finding | Review change authorization and effect |
| Router unreachable or backup incomplete | Collection failure | Restore management visibility before claiming health |
| Difference limited to known harmless formatting | Normalized comparison | Keep the raw capture and auditable filter rule |
The table describes an improved reporting contract, rather than a claim that the original lab implemented all five result categories as dedicated GitLab jobs.
06 / What I would harden before relying on it at scale
The OSPF lab demonstrated a useful foundation. It was not a production change-management platform, and I would not treat the April checkpoint as one.
The next version should record which baseline commit was used for every test, preserve raw configuration snapshots alongside normalized copies, and stop the workflow when any required device is missing. The normalizer should have explicit test fixtures showing that volatile formatting is removed while real interface, routing, or security changes remain visible. The comparison should also check both baseline and current file inventories so an unexpected new file cannot disappear from the assessment.
Access control deserves the same care. Read-only collection credentials should be separate from configuration-deployment credentials. GitLab Runner privileges and inventory secrets should be limited. A detected drift should produce evidence and a review decision, not an automatic overwrite of the router with whatever happened to be stored in Git.
These are engineering improvements to make after testing a small system honestly. The lab exposed the places where state collection, validation, and comparison could fail independently. That information is more valuable than claiming every green pipeline result proves the network is healthy.
07 / What this first project established
By the end of Phase 1, I had a four-router OSPF lab with a working automation path, Ansible collection and validation playbooks, GitLab integration, and an intentional configuration-drift test that surfaced in the validation workflow. I had also found and corrected a normalization issue that made harmless separator differences look significant.
Just as important, I had worked through the management-path dependencies that made collection unreliable in the first place. The technical lesson carried directly into the next projects: establish a trustworthy baseline, verify that the observation is complete, distinguish a device failure from a collector failure, and preserve the evidence behind a pass or fail result.
The next phase added BGP redundancy, Prometheus monitoring, Alertmanager, and incident collection. That would let the same discipline move from periodic configuration and state checks toward capturing what happened when a network event occurred.
Source record and publication notes
This article describes my independently operated Cisco CML OSPF Phase 1 lab, not a customer production environment. The following source references anchor the article:
- Repository:
network-lab/ospf-sre-lab, reviewed local path/home/byoung/netlab/ospf-sre-labonautomation-1. - Last reviewed Phase 1 checkpoint:
bb0df1d(April 17, 2026), Finalize drift normalization and clean Phase 1 baseline. - Evidence-bearing components:
README-lab-specific.md,playbooks/backup_configs.yml,playbooks/collect_state.yml,playbooks/validate_ospf.yml,inventory/hosts.yml,.gitlab-ci.yml, and the configuration comparison script. - Specific documented findings: management-side OSPF return reachability; a deliberate R3 description drift; R1 separator/metadata noise; resource pressure from unreachable-device collection.
Evidence note: The diagrams and drift comparison graphic are explanatory reconstructions, not raw device screenshots. The original repository remains the authoritative record of the code and CI logs.