A switch can be healthy while its running configuration has drifted from the approved design. A GitLab pipeline can finish successfully while an important test was never included. And an engineer can make a legitimate emergency change that the repository does not yet recognize.
I have spent plenty of time on Tier 3 escalation bridges reconstructing a network change after the fact: what changed, when, on which device, and which packet path was actually affected. Getting those answers depends on clean evidence. In large environments, no one wants to discover that a green automation dashboard only checked half the things everyone assumed it checked.
The earlier articles in this series established the routing and fabric side of that problem. The OSPF and BGP labs examined operating state and incident evidence. The Cisco CML VXLAN EVPN lab joined two spines, three leaves, three endpoints, and a Linux automation host. For this article I am following the operational record one level higher, into GitLab: how a version-controlled workflow can test the fabric, compare configurations, and support a controlled change.
The lab did run GitLab jobs, use pyATS and Ansible against the CML fabric, collect configurations, and compare snapshots. It did not demonstrate a complete device-change-webhook-to-GitLab production workflow or an independently verified automatic rollback. Those boundaries matter because the distance between a working lab and a trusted change system is mostly in the failure paths.
01 / What Git can be authoritative about
I use Git as the authoritative record of approved intent: configuration definitions, automation code, expected baselines, review history, and change decisions. It is not an authoritative measurement of what an NX-OS switch is currently forwarding. That information has to come from the switch and, where possible, a real endpoint test.
There are three records to keep straight:
| Record | What it answers | Where the lab looked |
|---|---|---|
| Intended configuration | What change or state was approved? | GitLab repository and stored baselines |
| Observed device state | What was actually configured and operational? | NX-OS running configuration, OSPF, NVE and VNI outputs |
| Validation result | Did the defined checks pass at a particular time? | GitLab Runner job logs, pyATS, Ansible and drift output |
A mismatch between the first two records is drift, but drift is not automatically an outage. A mismatch between the second and third can expose a weak validator: it may fail to test an important dependency, mis-handle an exception, or accept stale state. The deployment process has to account for both conditions.
That distinction was especially important in the lab's EVPN environment. A successful OSPF adjacency check could coexist with a broken L3 VNI. Article 04 documented the practical example of Vlan999 and its required ip forward configuration for L3 VNI 99999. Every automation gate has to say precisely which network behavior it verifies.
02 / The environment behind the pipeline
The implementation used Cisco NX-OS devices running inside Cisco Modeling Labs: two spines and three leaf VTEPs. OSPF carried the underlay, while MP-BGP EVPN and VXLAN supported the overlay. The validation host, automation-1, provided GitLab, a Linux GitLab Runner using the shell executor, and the Ansible, Python, Bash, and pyATS tooling.
A successful runner requires two independent capabilities. GitLab has to schedule a job and the runner has to execute it; separately, the runner's host must reach and authenticate to the NX-OS management interfaces. An application pipeline can succeed while the network is offline because it never needs that second path. A network CI job cannot make the same assumption.
The reviewed VXLAN repository contained four stages named validate, deploy, baseline, and post_validate. deploy and baseline were configured as manual operations. Those details describe the reviewed lab definition. Whether a manual job blocks later stages depends on its GitLab configuration, so the presence of a manual button alone is not equivalent to an enforced approval gate.
.gitlab-ci.yml stage order. The risk annotation is an engineering review finding; it is not a claim that a bad baseline was actually promoted.03 / The real job: what PASS established
One retained execution reference is concrete enough to examine. The prior project review identifies GitLab pipeline #124, Job #190, merge-request commit 31d7f689, which ran ./scripts/run_validation.sh through the Linux shell executor. The excerpt preserved for Article 04 reported all three leaves passing the pyATS assertions for OSPF neighbor state, NVE interface state, and required VNI state.
=== Running pyATS VXLAN validation ===
LEAF1: OK - OSPF neighbors are FULL
LEAF1: OK - NVE interface is up
LEAF1: OK - All required VNIs are Up
LEAF2: OK - OSPF neighbors are FULL
LEAF2: OK - NVE interface is up
LEAF2: OK - All required VNIs are Up
LEAF3: OK - OSPF neighbors are FULL
LEAF3: OK - NVE interface is up
LEAF3: OK - All required VNIs are Up
LEAF1: PASS
LEAF2: PASS
LEAF3: PASS
Overall Result: PASS
In that same reviewed execution, Ansible collected VXLAN information and running configurations from five devices. The recap recorded unreachable=0 and failed=0 for each spine and leaf. The drift step reported No drift detected, and GitLab ended the job successfully.
I would show that result to an operations team as evidence of a functioning collection and validation path. I would not show it as proof that every EVPN route, every endpoint flow, and every possible configuration difference had been checked. The printed assertions establish OSPF/NVE/VNI status on the leaves. They do not show a distinct Type 2/Type 3 route assertion or an endpoint ping assertion in that retained excerpt.
The later review of the comparison code identified a coverage defect. That means the successful job and the defect can both be true. A runner doing exactly what the code asks is still capable of giving an incomplete operational answer.
04 / Configuration drift needs a complete inventory
A text comparison sounds like an easy control until you put actual device output into it. Timestamps and regenerated comments create noise. Some lines should be normalized; protocol configuration, neighbor shutdowns, VLAN/VNI relationships, and route policies generally should not. Aggressive normalization can conceal a real defect just as easily as weak normalization produces false positives.
The lab's Python-based drift check normalized selected volatile content and compared the current collected configuration against stored baseline files. The October 13 code review identified a specific asymmetry: it found a baseline file that was absent from current collection, but did not reliably flag a new current file that had no counterpart in the baseline.
Suppose the approved inventory contains LEAF1.cfg, LEAF2.cfg, and LEAF3.cfg. The collector now produces a fourth file for an additional device. Iterating only over baseline filenames can find a missing LEAF2.cfg; it can miss the unexpected fourth file entirely. A completed comparison should reconcile the union of filenames across both sets, report additions and deletions separately, and then diff content for common files.
The review also found duplicate normalization logic. That is not a headline outage, but it becomes a real maintenance hazard: two code paths can quietly disagree about what constitutes harmless volatile output. Before using a drift result as a mandatory change gate, I would consolidate normalization, test known-good and intentionally modified fixtures, test missing and added files, and make incomplete collection a failure rather than a clean PASS.
This is exactly where AI-assisted scripting deserves an engineering review. I directed the lab's network objectives and operated and validated the environment, while portions of the Python automation were AI-assisted. The useful discipline was testing the resulting behavior rather than treating generated code as correct because it executed.
05 / A baseline should be promoted after verification
The strongest operational finding was the order of the original pipeline. The reviewed stage list put baseline before post_validate. The baseline promotion logic used rsync --delete to align the stored baseline with collected configuration files.
That command is legitimate for synchronizing directory trees. It is also unforgiving when the input is incomplete, misdirected, or captured during a degraded state. If baseline promotion runs before post-change validation, the system risks accepting an unhealthy configuration as the new reference. The repository would then report agreement with a baseline that should never have been accepted.
This was a design and sequencing risk, not evidence that the lab suffered an actual bad promotion. The safer lifecycle is to take and retain a pre-change snapshot, validate the existing fabric, review and approve the intended change, deploy through an explicitly authorized action, validate the result, and then promote a new baseline with a record of the reviewed evidence.
I would require baseline promotion to verify collection completeness and the expected inventory before modifying anything. Preserve the prior baseline, generate a reviewed diff, tie promotion to an approved change, and archive which pipeline, commit, devices, and test results justified the update. If post-validation fails, freeze promotion and begin investigation. Restoring service and updating the baseline are separate decisions.
There is another subtle issue: when: manual does not automatically mean the job has enterprise-grade change approval. A protected environment, properly configured blocking behavior, protected branches, required reviews, and controlled runner access all contribute to governance. Some of those GitLab capabilities are edition-dependent. They need to be selected and tested against the deployment actually in use.
06 / Webhooks: connect events to verification, carefully
The lab included an Alertmanager-to-Python incident webhook in a separate monitoring workflow. Its job was to receive an alert and begin incident-oriented evidence handling. The reviewed GitLab VXLAN pipeline, by contrast, was commit/push driven. I have not established from the retained material that Cisco device configuration-change events were already triggering GitLab through its pipeline API.
That is the next integration I would consider, and I would keep it read-only at first. NX-OS syslog, AAA accounting, or another trusted change-event source can tell an intermediary that something relevant may have happened. The intermediary authenticates the event, identifies affected devices, deduplicates repeat notifications, and requests a scoped GitLab validation pipeline. The runner collects observed state and tests the defined assertions. A difference is recorded for human review.
There is no need to trust every inbound event or give it permission to alter the network. The event is a reason to investigate. It should not become an arbitrary command or a branch name supplied unvalidated to a shell. Credentials belong in a protected secret-management mechanism; GitLab trigger tokens must be narrowly controlled, rotated, and prevented from leaking into logs. Use fixed, allowlisted pipeline inputs and device identifiers, rate limits, correlation IDs, and queueing to prevent an event storm from exhausting the runner.
GitLab documents the pipeline trigger API and the handling of trigger tokens. One important detail is that tokens can confer the originating user's project permissions, and trigger variables can override pipeline values. For a design that must fail safely, I would prefer constrained inputs, scoped credentials, and a validation-only entry point. An alert must never be allowed to select a privileged deployment path by setting a convenient variable.
07 / The runner is inside the trust boundary
A GitLab shell-executor runner is an operationally useful choice for a lab: tools and SSH reachability are directly available on the Linux host. That convenience changes the security model. The job runs within the host's process environment, so anyone who can change executable pipeline content may influence what happens on that machine, subject to its permissions.
For an enterprise deployment I would separate collection from configuration deployment and isolate runners according to trust level. A validation job needs read-only device accounts. A deployment job needs tightly scoped elevated access, maintenance approval, a clear device target, and auditing. Protected branches and merge-request review help control what code reaches those jobs; network ACLs and credential boundaries reduce the damage if the controls fail.
I would also keep raw running configurations out of broadly readable artifacts. They can expose addressing, management design, usernames, community strings, and occasionally secrets. Store only the evidence needed for the decision, apply artifact access controls and retention policies, and sanitize material before putting it into a public portfolio.
08 / Failure states deserve explicit names
The production gate I would build around this work needs more than PASS and FAIL. It should distinguish an actual negative network assertion from the absence of a trustworthy answer.
| Result | Example | Operational action |
|---|---|---|
| PASS | All required tests executed on the expected inventory and met their criteria | Permit the next authorized stage |
| FAIL | A required VNI is down or an unauthorized config difference is confirmed | Stop the change and investigate |
| UNKNOWN / ERROR | Device unreachable, collector incomplete, parser exception, artifact unavailable | Fail closed; do not promote baseline |
| REQUIRES REVIEW | Emergency change restored service but differs from approved intent | Preserve evidence and reconcile through approval |
A missing file, a skipped test, or a parser timeout does not become a healthy fabric because the Python process exited zero. I want test completeness, inventory completeness, and unexpected exceptions included in the job's final exit status. For parallel checks, each device's outcome must survive the thread pool and appear in the report.
This is also where meaningful rollout gates belong: precheck assertions for the target device set; a positive authorization record; change execution with a known stopping condition; postcheck assertions at routing, overlay, and endpoint layers; and a human-approved recovery path if one of those assertions fails. There is no universal automatic rollback for arbitrary network config. Any rollback mechanism needs its own tests and guardrails.
09 / Evidence I would expect on a change ticket
A network CI job is most valuable when another engineer can reconstruct the decision without joining a call and asking what the green box meant. At minimum, I would want the proposed change and reviewer, source commit, target devices, runner identity, timestamps, pre- and post-check results, normalization method, full device inventory reconciliation, and the exact artifact that failed or passed.
For this lab, the retained Job #190 console excerpt provides a good starting point: three explicit pyATS leaf assertions, Ansible collection on five NX-OS devices, and a successful job exit. To publish a stronger operational claim, the next evidence pass needs the original .gitlab-ci.yml, run_validation.sh, the detector code and fixtures, real archived GitLab artifacts, and the actual state of the repository at the referenced commit. The diagrams in this article are original engineering reconstructions. None is a synthetic GitLab screenshot presented as a genuine one.
10 / What I would carry into a production network
The lab established a working control path from a GitLab job to an NX-OS fabric. It showed that checks could run across three leaves, that Ansible could collect from the five-device topology, and that GitLab could surface an outcome a network engineer could act on. That work is worth keeping.
The review exposed the more important next steps. Drift detection needed symmetric inventory coverage. Baseline promotion needed to move behind successful post-change verification. The runner required clearer trust separation before it could be allowed near privileged deployment. A future device-change webhook required authentication, event filtering, and a deliberately validation-only action.
Those are reasonable findings for an independent lab. They are also the sort of details that decide whether a larger operations team trusts an automation system. When a change produces an outage at 2:00 a.m., the useful record is a defensible timeline: approved intent, actual device state, exactly which tests ran, and which engineer authorized the next action.
My standard for promoting network automation is simple: the system should make its evidence and its blind spots visible before someone stakes service availability on the result.
Evidence and implementation status
This article describes an independent Cisco CML / NX-OS lab. The reviewed implementation includes GitLab commit-driven CI, a Linux shell-executor runner, pyATS OSPF/NVE/VNI checks, Ansible state and configuration collection, normalization-based drift comparison, and manual deploy/baseline stages. The separate Alertmanager-to-Python incident webhook existed in the monitoring workflow. The device-change-event → GitLab trigger API integration, hardened baseline lifecycle, complete symmetric drift fixture suite, and automated rollback are recommendations, not demonstrated features.
The execution identifiers and assertions quoted above were recovered in the October 2026 editorial/repository review and Article 04 source record. The original raw Job #190 log and the exact referenced repository revision should be rechecked before public release. The article is a review draft, not publication authorization.
Primary technical references: GitLab CI/CD YAML reference, GitLab manual jobs and job controls, GitLab pipeline triggers, GitLab job artifacts, Cisco pyATS documentation.