GitLab once reported a successful validation of my Cisco NX-OS VXLAN EVPN lab. The runner reached the switches. The three leaves passed their reported pyATS checks. Ansible collected state from five devices, and the drift detector printed No drift detected. That was a useful working result. Later code review found a gap: a newly collected configuration file with no matching baseline could escape the comparison.
A separate BGP incident-response lab exposed the same class of problem from another angle. Its early automation created incident folders and saved monitoring information, but the device CLI snapshots were empty because unattended SSH collection had not actually completed. The folder existed; the evidence inside it did not. A later software test also produced a stale output file that initially looked like a fresh success.
Those problems are worth writing about because engineers increasingly use automated checks to approve changes, triage incidents, and report reliability. The tool output becomes part of the decision. I want the tool to be as skeptical of itself as I would be on a Tier-3 escalation bridge: which devices did it contact, which commands ran, what failed, what was skipped, and how do I know this record belongs to the current incident?
The labs proved that the component workflows could operate. They also exposed specific defects. This article uses those documented findings to develop a recommended validation quality framework. The full proposed framework was not installed and demonstrated as a single system in the original Cisco CML environments. That boundary matters.
01 / Reconstruct what the green job actually verified
The VXLAN EVPN lab used two NX-OS spines, three leaf VTEPs, OSPF in the underlay, and MP-BGP EVPN across the VXLAN overlay. The automation host, automation-1, ran GitLab and a Linux shell-executor runner along with Ansible, Bash, Python, and pyATS. This was an independent Cisco Modeling Labs environment, separate from the six-router BGP project.
The retained review record identifies GitLab pipeline #124, Job #190, commit 31d7f689. Its ./scripts/run_validation.sh job reported OSPF neighbors FULL, the NVE interface up, and required VNIs up on LEAF1, LEAF2, and LEAF3. Ansible collection completed from the three leaves and two spines, with unreachable=0 and failed=0 in the recorded recap. The job then reported No drift detected and exited successfully.
I would preserve that job as a legitimate test record. But it only proves what the captured assertions checked. The output did not separately prove every EVPN route-type assertion, every tenant endpoint flow, or all possible baseline-file inventory changes. Nor does a successful shell exit prove that an artifact was recent unless its freshness is tested.
For engineering reviews, I now think about four dimensions of validation quality: execution (did the command finish?), coverage (did it check the intended objects and conditions?), freshness (is the result from this run?), and behavior (did the actual service path work?). A PASS should identify the scope it covers instead of borrowing credibility from tests that were not run.
The separation has real operational value. The earlier VXLAN work found that an L3 VNI, 99999, remained down until the routing SVI Vlan999 had the required ip forward configuration. A validator that stops at OSPF and NVE interface checks would miss that tenant dependency. A more complete test includes VNI state and, for service acceptance, a representative endpoint path. The exact assertions belong in the test contract rather than in an implied definition of “fabric healthy.”
02 / A directory is not a completed incident capture
The BGP incident lab used ISP1, R1, R2, R3, R4, and ISP2 with alternative transit paths through R3 and R2. On automation-1, I combined Prometheus, Alertmanager, syslog, a Python webhook listener, shell scripts, and Ansible to move from an alert to an incident evidence bundle.
The first collection iterations exposed an uncomfortable result: directories and alert/Prometheus material appeared, while router CLI output files remained empty. Interactive SSH authentication had interrupted unattended collection. The automation had completed enough steps to create a plausible case folder but not enough to support a routing diagnosis.
After the SSH/Ansible path was corrected, the April 23 exercise captured data from all six routers. The recorded Ansible result showed zero failed hosts. This demonstrated progress in the collection path; it did not magically make every future evidence bundle complete or prove a universal retry/recovery policy.
I would have the collector write a capture manifest that names the expected devices and the minimum artifacts for each one. Each artifact needs an explicit outcome: CAPTURED, EMPTY, FAILED, TIMED_OUT, or NOT_APPLICABLE. The collector must distinguish an optional item from a required one before execution. A required output that is empty or unavailable should make the evidence-gathering result incomplete, whatever the Alertmanager notification said.
That becomes especially important during an outage. If R2 was unreachable, the collector might have useful logs from five other routers. I would keep them. But I would not turn five of six into a silent six-of-six PASS. Capture completeness and the network's operational health should be reported on separate lines.
03 / Count the objects, not just the successful commands
The VXLAN configuration-drift review on October 13 found a specific inventory defect. The comparison process could identify an expected baseline configuration missing from the current snapshot, but it did not reliably flag a new current-only file absent from the baseline.
This is the difference between asking “did every old file get compared?” and asking “do these two inventories agree?” The latter requires examining both sides. Given baseline filenames B and collected filenames C, a complete inventory comparison must examine missing files B − C, unexpected files C − B, and common files B ∩ C. Content comparison then applies to the common set. For device inventories, omissions or extra entries must be classified according to approved scope, not ignored as normal noise.
LEAF4.cfg is not a claim that a fourth leaf was deployed in the actual three-leaf fabric.A collector also needs an independent expected device inventory. Otherwise, if one switch silently disappears from both a faulty inventory feed and the current snapshot, the two available file sets could agree while both are wrong. The source-of-truth device list, actual collection targets, collected-file inventory, and comparison results should be reconciled together.
The implementation review also found duplicated normalization code. Normalization is useful for filtering volatile lines such as generated timestamps, but it must be narrow. A filter that removes too much can hide a changed routing policy or a neighbor shutdown. I would maintain a small fixture suite with stable expected diffs, including a legitimate config difference that the normalizer must never erase.
The defect was identified in the reviewed code. The available editorial record does not establish that a corrected symmetric detector was merged, deployed, and revalidated. The recommended algorithm belongs to the next engineering iteration, not to the historical results column.
04 / Freshness is a property you have to prove
In the BGP incident-processing exercises, an early revision produced a stale output file that initially looked like a valid result. The same project exposed a NameError in Python before subsequent corrections. These were software defects inside an automation system that existed to diagnose network faults. During later processing, an InterfaceOperDown alert for R2 Ethernet0/0 was investigated after the live interface had returned to up/up; the workflow classified the live condition as RESOLVED. A separate test-mode case with a deliberately down/down interface produced OPEN.
Those outcomes are not contradictory. The alert describes an event at one time. The command output describes device state at a later time. A stale file introduces a third possibility: the displayed result may not correspond to either current observation. A trustworthy report needs to carry at least the alert event time, collection time, runner job identifier, target, command, and artifact identity. Where device clocks differ, include clock-source context and the known uncertainty rather than manufacturing subsecond causality.
A fresh run should write to a new immutable incident or job directory, not reuse the last successful result.json as a fallback. The publisher should validate that the expected output files were created by the current run, have nonempty content when required, and can be linked to the exact commit or incident identifier. Hashes are useful for later integrity checks, but a checksum alone does not prove that the right command was run against the right device. The manifest must supply that context.
The archived April 22 BGP incident template preserves R1 (1.1.1.1) syslog reporting neighbor 10.20.0.10 reset at 18:42:00 UTC and Ethernet0/2 administratively down at 18:42:02. This is a different event from the previously described R4 ten-second timeline: the precise R4 18:18:06–18:18:16 event sequence was not independently corroborated by the retained records inspected during the publication audit. These R1 syslog entries establish device-reported events, not end-to-end packet loss or an exact recovery duration. A later healthy neighbor snapshot does not erase the event, and a saved alert does not prove the service remained down when collection began.
05 / Give the validator a third result: UNKNOWN
Traditional job summaries invite a binary answer, PASS or FAIL. That is inadequate when the collector loses access to the target or its parser cannot interpret the response. A blank CLI file is not a passing interface. A timeout is not an established BGP adjacency. A device that was not in the expected inventory is not automatically outside scope.
For a proposed production-oriented validator, I would separate test outcomes from collection quality and expose three operational decisions:
| Decision | Meaning | Typical example | Change-gate action |
|---|---|---|---|
PASS |
Required checks completed on the expected targets and satisfied their assertions | Expected devices collected; all required tests passed | Eligible to proceed, subject to approval |
FAIL |
A required test completed and observed an unacceptable condition | Required VNI down, unexpected configuration difference, failed endpoint probe | Block promotion; investigate |
UNKNOWN |
The system cannot establish the required claim | SSH timeout, parser error, incomplete manifest, missing fresh evidence | Block promotion; repair evidence path |
This is a deliberately strict rule for change authorization. An incident collector can still deliver partial evidence to an engineer, and an operator can make a separately approved emergency decision under uncertain conditions. But a machine should not automatically authorize baseline promotion because it could not measure a required prerequisite. That is how missing data becomes a false green.
There is another trap in parallel testing. The VXLAN review found ThreadPoolExecutor(max_workers=3) checking the three leaves concurrently. Parallelism reduced serial waiting, but the final result must combine all leaf outcomes, attach device identity to every exception, and fail the job when any required task errors. A complete aggregate includes the expected number of test records, not simply the number of PASS strings the program managed to print.
06 / Test the checker with failures it is required to catch
For each validation gate, I would maintain a small test suite of known-good and intentionally broken fixtures before connecting it to network change approval. The principle is familiar from networking: you do not know that redundancy works until you remove a path. You also do not know that a checker detects an error until you feed it an error with a known expected result.
The original labs supplied some real-world inputs for that suite. Others are test cases I would add, not historical runs I am claiming occurred:
| Checker fixture | Expected outcome | Source |
|---|---|---|
| All three expected leaf outputs present, parsed and healthy | PASS |
Recreates the reported scope of Job #190 |
| Expected baseline file absent in current collection | FAIL or UNKNOWN by collection policy |
Missing-current coverage from reviewed detector |
| New current-only device file exists | FAIL pending authorized inventory review |
Fixes identified symmetric-drift blind spot |
| Required router CLI capture is zero bytes | UNKNOWN |
Based on documented early BGP collector failure |
| Reused result file carries previous job ID | UNKNOWN |
Based on documented stale-output defect |
| Parser raises exception during one leaf check | UNKNOWN and aggregate job failure |
Proposed regression test for parallel validation |
| Required L3 VNI operational state is down | FAIL |
Based on documented VNI 99999 troubleshooting |
| Protocol checks pass but endpoint service probe fails | FAIL for service gate |
Proposed extension beyond printed Job #190 assertions |
In practice I would run these fixtures as part of the automation repository's own CI, without reaching into the live fabric. The fixture tests need to check exit codes, the machine-readable decision, completeness counters, and artifact creation. After that, a separate integration test can confirm the collector behaves the same way against Cisco CML devices. If the test harness is too permissive, merely asserting that python script.py exited zero proves very little.
07 / The promotion gate is only as strong as its prerequisites
The reviewed .gitlab-ci.yml used the stage order validate → deploy → baseline → post_validate, with deploy and baseline manual. The baseline operation used rsync --delete to synchronize collected files into the approved baseline location. Both the order and command deserved attention. If the preexisting network was unhealthy, the snapshot incomplete, or a post-deployment defect had not yet been detected, premature promotion could redefine a bad state as accepted intent.
I would build a safer target lifecycle: prepare an approved change, establish clean prechecks, execute a scoped deployment, confirm required network and service postchecks, review the artifact manifest, and only then promote a new versioned baseline. A promotion action must be attributable to an authorized reviewer, preserve the previous baseline for recovery, and reject incomplete collections. A baseline correction after a justified emergency action still requires review; the system should not automatically overwrite a running network just because a drift alarm fired.
A manual stage is not automatically a blocking control. GitLab distinguishes optional and blocking manual jobs based on configuration, including allow_failure and the context of rules. I would verify the exact pipeline definition and permissions instead of presenting a “Run” button as an approval policy. GitLab's manual-job documentation makes this distinction explicit.
For auditability, retain logs and test reports on failure as well as success. GitLab supports artifacts: when: always for this purpose, with documented exceptions such as job timeouts. It is a useful mechanism, not a substitute for deliberately producing the correct evidence. GitLab job artifacts documentation.
08 / Protect the runner and the evidence path
A network validator often holds access that ordinary application CI does not: management routing to switches, parser dependencies, device credentials, and sometimes configuration-changing privileges. The original lab's shell runner executed on automation-1, so a job's security boundary included the Linux host itself. GitLab explicitly warns that shell-executor jobs run with the runner user's permissions and are appropriate only for trusted jobs and hosts. GitLab shell executor security.
I would separate read-only evidence collection from configuration deployment. The collector should reach only approved management targets, use narrowly scoped accounts and SSH keys, and be unable to run privileged changes. Deployment credentials belong behind a separately controlled manual gate or runner boundary. Protected references, restricted runners, and reviewed merge permissions reduce the chance that an untrusted pipeline edit turns a monitoring job into arbitrary commands on the automation server.
The input path deserves the same care. Alertmanager JSON, syslog strings, device names, and webhook parameters are untrusted input. Parse fields against an allowlist, cap request sizes, authenticate the caller, and never interpolate event text directly into shell commands. Store raw incident evidence with access controls and retention rules; redact secrets and sensitive network metadata before sharing artifacts outside the lab.
Security also affects reliability. A compromised validation runner can falsify the evidence used to approve a change. Signed or hashed artifacts can help establish integrity after collection, but strong permissions, trustworthy execution, separation of duties, and audit logs remain essential.
09 / Make automation health visible beside network health
Article 07 examined path-level SLIs and the danger of treating a green protocol dashboard as proof of service. The same measurement discipline applies to the automation itself. I would track required capture completeness, fresh artifact rate, parser error rate, inventory reconciliation coverage, and time from event to usable incident bundle. Each metric needs a precise denominator and a defined treatment of skipped, timed-out, or unavailable targets.
Those metrics were not produced as a validated historical SLO series by the earlier labs. They are recommendations shaped by the failures the projects exposed. For example, a six-router incident workflow should report expected six, attempted six, completed six, failed zero, plus the status of each mandatory artifact. A result claiming “100% collection” with four devices silently excluded from inventory would still be misleading.
I also want an operational owner for the validator. If its collector stops authenticating or its parser breaks after a device OS upgrade, someone needs to know before the next outage. A synthetic collection check against known-good lab devices can detect that kind of decay. Alert fatigue still applies: a broken evidence pipeline should page when it destroys visibility required for an active service, while a lower-priority parser warning may belong in a maintenance queue.
A generated report should carry enough metadata for another engineer to reproduce its conclusion. At minimum: source repository and revision, job or incident ID, execution time, expected/observed target lists, executed checks, decisions per target, all collection errors, artifact paths or hashes, and the identity of any approval that promoted a baseline. I would rather see one UNKNOWN with a clear explanation than twenty apparently successful panels backed by old data.
10 / The operating standard I would take forward
The lesson from these labs is not that automation cannot be trusted. I watched it collect from six BGP routers after the SSH issue was fixed. I saw a real GitLab job verify OSPF, NVE, and VNI state on three NX-OS leaves and retrieve configurations from five switches. Those are useful foundations. The important part is knowing where each foundation ends.
The original project also gave me defects I could explain and turn into tests: empty incident snapshots, a stale report, a Python exception, a one-sided drift inventory comparison, and a baseline stage placed before post-validation. None requires a grand redesign of routing. They require modest, hard rules: verify the target count, fail on absent required evidence, separate current observation from prior event, test the test code with known-bad inputs, and make baseline promotion depend on proven post-change state.
That is how I would make the next implementation more dependable. When a pipeline reports PASS, it should say what it observed, what it did not observe, and which evidence supports the decision. When it cannot answer those questions, it should report UNKNOWN and preserve the useful partial evidence for the engineer investigating the network.
A green check earns its meaning from the tests and records behind it. I want to be able to open them, trace the device and time, and defend the operational decision they supported.
Evidence record and source limitations
Historical lab findings. The BGP incidents and collector failures come from the separate bgp-sre-lab project (April–May 2026), including the April 22 R4 interface/BGP reset, April 23 six-router collection, and later alert-parser/live-state exercises. The VXLAN EVPN findings come from vxlan-evpn-sre-lab, the retained GitLab pipeline #124 / Job #190 / commit 31d7f689 excerpt, the L3 VNI 99999 correction, and the October 13 review of drift-detection and baseline-promotion logic. These details were cross-checked against the earlier approved article materials; the underlying live GitLab repositories and raw logs were not re-accessed for this draft.
Not demonstrated as a unified deployment. Tri-state results, artifact manifests, a symmetric drift fix, checker mutation tests, collected automation SLIs, and re-ordered approval gates are proposed engineering improvements. Figures are authored diagrams, not recreated screenshots of completed tests. No convergence metric, service-impact duration, CI failure count, production incident, or corrected code merge is invented.
Vendor documentation. GitLab manual and blocking jobs; GitLab job artifacts; GitLab Runner security; GitLab pipeline controls.