01 / The fabric: two spines, three leaves, three endpoints
The CML environment contained two NX-OS spines and three NX-OS leaves. Each leaf connected to both spines. Three lab endpoints attached to the leaves to test traffic within a Layer 2 segment and between VLANs. The Linux automation host, automation-1, ran GitLab Runner jobs, Ansible collection, and Python-based validation.
The essential inventory is small enough to list without turning it into an architecture novel:
| Element | Role in the lab | Validation question |
|---|---|---|
| SPINE1 and SPINE2 | Redundant underlay next hops and EVPN control-plane infrastructure | Are the intended routing adjacencies and loopback paths reachable? |
| LEAF1, LEAF2, LEAF3 | VTEPs, access switching, VLAN/VNI mapping, distributed gateways | Are the expected NVE peers and VNIs operational? |
| SERVER-0, SERVER-1, SERVER-2 | Test endpoints on two VLANs | Can traffic reach the intended host within and across subnets? |
automation-1 |
GitLab Runner, Ansible, Python, pyATS execution | Can we repeat a check and preserve an unambiguous pass/fail result? |
The endpoint plan recorded in the project was SERVER-0 = 10.10.10.101 and SERVER-1 = 10.10.10.102 on VLAN 10, with SERVER-2 = 10.10.20.103 on VLAN 20. These are private addresses from the independent lab. The objective was to test both same-subnet bridging over VXLAN and inter-subnet forwarding through distributed gateway behavior.
The fabric used VLAN 10 / L2 VNI 10100, VLAN 20 / L2 VNI 10200, and L3 VNI 99999 for tenant routing. Those values are not interchangeable: a VLAN defines a local bridge domain, a Layer 2 VNI identifies the corresponding VXLAN segment, and a Layer 3 VNI connects the distributed routing context across VTEPs.
02 / Underlay reachability is the first dependency, not the final test
OSPF carried underlay routes between the leaves and spines. It supplied IP reachability to loopback addresses used by the overlay. Without those routes, a VTEP cannot reliably reach a remote VTEP across the IP transport, regardless of how tidy the EVPN configuration appears.
My first check was therefore straightforward: confirm the expected OSPF neighbors reached FULL, verify routes to the required loopbacks, and check that the NVE source interfaces had the intended addresses and reachability. A missing adjacency sends the investigation toward interface state, IP addressing, OSPF authentication or network-type mismatches, routing policy, and next-hop installation. No amount of reconfiguring a tenant VNI repairs an absent underlay route.
But OSPF FULL means exactly what it says. It establishes the adjacency relationship. It does not show whether EVPN route exchange is healthy, whether the VNI is up, whether MAC/IP routes were learned, or whether a host can reach another host.
On a real outage bridge, these distinctions shorten troubleshooting. If the VTEP loopback is unreachable, start with IP transport. If the underlay is healthy but show bgp l2vpn evpn summary reports a problem, shift to EVPN peering and address-family configuration. If both are healthy and traffic fails, inspect VNI state, local/remote endpoint learning, gateway state, and the actual forwarding path.
03 / What the EVPN control plane carried
The overlay used MP-BGP EVPN to advertise reachability between VXLAN tunnel endpoints. In the broader lab validation record, EVPN sessions were established, NVE peers were up, and Type 2 and Type 3 EVPN routes were observed. Those are distinct observations rather than a single boolean called “EVPN healthy.”
Type 2 MAC/IP advertisement routes associate a learned host MAC address, and when appropriate an IP address, with EVPN reachability information. For a remote VTEP, that information helps determine where an endpoint lives. Type 3 inclusive multicast Ethernet tag (IMET) routes establish participation in a broadcast domain and support building the membership needed for BUM handling, depending on the replication design. They do not by themselves prove an application flow reached its destination.
A VTEP also needs the corresponding VXLAN NVE and VNI state. For the lab I used checks such as show bgp l2vpn evpn summary, show bgp l2vpn evpn route-type 2, show bgp l2vpn evpn route-type 3, show nve peers, and show nve vni to compare EVPN signaling with VXLAN operational state. The exact command availability and output format depend on NX-OS version and device image.
A BGP session reported as established answers an important question, but it cannot certify that the expected MAC/IP route was advertised, imported into the correct VNI, or programmed into forwarding. Operational validation has to test those assertions separately.
04 / Map the VLANs, VNIs, VRF, and endpoints before testing traffic
The fabric's VLAN/VNI plan is small, but the dependencies are representative of a larger tenant fabric. Both servers on VLAN 10 should communicate within the same L2 overlay segment even when attached to different leaves. Reaching the server on VLAN 20 tests a different path: a distributed gateway and an inter-VLAN routing lookup associated with the tenant VRF and L3 VNI.
With distributed anycast gateways, leaves participating in the same routed tenant can present a consistent first-hop gateway identity to hosts. That reduces the need to hairpin local routing to a centralized gateway, but it introduces state that must be consistent across leaf nodes: gateway MAC/IP, VRF assignment, SVI state, L2/L3 VNI associations, and the necessary EVPN routing information.
A useful operator distinction is between a configured relationship and an installed forwarding relationship. A correct-looking vn-segment line is necessary configuration; it does not prove that nve1 has an active VNI or that a remote MAC resolves to the correct tunnel endpoint. Every test should be worded to match the evidence it sees.
05 / The L3 VNI that existed in configuration but remained down
This was one of the most useful problems encountered in the CML build. The tenant's L3 VNI, 99999, remained down even though the surrounding VXLAN configuration had been assembled. The relevant routing SVI, interface Vlan999, needed ip forward. After correcting that configuration, the L3 VNI could transition to its expected state.
That behavior aligns with Cisco's documented NX-OS configuration pattern for a Layer 3 VNI: associate a VLAN and tenant VRF, use a forwarding-enabled SVI, and associate the L3 VNI with the NVE interface. The exact configuration below is a schematic illustration, not a verbatim extraction from this lab's saved running configuration:
vlan 999
vn-segment 99999
!
interface Vlan999
no shutdown
vrf member <tenant-vrf>
ip forward
!
interface nve1
member vni 99999 associate-vrf
In practice I would isolate this fault with a short sequence: verify the tenant VRF and VLAN-to-VNI association; check the SVI's administrative and operational state; examine show nve vni; check nve1 membership and VRF association; and confirm the expected route and forwarding state after the change. A generic “VXLAN configured” checklist would have missed the failure that mattered.
The bigger lesson is not the syntax of ip forward. It is that VNI state belongs in the post-change acceptance criteria. A validated OSPF or BGP session would have given this defect a pass even though the tenant routing function was not ready.
06 / Traffic tests revealed one more layer of state
Once the control plane was in place, I tested endpoint connectivity within VLAN 10 and between the VLAN 10 and VLAN 20 segments. The retained project notes describe successful host reachability and a first-packet effect while ARP and EVPN endpoint-learning entries populated. That is a familiar pattern in a freshly started lab, but it should not become a blanket excuse for packet loss.
On a new path, an initial ping can coincide with ARP resolution, MAC learning, or EVPN updates. A follow-up success is informative only if the engineer checks the relevant table entries and repeats the test. A repeated failure, asymmetry, or sustained loss would demand a different explanation. I therefore separated the first probe, the learned-state check, and the repeat probe instead of assuming the first failure was harmless.
The useful checks include show ip arp vrf <tenant-vrf>, show mac address-table, show bgp l2vpn evpn route-type 2, show nve peers, and host-level ping or trace results, chosen according to the endpoint and fault. A single successful ping is a point-in-time check for one flow; it is not a proof of all possible paths or ECMP selections.
This is where the lab felt closest to incident response. A route can be present in the control plane while the endpoint is not yet fully resolved. Operators need a timeline showing when the route appeared, when the endpoint was learned, and when forwarding became stable. That timeline is more credible than declaring success based on the routing protocol alone.
07 / Make the checks repeatable: pyATS and GitLab
The automation phase connected the fabric to GitLab and a Linux Runner on automation-1. The repository combined pyATS, Ansible, Bash, and Python to perform tests, collect device state, save running configurations, and compare snapshots. The reviewed validation implementation used ThreadPoolExecutor(max_workers=3) to check LEAF1, LEAF2, and LEAF3 concurrently, while aggregating the device results.
One original execution record is especially useful because it shows what the pipeline actually printed. GitLab pipeline #124, Job #190, on merge-request commit 31d7f689, invoked ./scripts/run_validation.sh using GitLab Runner's shell executor. The preserved log excerpt reports the following three-leaf result:
=== Running pyATS VXLAN validation ===
LEAF1: OK - OSPF neighbors are FULL
LEAF1: OK - NVE interface is up
LEAF1: OK - All required VNIs are Up
LEAF2: OK - OSPF neighbors are FULL
LEAF2: OK - NVE interface is up
LEAF2: OK - All required VNIs are Up
LEAF3: OK - OSPF neighbors are FULL
LEAF3: OK - NVE interface is up
LEAF3: OK - All required VNIs are Up
LEAF1: PASS
LEAF2: PASS
LEAF3: PASS
Overall Result: PASS
The same execution used Ansible to collect VXLAN state and NX-OS running configurations from all five devices; its recap showed unreachable=0 and failed=0 on each spine and leaf. Drift detection reported No drift detected, and the job ended with Job succeeded.
This job establishes several concrete operational facts: the runner reached the fabric, pyATS obtained acceptable reported state from three leaves, Ansible collected from five NX-OS devices, and the checked pipeline exited successfully. It does not, by itself, show an explicit BGP EVPN summary assertion, a Type 2/3 route count assertion, or an endpoint ping assertion in that job's captured output. Those observations belong to the broader lab record and need their own preserved evidence if presented as separate automated gates.
A green result also inherits the weaknesses of the checks behind it. During later code review, we identified a drift-detector blind spot: it could detect a missing current file but not every new file absent from the baseline. The No drift detected line therefore cannot be treated as proof of complete baseline reconciliation. This is a specific limitation, not a reason to dismiss the working checks.
08 / A validation matrix for a failure, a change, and a recovery
The most useful way to operate this fabric is to define the expected outcome at each layer before pulling a link or changing a VNI. That helps avoid a common trap: testing the one condition that still works and reporting that the network survived the fault.
| Layer | Expected baseline | Investigation during fault | Acceptance after recovery |
|---|---|---|---|
| Underlay | Intended OSPF neighbors FULL; VTEP routes present | Identify missing adjacency or loopback path | Required paths and adjacencies restored |
| EVPN control plane | Expected BGP neighbors and relevant route types present | Inspect neighbor state, updates and route import | Expected EVPN route visibility restored |
| Tunnel/VNI | nve1, peers and required VNIs operational |
Identify tunnel or VNI state changes | Required NVE peers and VNIs up |
| Endpoint forwarding | Same-VLAN and inter-VLAN probes succeed | Compare source/destination and actual path | Repeat representative traffic tests |
| Automation evidence | Job returns a truthful result with saved output | Preserve failed device, command errors and timestamps | Re-run assertions and archive the result |
The physical or logical fault-injection scenario matters. A leaf-to-spine link failure primarily exercises redundancy and underlay reconvergence. Loss of a VTEP loopback route can disrupt tunnels while other OSPF adjacencies remain up. A missing VRF-to-L3-VNI forwarding relationship can leave the underlay and EVPN session seemingly normal while tenant routing fails. Each class needs a different acceptance test.
The lab records establish successful baseline validation and specific configuration/forwarding troubleshooting. For the public article, I will not invent convergence times, exact packet-loss counts, or a completed fault matrix unless the corresponding CML logs and test outputs are available. The matrix above is an operator-oriented acceptance plan, not a claim that each row was executed as one archived end-to-end test.
09 / Reliability controls have to include the validator itself
An automated test is part of the infrastructure's operating system in the human sense: people will make decisions based on its result. It needs predictable failure semantics. If a leaf cannot be reached, the result should be unknown or failed, not PASS because an SSH call returned no parseable data. If three devices are checked concurrently, an exception on the third must still change the aggregate exit code and retain the affected device's name.
I would also keep collection credentials read-only, protect secrets outside the repository, pin validated Python and automation dependencies, retain timestamped artifacts, and require approval before any change capable of altering the fabric. A GitLab Runner using the shell executor runs commands in the host environment; runner isolation, project trust boundaries, and token scope matter. Configuration checks should never become an unattended remediation path merely because the network emits an alert.
The reviewed GitLab pipeline defined validate, deploy, baseline, and post_validate, with deployment and baseline controls marked manual. Another later review found a sequencing risk: baseline promotion appeared before post-validation. In a production design I would reverse that governance relationship. Verify the changed network first, preserve the pre-change baseline, and promote a new approved reference only after the relevant post-change checks pass.
None of this requires automation to be self-healing. During incidents, I want an accurate, well-bounded collector more than a script that confidently changes a configuration while everyone is still trying to determine the affected path.
10 / What this fabric proved, and what remains to verify
The project demonstrated an operating fabric in CML and exposed a class of problems that configuration-only validation misses. OSPF supplied the transport reachability, EVPN distributed endpoint information, VXLAN NVE/VNI state exposed the overlay's operational condition, and host probes tested actual forwarding. The L3 VNI issue was a concrete example of one control-plane layer passing while another feature remained incomplete.
The retained GitLab job provides unusually clear evidence for a subset of this story. It is sufficient to say the three leaves passed the printed OSPF/NVE/VNI assertions and that collection completed from five NX-OS devices. It is not sufficient to claim that every endpoint and EVPN route test was automatically exercised in that job. That difference matters if an engineering leader is using the article to judge what could safely become a post-change gate.
The production relevance is straightforward. When a new fabric is built, when a tenant is added, or when an outage breaks an application path, engineers need layer-specific state together with forwarding evidence and a trustworthy automated record. A single successful command is too narrow to support that conclusion.
The next article in the Network SRE series will take a closer look at GitLab CI/CD and configuration drift: how approved intent differs from observed device state, what Job #190 did correctly, and where baseline promotion and new-file detection needed stronger safeguards.
Source record and editorial verification
This is an independent Cisco CML/NX-OS lab, not a representation of a production customer topology. Primary project: network-lab/vxlan-evpn-sre-lab on automation-1. Last identified local repository checkpoint: f09af71 (June 16, 2026, Complete Phase 9 service dependency graph). The article refers to the earlier implemented fabric and the recovered GitLab pipeline #124 / Job #190 / commit 31d7f689, which is supplied as a preserved excerpt under evidence/vxlan-job-190-excerpt.txt.
Source code and captures to reconcile before publication: inventory/hosts.yml, pyats/testbed.yml, pyats/validate_vxlan.py, scripts/run_validation.sh, the current .gitlab-ci.yml, NVE/VNI and BGP device state, the original L3 VNI fix, endpoint probe outputs, and the saved job log. The accompanying diagrams are original explanatory illustrations, not raw CML, NX-OS, GitLab, or Grafana screenshots. No unverified latency, packet-loss, or convergence statistics are assigned.
Vendor technical references: Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 10.1(x) (L3 VNI, ip forward, NVE/VRF patterns); Cisco VXLAN BGP EVPN Configuration Guide (OSPF/IS-IS underlay and EVPN design); Cisco DevNet pyATS: Parsing Device Output (programmatic device-state collection). Device-image-specific behavior still governs the lab itself.