Generic VXLAN troubleshooting checklists are easy to write and not much use, because they list everything that could theoretically be wrong in an order nobody follows under pressure. This one is built the other way round: these are the faults that actually happened while building the two-spine, two-leaf Nexus fabric behind the data center fabric guide, with the real output that found each.
Two were hit unintentionally and deserve the most attention. Two more were produced deliberately during other tests. A final group is explained without output because the failure was never induced, and that section is labelled as such rather than padded with invented terminal text.
Problem 1: the MTU black hole
Nothing was broken on purpose here. The fabric was built with default 1500-byte underlay interfaces, the single most common real-world VXLAN mistake, and it bit. The symptom is the nastiest kind: partial success. Small packets are fine, large packets vanish with no error, no ICMP unreachable, nothing in a log. Here is the boundary, from a server pinging across the fabric with the do-not-fragment bit set.
SRV1# ping -c 2 -M do -s 1422 10.10.20.12
1430 bytes from 10.10.20.12: icmp_seq=1 ttl=62 time=39.9 ms
1430 bytes from 10.10.20.12: icmp_seq=2 ttl=62 time=20.4 ms
2 packets transmitted, 2 received, 0% packet loss
SRV1# ping -c 2 -M do -s 1423 10.10.20.12
2 packets transmitted, 0 received, 100% packet loss, time 1026ms
SRV1# ping -c 2 -M do -s 1424 10.10.20.12
2 packets transmitted, 0 received, 100% packet loss, time 1003ms
SRV1# ping -c 2 -M do -s 1472 10.10.20.12
2 packets transmitted, 0 received, 100% packet loss, time 1024msOne byte of payload separates total success from total loss. 1472 is the figure people habitually test with, because 1472 plus 8 plus 20 is a 1500-byte IP packet, and on an ordinary network it passes. Here it does not. The arithmetic lands exactly on the nose.
1422 ICMP payload
+8 ICMP header
+20 inner IP header
+14 inner Ethernet header
+8 VXLAN header
+8 UDP header
+20 outer IP header
----
1500 = the underlay interface MTUWork it from the top and it is the same story. A 1450-byte inner IP packet, wrapped in its original Ethernet header plus VXLAN, UDP and outer IP, hands the spine a 1500-byte outer packet. One more byte from the server makes it 1501, which the spine cannot forward.
The reason no ICMP unreachable comes back is the important part. Path MTU Discovery works because a router that cannot forward an oversized DF packet sends "fragmentation needed" to the source. In a VXLAN fabric the dropped packet is the outer one, whose source is the ingress leaf's VTEP loopback, so the spine's error message, if any, goes to the leaf and not the server. The server learns nothing and keeps retransmitting.
That is why the real-world symptom misleads. Ping works, because ping is small. SSH connects and the login prompt appears, because the handshake is small. Then a file transfer stalls, a TLS handshake with a large certificate chain hangs, an HTTP response with big headers never arrives. Everything small works, everything large hangs, and the network gets blamed last.
The fix is mtu 9216 on every underlay-facing interface, spines and leaves, including the port-channels between them. Jumbo frames in the underlay are not an optimisation, they are a requirement.
An honesty note. The fix was applied to LEAF1's Ethernet1/1-2 before the lab's console subsystem stalled, and the post-fix confirmation ping was not captured in this session. The failure above is real and reproduced at four sizes; the verification that 9216 resolves it is not in the captures, so this article does not claim to have proved it. Do not take it on faith in your own fabric either: re-run the ping ladder afterwards and check every hop, because one 1500-byte interface anywhere recreates the problem.
Problem 2: show nve peers is empty, and both leaves are right
Not planned, and the most instructive thing the lab produced. The fabric was configured, BGP EVPN sessions were up, the VNIs were up, and the tunnel list was empty.
LEAF1# show nve peers
<empty>An empty NVE peer list with a healthy control plane is normally a serious fault. Here it was correct behaviour, and the diagnosis applies to any fabric with vPC leaf pairs. Start with what each leaf advertises: both were originating their Type-3 route with the same next hop.
LEAF1# show bgp l2vpn evpn
Route Distinguisher: 10.255.0.11:32777 (L2VNI 10010)
*>l[3]:[0]:[32]:[10.255.1.100]/88
10.255.1.100 100 32768 i
LEAF2# show bgp l2vpn evpn
Route Distinguisher: 10.255.0.12:32777 (L2VNI 10010)
*>l[3]:[0]:[32]:[10.255.1.100]/88
10.255.1.100 100 32768 iDifferent route distinguishers, derived per leaf. Identical next hop, 10.255.1.100, from both. That is the tell, and the reason is one line on the NVE interface.
LEAF2# show nve interface nve1
VPC Capability: VPC-VIP-Only [notified]
Source-Interface: loopback1 (primary: 10.255.1.12, secondary: 10.255.1.100)The secondary address on Loopback1 is the vPC anycast VTEP address, shared by both peers. VPC Capability: VPC-VIP-Only [notified] means NX-OS has detected vPC and switched the VTEP to source VXLAN from that shared address instead of its own primary. That is right: to the rest of the fabric a vPC pair must look like one VTEP, so traffic to a dual-homed host can arrive at either peer.
And a VTEP does not build a tunnel to itself. Both leaves advertise 10.255.1.100, so when LEAF1 evaluates LEAF2's routes it sees its own VTEP address as the next hop and declines to peer. The empty list is two switches agreeing they are one VTEP. Traffic between hosts on the two leaves crosses the vPC peer-link as ordinary VLAN traffic, never VXLAN.
The lesson: on a vPC leaf pair, show nve peers should never list your own peer switch, and if it does the anycast VTEP address is misconfigured. Getting two independent VTEPs means removing what makes them one. In the lab that was no feature vpc, removing the 10.255.1.100/32 secondary from Loopback1, and shutting Ethernet1/3-4 so the only path between leaves ran through the spines.
LEAF1# show nve interface nve1
VPC Capability: VPC-VIP-Only [not-notified]
Source-Interface: loopback1 (primary: 10.255.1.11, secondary: 0.0.0.0)
LEAF1# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- -------------------------------------- ----- --------- -------- -----------------
nve1 10.255.1.12 Up CP 01:14:14 n/a[not-notified], a zeroed secondary, and one peer at the remote leaf's real loopback. LearnType CP confirms the peer came from the control plane rather than data plane flooding. The mechanics behind the shared address are in the vPC explainer.
Problem 3: a port-channel member that is down but administratively up
Produced deliberately while testing LACP, by removing channel-group from the far end so LEAF1's Ethernet1/4 had no partner.
LEAF1# show port-channel summary
1 Po1(SU) Eth LACP Eth1/3(P) Eth1/4(s)
LEAF1# show interface Ethernet1/4 | include "is down|is up|Members"
Ethernet1/4 is down (suspended(no LACP PDUs))
admin state is up, Dedicated InterfaceTwo details make this quick to recognise. (s) is suspended, not down: the flag legend distinguishes D - Down from s - Suspended, and people read past it. And the interface reports is down while admin state is up in the same breath. The port is fine; LACP is refusing to let it forward, having heard no PDUs from the other end.
In a fabric this silently halves your bandwidth to a spine or a peer while the counters look healthy. The causes are always the same three: the far end is not in the channel, both ends are passive so neither initiates, or a mismatch means the far end never answers.
Problem 4: vPC Type-1 consistency failure
Also deliberate, by setting spanning-tree mode mst on LEAF2 only.
LEAF1# show vpc brief | head lines 14
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : failed
Per-vlan consistency status : success
Configuration inconsistency reason: vPC type-1 configuration incompatible - STP Mode inconsistent
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 0
Peer Gateway : Enabled
Dual-active excluded VLANs : -
LEAF1# show vpc consistency-parameters global | include "STP Mode"
STP Mode 1 Rapid-PVST MSTConfiguration inconsistency reason names the offending parameter directly, saving you the full consistency-parameters output, and piping that command with include confirms the two values side by side.
Note what did not happen. The peer adjacency stayed up and the keepalive stayed alive. A Type-1 mismatch does not tear down the vPC peering, it suspends vPC member ports. "vPC is down" and "vPCs are suspended" are different failures. This lab had zero vPCs configured, so there was nothing to suspend, which made the test safe.
Explained, not demonstrated
The next four are real failure modes worth knowing, but they were not induced in this lab and there is no captured output for them. They are described in prose deliberately, with no terminal blocks, so nothing is mistaken for something observed.
Missing retain route-target all on a spine reflector. The command is present in the lab's spine configuration, so this failure never occurred, but the reason it is there is worth spelling out. Default behaviour on a VPN address family is to discard any received route whose route targets are not imported into a local VRF. On a PE that is sensible. On a pure route reflector it is fatal: a spine has no VRFs, imports no route targets, and would drop every EVPN route before reflecting it. The signature is deceptive, because BGP looks perfect: sessions established, peers capable, prefix counts of zero. If a leaf shows healthy sessions with nothing received while another leaf is definitely originating, check the reflector for that line. The control plane build is in the EVPN route types article.
VNI-to-VLAN mapping mismatch between leaves. The VNI travels in the packet; the VLAN ID is locally significant only. LEAF1 mapping VLAN 10 to VNI 10010 while LEAF2 maps VLAN 110 to VNI 10010 is legal and works. But if LEAF2 maps its VLAN 10 to VNI 10011 instead, the leaves run separate broadcast domains that look identical in every per-device show command. Compare show nve vni across leaves, VNI column against [BD/VRF], rather than assuming matching VLAN numbers mean matching segments. That mapping is covered in the L2VNI configuration walkthrough.
Route-target import and export mismatch. This lab used auto route targets everywhere, deriving values from the AS number and the VNI so every leaf computes the same thing. Hand-assigned route targets remove that guarantee. If one leaf exports a route target nobody imports, its routes reach the reflector, get reflected, and are discarded on receipt: present in the fabric, invisible in the tenant.
Underlay reachability or a down NVE source loopback. The overlay cannot work if VTEP loopbacks cannot reach each other, and NVE will not come up at all if its source loopback is down or missing from the routing protocol. Never induced here, but it belongs at the top of the method below because it is cheap to rule out.
An order that works
Faults are layered, and checking the overlay while the underlay is broken wastes time. Work upward.
1. Underlay reachability, VTEP to VTEP, sourced from the VTEP loopback. Not the console's default source, and not the management address.
LEAF1# ping 10.255.1.12 source 10.255.1.11 count 5
PING 10.255.1.12 (10.255.1.12) from 10.255.1.11: 56 data bytes
64 bytes from 10.255.1.12: icmp_seq=0 ttl=253 time=17.458 ms
64 bytes from 10.255.1.12: icmp_seq=1 ttl=253 time=10.176 ms
64 bytes from 10.255.1.12: icmp_seq=2 ttl=253 time=7.459 ms
64 bytes from 10.255.1.12: icmp_seq=3 ttl=253 time=7.216 ms
64 bytes from 10.255.1.12: icmp_seq=4 ttl=253 time=7.075 ms
--- 10.255.1.12 statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 7.075/9.876/17.458 msIf this fails, stop and fix routing. ttl=253 also confirms two hops, so the path runs through a spine rather than a direct link. The underlay is built in the OSPF underlay article, with protocol mechanics in the OSPF guides.
2. The control plane. show bgp l2vpn evpn summary tells you whether sessions are established and whether routes are arriving. Its second table breaks received prefixes out per route type per neighbour, the fastest triage in the fabric. Sessions up with zero prefixes points at the reflector, send-community extended, or route targets.
3. show nve peers. Are the tunnels built, to the right addresses? Remember Problem 2: on a vPC pair an empty result for your own peer switch is correct. Elsewhere, missing peers with a healthy BGP table means Type-3 routes are not becoming tunnels.
4. show nve vni. Confirms each VNI is Up, in CP mode, and mapped to the bridge domain or VRF you expect. This is where a VNI-to-VLAN mismatch shows up, and where a VNI you thought you configured turns out missing.
5. The MAC and route tables. For Layer 2, show mac address-table dynamic and show l2route evpn mac all: a remote MAC should appear flagged C with a port of nve1(remote-VTEP). For Layer 3, show ip route vrf TENANT-A, where a remote host route should carry segid, tunnelid and encap: VXLAN. Symmetric IRB also needs a learned router MAC from show nve peers detail, covered in the L3VNI and anycast gateway build.
6. MTU. Last in the order but first in likelihood when the symptom is "some traffic works". Run the ping ladder with the DF bit set and find the boundary; if it sits 50 bytes below your underlay MTU, you have found it. The encapsulation packet walk takes a real capture apart field by field, and the packet analysis guides cover the capture technique.
Most fabric faults resolve inside those six steps, and the build order they mirror is in the data center fabric pillar guide.
Key Takeaways
- VXLAN adds 50 bytes. A 1500-byte underlay leaves tenants 1450 bytes of IP MTU, and the lab put the boundary at exactly 1422 bytes of ICMP payload.
- Oversized packets die with no ICMP unreachable reaching the host, because the dropped packet is the outer one. Ping and SSH work, transfers and TLS hang.
- The fix is
mtu 9216on every underlay-facing interface. The confirming ping was not captured here, so treat it as unverified and re-test in your own fabric. - On vPC leaves, an empty
show nve peersfor your own peer is correct: both peers source VXLAN from the shared secondary address and are one logical VTEP. VPC Capability: VPC-VIP-Only [notified]plus identical next hops on both leaves is the diagnostic pair for that.Eth1/4(s)is suspended, not down.is downalongsideadmin state is upmeans LACP is hearing no PDUs from the far end.Configuration inconsistency reasonnames the failing vPC parameter directly. A Type-1 mismatch suspends vPCs, it does not drop the peering.- Work upward: underlay ping from the VTEP loopback, EVPN summary, NVE peers, NVE VNIs, MAC and route tables, then MTU.