An L2VNI is the simplest thing you can build on a VXLAN fabric and the fastest way to prove the fabric actually works. You map a VLAN to a VNI, point an NVE interface at a loopback, let BGP EVPN handle reachability, and two hosts on different leaves start behaving as though they are plugged into the same switch. This walkthrough builds one on the two-spine, two-leaf Nexus 9000v topology from the data center fabric guide, and it spends most of its time on the mistake that stopped it working the first time.
That mistake is worth more than the configuration. The configuration is in every vendor guide. The failure mode below is in almost none of them, it produces no error message at all, and it will waste an afternoon of your life if nobody has warned you.
The trap: two vPC leaves are one VTEP
The fabric started out with vPC configured between LEAF1 and LEAF2, which is completely normal. Part of a standard vPC VXLAN build is an anycast VTEP address: a secondary IP on the VTEP loopback, shared by both peers, so the rest of the fabric sees one logical VTEP behind the vPC pair. Here is the relevant part of Loopback1 on LEAF1:
interface loopback1
ip address 10.255.1.11/32
ip address 10.255.1.100/32 secondary
ip router ospf UNDERLAY area 0.0.0.0LEAF2 had the same secondary, 10.255.1.100. With source-interface loopback1 on the NVE, NX-OS does exactly what vPC tells it to do and sources VXLAN from the shared address. You can see it admit this:
LEAF2# show nve interface nve1
VPC Capability: VPC-VIP-Only [notified]
Source-Interface: loopback1 (primary: 10.255.1.12, secondary: 10.255.1.100)VPC-VIP-Only [notified] means vPC has told the NVE process to use the virtual IP, and the secondary address is the one that will land in the outer IP header. Now look at what each leaf advertised into EVPN. These are the Type-3 Inclusive Multicast routes, the ones that build the flood list:
LEAF1# show bgp l2vpn evpn
Route Distinguisher: 10.255.0.11:32777 (L2VNI 10010)
*>l[3]:[0]:[32]:[10.255.1.100]/88
10.255.1.100 100 32768 i
LEAF2# show bgp l2vpn evpn
Route Distinguisher: 10.255.0.12:32777 (L2VNI 10010)
*>l[3]:[0]:[32]:[10.255.1.100]/88
10.255.1.100 100 32768 iTwo different leaves, two different route distinguishers, identical originating IP and next hop. The distinguishers keep the routes from colliding in the BGP table, which is why nothing looked broken at the control plane. But every leaf now believes there is one VTEP at 10.255.1.100. Here is the consequence, and it is the least helpful output in NX-OS:
LEAF1# show nve peers
<empty>No peers. No error, no syslog, no consistency check failure. Everything else looked healthy: OSPF adjacencies full, EVPN sessions up to both reflectors, VNIs Up. The tunnel simply did not exist.
Why it happens
A VTEP does not build a tunnel to itself. LEAF1's NVE process sees a Type-3 route for 10.255.1.100, compares that next hop to its own VTEP source address, finds they are the same, and correctly discards it as local. So does LEAF2. The two leaves are one logical VTEP to the rest of the fabric and therefore they never form an NVE peering with each other.
In a real vPC fabric this is not a bug, it is the design. Traffic between a host on LEAF1 and a host on LEAF2 does not need a VXLAN tunnel, because the two leaves already have a peer-link and that traffic crosses it as ordinary VLAN traffic. The anycast VTEP exists so remote leaves see a single next hop for the pair. The design only breaks down when, as here, the two leaves are the only two leaves and you expected VXLAN between them.
show nve peers is empty
show nve interface nve1, secondary field
The fix
Because the goal here was to study VXLAN between two leaves rather than to run a production vPC pair, the fix was to break the pair apart and give each leaf its own distinct VTEP identity. Three changes, on both leaves:
no feature vpc, removing the vPC domain entirely.no ip address 10.255.1.100/32 secondaryon Loopback1, removing the shared anycast VTEP address.shutdownon Ethernet1/3-4, the peer-link members, so the only remaining path between the leaves runs through the spines.
That last one matters more than it looks. Without it the leaves keep a direct Layer 2 path, and any successful ping afterwards proves nothing because the frames could have taken the short way round. If you want a test that proves encapsulation, remove the path that would let you cheat.
After the change, the NVE interface reports a clean single-address source and the peer appears:
LEAF1# show nve interface nve1
VPC Capability: VPC-VIP-Only [not-notified]
Source-Interface: loopback1 (primary: 10.255.1.11, secondary: 0.0.0.0)
LEAF1# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- -------------------------------------- ----- --------- -------- -----------------
nve1 10.255.1.12 Up CP 01:14:14 n/a[not-notified] and secondary: 0.0.0.0 are the two fields to compare against the broken state. LearnType CP means the peer was learned from the control plane, from a BGP EVPN route, not from a data-plane flood.
If you are running vPC in production and genuinely need the anycast VTEP, the lesson is not "remove vPC". It is that an empty show nve peers between two vPC peers is expected behaviour rather than a fault.
The L2VNI configuration
With two distinct VTEPs, the L2VNI itself is short. This is LEAF1 and LEAF2 is identical apart from its loopback addressing.
vlan 10
name TENANT-A-WEB
vn-segment 10010
vlan 20
name TENANT-A-APP
vn-segment 10020
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 10010
ingress-replication protocol bgp
member vni 10020
ingress-replication protocol bgp
evpn
vni 10010 l2
rd auto
route-target import auto
route-target export auto
vni 10020 l2
rd auto
route-target import auto
route-target export auto
interface Ethernet1/5
switchport
switchport mode access
switchport access vlan 10
spanning-tree port type edge
no shutdownFour blocks, each doing one job.
The VLAN to VNI mapping. vn-segment 10010 is what turns a locally significant VLAN ID into a fabric-wide segment identifier. VLAN 10 is a number that only means anything on this switch, and the fabric never carries it. VNI 10010 is the number that travels. The convention of VLAN 10 to VNI 10010 is for human sanity, not a requirement, and the two leaves need not use the same VLAN ID as long as both map to VNI 10010. The VLANs and Layer 2 switching guides cover the 12-bit tag and its 4,094 usable values, which is exactly the limit a 24-bit VNI was invented to lift.
The NVE interface. This is the tunnel endpoint. host-reachability protocol bgp is the line that makes this an EVPN fabric rather than a flood-and-learn one, telling the NVE to get its MAC-to-VTEP mapping from BGP. source-interface loopback1 sets the outer source IP for every encapsulated packet, which is why Loopback1 must be advertised into the underlay and why the anycast trap above was possible. Loopback1 is the VTEP source, Loopback0 is the BGP peering source, and keeping them separate makes that failure far easier to reason about.
The member VNIs. ingress-replication protocol bgp tells the leaf how to handle broadcast, unknown unicast and multicast. Instead of joining a multicast group in the underlay, the leaf keeps a list of every other VTEP that has advertised interest in this VNI and sends a separate unicast copy to each. The list comes from EVPN Type-3 routes, which is why the alternative name for those routes is the flood list.
The evpn stanza. rd auto and route-target import/export auto derive the route distinguisher and route targets from the router ID and the VNI, which is where the 10.255.0.11:32777 distinguisher you saw earlier comes from. Manual values are still valid and you will want them for multi-AS designs, but on a single-AS fabric auto is fewer things to get wrong. The details of what those routes carry are in the EVPN route types walkthrough.
The warning you will see
Mapping the second VLAN to a VNI produced this:
Warning: Enable double-wide arp-ether tcam carving if igmp snooping/Hsrp over vxlan is
enabled. Ignore if tcam carving is already configured.NX-OS is reminding you that ARP suppression, IGMP snooping over VXLAN and HSRP over VXLAN need a wider TCAM region than the default carving provides, and that carving TCAM requires a reload. Nothing here used those features, so the warning was correctly ignored. On real hardware, read it rather than scrolling past: discovering you need a reload after the fabric is carrying traffic is not a good afternoon.
Verifying the overlay
Start with the VNIs themselves.
show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
...
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 10010 UnicastBGP Up CP L2 [10]
nve1 10020 UnicastBGP Up CP L2 [20]Both VNIs Up, both in CP mode. UnicastBGP in the Multicast-group column is how ingress replication announces itself, and on a PIM-based fabric that column holds an actual group address instead. L2 [10] confirms the bridge domain mapping.
Next, the MAC address table, which is where a VXLAN fabric stops resembling a normal switch:
show mac address-table dynamic
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
C 10 4e69.d07b.10f7 dynamic NA F F nve1(10.255.1.12)
* 10 5254.0087.1ddf dynamic NA F F Eth1/5
C 10 5254.00b7.2c84 dynamic NA F F nve1(10.255.1.12)The port for two of these MACs is nve1(10.255.1.12), which is not a physical interface, it is the remote VTEP address. The flag C means control plane: that MAC arrived in a BGP update, not in a frame. The age column reads NA because a control-plane MAC is not aged out by an idle timer, it leaves the table when BGP withdraws the route. * on the middle entry is SRV1 on Ethernet1/5, learned the ordinary way.
The Layer 2 route table shows the same information with the producer named explicitly:
show l2route evpn mac all
Topology Mac Address Prod Flags Seq No Next-Hops
----------- -------------- ------ ------------------ ---------- --------------------------
10 4e69.d07b.10f7 BGP Rcv 0 10.255.1.12 (Label: 10010)
10 5254.0087.1ddf Local L, 0 Eth1/5
10 5254.00b7.2c84 BGP Rcv 0 10.255.1.12 (Label: 10010)Prod BGP with flag Rcv against Prod Local with flag L is the cleanest statement of what EVPN actually changed about switching. Half this table was learned by looking at a frame's source address. The other half was learned by a routing protocol. (Label: 10010) is the VNI that will go in the VXLAN header when a frame heads for that MAC. The companion post takes a real VXLAN packet apart header by header and finds VNI 10010 sitting exactly where this table says it will be.
Proof: the VLAN is stretched
SRV1 hangs off LEAF1's Ethernet1/5 in VLAN 10 as 10.10.10.11. SRV2 hangs off LEAF2 in VLAN 10 as 10.10.10.12. Same subnet, same broadcast domain, different physical switches, and with the peer-link shut the only path between them runs through a spine.
SRV1# ping -c 5 10.10.10.12
PING 10.10.10.12 (10.10.10.12) 56(84) bytes of data.
64 bytes from 10.10.10.12: icmp_seq=1 ttl=64 time=52.1 ms
64 bytes from 10.10.10.12: icmp_seq=2 ttl=64 time=15.2 ms
64 bytes from 10.10.10.12: icmp_seq=3 ttl=64 time=32.8 ms
64 bytes from 10.10.10.12: icmp_seq=4 ttl=64 time=20.3 ms
64 bytes from 10.10.10.12: icmp_seq=5 ttl=64 time=15.7 ms
--- 10.10.10.12 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 15.183/27.196/52.050/13.955 msWhy ttl=64 is the interesting number
Not the round-trip times, which are lab noise on a virtual platform, and not the zero packet loss. The number that proves the stretch is ttl=64.
Linux sends ICMP echo requests with an initial TTL of 64, and every router that forwards an IP packet decrements it by one. The reply arriving back at SRV1 with 64 intact means that from SRV2's point of view nothing routed the packet, so SRV2 replied with a full initial TTL and nothing decremented it on the way back either. Both hosts believe they are on the same wire.
They are not. Those frames went from SRV1 into LEAF1, got wrapped in a UDP packet addressed from 10.255.1.11 to 10.255.1.12, were routed across a spine as ordinary underlay IP traffic, arrived at LEAF2, were unwrapped, and were delivered onto SRV2's access port. The underlay TTL absolutely was decremented: a VTEP-to-VTEP ping across the same path returns ttl=253, two hops down from 255.
So two independent TTL counters run at once. The outer one counts real routed hops. The inner one never moves, because encapsulation hides those hops from the payload entirely. That is the promise of network virtualization in one header field: an arbitrary routed topology underneath, and a flat Layer 2 segment on top that hosts cannot tell apart from a physical switch.
The contrast with an L3VNI is sharp. Route between two VNIs with a distributed anycast gateway and the inner TTL drops to 62, two decrements, one per leaf, as the L3VNI and anycast gateway post shows. The difference between 64 and 62 tells you whether the fabric bridged your traffic or routed it, and it costs nothing to check.
What this build did not test
VNI 10020 came up but carried no test traffic, and with both servers single-homed there were no vPC member ports anywhere, so nothing here validates a dual-homed host. The underlay is also worth flagging: it runs default 1500-byte interfaces, and 50 bytes of VXLAN overhead means anything over 1450 bytes of inner IP is silently dropped. Small pings like the one above work perfectly, which is exactly what makes it dangerous.
Key Takeaways
- Two vPC peers sharing an anycast VTEP secondary address are one logical VTEP, so they never form an NVE peering with each other and
show nve peersbetween them is legitimately empty. - The tell is two Type-3 routes from different route distinguishers carrying the identical next hop, plus
VPC-VIP-Only [notified]and a non-zero secondary inshow nve interface nve1. vn-segmentmaps a locally significant VLAN ID to a fabric-wide VNI. The VLAN ID never leaves the switch, the VNI is what travels.host-reachability protocol bgpis the single line that makes a fabric EVPN-driven instead of flood-and-learn.UnicastBGPin the Multicast-group column ofshow nve vnimeans ingress replication, and aCflagged MAC pointing atnve1(remote-VTEP)was learned from BGP rather than from a frame.- An unchanged inner
ttl=64across two leaves proves the hosts are bridged, not routed. Compare withttl=62for an L3VNI andttl=253in the underlay on the same path. - Shut the alternative path before you declare a stretch proven, or your test proves nothing.
The L2VNI is the foundation everything else in the overlay sits on, and the rest of the cluster builds outward from here through the Data Center Fabric guide.