In a traditional data centre the default gateway for a VLAN lives in one place: a pair of aggregation switches, a firewall, or a core routing block. Every packet leaving its subnet goes there and comes back, even when source and destination are two racks apart on the same leaf pair. That hairpin is fine at low volume and awful at scale, and it is why east-west traffic in a badly designed fabric spends most of its life going north and south. The distributed anycast gateway removes it by putting the same gateway on every leaf, and this article builds one on the two-leaf Nexus fabric from the data center fabric guide.
Everything below comes off that lab. The interesting part is not the config, which is short, but what a single TTL value in a ping tells you about where the routing actually happened.
Identical config on every leaf is the entire point
Here is the L3VNI and anycast gateway configuration, applied to LEAF1 and LEAF2 unchanged, character for character.
fabric forwarding anycast-gateway-mac 0000.2222.3333
vlan 3900
name L3VNI-TENANT-A
vn-segment 50001
vrf context TENANT-A
vni 50001
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
interface Vlan3900
no shutdown
vrf member TENANT-A
ip forward
interface nve1
member vni 50001 associate-vrf
interface Vlan10
no shutdown
vrf member TENANT-A
ip address 10.10.10.1/24
fabric forwarding mode anycast-gateway
interface Vlan20
no shutdown
vrf member TENANT-A
ip address 10.10.20.1/24
fabric forwarding mode anycast-gateway
route-map ALL permit 10
router bgp 65001
vrf TENANT-A
address-family ipv4 unicast
redistribute direct route-map ALLRead that again with the duplication in mind. Both leaves own 10.10.10.1/24 on Vlan10 and 10.10.20.1/24 on Vlan20, and both answer ARP for them with the same MAC, 0000.2222.3333, because fabric forwarding anycast-gateway-mac is global and every leaf uses the same value.
On a classic network that is a duplicate IP and a duplicate MAC, and it would be an outage. Here it is the design. A VM in VLAN 10 on LEAF1 has ARPed for 10.10.10.1 and cached 0000.2222.3333. Move it to a host under LEAF2 and its ARP cache is still correct, because LEAF2 answers to exactly the same address and MAC. No gratuitous ARP, no gateway relearn, no traffic blackholed while the old entry ages out. The gateway is wherever the workload is, which is what "distributed" means. It also means the first hop is always one hop: a packet from VLAN 10 to VLAN 20 is routed by the leaf it arrives on, never sent to a central router to be turned around.
The pieces, one at a time
The L3VNI itself. VLAN 3900 exists only to carry VNI 50001 and has no host ports. This is the transit VNI routed traffic rides between leaves, and it is the structural difference between the two kinds: an L2VNI extends a broadcast domain, an L3VNI carries packets already routed. The L2VNIs for this tenant are built in the VXLAN L2VNI configuration walkthrough and sit on the same NVE interface.
The VRF binding. vrf context TENANT-A with vni 50001 ties the tenant routing table to that transit VNI. route-target both auto and route-target both auto evpn derive import and export route targets from the AS number and the VNI, so every leaf computes the same value without anybody typing it. rd auto does the same for the route distinguisher, deriving it per leaf so routes stay unique.
The core SVI. interface Vlan3900 is a member of TENANT-A and carries ip forward rather than an IP address. It exists so the tenant VRF has an interface in the L3VNI's VLAN to route through, and ip forward tells NX-OS to route packets arriving on it rather than expecting them to be locally destined.
The NVE binding. member vni 50001 associate-vrf makes NVE1 treat 50001 as a routing VNI rather than a bridging one. Miss it and the VRF is defined, the SVIs are up, and nothing routes across the fabric.
The gateway mode. fabric forwarding mode anycast-gateway turns an ordinary SVI into an anycast gateway sharing its address with every other leaf. Without it, the duplicate addresses are exactly the outage you would expect.
Two things NX-OS will tell you while you type this
The first is a deprecation notice, and it matters because a large fraction of the VXLAN tutorials still on the internet include the command.
LEAF1(config-router-vrf-af-ipv4)# advertise l2vpn evpn
Command 'advertise l2vpn evpn' has been deprecated and no longer has any effect.On older NX-OS this line was how you got tenant VRF routes into the EVPN address family as Type-5 routes. On 10.6(2) it is a no-op. If you are following an older guide and it works anyway, this is why, and if somebody added it recently expecting it to fix something, this is why it did not.
The second is more dangerous, and it is a config-order trap rather than a cosmetic warning.
LEAF1(config-if)# vrf member TENANT-A
Warning: Deleted all L3 config on interface Vlan10Moving an interface into a VRF wipes its Layer 3 configuration. That is correct behaviour, because an address valid in the default VRF has no guaranteed meaning in the tenant VRF. But paste a block that sets the IP address and then adds vrf member, and you get an SVI in the right VRF with no address, quietly. VRF membership first, addressing second, every time. Note the ordering in the config above: vrf member TENANT-A, then ip address, then fabric forwarding mode anycast-gateway.
The proof: ttl=62
To test routing rather than bridging, SRV2 was moved into VLAN 20 and readdressed, putting the servers in different subnets.
SRV2# ip address flush dev eth0; ip address add 10.10.20.12/24 dev eth0; ip route replace default via 10.10.20.1
eth0@if272 UP 10.10.20.12/24
default via 10.10.20.1 dev eth0
10.10.20.0/24 dev eth0 proto kernel scope link src 10.10.20.12SRV2's default gateway is 10.10.20.1, which is LEAF2's Vlan20 SVI and equally LEAF1's. SRV2 neither knows nor cares which answered. Then the ping from SRV1, in VLAN 10 under LEAF1.
SRV1# ping -c 5 10.10.20.12
PING 10.10.20.12 (10.10.20.12) 56(84) bytes of data.
64 bytes from 10.10.20.12: icmp_seq=2 ttl=62 time=47.0 ms
64 bytes from 10.10.20.12: icmp_seq=3 ttl=62 time=19.7 ms
64 bytes from 10.10.20.12: icmp_seq=4 ttl=62 time=20.3 ms
64 bytes from 10.10.20.12: icmp_seq=5 ttl=62 time=21.2 ms
--- 10.10.20.12 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4036ms
rtt min/avg/max/mdev = 19.733/27.043/46.964/11.512 msSequence 1 is missing (the first packet is lost to ARP, as always). The value to stare at is ttl=62.
Linux starts an ICMP echo reply at TTL 64. It arrived at 62, so exactly two devices decremented it. Not one, not four. The ingress leaf routed the packet out of the source subnet and into the L3VNI, and the egress leaf routed it out of the L3VNI and into the destination subnet. The two spines in the path contributed nothing, because to a spine this is an outer IP packet between two VTEP loopbacks and the inner packet is payload.
Compare that with the pure Layer 2 case in the same fabric, where a ping between two hosts in one stretched VLAN came back with ttl=64 untouched even though the frames crossed two routed underlay hops. Bridged traffic is not decremented. Routed traffic is decremented once per leaf. Two decrements is the signature of symmetric IRB, and it is the fastest read you will get on whether routing is happening where you think it is.
The route table, read properly
show ip route vrf TENANT-A
10.10.10.0/24, ubest/mbest: 1/0, attached
*via 10.10.10.1, Vlan10, [0/0], 00:04:32, direct
10.10.10.1/32, ubest/mbest: 1/0, attached
*via 10.10.10.1, Vlan10, [0/0], 00:04:32, local
10.10.10.11/32, ubest/mbest: 1/0, attached
*via 10.10.10.11, Vlan10, [190/0], 00:02:05, hmm
10.10.20.0/24, ubest/mbest: 1/0, attached
*via 10.10.20.1, Vlan20, [0/0], 00:04:31, direct
10.10.20.1/32, ubest/mbest: 1/0, attached
*via 10.10.20.1, Vlan20, [0/0], 00:04:31, local
10.10.20.12/32, ubest/mbest: 1/0
*via 10.255.1.12%default, [200/0], 00:02:05, bgp-65001, internal, tag 65001,
segid: 50001 tunnelid: 0xaff010c encap: VXLANThree fields in there are worth learning to read on sight.
hmm on the 10.10.10.11/32 entry stands for Host Mobility Manager. That is the locally attached host, SRV1, learned on this leaf and installed as a /32. HMM tracks host location and is what makes VM mobility work: when a host appears somewhere else, HMM's view changes and the EVPN advertisement follows it. Seeing hmm tells you the host is local; a BGP-learned /32 tells you it is not.
%default on the remote entry is the piece people trip over. The route to 10.10.20.12/32 lives in the TENANT-A table, but its next hop, 10.255.1.12, is LEAF2's VTEP loopback in the default VRF underlay. The percent notation is NX-OS saying "resolve this next hop in that other VRF". Overlay routes in the tenant, next hops in the underlay, and the recursion between them is the structure of a VXLAN fabric in one syntax quirk. If those next hops stop resolving, this field tells you the problem is underlay, not overlay.
segid: 50001 tunnelid: 0xaff010c encap: VXLAN is the forwarding instruction: encapsulate in VXLAN, use VNI 50001, send it to that tunnel. 0xaff010c is 10.255.1.12 in hex, handy when matching tunnel IDs to peers. This is the field proving routed traffic between subnets rides the L3VNI, not 10010 or 10020.
Symmetric mode, confirmed on the peer
show nve peers detail
Peer-Ip: 10.255.1.12
NVE Interface : nve1
Peer State : Up
Peer Uptime : 01:23:00
Router-Mac : 521f.ff1f.1b08
Peer First VNI : 10010
Time since Create : 01:23:00
Configured VNIs : 10010,10020,50001
Provision State : peer-add-complete
Learnt CP VNIs : 10010,10020,50001
vni assignment mode : SYMMETRICTwo fields confirm symmetric IRB is operational rather than merely configured. vni assignment mode : SYMMETRIC is the explicit statement. Router-Mac : 521f.ff1f.1b08 is the working part: this leaf has learned the remote leaf's router MAC through the control plane. Learnt CP VNIs also includes 50001 alongside the two L2VNIs, so the peer is advertising participation in the L3VNI.
Where that MAC comes from is visible in the Type-5 route detail.
show bgp l2vpn evpn route-type 5
Route Distinguisher: 10.255.0.11:4 (L3VNI 50001)
BGP routing table entry for [5]:[0]:[0]:[24]:[10.10.10.0]/224, version 50
Paths: (2 available, best #2)
Path type: internal, path is valid, not best reason: Weight, no labeled nexthop
Imported from 10.255.0.12:4:[5]:[0]:[0]:[24]:[10.10.10.0]/224
Gateway IP: 0.0.0.0
10.255.1.12 (metric 81) from 10.255.0.1 (10.255.0.1)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 50001
Extcommunity: RT:65001:50001 ENCAP:8 Router MAC:521f.ff1f.1b08
Originator: 10.255.0.12 Cluster list: 10.255.0.1
Advertised path-id 1
Path type: local, path is valid, is best path, no labeled nexthop
10.255.1.11 (metric 0) from 0.0.0.0 (10.255.0.11)
Origin incomplete, MED 0, localpref 100, weight 32768
Received label 50001
Extcommunity: RT:65001:50001 ENCAP:8 Router MAC:529a.2a3a.1b08
Path-id 1 advertised to peers:
10.255.0.1 10.255.0.2Received label 50001 is the L3VNI carried as the route's label. ENCAP:8 is the encapsulation type extended community, value 8 meaning VXLAN. And Router MAC:521f.ff1f.1b08 is the field that ties the whole mechanism together.
When LEAF1 routes a packet into the L3VNI toward LEAF2, it must build an inner Ethernet header. The inner source MAC is LEAF1's own router MAC and the inner destination MAC is LEAF2's, learned from exactly this extended community. LEAF2 receives the VXLAN packet, sees its own router MAC as the inner destination, and knows the packet is addressed to it for routing rather than to be bridged onward. That is why routing works without the leaves having any adjacency in the tenant subnets: the router MAC is the handoff. The route type carrying it is covered in the EVPN route types breakdown.
Both paths also carry the same route target, RT:65001:50001, derived from the AS and the VNI on both leaves. That is route-target both auto evpn doing its job.
Symmetric versus asymmetric, briefly
There are two ways to do inter-VNI routing. This lab ran symmetric.
Symmetric scales better and is what NX-OS builds with associate-vrf, which is why SYMMETRIC appears in the peer detail without anyone asking for it. The cost of asymmetric is configuration sprawl: adding a subnet anywhere means touching every leaf, whether or not that leaf hosts anything in it. The cost of symmetric is one extra VLAN and VNI per tenant, which is cheap. The network virtualization guides cover the wider overlay family, and the recursion in that route table is ordinary IP routing next-hop resolution wearing a VRF label.
Verifying in order when it does not work
If inter-subnet traffic fails here, the fastest order is: confirm the tenant SVIs are in the VRF and still have their addresses (remember the wipe warning), confirm member vni 50001 associate-vrf is on NVE1, then check show nve peers detail for SYMMETRIC and a learned Router-Mac. A missing router MAC means the Type-5 routes are not arriving, or arriving stripped, and show bgp l2vpn evpn route-type 5 shows which. If the routes are correct and traffic still fails, the problem has moved to the underlay or to MTU, both covered in the VXLAN EVPN troubleshooting guide. The build order for the whole fabric is in the data center fabric pillar.
Key Takeaways
- The anycast gateway config is identical on every leaf on purpose: same SVI addresses, same
fabric forwarding anycast-gateway-mac, so a workload keeps its gateway and ARP entry when it moves. - An L3VNI is a transit VNI for already-routed traffic: a VLAN, a
vn-segment, a VRF with a matchingvni, a core SVI withip forward, andmember vni 50001 associate-vrfon NVE1. vrf memberdeletes all Layer 3 config on the interface. VRF membership before addressing, or you get an SVI with no IP.advertise l2vpn evpnis deprecated on current NX-OS and has no effect, despite still appearing in many published configs.ttl=62on a cross-subnet ping is two decrements, one per leaf. That is symmetric IRB: ingress leaf routes in, egress leaf routes out, spines never see the inner packet.- In the tenant route table,
hmmis a local host,%defaultmeans the next hop resolves in the underlay VRF, andsegid: 50001 ... encap: VXLANis the L3VNI the packet leaves in. - The
Router MACextended community on a Type-5 route becomes the inner destination MAC when the ingress leaf routes into the L3VNI. - Use
route-target both autoandrd autoso every leaf derives the same values. Mismatched hand-typed route targets are a slow fault to find.