The underlay of a VXLAN fabric is the part nobody should ever have to talk about. It exists to get one loopback address to another loopback address over every available path, quickly, and then get out of the way. If you find yourself explaining your underlay design in an incident review, something has gone wrong. This walkthrough builds that deliberately boring underlay on four Nexus 9000v switches, two spines and two leaves, and it is the foundation everything else in the Data Center Fabric guide sits on top of.
All output here is verbatim from the lab. Where something was not tested, this post says so.
What the underlay actually has to do
Strip away the marketing and an underlay has three jobs. Advertise every VTEP source loopback to every other VTEP. Provide equal-cost paths across all spines so traffic hashes evenly instead of pinning to one uplink. Converge fast enough that a link failure is a blip rather than an outage.
Everything else is optional. No summarisation, no route filtering, no redistribution, no multiple areas. A fabric underlay that has grown policy is one that will eventually black-hole an overlay you cannot see from the underlay CLI.
This lab uses OSPF in a single area, area 0.0.0.0, with a named process. IS-IS and eBGP are both fine alternatives and at very large scale eBGP-to-the-leaf has real advantages, but at two spines and two leaves OSPF is the fastest thing to stand up and the easiest thing to read. If you want the protocol fundamentals rather than the fabric application, the OSPF guides cover areas, LSA types and timers in depth.
/31 links on every fabric port
Fabric links are point-to-point by definition. There are exactly two routers on the wire and there will never be a third, so a /30 wastes two addresses per link for a network and broadcast address nothing will ever use. A /31 gives you both usable addresses and halves your underlay address consumption.
The addressing plan in this lab is deliberately mechanical, because a mechanical plan is one you can read from a route table without a spreadsheet.
10.0.1.1/31
10.0.2.1/31
10.255.0.11/32
10.255.1.11/32
0.0.0.0
Two loopbacks per leaf, not one. Loopback0 is the routing and control plane identity: it is the OSPF router ID and it is the BGP update source for the EVPN sessions. Loopback1 is the VXLAN tunnel endpoint source. Keeping them separate costs you one extra /32 per switch and buys you the ability to move, renumber or share the VTEP address without touching your BGP peerings. On a vPC pair the VTEP loopback carries a shared secondary address while the control plane loopback stays unique, which is exactly why splitting them is not optional in a vPC design.
The configuration, in full
Here is LEAF1 as applied. The spines are the mirror image with their own router IDs and the far end of each /31.
feature ospf
router ospf UNDERLAY
router-id 10.255.0.11
log-adjacency-changes
interface loopback0
ip address 10.255.0.11/32
ip router ospf UNDERLAY area 0.0.0.0
interface loopback1
ip address 10.255.1.11/32
ip address 10.255.1.100/32 secondary
ip router ospf UNDERLAY area 0.0.0.0
interface Ethernet1/1
no switchport
ip address 10.0.1.1/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/2
no switchport
ip address 10.0.2.1/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdownA few NX-OS specifics worth flagging. feature ospf comes first, because on NX-OS the OSPF commands do not exist until the feature is enabled. The process is named UNDERLAY rather than numbered, which is an NX-OS convention and a genuinely nice one when you are reading someone else's config at 2am. And no switchport is required on every fabric port, because Nexus ports come up as switchports in VLAN 1 by default.
The secondary address on Loopback1, 10.255.1.100/32, is the anycast VTEP address for the vPC pair. It was configured for the vPC phase of this lab and later removed so that the two leaves would present as two distinct VTEPs. It is shown here because it is what was applied, not because you need it in a non-vPC build.
Why ip ospf network point-to-point is the line that matters
Every Ethernet interface OSPF sees is, by default, a broadcast network. Broadcast networks elect a designated router and a backup designated router. That election is not free. The router waits out the dead interval before deciding the election is settled, and while it waits, adjacencies sit in 2WAY or DROTHER. On default timers that is a 40-second window where the link is physically up, OSPF is talking, and no routes have been exchanged.
On a link with exactly two routers, a DR election is pure ceremony. There is no multi-access segment to optimise flooding across, so the DR/BDR machinery buys you nothing and costs you 40 seconds on every link event. Setting the network type to point-to-point deletes that machinery entirely. No DR, no BDR, no waiting.
Here is what that looks like on the neighbour table.
show ip ospf neighbors
OSPF Process ID UNDERLAY VRF default
Total number of neighbors: 2
Neighbor ID Pri State Up Time Address Interface
10.255.0.1 1 FULL/ - 00:00:30 10.0.1.0 Eth1/1
10.255.0.2 1 FULL/ - 00:00:29 10.0.2.0 Eth1/2Read the State column carefully. It says FULL/ -. On a broadcast segment you would see FULL/DR, FULL/BDR or FULL/DROTHER. The hyphen means there is no DR role because there is no DR election. That single character is your confirmation that the network type took effect, and it is far more reliable than reading it back out of the running config.
The Up Time column is the other half of the story: 30 and 29 seconds. Both adjacencies reached FULL in about half a minute, and that includes the time to bring the interfaces up. Notice also that the neighbour addresses are 10.0.1.0 and 10.0.2.0. Those are the .0 halves of the /31s, addresses a /30 would have burned as network addresses.
Confirm the interface roles as well.
show ip ospf interface brief
OSPF Process ID UNDERLAY VRF default
Total number of interface: 4
Interface ID Area Cost State Neighbors Status
Eth1/1 3 0.0.0.0 40 P2P 1 up
Eth1/2 4 0.0.0.0 40 P2P 1 up
Lo0 1 0.0.0.0 1 LOOPBACK 0 up
Lo1 2 0.0.0.0 1 LOOPBACK 0 upFour interfaces, all in area 0.0.0.0. The two fabric ports are in state P2P with one neighbour each. The two loopbacks are in state LOOPBACK with zero neighbours, which is correct and not a problem: a loopback has nobody to talk to, it is there to be advertised. Cost 40 on the fabric links, cost 1 on the loopbacks. Hold on to those numbers, they are about to explain a route metric.
Cost 40 is not arbitrary. OSPF derives interface cost from a reference bandwidth divided by the interface bandwidth, and these virtual fabric ports report Speed 1000 in show interface status, which against the NX-OS default reference bandwidth lands on 40. The number itself does not matter much. What matters is that every fabric link in a Clos fabric lands on the same number, because the instant one does not, your equal-cost paths stop being equal.
Advertising the loopbacks, and why that is the whole point
Both loopbacks go into OSPF with ip router ospf UNDERLAY area 0.0.0.0 on the interface, and that is all the underlay is really being asked to distribute. The /31 transit links get advertised too, which is harmless, but nothing in the overlay ever needs to reach a transit address. What the overlay needs is every leaf reaching every other leaf's Loopback1, and every leaf reaching the spines' Loopback0 for BGP peering.
This is why a fabric underlay stays flat. The moment you summarise loopbacks at an area boundary you have created a place where a VTEP can be unreachable while the summary still looks healthy. Advertise host routes and let the fabric be dumb. The IP routing fundamentals material covers the next-hop resolution mechanics OSPF is feeding here.
ECMP over both spines
Here is the payoff. LEAF1 looking up LEAF2's VTEP address:
show ip route 10.255.1.12
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.255.1.12/32, ubest/mbest: 2/0
*via 10.0.1.0, Eth1/1, [110/81], 00:00:20, ospf-UNDERLAY, intra
*via 10.0.2.0, Eth1/2, [110/81], 00:00:28, ospf-UNDERLAY, intraubest/mbest: 2/0 is the number to check. Two unicast best paths, both marked with the leading asterisk, one out Eth1/1 through SPINE1 and one out Eth1/2 through SPINE2. If that said 1/0, you would have a fabric where every flow between these two leaves rides one spine and the other spine is a very expensive standby.
The metric arithmetic is worth doing by hand once, because it teaches you to spot an asymmetric cost by eye.
LEAF1 Ethernet1/1 outbound, the cost shown in show ip ospf interface brief.
The spine's fabric-facing Ethernet interface toward LEAF2, an identical link and therefore an identical cost.
LEAF2's Loopback1 itself, which OSPF costs at 1.
Total metric on both paths, which is why they are equal cost and both installed.
Both paths cost 81 because both cross exactly one spine and every fabric link is the same speed. That is not an accident of this lab, it is the property a Clos fabric is built to have: every leaf is the same distance from every other leaf. When you see one path at 81 and another at 121, someone has either configured a manual cost or plugged a slower link into the fabric, and your ECMP has quietly become a primary and a backup.
The summary view confirms the same thing at process level.
show ip ospf route summary | head lines 8
OSPF Process ID UNDERLAY VRF default, Routing Table
(D) denotes route is directly attached (R) denotes route is in RIB
(L) denotes route label is in ULIB (NHR) denotes next-hop is in RIB
Total number of routes: 11
Total number of paths: 13Eleven routes, thirteen paths. The two extras are the second next hop on each of the two ECMP destinations. On a multi-spine fabric, paths should always exceed routes, and the moment that number equals the route count you have lost your ECMP fabric-wide.
The only underlay test that matters
You can admire route tables all day. Before you configure a single line of overlay, run exactly one test: ping the remote VTEP source address from your own VTEP source address.
LEAF1# ping 10.255.1.12 source 10.255.1.11 count 5
PING 10.255.1.12 (10.255.1.12) from 10.255.1.11: 56 data bytes
64 bytes from 10.255.1.12: icmp_seq=0 ttl=253 time=17.458 ms
64 bytes from 10.255.1.12: icmp_seq=1 ttl=253 time=10.176 ms
64 bytes from 10.255.1.12: icmp_seq=2 ttl=253 time=7.459 ms
64 bytes from 10.255.1.12: icmp_seq=3 ttl=253 time=7.216 ms
64 bytes from 10.255.1.12: icmp_seq=4 ttl=253 time=7.075 ms
--- 10.255.1.12 statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 7.075/9.876/17.458 msThe source 10.255.1.11 is the entire point of the test. A ping without it sources from the outgoing interface, which is a /31 transit address, and a /31-to-loopback ping proves almost nothing about whether VXLAN will work. VXLAN encapsulation puts Loopback1 in the outer source IP and the remote Loopback1 in the outer destination IP. If that specific pair cannot reach each other, your tunnels will not come up no matter how correct the EVPN configuration is. Test the exact addresses the data plane will use.
ttl=253 is the small confirmation to notice. The reply left LEAF2 at 255 and arrived two decrements down: one at the spine, one at LEAF1 itself. Two routed hops, leaf to spine to leaf, which is exactly the Clos promise. A ttl=252 would mean three hops, and in a two-tier fabric that means your topology is not what you think it is, not that your protocol is misbehaving.
The first ping is slower than the rest at 17.458 ms against a 7 ms floor (the first packet pays for address resolution and a control plane punt, as always). That pattern is normal and is not a fabric problem.
What this underlay does not have
Being honest about scope is more useful than pretending completeness. This underlay was built with the default 1500-byte interface MTU, which is the wrong answer for a production VXLAN fabric: 50 bytes of VXLAN overhead means tenants silently lose 50 bytes of usable MTU, and oversized packets are dropped inside the fabric with no ICMP message coming back. Jumbo MTU on every underlay-facing interface belongs in the same config block as the /31 and the point-to-point network type.
There is also no BFD here, no authentication on the adjacencies and no tuned timers. All three are reasonable production additions, none were configured or tested in this lab, so no output is shown for them.
Where the underlay hands off
Once ping 10.255.1.12 source 10.255.1.11 returns five for five, the underlay's job is finished and it should never appear in a conversation again. The next layer up is BGP EVPN, which peers over Loopback0 rather than Loopback1 and uses the underlay purely as transport for its sessions. Those sessions are ordinary iBGP with the spines as route reflectors, so everything in the BGP fundamentals guides about reflection, next-hop behaviour and update sources applies unchanged. From there you can walk the control plane in BGP EVPN route types explained, then put tenant traffic on the wire with VXLAN L2VNI configuration. Both assume exactly the underlay built here, and the full build order across every layer is laid out in the data center fabric pillar guide.
Key Takeaways
- Use /31s on fabric links. Two hosts, two usable addresses, no waste, and the neighbour addresses read cleanly in
show ip ospf neighbors. ip ospf network point-to-pointremoves the DR election and the 40-second 2WAY/DROTHER window. Confirm it worked by looking forFULL/ -with no DR role, not by reading the config back.- Run two loopbacks per leaf: Loopback0 for router ID and BGP peering, Loopback1 as the VTEP source. Advertise both as host routes and never summarise them.
ubest/mbest: 2/0with two equal-cost*viaentries at metric 81 is the proof your spine layer is load sharing. One best path means half your fabric bandwidth is idle.- Metric 81 breaks down as 40 plus 40 plus 1. Equal costs everywhere is a design property of a Clos fabric, not luck.
- Before any overlay config, ping the remote VTEP loopback sourced from your own VTEP loopback. It is the only underlay test that reflects what VXLAN will actually do.
- Set jumbo MTU on every underlay-facing interface. This lab did not, and the resulting silent drops are a real and common failure.