Unicast in a VXLAN fabric is the easy half. BGP EVPN tells every leaf where every MAC address lives, the ingress VTEP wraps the frame, and it goes to exactly one destination. The hard half is everything that is not addressed to a single known destination: broadcast, unknown unicast and multicast, collectively BUM. Those frames have to reach every VTEP participating in the VNI, and how your fabric does that is a design decision you make once and live with. This post walks the ingress-replication answer with real output and is a companion to the broader data center fabric build guide.
What BUM actually is, in a fabric that has EVPN
People assume EVPN makes BUM traffic go away. It does not, it shrinks it.
Broadcast is the obvious one and ARP is most of it. Even with a control plane distributing MAC addresses, a host that wants to talk to a neighbour it has not seen still broadcasts an ARP request. DHCP discovers, IPv6 neighbour discovery and various discovery protocols pile in behind it.
Unknown unicast is where EVPN genuinely helps. On a classic flood-and-learn fabric, any frame for a MAC the switch has not learned gets flooded everywhere. With EVPN the control plane has already advertised remote MACs before any traffic arrives, so the switch usually knows the destination and does not flood at all. Silent hosts and genuine unknowns still fall through to flooding, so the path has to exist even when it is rarely used.
Multicast inside the tenant VLAN is the third category, and without IGMP snooping over VXLAN it is treated as broadcast and flooded to every VTEP in the VNI.
Two ways to get a frame everywhere
There are only two mechanisms, and they differ in where the work happens.
Ingress replication puts the work on the ingress VTEP. It holds a list of every remote VTEP participating in the VNI, and when a BUM frame arrives, it makes one VXLAN-encapsulated copy per remote VTEP and unicasts each one. The underlay sees ordinary unicast traffic and needs to know nothing about flooding. This is also called head-end replication, and it is what this lab runs.
Multicast replication puts the work on the underlay. Each VNI is mapped to an underlay multicast group, every VTEP in the VNI joins that group, and the ingress VTEP sends a single copy to the group address. The underlay's PIM tree replicates it at the branch points, which on a Clos fabric means the spines. It needs multicast in the underlay: PIM enabled everywhere, a rendezvous point, and in any serious build an anycast RP so that RP failure is not a fabric outage.
To be clear up front, because this matters for how you read the rest of this article: the PIM anycast RP option was not configured in this lab. There is no PIM output below because none was captured. Everything shown is the ingress-replication build, and the PIM discussion is design reasoning, not measurement.
What ingress replication looks like on the box
The configuration is one line per VNI under the NVE interface. Here is the L2VNI setup from LEAF1, both VNIs included.
vlan 10
name TENANT-A-WEB
vn-segment 10010
vlan 20
name TENANT-A-APP
vn-segment 10020
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 10010
ingress-replication protocol bgp
member vni 10020
ingress-replication protocol bgp
evpn
vni 10010 l2
rd auto
route-target import auto
route-target export auto
vni 10020 l2
rd auto
route-target import auto
route-target export autoTwo commands carry the whole design. host-reachability protocol bgp says unicast forwarding comes from the control plane, not data plane learning. ingress-replication protocol bgp says the flood list comes from EVPN too, rather than being statically configured or derived from a multicast group. The word bgp in both is the difference between a fabric that discovers its own flood list and one where you maintain peer lists by hand.
Verify it with a single command.
show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
...
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 10010 UnicastBGP Up CP L2 [10]
nve1 10020 UnicastBGP Up CP L2 [20]The Multicast-group column is the tell. It says UnicastBGP, which is NX-OS telling you there is no multicast group because this VNI floods by unicast replication and BGP supplies the peer list. On a PIM-based fabric that same column holds a real multicast group address, something in your underlay's group range, and seeing an actual group there when you thought you configured ingress replication is your fastest possible catch of a mismatched build.
The Mode column says CP, control plane, on both VNIs. DP there would mean data plane learning, flood-and-learn VXLAN without EVPN doing the work. Two columns, and between them you know exactly which forwarding model the box is running.
Type-3 routes are the flood list
If the flood list comes from BGP, there has to be a route that carries it. That route is EVPN Route Type 3, the Inclusive Multicast Ethernet Tag route. Here is LEAF1's EVPN table with Type-2 and Type-3 routes side by side.
show bgp l2vpn evpn
Route Distinguisher: 10.255.0.11:32777 (L2VNI 10010)
*>i[2]:[0]:[0]:[48]:[4e69.d07b.10f7]:[0]:[0.0.0.0]/216
10.255.1.12 100 0 i
*>l[2]:[0]:[0]:[48]:[5254.0087.1ddf]:[0]:[0.0.0.0]/216
10.255.1.11 100 32768 i
*>l[3]:[0]:[32]:[10.255.1.11]/88
10.255.1.11 100 32768 i
*>i[3]:[0]:[32]:[10.255.1.12]/88
10.255.1.12 100 0 i
Route Distinguisher: 10.255.0.11:32787 (L2VNI 10020)
*>l[3]:[0]:[32]:[10.255.1.11]/88
10.255.1.11 100 32768 i
*>i[3]:[0]:[32]:[10.255.1.12]/88
10.255.1.12 100 0 iRead the [3] prefixes. Each one is a VTEP saying "I participate in this VNI, send me the flooded traffic, and here is my tunnel address". The [32] is the prefix length of the originating router's IP, and the address that follows is the VTEP source loopback. l marks a locally originated route and i marks one learned internally from the route reflector, so in VNI 10010 LEAF1 sees its own Type-3 for 10.255.1.11 and LEAF2's for 10.255.1.12.
That pair of routes is the entire flood list for VNI 10010 here: two VTEPs, so one remote entry from LEAF1's point of view. Add a third leaf and a third Type-3 appears with no configuration change on the existing leaves. That is what makes ingress replication over EVPN operationally pleasant, because the flood list maintains itself.
Notice the second block, VNI 10020, has Type-3 routes but no Type-2 routes. Nothing had been learned in VLAN 20 at that point, yet both VTEPs still announced participation. Flood membership is independent of whether any host exists, which is exactly right: a VLAN with no hosts still needs a working flood path the moment the first host powers on and ARPs. For the full route type walkthrough, the EVPN route types breakdown takes each one apart with output.
Watching a broadcast get replicated
The theory is fine, but here is the concrete case. A packet capture was taken on the SPINE1 to LEAF1 fabric link with a BPF filter of udp port 4789, while the two servers pinged each other. Ten packets, and the last one is the interesting one.
no time source destination len proto info
1 0.000000 10.10.10.11 10.10.10.12 148 ICMP Echo (ping) request id=0x0057, seq=1/256, ttl=64
2 1.015157 10.10.10.11 10.10.10.12 148 ICMP Echo (ping) request id=0x0057, seq=2/512, ttl=64
3 2.002264 10.10.10.11 10.10.10.12 148 ICMP Echo (ping) request id=0x0057, seq=3/768, ttl=64
4 2.013175 10.10.10.12 10.10.10.11 148 ICMP Echo (ping) reply id=0x0057, seq=3/768, ttl=64
...
10 6.246931 52:54:00:87:1d:df 52:54:00:b7:2c:84 92 ARP Who has 10.10.10.12? Tell 10.10.10.11Packet 10 is an ARP request, inside a VXLAN packet, crossing a routed underlay link between a leaf and a spine. That is the broadcast half of BUM traffic caught on the wire: SRV1 ARPed for its neighbour in VLAN 10, LEAF1 consulted its Type-3 derived flood list for VNI 10010, found one remote VTEP at 10.255.1.12, encapsulated the frame and unicast it across the fabric. No multicast was involved anywhere in that path.
One honest caveat, because the decoder is showing the inner addresses. Those source and destination columns hold MACs, and the destination is 52:54:00:b7:2c:84, SRV2's own MAC rather than the all-ones broadcast address, so this particular frame is a unicast ARP refresh rather than a first-contact broadcast. What it proves is the encapsulation path, which a broadcast ARP takes identically with the flood list supplying the destinations instead of one learned MAC. With two VTEPs the replication list has one entry either way, so nothing captured here demonstrates fan-out.
Worth pausing on how ordinary this looks from the underlay's perspective. The spine treated it like any other unicast VXLAN packet and has no idea it just forwarded a broadcast. That opacity is the feature: the underlay never needs a flooding mechanism of its own.
The cost, stated honestly
Ingress replication is simple, and it is simple because it moved the expensive part onto the ingress VTEP. With N VTEPs in a VNI, one broadcast frame arriving at a leaf becomes N minus 1 encapsulated copies generated by that leaf and N minus 1 packets injected into the fabric. The work scales linearly with VTEP count, and it is per VNI, so a leaf carrying many stretched VLANs multiplies it again.
On this two-leaf lab that is one copy, which is not a scaling story. On a fabric with 40 leaves in a VNI it is 39 copies of every broadcast, and a broadcast storm in one tenant VLAN becomes 39 times the traffic on the ingress uplinks. That is the number to hold in your head when someone tells you ingress replication is always the right answer.
Ingress replication: the source leaf. PIM: the underlay, at the branch points, which on a Clos fabric means the spines.
Ingress replication grows linearly with VTEP count per VNI. PIM sends one copy regardless of how many VTEPs joined.
Ingress replication needs plain unicast routing. PIM needs multicast routing everywhere, RP placement, anycast RP and a way to share source state between the RPs.
Ingress replication learns it from EVPN Type-3 routes automatically. PIM builds it from group membership in the underlay.
Ingress replication fails in BGP, where you already look. PIM adds RP reachability, group mapping and tree state to the list of suspects.
show nve vni for both. UnicastBGP in the Multicast-group column means ingress replication, a group address means PIM.
Why modern builds pick ingress replication anyway
Given that PIM scales better, why has ingress replication become the default in new fabrics? Because the multicast option costs you an entire additional protocol domain in exchange for optimising traffic that EVPN has already made rare.
Consider what running PIM in the underlay actually means. You enable PIM on every fabric-facing interface and every SVI that participates. You choose an RP, and since a single RP is a single point of failure for all fabric flooding, you deploy anycast RP: the same RP address configured on both spines, advertised by both, with every leaf reaching whichever one the underlay says is closer. The two RPs then have to share source state, either through MSDP or through the PIM-native anycast RP mechanism, or a source registered at one RP stays invisible to receivers joining at the other. You map VNI ranges to multicast groups and keep that mapping consistent across every leaf, because a leaf with the wrong group for a VNI joins the wrong tree and silently receives nothing. Then you maintain all of it as the fabric grows.
That is a lot of machinery, and it is machinery your NOC has to troubleshoot at 3am. The counter-argument is real at large scale: with hundreds of VTEPs in a stretched VNI the linear cost of head-end replication genuinely hurts, and pushing replication into hardware that is already excellent at it is the correct engineering call. If you are heading that way, or want to understand sparse mode and anycast RP properly before deciding, the multicast routing guides cover RP selection, shared trees and MSDP in detail.
This is a scale decision with a complexity price attached. Most enterprise fabrics never reach the VTEP count where PIM pays for itself, and they benefit every day from having one fewer protocol to operate.
Reducing BUM before you replicate it
The best optimisation for either model is to have less BUM traffic in the first place, and EVPN gives you the tool. ARP suppression lets a leaf answer ARP requests locally from the MAC/IP bindings it already learned through Type-2 routes, so the request never becomes a flooded frame at all. The legend in show nve vni lists SA - Suppress ARP as a flag for exactly this reason.
It was not enabled in this lab. The Flags column in the show nve vni output above is empty on both VNIs, which is consistent with that, and packet 10 is an ARP request that did cross the fabric encapsulated rather than being answered locally by the leaf. Turning it on is usually the highest-value change you can make to BUM load, and it applies whichever replication model you chose.
Where this fits in the build
Replication mode is a decision you make while configuring the NVE interface, at the same moment you set the VNI members, so it belongs with the L2VNI work rather than being a separate project. The full VNI build including the VLAN mapping, the EVPN stanza and the verification chain is in VXLAN L2VNI configuration, and the layer-by-layer fabric build order lives in the data center fabric pillar. Pick ingress replication unless you have a concrete scale reason not to, and if you do pick PIM, commit to anycast RP properly rather than leaving a single RP as a fabric-wide dependency.
Key Takeaways
- EVPN reduces BUM traffic but does not eliminate it. ARP, DHCP, silent hosts and tenant multicast still need a flood path.
ingress-replication protocol bgpunder the VNI makes the ingress VTEP produce one unicast copy per remote VTEP, using a flood list built from BGP.UnicastBGPin the Multicast-group column ofshow nve vniconfirms ingress replication. A real group address in that column means the VNI is running PIM instead.- EVPN Route Type 3, Inclusive Multicast Ethernet Tag, is the route that builds the flood list. It appears per VNI per VTEP and exists even when no hosts have been learned yet.
- Ingress replication scales linearly: N VTEPs means N minus 1 copies per BUM frame per VNI, generated by the ingress leaf.
- PIM sends one copy and lets the underlay replicate, which scales better, at the cost of multicast routing, RP placement, anycast RP and MSDP. This lab did not configure PIM, so no PIM output is shown here.
- ARP suppression cuts BUM at the source and helps either model. It was not enabled in this lab, and an ARP request was duly captured being ingress-replicated across the fabric.