VXLAN Deep Dive: VNIs, VTEPs, and the UDP Encapsulation

Every modern data centre fabric, and the campus fabric of SD-Access, is built on VXLAN. It is the technology that lets you stretch a Layer 2 segment across a routed network, put a hundred thousand tenants where VLANs gave you four thousand, and move a virtual machine from one rack to another without changing its IP. Underneath the marketing, VXLAN is a genuinely simple idea: wrap the original Ethernet frame inside a UDP packet and route it like any other traffic.

This article takes VXLAN apart to the byte level, using a real crafted-and-decoded packet rather than a diagram. It extends the Network Virtualization and Overlays cluster guide.

The Problems VXLAN Solves

Three limitations of traditional VLANs drove VXLAN:

The Two Terms That Matter: VTEP and VNI

A packet's journey: host A sends a normal Ethernet frame; its local VTEP looks up the destination, wraps the whole frame in VXLAN + UDP + IP addressed to the remote VTEP, and routes it across the underlay; the remote VTEP strips the wrapper and delivers the original frame to host B. Host A and host B believe they are on the same switch. They may be in different buildings.

The Encapsulation, Byte by Byte

Rather than draw the header, here is a real VXLAN packet built with Scapy and dumped as raw bytes. The inner frame is a ping from tenant host 10.10.10.1 to 10.10.10.2; the outer wrapper carries it between VTEPs 10.0.0.1 and 10.0.0.2 in VNI 10010:

###[ IP ]###          (outer - routed across the underlay)
  proto     = udp
  src       = 10.0.0.1        <- source VTEP
  dst       = 10.0.0.2        <- destination VTEP
###[ UDP ]###
     sport     = 51234        <- hash of the inner flow (entropy for ECMP)
     dport     = 4789         <- the VXLAN port
###[ VXLAN ]###
        flags     = Instance
        vni       = 0x271a     <- 10010
###[ Ethernet ]###     (inner - the original tenant frame, intact)
           dst   = 00:11:22:33:44:02
           src   = 00:11:22:33:44:01
###[ IP ]###
              src = 10.10.10.1
              dst = 10.10.10.2
###[ ICMP ]###  echo-request

And the actual bytes on the wire, which make the structure undeniable:

0000  45 00 00 4E 00 01 00 00 40 11 66 9C 0A 00 00 01   E..N....@.f.....
0010  0A 00 00 02 C8 22 12 B5 00 3A 19 ED 08 00 00 00   ....."...:......
0020  00 27 1A 00 00 11 22 33 44 02 00 11 22 33 44 01   .'...."3D..."3D.
0030  08 00 45 00 00 1C 00 01 00 00 40 01 52 CA 0A 0A   ..E.......@.R...
0040  0A 01                                             ..

Trace it:

Two design details fall out of those bytes. The UDP source port is a hash of the inner flow, which is deliberate: it gives the underlay entropy so equal-cost paths (ECMP) load-balance different tenant flows across different links, even though every VXLAN packet has the same UDP destination port. And the encapsulation adds 50 bytes of overhead (outer Ethernet + IP + UDP + VXLAN), which is why VXLAN fabrics run jumbo MTU (9216) on the underlay; a 1500-byte tenant frame plus 50 bytes exceeds 1500 and would fragment or drop.

The Missing Piece: How Does a VTEP Know Where to Send?

The encapsulation above assumes the source VTEP already knows that host 10.10.10.2's MAC lives behind VTEP 10.0.0.2. But how did it learn that? This is the control-plane question, and it is where the two generations of VXLAN differ:

The data-plane encapsulation is identical either way, the bytes above are the same. What changes is how the mapping table gets populated. Flood-and-learn does it reactively by flooding; EVPN does it proactively by advertising. The control plane is a big enough topic that it has its own article: VXLAN with BGP EVPN, which builds on the BGP and MP-BGP knowledge you already have.

L2VNI and L3VNI

One more distinction that trips people up. A fabric uses two kinds of VNI:

• L2VNI maps to a VLAN and carries a Layer 2 segment. Hosts in the same L2VNI are on the same subnet and bridge to each other across the fabric. The packet above is an L2VNI packet.
• L3VNI maps to a VRF and carries routed traffic between subnets. When host A in one subnet talks to host B in another, the traffic is routed at the ingress leaf (anycast gateway) and carried across the fabric in the L3VNI. This is symmetric IRB (Integrated Routing and Bridging), the standard modern design.

The practical upshot: every leaf shares the same anycast gateway IP and MAC for each subnet, so a host's default gateway is always its local leaf, wherever the host moves. That is what makes VM mobility seamless, and it is pure VXLAN-EVPN mechanics.

FAQ

What port does VXLAN use?

UDP 4789 (IANA-assigned). You can see it as 0x12B5 in the real capture above. Some older Cisco deployments used 8472; standardise on 4789.

How much overhead does VXLAN add?

50 bytes (outer Ethernet 14 + IP 20 + UDP 8 + VXLAN 8). This is why underlay MTU must be raised, typically to 9216, or tenant frames near 1500 bytes will fragment or drop.

Why is the UDP source port randomised?

It is a hash of the inner flow, giving the underlay entropy so ECMP can load-balance different tenant flows across parallel links. The destination port is always 4789, so without this, all VXLAN traffic between two VTEPs would pin to one path.

Do I still use VLANs with VXLAN?

Yes, locally. A leaf maps a local VLAN to an L2VNI. The VLAN is significant only on that switch; the VNI is what is significant across the fabric. VLANs did not go away, they became leaf-local.

Is VXLAN the same as a GRE tunnel?

Conceptually similar (encapsulate and route), but VXLAN uses UDP (better for ECMP and hardware offload), carries a 24-bit segment ID, and is purpose-built for L2-over-L3 at fabric scale. See the GRE cluster for the comparison.

Key Takeaways

• VXLAN wraps the original Ethernet frame in UDP (port 4789) + IP and routes it between VTEPs across a Layer 3 underlay.
• A VTEP encapsulates/decapsulates; a VNI is the 24-bit segment ID (16M segments vs the VLAN's 4094).
• The real byte dump proves it: outer VTEP IPs, UDP dport 0x12B5 (4789), the 8-byte VXLAN header with the Instance flag and VNI 0x271A (10010), then the tenant frame intact.
• The UDP source port is a per-flow hash for ECMP entropy; the encapsulation adds 50 bytes, so raise the underlay MTU.
• The data plane is identical whether the control plane is flood-and-learn or BGP EVPN. Only how VTEPs learn reachability differs.
• L2VNI bridges within a subnet; L3VNI routes between subnets via anycast gateways, which is what makes VM mobility seamless.

Next: VXLAN with BGP EVPN for the control plane, or the Network Virtualization cluster guide.