Almost every VXLAN explanation you have read uses the same diagram: a coloured rectangle labelled "outer headers", another labelled "VXLAN", and a third labelled "original frame". It is a fine diagram and it teaches you nothing about what is actually on the wire. This post takes a real VXLAN packet, captured on a real spine-to-leaf link while a real ping was in flight, and walks every byte of it. The fabric it came from is the one built in the Data Center Fabric guide.
Captures like this are rare because you cannot get one from either end. Tap a server and you see a plain Ethernet frame. Log into a leaf and you see MAC tables and BGP routes. The encapsulated packet only exists on the fabric links between the switches, and those are usually inside a chassis, a data centre, or somebody else's cloud.
How the capture was taken
CML can capture on a virtual link, which is the piece that makes this possible. The capture point was the link between SPINE1's Ethernet1/1 and LEAF1's Ethernet1/1, with the BPF filter udp port 4789 applied so only VXLAN-encapsulated traffic was recorded. It ran while SRV1 pinged SRV2, two hosts in the same stretched VLAN 10 on opposite leaves.
udp port 4789
The resulting pcap opens cleanly in Wireshark with no coaxing, and if you want to follow along with your own tooling the packet analysis guides cover the general workflow. The important detail about how a decoder presents this file comes up immediately, in the summary view.
The summary view lies to you, helpfully
no time source destination len proto info
1 0.000000 10.10.10.11 10.10.10.12 148 ICMP Echo (ping) request id=0x0057, seq=1/256, ttl=64
2 1.015157 10.10.10.11 10.10.10.12 148 ICMP Echo (ping) request id=0x0057, seq=2/512, ttl=64
3 2.002264 10.10.10.11 10.10.10.12 148 ICMP Echo (ping) request id=0x0057, seq=3/768, ttl=64
4 2.013175 10.10.10.12 10.10.10.11 148 ICMP Echo (ping) reply id=0x0057, seq=3/768, ttl=64
...
10 6.246931 52:54:00:87:1d:df 52:54:00:b7:2c:84 92 ARP Who has 10.10.10.12? Tell 10.10.10.11Look at the source and destination columns. 10.10.10.11 and 10.10.10.12 are SRV1 and SRV2, the inner hosts. Nowhere in that summary do you see the VTEP addresses 10.255.1.11 and 10.255.1.12, and nowhere does the protocol column say VXLAN. A decoder that understands VXLAN dissects straight through to the innermost protocol and summarises that, which is genuinely what you want most of the time and genuinely confusing the first time you meet it.
The giveaway is the length column. A standard ping with 56 bytes of payload produces an 84-byte IP packet and a 98-byte Ethernet frame. These frames are 148 bytes. Fifty extra bytes appeared between the host and the wire, and those fifty bytes are the encapsulation. If you ever find yourself staring at a capture wondering whether something is tunnelled, the frame length against the payload you expected is the fastest check there is.
The arithmetic, before the hex
Do the sum first and the hex stops being intimidating, because you already know how many bytes each layer should take.
84 inner IP packet (56 bytes ICMP payload + 8 ICMP + 20 IP)
+14 inner Ethernet header
----
98 the original frame as SRV1 sent it
98 original frame
+50 VXLAN encapsulation overhead
----
148 the frame on the spine-to-leaf linkAnd the fifty bytes break down into four headers:
That fifty is not trivia. It is the number that decides your underlay MTU, and getting it wrong produces the nastiest failure mode in the whole technology: small packets fine, large packets vanishing with no ICMP unreachable, which the VXLAN and EVPN troubleshooting post covers in detail.
Packet 1, raw
Here are the first 96 bytes of packet 1, which is enough to reach the ICMP payload. The rest of the frame is echo payload and is not interesting.
0000 5247 fee9 1b08 529a 2a3a 1b08 0800 4500
0010 0086 0000 0000 fe11 a452 0aff 010b 0aff
0020 010c f3a0 12b5 0072 0000 0800 0000 0027
0030 1a00 5254 00b7 2c84 5254 0087 1ddf 0800
0040 4500 0054 ea55 4000 4001 2829 0a0a 0a0b
0050 0a0a 0a0c 0800 1826 0057 0001 3572 7f6aAnd the same packet dissected:
outer dst mac 52:47:fe:e9:1b:08
outer src mac 52:9a:2a:3a:1b:08
ethertype 0800 IPv4
outer ip 0x45, total len 134
outer ttl 254
outer proto 17 UDP
outer src ip 10.255.1.11 LEAF1 Loopback1 (VTEP source)
outer dst ip 10.255.1.12 LEAF2 Loopback1 (VTEP destination)
udp src port 62368 entropy hash of the inner flow
udp dst port 4789 VXLAN
udp length 114
udp checksum 0x0000 not computed, as the RFC allows
vxlan flags 0x08 I bit set: VNI field is valid
vxlan vni 10010 L2VNI for VLAN 10
inner dst mac 52:54:00:b7:2c:84 SRV2
inner src mac 52:54:00:87:1d:df SRV1
inner ethertype 0800 IPv4
inner ttl 64 untouched by the fabric
inner src ip 10.10.10.11 SRV1
inner dst ip 10.10.10.12 SRV2
icmp type 8 code 0 echo requestWalking the fields
Outer Ethernet, bytes 0 to 13
5247 fee9 1b08 is the destination MAC, SPINE1's interface on this link. 529a 2a3a 1b08 is the source, LEAF1's. Then 0800 for IPv4. This header is ordinary and disposable: SPINE1 strips it, decrements the outer TTL and writes a fresh one toward LEAF2. Nothing about it says VXLAN, which is the point. Every device between the two VTEPs treats this as a normal routed IP packet, and that is what lets you build a fabric out of switches that have never heard of the overlay.
Outer IP, bytes 14 to 33
45 is version 4 with a 5-word header, so 20 bytes, no options. 0086 is 134, the total IP length, which is our 148-byte frame minus the 14-byte Ethernet header. The addresses are 0aff010b and 0aff010c, which are 10.255.1.11 and 10.255.1.12, the two leaf Loopback1 interfaces. Those are the only IP addresses the underlay ever sees or routes on.
The byte to dwell on is fe, the outer TTL, 254. This packet has just left LEAF1 and has not yet been forwarded by anything. In the return direction of the same capture, the outer TTL reads 253, one lower, because those packets had already crossed SPINE1 before reaching this link. That single decrement is the entire visible cost of the fabric hop.
Meanwhile the inner TTL, at byte 72, is 40, decimal 64. Original, untouched, and it stays that way all the way to SRV2. Two TTL counters ticking independently in one packet is the clearest illustration of network virtualization you will find: real routed hops counted in the outer header, zero apparent hops in the inner one.
UDP, bytes 34 to 41
Destination port 12b5 is 4789, the IANA-assigned VXLAN port, and the one thing everybody remembers. The interesting field is the source port, f3a0, 62368.
That number is not random and it is not a socket. The encapsulating VTEP computes it as a hash over the inner packet's headers, typically the inner source and destination MAC and IP and the inner Layer 4 ports, and stuffs the result into the outer UDP source port. It exists purely to be hashed again, by somebody else.
Here is why that matters. A spine forwarding this packet has no idea it is VXLAN. It does an ECMP hash over the fields it always uses: outer source IP, outer destination IP, protocol, and the two UDP ports. But in a VXLAN fabric every packet between the same pair of leaves has identical outer IPs and an identical destination port. If the source port were fixed too, every flow between LEAF1 and LEAF2 would hash to the same value and pin itself to a single spine. You would have built a beautiful multi-path fabric and then funnelled all of it down one link.
The entropy source port fixes this without the spine understanding anything. Different inner flows produce different hashes, different source ports, different ECMP results, and the load spreads. The intelligence sits entirely in the VTEP and the transit devices stay dumb and fast. It also means a single long-lived inner flow still pins to one spine, because its hash never changes, which is why elephant flows are a real problem in fabrics.
Then 0072, UDP length 114, which is the 134-byte IP packet minus its 20-byte header. And 0000, the UDP checksum, or rather the absence of one.
A zero UDP checksum over IPv4 is explicitly permitted, and RFC 7348 recommends transmitting VXLAN packets that way. Computing a checksum over the entire encapsulated frame at line rate costs real silicon for a guarantee you already have twice over: Ethernet's FCS protects each hop, and the inner IP and transport checksums protect the payload end to end. If you see 0x0000 here, the packet is fine and your receiver will not validate it.
VXLAN, bytes 42 to 49
Eight bytes, and six of them are wasted. 08 is the flags byte, and the only bit that matters is the I bit, which must be set to indicate the VNI field is valid. Set it to zero and a compliant receiver discards the packet. The remaining flag bits are zero, then three reserved bytes, all zero.
Then 0027 1a, three bytes, decimal 10010. That is the VNI, and it is the whole reason this header exists. Twenty-four bits gives 16,777,216 possible segments against the 4,094 usable VLANs a 12-bit tag allows, and lifting that ceiling was the original motivation for the entire technology. One more reserved byte, and the header is done.
What makes 10010 satisfying rather than merely correct is that you can trace it back to the control plane. On LEAF1, show l2route evpn mac all lists SRV2's MAC with next hop 10.255.1.12 (Label: 10010). BGP said the VNI would be 10010 and the tunnel destination would be 10.255.1.12. The packet on the wire carries exactly those two values, in those two fields. Control plane and data plane, closed loop, verified. The L2VNI configuration walkthrough shows that control-plane side and the config that produced it.
Inner Ethernet, bytes 50 to 63
From here down, this is SRV1's frame exactly as it left the server's NIC. Destination 5254 00b7 2c84 is SRV2, source 5254 0087 1ddf is SRV1, ethertype 0800.
Note what is not here. There is no 802.1Q tag. VLAN 10 exists on LEAF1 and on LEAF2, but the tag itself never enters the tunnel, because the VNI has replaced it. That is precisely why the two leaves would not have to use the same VLAN ID as long as both map to VNI 10010.
Note also that SRV2's MAC is here at all. Neither server ARPed across a broadcast for this: LEAF1 learned that MAC from a BGP EVPN Type-2 route, installed it in the VLAN 10 MAC table pointing at nve1(10.255.1.12), and could therefore encapsulate directly.
Inner IP and ICMP, bytes 64 onward
45 again, 0054 is 84 bytes total, matching our arithmetic. 4000 sets the Don't Fragment bit, which is what makes the MTU failure mode a hard drop rather than silent fragmentation. TTL 40 is 64. Protocol 01 is ICMP. Addresses 0a0a 0a0b and 0a0a 0a0c are 10.10.10.11 and 10.10.10.12.
Then 0800: ICMP type 8, code 0, echo request. 0057 is the identifier, matching the id=0x0057 in the summary view, and 0001 is sequence number 1. This is packet 1 of the ping, and everything after it is timestamp and filler payload.
The last packet: broadcast inside a tunnel
Packet 10 is the one worth staying for.
10 6.246931 52:54:00:87:1d:df 52:54:00:b7:2c:84 92 ARP Who has 10.10.10.12? Tell 10.10.10.11An ARP request, encapsulated in VXLAN, on a routed spine-to-leaf link. This is what BUM traffic looks like on the wire when the fabric handles it by ingress replication: LEAF1 took a broadcast frame, consulted the flood list it built from EVPN Type-3 routes, and sent a unicast copy to each VTEP on that list, each one wrapped in exactly the same 50 bytes of overhead as a unicast packet. No multicast in the underlay, no PIM, no rendezvous point.
The length works out too. Ninety-two bytes minus 50 of overhead leaves a 42-byte inner frame, which is a 14-byte Ethernet header plus a 28-byte ARP payload, the minimum an ARP request can be.
The tradeoff is visible in that arithmetic. Ingress replication scales linearly: ten VTEPs in a VNI means ten copies generated by the ingress leaf for every broadcast frame. Multicast in the underlay pushes that replication into the spines instead. For a small fabric like this one, ingress replication is far simpler and the cost is negligible.
Key Takeaways
- VXLAN adds exactly 50 bytes: 14 outer Ethernet, 20 outer IP, 8 UDP, 8 VXLAN. An 84-byte inner IP packet plus 14 bytes of inner Ethernet becomes a 148-byte frame, which is what the capture shows.
- A VXLAN-aware decoder shows the inner addresses in its summary column, so the frame length against the payload you expected is the real giveaway that encapsulation is happening.
- The outer TTL counts real fabric hops (254 leaving the VTEP, 253 after one spine) while the inner TTL stays at 64 the whole way.
- The UDP source port is a hash of the inner flow, not a socket. It exists so transit spines can ECMP on fields they already hash without understanding VXLAN at all.
- A single long-lived flow always hashes to the same source port and therefore pins to one spine, which is why elephant flows are a fabric problem.
- UDP checksum
0x0000is correct and expected. The RFC allows it because the Ethernet FCS and the inner checksums already cover the same bytes. - VXLAN flags
0x08is the I bit, meaning the VNI is valid. Six of the eight header bytes are flags and reserved space. - The VNI on the wire matches the
(Label: 10010)inshow l2route evpn mac all. That match is the cleanest proof that your control plane and data plane agree. - Ingress-replicated broadcast is just a unicast VXLAN packet with an ARP request inside, and one is generated per remote VTEP in the VNI.
If you can read this packet, you can troubleshoot a fabric, because every VXLAN failure is ultimately one of these fields being wrong or one of these packets not arriving. The rest of the cluster, from underlay to overlay, is indexed in the Data Center Fabric guide.