Expert Layer 2 Troubleshooting: Five Broken Scenarios, Ticket Style

Layer 2 problems have a special quality: the switches all say everything is fine, and yet nothing works. There is no routing table to inspect, no neighbour state to compare, just VLANs, trunks, and a spanning tree that is quietly blocking the exact port you need. The diagnosis lives in a handful of show commands and, above all, in the syslog - because Layer 2, unlike OSPF, is actually fairly talkative when it detects a mismatch.

This article is five Layer 2 faults, each built and broken in a CML lab, with the real output and the real fix. For the theory, see the VLAN and switching guide and the spanning tree guide.

The method

Work Layer 2 in this order:

Ticket 1: "This VLAN won't cross one trunk"

Symptom. VLAN 10 works everywhere except across the SW3-SW2 trunk. The trunk is up. Other VLANs cross it fine.

Diagnosis. The logs tell you immediately:

SW3#show logging | include NATIVE_VLAN
*Jul 12 08:10:33.701: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on
                      Ethernet0/0 (88), with SW2 Ethernet0/1 (99).

And earlier, when the native VLANs were 1 vs 99:

*SPANTREE-2-BLOCK_PVID_PEER: Blocking Ethernet0/0 on VLAN0099. Inconsistent peer vlan.
*SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Ethernet0/0 on VLAN0001. Inconsistent local vlan.

Cause. A native VLAN mismatch. One side of the trunk has native VLAN 99, the other has something else. This produces two distinct symptoms, and it is important to understand both:

• CDP logs a warning (%CDP-4-NATIVE_VLAN_MISMATCH). This is cosmetic - CDP noticed and is telling you.
• Spanning tree actually blocks the native VLAN (%SPANTREE-2-BLOCK_PVID). This is not cosmetic. STP detects that the untagged (native) traffic on the two ends belongs to different VLANs - a security and looping hazard - and blocks the PVID on that segment. The VLAN genuinely stops passing.

A native VLAN mismatch is also a real security concern: it can enable VLAN hopping, because traffic in one side's native VLAN pops out into the other side's native VLAN untagged.

Fix: make the native VLAN match on both ends.

SW3(config-if)#switchport trunk native vlan 99

Lesson: a native VLAN mismatch gives you two log messages from two subsystems. The CDP one is a warning; the SPANTREE BLOCK_PVID one is the actual outage. Both point at the same fix.

Ticket 2: "MST is behaving strangely after a change"

Symptom. After someone edited the MST configuration, two switches that should be in the same region are suddenly not, and ports are blocking unexpectedly.

Diagnosis. The MST region digest is the entire answer:

SW1#show spanning-tree mst configuration digest
Digest          0xE05029508B8AC96367E964B97EE7DA9B

SW2#show spanning-tree mst configuration digest
Digest          0xDA7E3D01C232F1CFDB7F72C25D414BAA

Cause. Different digests mean different regions. Two switches are in the same MST region only if their name, revision, and complete VLAN-to-instance map all match exactly. In the lab we moved a single VLAN from one instance to another on just one switch - and that was enough to change the digest and split the region.

When a region splits, the link between the two "regions" becomes a boundary, spanning tree recalculates as if crossing between separate domains, and ports that were forwarding can start blocking.

Fix: make the MST configuration byte-identical on every switch in the region. Compare the name, revision, and VLAN map, and copy-paste rather than retype.

spanning-tree mst configuration
 name PINGLABZ
 revision 1
 instance 1 vlan 10
 instance 2 vlan 20,99

Lesson: for any MST oddity, show spanning-tree mst configuration digest on both switches is the first command. Same digest = same region. Different digest = you have found your problem. Full detail in MST and PVST+ interoperation.

Ticket 3: "A port is dead and won't come back"

Symptom. An access port is down. The cable is fine. no shutdown does not fix it.

Diagnosis.

SW3#show interfaces Ethernet0/2 status
Port         Name               Status       Vlan
Et0/2                           err-disabled 10

err-disabled, not down. That is a protection feature, not a physical fault. Find which one:

SW3#show port-security interface Ethernet0/2
Port Status                : Secure-shutdown
Security Violation Count   : 1

SW3#show logging | include SECURE
*Jul 12 08:08:40.999: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred,
                       caused by MAC address 02aa.bbcc.ddee on port Ethernet0/2.

Cause. A port security violation. A frame with an unauthorised source MAC (here 02aa.bbcc.ddee - a MAC that changed on the connected host) arrived on a port configured to allow only one, sticky-learned MAC. Port security did exactly what it was told: errdisabled the port.

Fix. A plain no shutdown does not clear errdisable - you must either bounce the port with shutdown then no shutdown, or configure automatic recovery:

SW3(config)#errdisable recovery cause psecure-violation
SW3(config)#errdisable recovery interval 30

SW3#show interfaces Ethernet0/2 status
Et0/2                           connected    10          <-- recovered

Lesson: err-disabled is a feature acting, not a fault. show interfaces status tells you it is errdisabled; the syslog tells you which feature and why. Fix the cause before you recover the port, or it just errdisables again.

Ticket 4: "The router-on-a-stick subinterfaces are all down"

Symptom. A router doing inter-VLAN routing has all its dot1q subinterfaces down. The config looks correct. The trunk on the switch side is up.

Diagnosis.

R1#show ip interface brief | include Ethernet0/1
Ethernet0/1            unassigned      YES unset  administratively down down
Ethernet0/1.10         10.10.10.1      YES TFTP   administratively down down
Ethernet0/1.20         10.10.20.1      YES TFTP   administratively down down
Ethernet0/1.99         10.10.99.1      YES TFTP   administratively down down

Cause. The parent physical interface is administratively down. A subinterface inherits the admin state of its parent - if the physical interface is shut, every subinterface on it is down regardless of its own config. In this case the parent came up admin-down at boot (a real quirk we hit in the lab: the day-0 config's no shutdown was applied before the subinterfaces existed, and the parent ended up shut).

Fix: bring up the parent.

R1(config)#interface Ethernet0/1
R1(config-if)#no shutdown

Lesson: when every subinterface on one physical port is down at once, look at the parent. A subinterface cannot be up while its parent is shut. The tell is that they are all down together - a config problem on one subinterface would only affect that one.

Ticket 5: "A static-IP host lost connectivity after we enabled DAI"

Symptom. After enabling Dynamic ARP Inspection, one server - the one with a manually-configured static IP - can no longer communicate. Every DHCP client on the same VLAN is fine.

Diagnosis. The DHCP-client hosts work; the static host does not. That pattern points straight at the binding table:

SW3#show ip dhcp snooping binding
MacAddress          IpAddress        Lease(sec)  Type           VLAN  Interface
52:54:00:4E:11:C0   10.10.10.10      86387       dhcp-snooping   10    Ethernet0/2
Total number of bindings: 1

The DHCP client (10.10.10.10) has a binding. The static server does not appear at all.

Cause. DAI validates ARP against the DHCP snooping binding table. A host with a static IP never did a DHCP transaction, so it has no binding, so DAI drops its ARP - and it goes dark. This is the single most common DAI deployment mistake: turning it on without first accounting for the static devices (servers, printers, network gear) that do not use DHCP.

Fix. Give the static host a binding, either a static snooping entry or an ARP ACL:

! Option A: static source binding
ip source-binding 0011.2233.4455 vlan 10 10.10.10.50 interface Ethernet0/5

! Option B: an ARP ACL that permits the pair
arp access-list STATIC-SERVERS
 permit ip host 10.10.10.50 mac host 0011.2233.4455
!
ip arp inspection filter STATIC-SERVERS vlan 10

Lesson: DAI (and IPSG) trust only the snooping binding table. Any host that does not use DHCP needs an explicit binding before you enable inspection. Inventory your static devices first. See DAI and IP Source Guard.

The commands, collected

! Logs first - L2 announces its mismatches
show logging | include NATIVE_VLAN|PVID|SECURE|err-disable|UDLD

! Trunks
show interfaces trunk
show interfaces <int> switchport

! Spanning tree
show spanning-tree
show spanning-tree mst configuration digest      (region membership)
show spanning-tree inconsistentports
show spanning-tree vlan <n>

! Ports
show interfaces status
show port-security interface <int>
show errdisable recovery

! MAC / bindings
show mac address-table
show ip dhcp snooping binding
show ip verify source

Try it yourself

Build a mixed MST/PVST square with a router-on-a-stick and some security features, then break it. Ten minutes per fault, and name the cause from show output before you look at the config:

1. Set a different native VLAN on one end of a trunk. Note that you get both a CDP warning and an STP block.
2. Move one VLAN to a different MST instance on one switch only. Diff the digests.
3. Change a host's MAC on a port with sticky port security allowing one MAC.
4. Shut the parent of a set of router-on-a-stick subinterfaces.
5. Enable DAI on a VLAN that has a static-IP host with no binding.

Key takeaways

• Read the logs first. Layer 2 announces native VLAN mismatches, PVID inconsistencies, port security violations, and errdisable causes. This is the opposite of OSPF, and you should exploit it.
• A native VLAN mismatch produces two messages: a cosmetic CDP warning and an actual STP BLOCK_PVID that stops the VLAN. Same fix for both.
• For any MST oddity, compare show spanning-tree mst configuration digest. Different digest = different region = your problem.
• err-disabled is a protection feature acting, not a physical fault. The syslog names the cause; fix it before recovering the port.
• All subinterfaces down at once = the parent physical interface is shut.
• DAI/IPSG blocking one specific host, while DHCP clients are fine, means that host has no snooping binding. Static devices need explicit bindings before you enable inspection.

That closes the expert Layer 2 series - and with it, switching on PingLabz is complete from CCNA fundamentals to CCIE depth. The full cluster indexes live on the VLAN and switching pillar and the spanning tree pillar.