vPC Failure Modes: Peer-Link, Keepalive, Type-1

The vPC questions people search for at 2am all have the same shape: the peer-link is down, or the keepalive is down, or both, and what is happening to traffic right now. This article answers that, and it opens with a disclosure. The two-leaf Nexus 9000v pair behind the data center fabric guide exercised exactly one of those failures for real. The other three are explained from first principles, with no CLI output, because inventing it would defeat the point of running a lab.

What was tested and what was not

Being precise up front saves you trusting the wrong paragraph later:

There is a second honest caveat that shapes everything below. This fabric had zero vPC member ports. SRV1 is single-homed to LEAF1 and SRV2 is single-homed to LEAF2, so nothing downstream was dual-attached and Number of vPCs configured read 0 throughout. That is why the consistency test was safe to run on a live domain, and also why running the peer-link failure case would have proved little: there were no member ports to suspend. The domain, its peer-link, its keepalive and its consistency checking were all fully operational, which is enough to demonstrate the control plane behaviour but not the data plane consequence.

Three health states, three independent failures

Every vPC diagnosis starts by separating the three things at the top of show vpc. Here they are in the healthy state on LEAF1:

vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success

Those three lines report on three different mechanisms that share almost nothing:

• Peer status is CFS across the peer-link. If the peer-link fails, this is the line that changes.
• Keep-alive status is a UDP heartbeat over the out-of-band management path. It knows nothing about the peer-link and does not care whether it is up.
• Configuration consistency status is a comparison of the two running configurations. It can fail while both of the above are perfectly healthy, which is precisely what the lab produced.

Most bad vPC troubleshooting comes from collapsing those three into one idea of "the vPC is broken". They fail independently, and the combination of the three tells you what the switches are about to do. For the build and a field-by-field read of the healthy output, see the vPC explainer.

The failure this lab did produce: a Type-1 mismatch

The break was one line, applied to LEAF2 only:

spanning-tree mode mst

STP Mode is a Type-1 consistency parameter, which the legend on show vpc consistency-parameters global describes as "vPC will be suspended in case of mismatch". Within seconds LEAF1 reported this:

LEAF1# show vpc brief | head lines 14

vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : failed
Per-vlan consistency status       : success
Configuration inconsistency reason: vPC type-1 configuration incompatible - STP Mode inconsistent
Type-2 consistency status         : success
vPC role                          : primary
Number of vPCs configured         : 0
Peer Gateway                      : Enabled
Dual-active excluded VLANs        : -

The Configuration inconsistency reason line is the gift here. NX-OS does not just tell you the check failed, it names the parameter class and the specific parameter. Confirming it against the comparison table takes one command:

LEAF1# show vpc consistency-parameters global | include "STP Mode"
STP Mode                    1     Rapid-PVST             MST

Local value Rapid-PVST, peer value MST, parameter type 1. That is the entire diagnosis in one line, and it is the command to run whenever consistency status goes to failed. Reverting LEAF2 with spanning-tree mode rapid-pvst cleared it.

What did not happen, which is the interesting part

Read the first three lines of that failure output again. Peer status: peer adjacency formed ok. vPC keep-alive status: peer is alive. Neither moved.

A Type-1 mismatch does not tear down the vPC peering. The peer-link stayed up, CFS stayed up, the heartbeat kept arriving, and the two switches carried on talking about a configuration they disagreed on. What a Type-1 mismatch does is suspend vPCs, meaning the vPC member ports, leaving the peering machinery running so that the ports come straight back when you fix the config.

This changes what you check first. If Configuration consistency status is failed, nothing is wrong with your cabling, peer-link or management network, and looking at them is wasted time: somebody changed a config on one switch and not the other. If the peer status or keepalive status is what moved, the configuration is not your problem.

Note also that Per-vlan consistency status stayed success. That is a separate scope, checking per-VLAN Type-1 parameters, and a failure there suspends only the affected VLANs rather than every vPC.

Why zero vPCs made this a safe test

Breaking STP mode on a production vPC pair is not a casual experiment. With Graceful Consistency Check enabled (the default, and it was enabled here), a Type-1 mismatch suspends the vPC member ports on the secondary and lets the primary keep forwarding, so a dual-homed server loses one of its two paths. With graceful consistency check disabled, both peers suspend and the server loses everything.

In this lab neither outcome was reachable, because there were no vPC member ports. The consistency engine ran, detected the mismatch, reported it accurately and had nothing to act on. That makes it a clean demonstration of the control-plane behaviour and an honest non-demonstration of the data-plane consequence, and it is worth saying which is which.

This is also why so many Type-1 parameters are spanning tree settings: STP mode, Bridge Assurance, Loopguard, MST region name, revision and instance mapping. Two switches presenting themselves downstream as one bridge must agree completely about how bridging works, or the loop prevention vPC relies on stops being trustworthy.

The rest of the matrix, from first principles

Everything from here to the orphan port section is explained, not demonstrated. It is standard vPC behaviour worth knowing cold, but this lab did not produce it and there is no output below because none was captured.

Peer-link down, keepalive up

This is the designed failure case, and the reason the keepalive exists at all. The peer-link fails, CFS stops, and each switch has to work out whether its peer is dead or just unreachable over the peer-link. The keepalive answers that: if the heartbeat is still arriving over the management path, the peer is alive.

The response is asymmetric and deterministic. The secondary suspends its vPC member ports and shuts down its vPC VLAN SVIs. The primary keeps forwarding on everything. A dual-homed device sees its member to the secondary go down, hashes all its traffic onto the member to the primary, and stays online. Half the bandwidth, zero downtime.

The logic: without a peer-link the two switches cannot synchronise MAC addresses or pass traffic between themselves, so both forwarding for the same emulated switch would duplicate frames and break MAC learning downstream. One side has to stop, and the role decides which. This is where the role priority behaviour bites, because the operational role is what the last election produced, not what the config says: role priority does not apply until you preempt or re-initialise the peer-link.

The other thing that matters here is orphan ports, which is a large enough problem to have its own section below.

Keepalive down, peer-link up

This is the benign one, and it is the one that causes the most unnecessary panic. Losing the keepalive while the peer-link is healthy has no data-plane impact at all. vPCs stay up, member ports stay up, SVIs stay up, traffic is untouched. NX-OS logs the loss and show vpc stops saying peer is alive, and that is the extent of it.

The keepalive is not a forwarding mechanism. Its only job is to answer one question, "is my peer alive?", and that question only gets asked when the peer-link goes down. While CFS is running over a healthy peer-link, the switches have a better channel for knowing the peer is there.

What it does cost you is your safety net. A pair running with a dead keepalive is one peer-link failure away from a dual-active situation, because when the peer-link drops there will be nothing left to distinguish "peer is gone" from "peer-link is gone". Treat keepalive loss as an urgent ticket, not an outage. The lab's keepalive parameters show how quickly this gets detected:

vPC Keep-alive parameters
--Destination                   : 192.168.100.12
--Keepalive interval            : 1000 msec
--Keepalive timeout             : 5 seconds
--Keepalive hold timeout        : 3 seconds
--Keepalive vrf                 : management
--Keepalive udp port            : 3200
--Keepalive tos                 : 192

One heartbeat per second on UDP 3200 in the management VRF, peer declared dead after 5 seconds of silence. Those defaults are sensible and rarely worth touching.

Both down: dual-active, also known as split brain

Peer-link and keepalive both gone. Each switch can see nothing of the other and has no way to tell a dead peer from a severed pair, so both assume they are now the only one left. Both keep the primary role, both keep forwarding, and both keep answering downstream with the same emulated LACP system ID and the same vPC identity.

The result is every classic Layer 2 pathology at once. Frames get duplicated, MAC addresses flap as downstream switches see the same source from two directions, and any first hop redundancy on top of the pair now has two active forwarders with the same virtual MAC (the interaction between vPC and FHRP protocols is one reason dual-active is so unpleasant).

The mitigations are preventative, because once you are in dual-active there is no clever recovery, only picking a switch and isolating it. Keep the keepalive on a genuinely independent path, which is what the management VRF and mgmt0 buy you. Build the peer-link as a port-channel with members on different modules. Understand auto-recovery, which was Disabled by default in this lab, before you enable it: it lets a switch that has lost both links bring vPCs up on its own after a timer, which is exactly what you want after a reload and exactly what you do not want during a true split.

Orphan ports, the failure nobody plans for

This part is not hypothetical in this lab. Both servers here are orphan ports.

An orphan port is a port on a vPC peer that carries vPC VLANs but is not part of a vPC: a single-homed server, an appliance with one NIC, an uplink to a legacy switch. SRV1 on LEAF1 and SRV2 on LEAF2 are textbook examples, and more representative of real data centers than a lab where everything is neatly dual-homed.

Orphan ports matter because they depend on the exact resource that vPC member ports are designed not to need. When the peer-link goes down and the secondary suspends its vPC member ports, the orphan ports on that secondary stay up by default. Their links are fine, nothing failed locally, and they carry on while cut off from anything reachable only across the peer-link. An orphan whose gateway or database server sits on the other side is isolated, and it does not know it.

The knob is vpc orphan-port suspend, applied per interface. With it configured, the orphan port is brought down alongside the vPC member ports when the peer-link fails. For a server with two NICs in an active/standby bond into both peers, that is exactly right: the port goes down, the server fails over to its other NIC, and traffic keeps flowing. For a genuinely single-attached device it is exactly wrong, because you have converted "might be partially isolated" into "definitely offline".

So it is a per-port judgement, not a blanket setting. Ask one question per orphan: if this port goes down, does the thing behind it have another way in? If yes, suspend it. If no, leave it up and accept the isolation risk.

A triage order for 2am

1. Run show vpc and read the first three lines separately. Peer status, keepalive status, consistency status. They point at three different investigations and the combination narrows it immediately.
2. Consistency failed, peering healthy? Someone changed a config. Read Configuration inconsistency reason, then confirm with show vpc consistency-parameters global filtered on the named parameter, exactly as the lab did for STP Mode.
3. Peer status down, keepalive alive? The peer-link is the problem. The secondary has already suspended its vPC member ports and SVIs, so you are running on the primary. Check the peer-link port-channel and its members first.
4. Keepalive down, peer status ok? Nothing is broken for traffic. Fix the management path with appropriate urgency, but do not go looking for a data-plane outage that is not there.
5. Both down? Assume dual-active and stop the duplication first by isolating one switch, then investigate.
6. Check the orphan ports last, and specifically whether the ones on the affected peer are isolated. Nothing in show vpc will volunteer this.

Where this fits

vPC failures are their own category because vPC is two independent switches maintaining a shared fiction, and the interesting failures are those where the fiction survives while something underneath it does not. That is a different problem from the overlay failures in the VXLAN and EVPN troubleshooting walkthrough, where the control plane is BGP and the symptoms are silent drops rather than suspended ports, and it sits underneath everything in the spine-leaf data center fabric series. If you have not built the domain yet, start with how vPC is configured and how to read its health.

Key Takeaways

• Only the Type-1 mismatch was demonstrated here. The peer-link, keepalive and dual-active rows are explained from first principles, with no output shown because none was captured.
• A Type-1 mismatch suspends vPCs, it does not tear down the peering. In the capture the peer adjacency and the keepalive both stayed healthy while consistency went to failed.
• Configuration inconsistency reason names the parameter, and show vpc consistency-parameters global filtered on that name shows both values side by side.
• Peer-link down with keepalive up: the secondary suspends its vPC member ports and SVIs, the primary keeps forwarding. This is the design working, not a fault to fight.
• Keepalive down with peer-link up: no data-plane impact. It costs you the safety net, not the traffic.
• Both down means dual-active, which is prevented by an independent keepalive path and a resilient peer-link, not recovered from cleverly.
• Orphan ports stay up by default when the peer-link fails and can be silently isolated. vpc orphan-port suspend helps only if the device behind the port has another way in.