Spanning tree and EtherChannel are two things that are easy to get mostly right by hand and then forget about. The root bridge ends up wherever the lowest MAC address happens to be, an access port without portfast waits for 30 seconds before it forwards, and a bundle with one member missing still says it is up. This post puts all three in the campus data model from part 10 and adds a check that fails when any of them is wrong.
It was built in my home lab on three IOS XE L2 lab switches: hq-dist1 and two access switches, hq-acc1 and hq-acc2. Each access switch has two uplinks to hq-dist1, and there is one extra link between the two access switches, so there is a loop for spanning tree to deal with. The previous post was part 15, the CDP and LLDP neighbor map.
Versions: ansible-core 2.21, cisco.ios 11.6.
Where the root was
Before any changes, the check playbook further down reported this for VLAN 10:
TASK [Show it] *****************************************************************
ok: [hq-dist1] => {
"msg": [
"root for VLAN 10: False",
"Et0/1 Root FWD 100 128.2 P2p ",
"Et0/2 Desg FWD 100 128.3 P2p"
]
}
ok: [hq-acc1] => {
"msg": [
"root for VLAN 10: True",
"Et0/1 Desg FWD 100 128.2 P2p ",
"Et0/2 Desg FWD 100 128.3 P2p"
]
}
ok: [hq-acc2] => {
"msg": [
"root for VLAN 10: False",
"Et0/1 Root FWD 100 128.2 P2p ",
"Et0/2 Desg FWD 100 128.3 P2p ",
"Et0/3 Desg FWD 100 128.4 P2p"
]
}
hq-acc1 is the root bridge. Nobody chose that. All three switches have the default priority, and hq-acc1 has the lowest MAC address, so it won. Traffic from hq-acc2 to anything behind hq-acc1 is fine here, but on a real campus this is how user traffic ends up crossing an access switch to reach the core. The second uplinks are not doing anything either, and the check fails because the bundles it expects do not exist:
[ERROR]: Task failed: Action failed: Po1 is not up with 2 members
failed: [hq-dist1] (item=Po1) => {"ansible_loop_var": "item", "assertion": "po_lines | select('search', 'Po' ~ item.id ~ '\\\\(SU\\\\)') | list | leng ...
failed: [hq-acc1] (item=Po1) => {"ansible_loop_var": "item", "assertion": "po_lines | select('search', 'Po' ~ item.id ~ '\\\\(SU\\\\)') | list | lengt ...
[ERROR]: Task failed: Action failed: Po2 is not up with 2 members
failed: [hq-acc2] (item=Po2) => {"ansible_loop_var": "item", "assertion": "po_lines | select('search', 'Po' ~ item.id ~ '\\\\(SU\\\\)') | list | lengt ...
failed: [hq-dist1] (item=Po2) => {"ansible_loop_var": "item", "assertion": "po_lines | select('search', 'Po' ~ item.id ~ '\\\\(SU\\\\)') | list | leng ...
The data model
This file sits next to vars/campus.yml from part 10. The access ports in that file (mode access) are the ones that get portfast.
---
# Part 16 additions to the campus data model
stp_mode: rapid-pvst
stp_root:
primary: hq-dist1 # priority 4096 on all campus VLANs
stp_vlans: [10, 20, 30, 90]
port_channels:
hq-dist1:
- {id: 1, members: [Ethernet0/1, Ethernet1/0], desc: "PO: hq-acc1 Po1"}
- {id: 2, members: [Ethernet0/2, Ethernet1/1], desc: "PO: hq-acc2 Po2"}
hq-acc1:
- {id: 1, members: [Ethernet0/1, Ethernet1/0], desc: "UPLINK PO: hq-dist1 Po1"}
hq-acc2:
- {id: 2, members: [Ethernet0/1, Ethernet1/0], desc: "UPLINK PO: hq-dist1 Po2"}
# access-to-access trunk: a deliberate loop, STP should block one end
extra_trunks:
hq-acc1: [{name: Ethernet1/1, desc: "TRUNK: hq-acc2 Et1/1"}]
hq-acc2: [{name: Ethernet1/1, desc: "TRUNK: hq-acc1 Et1/1"}]
The playbook
---
- name: STP root, edge ports and LACP uplinks on the HQ campus
hosts: campus
gather_facts: false
vars_files:
- ../vars/campus.yml
- ../vars/campus-l2.yml
vars:
my_pos: "{{ port_channels[inventory_hostname] | default([]) }}"
my_trunks: "{{ extra_trunks[inventory_hostname] | default([]) }}"
edge_ports: "{{ ports[inventory_hostname] | selectattr('mode', 'eq', 'access') | map(attribute='name') | list }}"
trunk_def:
encapsulation: dot1q
allowed_vlans: "{{ trunk_vlans | map('string') | list }}"
native_vlan: "{{ native_vlan }}"
tasks:
- name: Data model into module input
ansible.builtin.set_fact:
lag_cfg: >-
{%- set out = [] -%}
{%- for po in my_pos -%}
{%- set _ = out.append({'name': 'Port-channel' ~ po.id, 'members': []}) -%}
{%- for m in po.members -%}{%- set _ = out[-1].members.append({'member': m, 'mode': 'active'}) -%}{%- endfor -%}
{%- endfor -%}
{{ out }}
trunk_names: "{{ (my_pos | map(attribute='members') | flatten) + (my_pos | map(attribute='id') | map('regex_replace', '^', 'Port-channel') | list) + (my_trunks | map(attribute='name') | list) }}"
desc_cfg: >-
{%- set out = [] -%}
{%- for po in my_pos -%}{%- set _ = out.append({'name': 'Port-channel' ~ po.id, 'description': po.desc, 'enabled': true}) -%}{%- endfor -%}
{%- for t in my_trunks -%}{%- set _ = out.append({'name': t.name, 'description': t.desc, 'enabled': true}) -%}{%- endfor -%}
{{ out }}
- name: Members, bundles and the extra link are trunks with the same settings
cisco.ios.ios_l2_interfaces:
config: >-
{%- set out = [] -%}
{%- for n in trunk_names -%}{%- set _ = out.append({'name': n, 'mode': 'trunk', 'trunk': trunk_def}) -%}{%- endfor -%}
{{ out }}
state: replaced
register: r_l2
- name: Bundle the members with LACP active
cisco.ios.ios_lag_interfaces:
config: "{{ lag_cfg }}"
state: merged
register: r_lag
when: lag_cfg | length > 0
- name: Descriptions, ports enabled
cisco.ios.ios_interfaces:
config: "{{ desc_cfg }}"
state: merged
register: r_desc
- name: STP mode everywhere, root priority on the distribution switch
cisco.ios.ios_config:
lines: >-
{{ ['spanning-tree mode ' ~ stp_mode]
+ (['spanning-tree vlan ' ~ stp_vlans | join(',') ~ ' priority 4096'] if inventory_hostname == stp_root.primary else []) }}
register: r_stp
- name: Portfast and BPDU guard on every access port
cisco.ios.ios_config:
parents: "interface {{ item }}"
lines:
- spanning-tree portfast
- spanning-tree bpduguard enable
loop: "{{ edge_ports }}"
register: r_edge
- name: Commands sent
ansible.builtin.debug:
msg: >-
{{ (r_l2.commands | default([])) + (r_lag.commands | default([])) + (r_desc.commands | default([]))
+ (r_stp.updates | default([])) + (r_edge.results | map(attribute='updates', default=[]) | flatten) }}
Some notes on it:
- The member ports get their trunk settings before they join the bundle. My first version did it the other way round, and the dry run showed
channel-groupgoing out beforeswitchport mode trunkon the new ports. A member that does not match the bundle gets suspended, so the order matters. - There is no resource module for spanning tree in cisco.ios, so the root priority and the edge port settings go through
ios_config. The bundles and trunks use the resource modules. - Priority 4096 is set explicitly rather than with
spanning-tree vlan ... root primary. Theroot primarymacro reads the current root's priority once and picks a lower one. It does not keep the switch as root if someone else is changed later, and the running config shows a number anyway. bpduguardgoes withportfast. Portfast makes a port forward straight away, which is what you want for a desk. If somebody plugs a switch into that desk port, BPDU guard shuts the port instead of letting it join the tree.
The run sent this to hq-dist1:
TASK [Commands sent] ***********************************************************
ok: [hq-dist1] => {
"msg": [
"interface Ethernet1/0",
"switchport trunk encapsulation dot1q",
"switchport mode trunk",
"switchport trunk native vlan 999",
"switchport trunk allowed vlan 10,20,30,90",
"interface Ethernet1/1",
"switchport trunk encapsulation dot1q",
"switchport mode trunk",
"switchport trunk native vlan 999",
"switchport trunk allowed vlan 10,20,30,90",
"interface Port-channel1",
"switchport trunk encapsulation dot1q",
"switchport mode trunk",
"switchport trunk native vlan 999",
"switchport trunk allowed vlan 10,20,30,90",
"interface Port-channel2",
"switchport trunk encapsulation dot1q",
"switchport mode trunk",
"switchport trunk native vlan 999",
"switchport trunk allowed vlan 10,20,30,90",
"interface Ethernet0/1",
"channel-group 1 mode active",
"interface Ethernet1/0",
"channel-group 1 mode active",
"interface Ethernet1/1",
"channel-group 2 mode active",
"interface Ethernet0/2",
"channel-group 2 mode active",
"interface Port-channel1",
"description PO: hq-acc1 Po1",
"interface Port-channel2",
"description PO: hq-acc2 Po2",
"spanning-tree vlan 10,20,30,90 priority 4096"
]
}
ok: [hq-acc2] => {
"msg": [
"interface Ethernet1/0",
"switchport trunk encapsulation dot1q",
"switchport mode trunk",
"switchport trunk native vlan 999",
"switchport trunk allowed vlan 10,20,30,90",
"interface Port-channel2",
"switchport trunk encapsulation dot1q",
"switchport mode trunk",
"switchport trunk native vlan 999",
"switchport trunk allowed vlan 10,20,30,90",
"interface Ethernet1/1",
"switchport trunk encapsulation dot1q",
"switchport mode trunk",
"switchport trunk native vlan 999",
"switchport trunk allowed vlan 10,20,30,90",
"interface Ethernet0/1",
"channel-group 2 mode active",
"interface Ethernet1/0",
"channel-group 2 mode active",
"interface Ethernet1/1",
"description TRUNK: hq-acc1 Et1/1",
"interface Port-channel2",
"description UPLINK PO: hq-dist1 Po2",
"interface Ethernet0/2",
"spanning-tree portfast",
"spanning-tree bpduguard enable",
"interface Ethernet0/3",
"spanning-tree portfast",
"spanning-tree bpduguard enable"
]
}
and this to hq-acc1:
ok: [hq-acc1]
The check
Reading show etherchannel summary and show spanning-tree vlan 10 and keeping only the lines that matter is enough here. No parser needed.
---
- name: Check bundles and spanning tree on the HQ campus
hosts: campus
gather_facts: false
vars_files:
- ../vars/campus-l2.yml
vars:
my_pos: "{{ port_channels[inventory_hostname] | default([]) }}"
tasks:
- name: Read the state
cisco.ios.ios_command:
commands:
- show etherchannel summary
- show spanning-tree vlan 10
register: r
- name: Keep the lines that matter
ansible.builtin.set_fact:
po_lines: "{{ r.stdout[0].splitlines() | select('search', '^\\d+\\s+Po') | list }}"
is_root: "{{ 'This bridge is the root' in r.stdout[1] }}"
stp_ports: "{{ r.stdout[1].splitlines() | select('search', '^(Et|Po)\\S+\\s+(Root|Desg|Altn|Back)') | list }}"
- name: Show it
ansible.builtin.debug:
msg: "{{ ['root for VLAN 10: ' ~ is_root] + po_lines + stp_ports }}"
- name: Every bundle is up (SU) and every member is bundled (P)
ansible.builtin.assert:
that:
- po_lines | select('search', 'Po' ~ item.id ~ '\\(SU\\)') | list | length == 1
- po_lines | select('search', 'Po' ~ item.id ~ '\\(') | first | regex_findall('\\(P\\)') | length == item.members | length
fail_msg: "Po{{ item.id }} is not up with {{ item.members | length }} members"
quiet: true
loop: "{{ my_pos }}"
loop_control:
label: "Po{{ item.id }}"
- name: The distribution switch is the root, and nobody else is
ansible.builtin.assert:
that: is_root == (inventory_hostname == stp_root.primary)
fail_msg: "root bridge in the wrong place (this bridge is root: {{ is_root }})"
quiet: true
After the change and 45 seconds for LACP and spanning tree to settle:
TASK [Show it] *****************************************************************
ok: [hq-dist1] => {
"msg": [
"root for VLAN 10: True",
"1 Po1(SU) LACP Et0/1(P) Et1/0(P) ",
"2 Po2(SU) LACP Et0/2(P) Et1/1(P)",
"Po1 Desg FWD 56 128.65 P2p ",
"Po2 Desg FWD 56 128.66 P2p"
]
}
ok: [hq-acc1] => {
"msg": [
"root for VLAN 10: False",
"1 Po1(SU) LACP Et0/1(P) Et1/0(P)",
"Et0/2 Desg FWD 100 128.3 P2p Edge ",
"Et1/1 Desg FWD 100 128.6 P2p ",
"Po1 Root FWD 56 128.65 P2p"
]
}
ok: [hq-acc2] => {
"msg": [
"root for VLAN 10: False",
"2 Po2(SU) LACP Et0/1(P) Et1/0(P)",
"Et0/2 Desg FWD 100 128.3 P2p Edge ",
"Et0/3 Desg FWD 100 128.4 P2p Edge ",
"Et1/1 Altn BLK 100 128.6 P2p ",
"Po2 Root FWD 56 128.65 P2p"
]
}
TASK [Every bundle is up (SU) and every member is bundled (P)] *****************
ok: [hq-dist1] => (item=Po1)
ok: [hq-acc1] => (item=Po1)
ok: [hq-acc2] => (item=Po2)
ok: [hq-dist1] => (item=Po2)
TASK [The distribution switch is the root, and nobody else is] *****************
ok: [hq-dist1]
ok: [hq-acc1]
ok: [hq-acc2]
PLAY RECAP *********************************************************************
hq-acc1 : ok=5 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
hq-acc2 : ok=5 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
hq-dist1 : ok=5 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Everything you would want to see is in there:
- hq-dist1 is the root, and both bundles are designated ports on it.
- Each bundle is
SU(layer 2, in use) with both membersP(bundled). - The link between the access switches is blocked at hq-acc2 (
Altn BLK). That is the loop, handled. - The desk ports show
Edge, which is portfast working. - Each bundle has a spanning tree cost of 56 against 100 for a single port, because the cost follows the combined bandwidth.
Running it again
A second run should change nothing. It did change nothing, but I could not get one clean run across all three switches: on each of three runs one switch timed out on SSH and the other two reported changed=0. Across the runs, every switch reported changed=0 at least once. The timeouts are a lab problem, not a playbook problem, but it is worth saying because an unattended job against real devices will meet the same thing, and a run that failed on one host should be retried before anyone trusts it.
A member goes missing
To check that the check is worth having, I shut Ethernet1/0 on hq-acc1, one of the two members of Po1, and ran it again 20 seconds later.
"1 Po1(SU) LACP Et0/1(P) Et1/0(P) ",
"1 Po1(SU) LACP Et0/1(P) Et1/0(D)",
The first line is hq-dist1, the second is hq-acc1. hq-acc1 shows the member down (D) and the check fails:
failed: [hq-acc1] (item=Po1) => {"ansible_loop_var": "item", "assertion": "po_lines | select('search', 'Po' ~ item.id ~ '\\\\(') | first | regex_finda ...
hq-dist1 still showed both members bundled. In this lab, shutting a port on one side does not take the link down on the other, so hq-dist1 only finds out when LACP stops hearing from the neighbor, and with the default slow timer that takes up to 90 seconds. Po1 stayed up the whole time, as it should, at half the bandwidth. That is why the check runs on both ends of every bundle: one side can be wrong for a minute and a half.
After no shutdown and 45 seconds, the check passed on all three switches again.
Where to take it
- Add
lacp rate faston the members if you want a dead member noticed in seconds instead of a minute and a half. - Put a second distribution switch in the model with priority 8192 as the backup root.
- Run the check on a schedule. A root bridge in the wrong place does not cause an outage, so nothing else will tell you.
Part 17 is a port security baseline for every access port. The rest of the series is on the series page.