Ansible rolling changes on IOS XE with serial and automatic rollback

This is part 14 of the Ansible for network engineers series. Part 13 built OSPF and BGP from a vars file and checked the routing before and after each run. That run touched every router at once. Here the same checks gate a rolling change: one router at a time, a health check before and after each one, an automatic rollback when the check fails, and nothing more touched after the first failure.

The targets are the three HQ IOS XE routers from part 13, hq-core1, hq-core2 and hq-edge1, running OSPF area 0 on point-to-point links.

Built in my home lab with ansible-core 2.21.5, cisco.ios 11.6.0, ansible.netcommon 8.7.1, ansible.utils 6.1.1 and ntc_templates 9.3.0, against Cisco IOS XE 17.18.2 (lab images).

Lab diagram: Ansible rolling an OSPF change across three IOS XE routers one at a time with serial: 1. In run A (hello_interval 5) hq-core1 fails its post-check with "ospf 0/2, lost" both loopback routes, the rescue restores the gathered config, 2 OSPF neighbors come back FULL and the rollout stops, so hq-core2 and hq-edge1 are never touched. In run B (cost 20) all three routers pass with 0 routes lost. — Run A stops at hq-core1 and rolls back. Run B rolls the cost change across all three routers.

The pieces

Four Ansible features do the work:

serial: 1 runs the whole play on one host before it starts the next.
max_fail_percentage: 0 ends the play when any host in a batch fails, so the next router never gets the change.
block and rescue run the rollback only when something inside the block fails.
The resource module's gathered state saves the current OSPF interface config as data, and replaced puts that data back.

The pre-check and post-check are the route-check.yml task file from part 13. It parses show ip ospf neighbor, show ip bgp summary and show ip route, and reduces them to neighbor counts and a list of OSPF and BGP routes.

The playbook

---
- name: Rolling OSPF interface change, one router at a time
  hosts: hq
  serial: 1
  max_fail_percentage: 0
  gather_facts: false
  vars_files:
    - ../vars/routing.yml
  vars:
    # the change, passed in with -e, for example '{"ospf_change": {"hello_interval": 5}}'
    ospf_change: {}
    core_links: "{{ ospf_links[inventory_hostname] | rejectattr('name', 'match', 'Loopback') | map(attribute='name') | list }}"
    settle: 45
  tasks:
    - name: Pre-check
      ansible.builtin.include_tasks: tasks/route-check.yml

    - name: Refuse to start on a router that is already unhealthy
      ansible.builtin.assert:
        that:
          - health.ospf_full | int == expect[inventory_hostname].ospf
          - health.bgp_up | int == expect[inventory_hostname].bgp
        fail_msg: "pre-check failed: ospf {{ health.ospf_full }} bgp {{ health.bgp_up }}"
        quiet: true

    - name: Keep the pre-change picture
      ansible.builtin.set_fact:
        pre: "{{ health }}"

    - name: Save the current OSPF interface config for rollback
      cisco.ios.ios_ospf_interfaces:
        state: gathered
      register: before

    - name: Change, verify, roll back on failure
      block:
        - name: Apply the change to the core links
          cisco.ios.ios_ospf_interfaces:
            config: "{{ before.gathered | selectattr('name', 'in', core_links) | map('combine', {'address_family': [item_af]}) | list }}"
            state: merged
          vars:
            item_af: "{{ {'afi': 'ipv4'} | combine(ospf_change) }}"

        - name: Let it settle for longer than a dead interval
          ansible.builtin.pause:
            seconds: "{{ settle }}"

        - name: Post-check
          ansible.builtin.include_tasks: tasks/route-check.yml

        - name: Assert nothing was lost
          ansible.builtin.assert:
            that:
              - health.ospf_full | int == expect[inventory_hostname].ospf
              - health.bgp_up | int == expect[inventory_hostname].bgp
              - pre.routes | difference(health.routes) | length == 0
            fail_msg: "ospf {{ health.ospf_full }}/{{ expect[inventory_hostname].ospf }}, lost {{ pre.routes | difference(health.routes) }}"
            success_msg: "ospf {{ health.ospf_full }}, {{ health.routes | length }} routes, 0 lost"

      rescue:
        - name: Roll back to the saved OSPF interface config
          cisco.ios.ios_ospf_interfaces:
            config: "{{ before.gathered | selectattr('name', 'in', core_links) | list }}"
            state: replaced

        - name: Wait for the neighbors to come back
          cisco.ios.ios_command:
            commands: show ip ospf neighbor
          register: w_ospf
          until: w_ospf.stdout[0] | regex_findall('FULL/') | length >= expect[inventory_hostname].ospf
          retries: 12
          delay: 10

        - name: Stop the rollout
          ansible.builtin.fail:
            msg: "{{ inventory_hostname }}: change failed its post-check and was rolled back, {{ w_ospf.stdout[0] | regex_findall('FULL/') | length }} OSPF neighbors FULL again"

The change comes in as an extra var, so one playbook handles any OSPF interface attribute the module supports. It's merged into the gathered config for the core links only. Loopback0 and the management interface are never in the config list, so neither the change nor the rollback can touch them.

The rollback uses replaced with the saved entries for those same links. replaced makes each listed interface match the saved data exactly, so it removes whatever the change added. merged can only add, so it can't undo anything.

A change that has to fail

The first change sets the OSPF hello interval to 5 seconds:

ansible-playbook playbooks/140-rolling-ospf.yml -e '{"ospf_change": {"hello_interval": 5}}'

Hello and dead intervals have to match on both ends of a link. With serial: 1, hq-core1 changes while its neighbors still run the default 10 seconds, so the adjacencies drop. That's what the safeguards are for:

TASK [Apply the change to the core links] **************************************
changed: [hq-core1]

TASK [Let it settle for longer than a dead interval] ***************************
Pausing for 45 seconds
ok: [hq-core1]

...

TASK [Assert nothing was lost] *************************************************
[ERROR]: Task failed: Action failed: ospf 0/2, lost ['10.66.255.12/32', '10.66.255.13/32']
fatal: [hq-core1]: FAILED! => {
    "assertion": "health.ospf_full | int == expect[inventory_hostname].ospf",
    "changed": false,
    "evaluated_to": false,
    "msg": "ospf 0/2, lost ['10.66.255.12/32', '10.66.255.13/32']"
}

TASK [Roll back to the saved OSPF interface config] ****************************
changed: [hq-core1]

TASK [Wait for the neighbors to come back] *************************************
ok: [hq-core1]

TASK [Stop the rollout] ********************************************************
[ERROR]: Task failed: Action failed: hq-core1: change failed its post-check and was rolled back, 2 OSPF neighbors FULL again
fatal: [hq-core1]: FAILED! => {"changed": false, "msg": "hq-core1: change failed its post-check and was rolled back, 2 OSPF neighbors FULL again"}

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
hq-core1                   : ok=15   changed=2    unreachable=0    failed=1    skipped=2    rescued=1    ignored=0

The assert caught it with zero of two neighbors FULL and both loopback routes gone. The rescue put the saved config back, both neighbors came back FULL on the first poll, and fail ended the run. hq-core2 and hq-edge1 were never touched. Afterwards no HQ router had an ip ospf hello-interval line.

The lesson here isn't about Ansible. Serial rollouts break any setting that must match between neighbors: OSPF timers, authentication keys, MTU, LACP settings. Change those per link, with both ends in the same batch, or plan a short outage window. serial protects you from changes that are safe one device at a time, and only from those.

A change that's safe to roll

Interface cost only matters locally, so it can roll one router at a time:

ansible-playbook playbooks/140-rolling-ospf.yml -e '{"ospf_change": {"cost": {"interface_cost": 20}}}'

PLAY [Rolling OSPF interface change, one router at a time] *********************
changed: [hq-core1]
    "msg": "ospf 2, 2 routes, 0 lost"
PLAY [Rolling OSPF interface change, one router at a time] *********************
changed: [hq-core2]
    "msg": "ospf 1, 3 routes, 0 lost"
PLAY [Rolling OSPF interface change, one router at a time] *********************
changed: [hq-edge1]
    "msg": "ospf 1, 5 routes, 0 lost"

PLAY RECAP *********************************************************************
hq-core1                   : ok=14   changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
hq-core2                   : ok=14   changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
hq-edge1                   : ok=16   changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Each router got its own play run, which is what serial looks like in the output, and each passed its post-check before the next one started.

The settle time matters

The pause before the post-check is easy to cut when you want a faster run. To see what happens without it, I ran the failing hello change again on hq-core1 only, with settle set to 1 second:

TASK [Apply the change to the core links] **************************************
changed: [hq-core1]

...

TASK [Assert nothing was lost] *************************************************
ok: [hq-core1] => {
    "changed": false,
    "msg": "ospf 2, 2 routes, 0 lost"
}

PLAY RECAP *********************************************************************
hq-core1                   : ok=14   changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

It passed. hq-core1 still had two FULL neighbors and every route, because an OSPF adjacency stays up until the dead timer runs out, and that's 40 seconds on the neighbor with default timers. 45 seconds later, the same router:

hq-core1# show ip ospf neighbor

hq-core1# show run | i ip ospf (hello|cost|dead)
 ip ospf hello-interval 5
 ip ospf cost 20
 ip ospf hello-interval 5
 ip ospf cost 20

No neighbors at all, and the playbook had already reported success. A post-check proves something only after the protocol has had time to notice the change. For OSPF, that means waiting longer than the dead interval. For BGP, it's the hold time, 180 seconds by default. Keep that in mind before shortening settle.

Lab diagram: A 0 to 60 second timeline for hq-core1 after its OSPF hello interval changes to 5 seconds. At 1 second the post-check passes with 2 neighbors FULL and 0 routes lost, because the neighbors keep the adjacency until their 40 second dead timer expires. By 45 seconds hq-core1 has 0 neighbors, so a snapshot taken anywhere in the first 40 seconds looks healthy. — A 1 second settle passes the post-check. The neighbors drop at 40 seconds, after the run has already passed.

After the tests, I removed the hello and cost lines from all three routers and checked that the neighbor counts were back to two, one and one.

Things I'd add before using this in production

Run the same health check on the neighbors as well as on the changed router. hq-core1 noticed its own failure here. A change that only breaks things one hop away needs the check to run on those devices too.
Save a full show running-config as well as the module data, in case the rollback itself fails.
Use serial: [1, "50%"] to run a canary first and then larger batches.

Part 15 Next in the series is [part 15, a CDP and LLDP neighbor map built with Ansible](https://www.pinglabz.com/ansible-cdp-lldp-neighbor-map-ios/). The rest of the series is on the series page.

Rolling Network Changes with Ansible: serial, Post-Checks and Automatic Rollback

The pieces

The playbook

A change that has to fail

A change that's safe to roll

The settle time matters

Things I'd add before using this in production

Read next

Run Ansible Network Playbooks from Semaphore UI: Templates, Vault Keys and Schedules

A Git Pipeline for Network Configs: Lint, Check-Mode Diff, Then Deploy

Build a Network Inventory from Ansible Facts and Parsed CLI Output