> ## Content Index > Fetch the complete content index at: https://www.pinglabz.com/llms.txt > Use this file to discover other available public pages before exploring further. # Rolling Network Changes with Ansible: serial, Post-Checks and Automatic Rollback - URL: https://www.pinglabz.com/ansible-rolling-change-serial-rollback-ios/ - Published: 2026-11-01T00:53:21.000Z - Updated: 2026-11-01T04:41:39.000Z - Description: One router at a time with serial: 1, a parsed health check after each, and a block/rescue rollback that stops the rollout. A hello timer change fails and rolls back on the first router, a cost change rolls clean, and a 1 second settle time produces a false pass. - Author: Jaime - Tags: Network Automation, Cisco IOS XE, Cisco This is part 14 of the [Ansible for network engineers series](https://www.pinglabz.com/ansible-network-automation/). [Part 13](https://www.pinglabz.com/ansible-ospf-bgp-ios/) built OSPF and BGP from a vars file and checked the routing before and after each run. That run touched every router at once. Here the same checks gate a rolling change: one router at a time, a health check before and after each one, an automatic rollback when the check fails, and nothing more touched after the first failure. The targets are the three HQ IOS XE routers from part 13, hq-core1, hq-core2 and hq-edge1, running OSPF area 0 on point-to-point links. Built in my home lab with ansible-core 2.21.5, cisco.ios 11.6.0, ansible.netcommon 8.7.1, ansible.utils 6.1.1 and ntc\_templates 9.3.0, against Cisco IOS XE 17.18.2 (lab images). [![Lab diagram: Ansible rolling an OSPF change across three IOS XE routers one at a time with serial: 1. In run A (hello_interval 5) hq-core1 fails its post-check with "ospf 0/2, lost" both loopback routes, the rescue restores the gathered config, 2 OSPF neighbors come back FULL and the rollout stops, so hq-core2 and hq-edge1 are never touched. In run B (cost 20) all three routers pass with 0 routes lost.](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/diagram-topology.png)](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/diagram-topology.png?ref=pinglabz.com) Run A stops at hq-core1 and rolls back. Run B rolls the cost change across all three routers. ## The pieces Four Ansible features do the work: - `serial: 1` runs the whole play on one host before it starts the next. - `max_fail_percentage: 0` ends the play when any host in a batch fails, so the next router never gets the change. - `block` and `rescue` run the rollback only when something inside the block fails. - The resource module's `gathered` state saves the current OSPF interface config as data, and `replaced` puts that data back. The pre-check and post-check are the `route-check.yml` task file from part 13\. It parses `show ip ospf neighbor`, `show ip bgp summary` and `show ip route`, and reduces them to neighbor counts and a list of OSPF and BGP routes. ## The playbook ```yaml --- - name: Rolling OSPF interface change, one router at a time hosts: hq serial: 1 max_fail_percentage: 0 gather_facts: false vars_files: - ../vars/routing.yml vars: # the change, passed in with -e, for example '{"ospf_change": {"hello_interval": 5}}' ospf_change: {} core_links: "{{ ospf_links[inventory_hostname] | rejectattr('name', 'match', 'Loopback') | map(attribute='name') | list }}" settle: 45 tasks: - name: Pre-check ansible.builtin.include_tasks: tasks/route-check.yml - name: Refuse to start on a router that is already unhealthy ansible.builtin.assert: that: - health.ospf_full | int == expect[inventory_hostname].ospf - health.bgp_up | int == expect[inventory_hostname].bgp fail_msg: "pre-check failed: ospf {{ health.ospf_full }} bgp {{ health.bgp_up }}" quiet: true - name: Keep the pre-change picture ansible.builtin.set_fact: pre: "{{ health }}" - name: Save the current OSPF interface config for rollback cisco.ios.ios_ospf_interfaces: state: gathered register: before - name: Change, verify, roll back on failure block: - name: Apply the change to the core links cisco.ios.ios_ospf_interfaces: config: "{{ before.gathered | selectattr('name', 'in', core_links) | map('combine', {'address_family': [item_af]}) | list }}" state: merged vars: item_af: "{{ {'afi': 'ipv4'} | combine(ospf_change) }}" - name: Let it settle for longer than a dead interval ansible.builtin.pause: seconds: "{{ settle }}" - name: Post-check ansible.builtin.include_tasks: tasks/route-check.yml - name: Assert nothing was lost ansible.builtin.assert: that: - health.ospf_full | int == expect[inventory_hostname].ospf - health.bgp_up | int == expect[inventory_hostname].bgp - pre.routes | difference(health.routes) | length == 0 fail_msg: "ospf {{ health.ospf_full }}/{{ expect[inventory_hostname].ospf }}, lost {{ pre.routes | difference(health.routes) }}" success_msg: "ospf {{ health.ospf_full }}, {{ health.routes | length }} routes, 0 lost" rescue: - name: Roll back to the saved OSPF interface config cisco.ios.ios_ospf_interfaces: config: "{{ before.gathered | selectattr('name', 'in', core_links) | list }}" state: replaced - name: Wait for the neighbors to come back cisco.ios.ios_command: commands: show ip ospf neighbor register: w_ospf until: w_ospf.stdout[0] | regex_findall('FULL/') | length >= expect[inventory_hostname].ospf retries: 12 delay: 10 - name: Stop the rollout ansible.builtin.fail: msg: "{{ inventory_hostname }}: change failed its post-check and was rolled back, {{ w_ospf.stdout[0] | regex_findall('FULL/') | length }} OSPF neighbors FULL again" ``` The change comes in as an extra var, so one playbook handles any OSPF interface attribute the module supports. It's merged into the gathered config for the core links only. Loopback0 and the management interface are never in the config list, so neither the change nor the rollback can touch them. The rollback uses `replaced` with the saved entries for those same links. `replaced` makes each listed interface match the saved data exactly, so it removes whatever the change added. `merged` can only add, so it can't undo anything. ## A change that has to fail The first change sets the OSPF hello interval to 5 seconds: ```bash ansible-playbook playbooks/140-rolling-ospf.yml -e '{"ospf_change": {"hello_interval": 5}}' ``` Hello and dead intervals have to match on both ends of a link. With `serial: 1`, hq-core1 changes while its neighbors still run the default 10 seconds, so the adjacencies drop. That's what the safeguards are for: ```text TASK [Apply the change to the core links] ************************************** changed: [hq-core1] TASK [Let it settle for longer than a dead interval] *************************** Pausing for 45 seconds ok: [hq-core1] ... TASK [Assert nothing was lost] ************************************************* [ERROR]: Task failed: Action failed: ospf 0/2, lost ['10.66.255.12/32', '10.66.255.13/32'] fatal: [hq-core1]: FAILED! => { "assertion": "health.ospf_full | int == expect[inventory_hostname].ospf", "changed": false, "evaluated_to": false, "msg": "ospf 0/2, lost ['10.66.255.12/32', '10.66.255.13/32']" } TASK [Roll back to the saved OSPF interface config] **************************** changed: [hq-core1] TASK [Wait for the neighbors to come back] ************************************* ok: [hq-core1] TASK [Stop the rollout] ******************************************************** [ERROR]: Task failed: Action failed: hq-core1: change failed its post-check and was rolled back, 2 OSPF neighbors FULL again fatal: [hq-core1]: FAILED! => {"changed": false, "msg": "hq-core1: change failed its post-check and was rolled back, 2 OSPF neighbors FULL again"} NO MORE HOSTS LEFT ************************************************************* NO MORE HOSTS LEFT ************************************************************* PLAY RECAP ********************************************************************* hq-core1 : ok=15 changed=2 unreachable=0 failed=1 skipped=2 rescued=1 ignored=0 ``` The assert caught it with zero of two neighbors FULL and both loopback routes gone. The rescue put the saved config back, both neighbors came back FULL on the first poll, and `fail` ended the run. hq-core2 and hq-edge1 were never touched. Afterwards no HQ router had an `ip ospf hello-interval` line. The lesson here isn't about Ansible. Serial rollouts break any setting that must match between neighbors: OSPF timers, authentication keys, MTU, LACP settings. Change those per link, with both ends in the same batch, or plan a short outage window. `serial` protects you from changes that are safe one device at a time, and only from those. ## A change that's safe to roll Interface cost only matters locally, so it can roll one router at a time: ```bash ansible-playbook playbooks/140-rolling-ospf.yml -e '{"ospf_change": {"cost": {"interface_cost": 20}}}' ``` ```text PLAY [Rolling OSPF interface change, one router at a time] ********************* changed: [hq-core1] "msg": "ospf 2, 2 routes, 0 lost" PLAY [Rolling OSPF interface change, one router at a time] ********************* changed: [hq-core2] "msg": "ospf 1, 3 routes, 0 lost" PLAY [Rolling OSPF interface change, one router at a time] ********************* changed: [hq-edge1] "msg": "ospf 1, 5 routes, 0 lost" PLAY RECAP ********************************************************************* hq-core1 : ok=14 changed=1 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 hq-core2 : ok=14 changed=1 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 hq-edge1 : ok=16 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ``` Each router got its own play run, which is what `serial` looks like in the output, and each passed its post-check before the next one started. ## The settle time matters The `pause` before the post-check is easy to cut when you want a faster run. To see what happens without it, I ran the failing hello change again on hq-core1 only, with `settle` set to 1 second: ```text TASK [Apply the change to the core links] ************************************** changed: [hq-core1] ... TASK [Assert nothing was lost] ************************************************* ok: [hq-core1] => { "changed": false, "msg": "ospf 2, 2 routes, 0 lost" } PLAY RECAP ********************************************************************* hq-core1 : ok=14 changed=1 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 ``` It passed. hq-core1 still had two FULL neighbors and every route, because an OSPF adjacency stays up until the dead timer runs out, and that's 40 seconds on the neighbor with default timers. 45 seconds later, the same router: ```text hq-core1# show ip ospf neighbor hq-core1# show run | i ip ospf (hello|cost|dead) ip ospf hello-interval 5 ip ospf cost 20 ip ospf hello-interval 5 ip ospf cost 20 ``` No neighbors at all, and the playbook had already reported success. A post-check proves something only after the protocol has had time to notice the change. For OSPF, that means waiting longer than the dead interval. For BGP, it's the hold time, 180 seconds by default. Keep that in mind before shortening `settle`. [![Lab diagram: A 0 to 60 second timeline for hq-core1 after its OSPF hello interval changes to 5 seconds. At 1 second the post-check passes with 2 neighbors FULL and 0 routes lost, because the neighbors keep the adjacency until their 40 second dead timer expires. By 45 seconds hq-core1 has 0 neighbors, so a snapshot taken anywhere in the first 40 seconds looks healthy.](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/diagram-topology-1.png)](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/diagram-topology-1.png?ref=pinglabz.com) A 1 second settle passes the post-check. The neighbors drop at 40 seconds, after the run has already passed. After the tests, I removed the hello and cost lines from all three routers and checked that the neighbor counts were back to two, one and one. ## Things I'd add before using this in production - Run the same health check on the neighbors as well as on the changed router. hq-core1 noticed its own failure here. A change that only breaks things one hop away needs the check to run on those devices too. - Save a full `show running-config` as well as the module data, in case the rollback itself fails. - Use `serial: [1, "50%"]` to run a canary first and then larger batches. [Part 15](https://www.pinglabz.com/ansible-cdp-lldp-neighbor-map-ios/) Next in the series is \[part 15, a CDP and LLDP neighbor map built with Ansible\](https://www.pinglabz.com/ansible-cdp-lldp-neighbor-map-ios/). The rest of the series is on the [series page](https://www.pinglabz.com/ansible-network-automation/).