An IOS upgrade is mostly checking. The copy and the reload take a few minutes; the work is proving the box is ready beforehand and proving nothing broke afterwards. That checking is exactly what Ansible is good at, so this post builds the workflow as four playbooks: pre-check, stage, post-check and rollback.
I built this in my home lab, and I need to be upfront about one thing. The lab runs a single IOS XE build, 17.18.2, on the lab image. There is no newer image to move to, and as you'll see, the lab image ignores the boot variable and refuses a reload over SSH. So no version changes in this post. What you get is the whole workflow run for real against six devices, a same-version staging rehearsal on one of them, and a plain list of what the lab could not prove.
This follows part 20's services standard and reuses the check mode habit from part 19.
Versions: ansible-core 2.21, cisco.ios 11.6, IOS XE 17.18.2 on five routers and one L2 switch image (br1-acc1).
The approved images as data
The first decision is which image each platform should run. That goes in a vars file keyed by the platform family string from show version, with the size and MD5 that came with the image. The playbooks never guess a filename.
# vars/upgrade.yml
---
# Part 21: approved images, one entry per platform family from show version
upgrade_target_version: "17.18.2"
upgrade_fs: "unix:"
upgrade_free_margin: 1.10 # need image size plus 10 percent free
upgrade_image_dir: /tmp/upg # where the approved images sit on the control node
upgrade_catalog:
X86_64BI_LINUX-ADVENTERPRISEK9-M:
file: x86_64_crb_linux-adventerprisek9-ms.iol
size: 293729288
md5: 34bf876c5b3c60f3ef4eb28eb8a176d7
X86_64BI_LINUX_L2-ADVENTERPRISEK9-M:
file: x86_64_crb_linux_l2-adventerprisek9-ms.iol
size: 246471208
md5: 0e460cbcfa9e0b76ca400162928fd989
On real hardware upgrade_fs would be bootflash: or flash:. The lab image only has unix:. The size and MD5 here are the real values for the lab binaries, and they match the iol.bin each device is running, which is why the target version equals the running version.
Pre-checks
The pre-check is read-only apart from saving the config. It answers four questions per device: what is it running, is the platform in the catalog, is there room for the image, and what does the network look like right now.
# playbooks/210-upgrade-precheck.yml
---
- name: Upgrade pre-checks (read-only apart from the saved config copy)
hosts: sites
gather_facts: false
vars_files:
- ../vars/upgrade.yml
tasks:
- name: Facts with filesystems
cisco.ios.ios_facts:
gather_subset: hardware
- name: Read version, boot and the state we will compare after
cisco.ios.ios_command:
commands:
- show version
- show boot
- show ip interface brief
- show ip ospf neighbor
- show ip bgp summary
- show ip route summary
register: pre
- name: Work out the platform family and the approved image for it
ansible.builtin.set_fact:
family: "{{ pre.stdout[0] | regex_search('Linux Software \\((\\S+)\\)', '\\1') | first }}"
- name: Pick the image
ansible.builtin.set_fact:
img: "{{ upgrade_catalog[family] }}"
free_bytes: "{{ (ansible_net_filesystems_info[upgrade_fs].spacefree_kb * 1024) | int }}"
- name: Gate on platform, space and saved config
ansible.builtin.assert:
that:
- family in upgrade_catalog
- free_bytes | int > (img.size * upgrade_free_margin) | int
fail_msg: "{{ inventory_hostname }}: not ready ({{ family }}, free {{ free_bytes }})"
success_msg: "{{ inventory_hostname }}: {{ family }}, free {{ free_bytes }} bytes, image needs {{ img.size }}"
- name: Save the config and keep a copy next to the image for rollback
cisco.ios.ios_command:
commands:
- write memory
# a second copy to the same name stops at an overwrite prompt, so clear it first
- "delete /force {{ upgrade_fs }}pre-upgrade.cfg"
- command: "copy running-config {{ upgrade_fs }}pre-upgrade.cfg"
prompt: 'Destination filename \[pre-upgrade.cfg\]\?'
answer: ""
when: not ansible_check_mode
- name: Snapshot to the control node
ansible.builtin.copy:
dest: "{{ playbook_dir }}/../snapshots/{{ inventory_hostname }}-pre.json"
content: "{{ snap | to_nice_json }}"
delegate_to: localhost
vars:
snap:
version: "{{ ansible_net_version }}"
image: "{{ ansible_net_image }}"
up_interfaces: "{{ pre.stdout[2].splitlines() | select('search', '\\sup\\s+up\\s*$') | map('split') | map('first') | list }}"
ospf_full: "{{ pre.stdout[3].splitlines() | select('search', 'FULL') | map('split') | map('first') | list }}"
bgp_established: "{{ pre.stdout[4].splitlines() | select('search', '^\\d+\\.\\d+\\.\\d+\\.\\d+\\s.*\\s\\d+$') | map('split') | map('first') | list }}"
route_total: "{{ pre.stdout[5] | regex_search('Total\\s+(\\d+)\\s+(\\d+)', '\\1', '\\2') | map('int') | sum }}"
- name: Verdict
ansible.builtin.debug:
msg: >-
{{ inventory_hostname }} runs {{ ansible_net_version }} from {{ ansible_net_image }},
target {{ upgrade_target_version }}:
{{ 'needs upgrade' if ansible_net_version != upgrade_target_version else 'already at target, nothing to stage' }}
The snapshot is the important part. It records the up interfaces, OSPF neighbors in FULL, established BGP peers and the route count, as JSON on the control node. After the upgrade, the post-check reads the same commands and compares against this file. Without it, "the network looks fine" is an opinion.
The run on all six:
ok: [hq-core1] => msg: hq-core1: X86_64BI_LINUX-ADVENTERPRISEK9-M, free 2147479552 bytes, image needs 293729288
ok: [br1-acc1] => msg: br1-acc1: X86_64BI_LINUX_L2-ADVENTERPRISEK9-M, free 2147479552 bytes, image needs 246471208
hq-core1 runs 17.18.2 from unix:/iol/iol.bin, target 17.18.2: already at target, nothing to stage
hq-edge1 runs 17.18.2 from unix:/iol/iol.bin, target 17.18.2: already at target, nothing to stage
br2-edge1 runs 17.18.2 from unix:/iol/iol.bin, target 17.18.2: already at target, nothing to stage
...
br1-acc1 : ok=8 changed=1 unreachable=0 failed=0
br2-edge1 : ok=8 changed=1 unreachable=0 failed=0
hq-core1 : ok=8 changed=1 unreachable=0 failed=0
The changed=1 is the config save. Everything else is reads.
A snapshot for hq-edge1 looks like this (the two BGP peers are its upstreams):
{
"bgp_established": ["10.66.103.2", "10.66.104.2"],
"image": "unix:/iol/iol.bin",
...
}
Two things that broke on the way
The overwrite prompt. The first version just ran copy running-config unix:pre-upgrade.cfg with a prompt answer for the filename. On the second run the file already existed, IOS asked a different question about overwriting, and the task sat there until:
fatal: [hq-edge1]: FAILED! => {"changed": false, "msg": "timeout value 120 seconds reached while trying to send command: b'copy running-config unix:pre-upgrade.cfg'"}
Answering a list of prompts did not fix it either. Deleting the old copy with delete /force first did, and it makes the task behave the same on every run.
The route count. show ip route summary puts networks and subnets in two columns on the Total line. My first regex kept one group and every device reported a route total of 1. Summing both groups gave 2, 8, 8, 8, 7 and 13, which match the routing tables.
The free-space number is not real here
Look at the free bytes again: 2147479552 on every device, equal to the total. The unix: filesystem on the lab image reports free equal to total no matter what is on it. I copied a 293 MB file onto br2-edge1 later and dir still said:
2147479552 bytes total (2147479552 bytes free)
So the free-space gate passes in the lab for the wrong reason. On a real bootflash: the number is accurate and this gate is the one that saves you from a copy dying at 90 percent. Keep the gate, but don't take the lab as proof that it works.
Staging the image
Staging is the first step that writes to the box, so it has two guards. It refuses to run without -e upgrade_confirm=yes, and it ends the host if the device already runs the target. It runs one device at a time.
# playbooks/211-upgrade-stage.yml
---
# Stage the approved image on one device at a time and point the boot variable at it.
# Run with --limit <host> -e upgrade_confirm=yes. No reload here.
- name: Stage image and set boot variable
hosts: sites
gather_facts: false
serial: 1
vars_files:
- ../vars/upgrade.yml
vars:
upgrade_http: "http://10.66.10.1:8080"
upgrade_force_stage: false # lab only: stage even when already at target
upgrade_mgmt_intf: Ethernet0/0 # management lives in a VRF; source the HTTP client there
tasks:
- name: Refuse to run without an explicit go-ahead
ansible.builtin.assert:
that: upgrade_confirm | default('no') == 'yes'
fail_msg: "Add -e upgrade_confirm=yes after reading the pre-check"
- name: Version and platform
cisco.ios.ios_command:
commands: show version
register: ver
- name: Pick the image for this platform
ansible.builtin.set_fact:
img: "{{ upgrade_catalog[ver.stdout[0] | regex_search('Linux Software \\((\\S+)\\)', '\\1') | first] }}"
running: "{{ ver.stdout[0] | regex_search('Version (\\S+),', '\\1') | first }}"
- name: The boot order we want
ansible.builtin.set_fact:
boot_want:
- "boot system {{ upgrade_fs }}{{ img.file }}"
- "boot system {{ ver.stdout[0] | regex_search('System image file is \"([^\"]+)\"', '\\1') | first }}"
- name: Stop here if nothing to do
ansible.builtin.meta: end_host
when: running == upgrade_target_version and not upgrade_force_stage | bool
- name: Is the image already on the device
cisco.ios.ios_command:
commands: "dir {{ upgrade_fs }}{{ img.file }}"
register: present
failed_when: false
- name: HTTP client sources from the management interface
cisco.ios.ios_config:
lines:
- "ip http client source-interface {{ upgrade_mgmt_intf }}"
- name: Pull the image from the control node
cisco.ios.ios_command:
commands:
- command: "copy {{ upgrade_http }}/{{ img.file }} {{ upgrade_fs }}{{ img.file }}"
prompt: 'Destination filename \[.*\]\?'
answer: ""
vars:
ansible_command_timeout: 900
# a failed dir leaves no stdout, so default it
when: (present.stdout | default(['']))[0] is not search(img.file ~ '\\s*$', multiline=True)
- name: Verify the MD5 on the device
cisco.ios.ios_command:
commands: "verify /md5 {{ upgrade_fs }}{{ img.file }} {{ img.md5 }}"
register: md5
vars:
ansible_command_timeout: 600
- name: Stop if the hash does not match
ansible.builtin.assert:
that: md5.stdout[0] is search('Verified')
fail_msg: "MD5 mismatch on {{ inventory_hostname }}, image not trusted"
- name: Current boot statements
cisco.ios.ios_command:
commands: show running-config | include ^boot system
register: boot_now
# "no boot system" in before: makes ios_config push the block every run,
# so only touch it when the order on the box is not the order we want
- name: Boot the new image first, the running one as fallback
cisco.ios.ios_config:
lines: "{{ boot_want }}"
before: no boot system
match: exact
save_when: modified
when: boot_now.stdout_lines[0] != boot_want
- name: Read back the boot variable
cisco.ios.ios_command:
commands:
- show boot
- show running-config | include ^boot system
register: boot
- name: Show it
ansible.builtin.debug:
var: boot.stdout_lines
The guards, run against br2-edge1:
# without upgrade_confirm
fatal: [br2-edge1]: FAILED! => msg: Add -e upgrade_confirm=yes after reading the pre-check
# with upgrade_confirm, already at target
br2-edge1 : ok=3 changed=0 unreachable=0 failed=0
To exercise the rest I set upgrade_force_stage=true. That stages the same 17.18.2 build under its release filename. It is a rehearsal of the mechanics, not an upgrade.
Moving the file
The device pulls the image over HTTP from the control node, which serves the image directory with python3 -m http.server 8080 --bind 10.66.10.1. I chose a pull because the automation account is privilege 1 with enable, and SCP to the device needs a privilege 15 login. In production use your normal image server.
The first attempt failed:
%Error opening http://10.66.10.1:8080/x86_64_crb_linux-adventerprisek9-ms.iol (HTTP error: Connection failure)
Management sits in a VRF, and the HTTP client was sourcing from the global table. ip http client source-interface Ethernet0/0 fixed it, which is why that task is in the playbook. The successful copy:
Accessing http://10.66.10.1:8080/x86_64_crb_linux-adventerprisek9-ms.iol...
Loading http://10.66.10.1:8080/x86_64_crb_linux-adventerprisek9-ms.iol !!!
293729288 bytes copied in 38.382 secs (7652787 bytes/sec)
Verified (unix:/x86_64_crb_linux-adventerprisek9-ms.iol) = 34bf876c5b3c60f3ef4eb28eb8a176d7
The ansible_command_timeout: 900 on the copy matters. The default 30 seconds would have killed it.
The MD5 check has to fail when it should
A check that always passes is worse than none. I ran verify /md5 with a deliberately wrong hash:
Computed signature = 34bf876c5b3c60f3ef4eb28eb8a176d7
%Error verifying unix:/x86_64_crb_linux-adventerprisek9-ms.iol
Submitted signature = 00000000000000000000000000000000
There is no "Verified" in that output, so the assert stops the play before the boot order is touched. The verify took about 17 seconds for 293 MB, so it gets its own long timeout too.
Boot order, and where the lab image stops cooperating
The boot block puts the new image first and the running image second as a fallback. After the push:
"commands": ["no boot system", "boot system unix:x86_64_crb_linux-adventerprisek9-ms.iol", "boot system unix:/iol/iol.bin"]
BOOT variable does not exist
CONFIG_FILE variable does not exist
BOOTLDR variable does not exist
Configuration register is 0x0
The boot system lines are in the running and startup config, but show boot says the BOOT variable does not exist. The lab image runs whatever binary the host launches it with (unix:/iol/iol.bin), and the bootstrap is Linux, not ROMMON. The config is right; the platform just doesn't read it. On a real ISR or Catalyst, show boot would list the new image here, and that read-back is the check you want before a reload.
The first version also had an idempotency bug. before: no boot system makes ios_config push the whole block on every run, so a second run reported changed=1 with nothing different. Comparing the current boot lines first and skipping when they match fixed it:
br2-edge1 : ok=11 changed=0 unreachable=0 failed=0 skipped=2
The skipped two are the copy (file already there) and the boot block (already in order).
Reload and post-check
The post-check playbook has an optional reload, behind its own flag, and then a comparison against the pre-check snapshot.
# playbooks/212-upgrade-reload-postcheck.yml
---
# Reload one device (only with -e upgrade_reload=yes), wait for it, then compare against the pre-check snapshot.
- name: Reload and post-check
hosts: sites
gather_facts: false
serial: 1
vars_files:
- ../vars/upgrade.yml
tasks:
- name: Reload
when: upgrade_reload | default('no') == 'yes'
block:
- name: Send reload and confirm
cisco.ios.ios_command:
commands:
- command: reload
prompt:
- 'Save\? \[yes/no\]'
- 'Proceed with reload\? \[confirm\]'
answer: ["no", ""]
check_all: false
# the session drops as the box goes down, which the module reports as an error
failed_when: false
- name: Drop the old SSH session
ansible.builtin.meta: reset_connection
- name: Wait for SSH to go away
ansible.builtin.wait_for:
host: "{{ ansible_host }}"
port: 22
state: stopped
timeout: 300
delegate_to: localhost
- name: Wait for SSH to come back
ansible.builtin.wait_for:
host: "{{ ansible_host }}"
port: 22
state: started
delay: 10
timeout: 900
delegate_to: localhost
- name: Give the control plane time to settle
ansible.builtin.pause:
seconds: 60
- name: Read the same state as the pre-check
cisco.ios.ios_command:
commands:
- show version
- show ip interface brief
- show ip ospf neighbor
- show ip bgp summary
- show ip route summary
register: post
- name: Load the pre-check snapshot
ansible.builtin.set_fact:
before: "{{ lookup('file', playbook_dir ~ '/../snapshots/' ~ inventory_hostname ~ '-pre.json') | from_json }}"
after:
version: "{{ post.stdout[0] | regex_search('Version (\\S+),', '\\1') | first }}"
image: "{{ post.stdout[0] | regex_search('System image file is \"([^\"]+)\"', '\\1') | first }}"
uptime: "{{ post.stdout[0] | regex_search('uptime is (.+)', '\\1') | first }}"
up_interfaces: "{{ post.stdout[1].splitlines() | select('search', '\\sup\\s+up\\s*$') | map('split') | map('first') | list }}"
ospf_full: "{{ post.stdout[2].splitlines() | select('search', 'FULL') | map('split') | map('first') | list }}"
bgp_established: "{{ post.stdout[3].splitlines() | select('search', '^\\d+\\.\\d+\\.\\d+\\.\\d+\\s.*\\s\\d+$') | map('split') | map('first') | list }}"
route_total: "{{ post.stdout[4] | regex_search('Total\\s+(\\d+)\\s+(\\d+)', '\\1', '\\2') | map('int') | sum }}"
- name: What changed
ansible.builtin.debug:
msg:
version: "{{ before.version }} -> {{ after.version }}"
image: "{{ before.image }} -> {{ after.image }}"
uptime: "{{ after.uptime }}"
interfaces_lost: "{{ before.up_interfaces | difference(after.up_interfaces) }}"
ospf_lost: "{{ before.ospf_full | difference(after.ospf_full) }}"
bgp_lost: "{{ before.bgp_established | difference(after.bgp_established) }}"
routes: "{{ before.route_total }} -> {{ after.route_total }}"
- name: Pass or fail
ansible.builtin.assert:
that:
- after.version == upgrade_target_version
- before.up_interfaces | difference(after.up_interfaces) | length == 0
- before.ospf_full | difference(after.ospf_full) | length == 0
- before.bgp_established | difference(after.bgp_established) | length == 0
- after.route_total | int >= before.route_total | int
fail_msg: "{{ inventory_hostname }} post-check failed, see the diff above"
Note the two wait_for tasks. Waiting only for port 22 to be open is a classic bug: the device is often still up when the wait starts, so it passes straight away and the post-check runs against a box that's about to go down. Wait for the port to close, then for it to open.
Here is what happened when I ran it with upgrade_reload=yes on br2-edge1:
ok: [br2-edge1] => stdout: Reload to the ROM monitor only allowed from console line
unless the configuration register boot bits are non-zero.
fatal: [br2-edge1 -> localhost]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 10.66.10.31:22 to stop."}
The config register on the lab image is 0x0, so a reload from a vty is treated as a reload to ROMMON and refused. I left the register alone; changing it on a lab image to make a demo work tells you nothing about real hardware, where the register is 0x2102 and the reload goes through. Two useful things came out of it anyway. failed_when: false on the reload hid the refusal, and only the "wait for SSH to go away" task caught it. In your own version, check the reload output for that message and fail early.
So I can't show a device booting a new image. What I can show is the comparison logic, run without a reload on all six:
TASK [What changed]
ok: [hq-core1] => msg:
bgp_lost: []
image: unix:/iol/iol.bin -> unix:/iol/iol.bin
interfaces_lost: []
ospf_lost: []
routes: 8 -> 8
uptime: 10 hours, 23 minutes
version: 17.18.2 -> 17.18.2
TASK [Pass or fail]
ok: [hq-core1] => msg: All assertions passed
br1-acc1 : ok=4 changed=0 unreachable=0 failed=0 skipped=4
br2-edge1 : ok=4 changed=0 unreachable=0 failed=0 skipped=4
hq-edge1 : ok=4 changed=0 unreachable=0 failed=0 skipped=4
After a real upgrade you'd expect the version line to change, the uptime to read a few minutes, and the three lost lists to stay empty. An empty list is the whole point. If an OSPF neighbor doesn't come back, its router ID shows up in ospf_lost and the assert fails for that one device, while serial: 1 stops you rolling the same image to the next box.
Rollback
Rollback has two levels. If the device never reloaded, rolling back means removing what staging added. If it reloaded onto a bad image, the boot system fallback line and the pre-upgrade.cfg copy are what you work from, normally from the console.
The first level as a playbook:
# playbooks/213-upgrade-rollback.yml
---
- name: Roll back a staged upgrade
hosts: "{{ target | default('none') }}"
gather_facts: false
serial: 1
vars_files:
- ../vars/upgrade.yml
vars:
upgrade_mgmt_intf: Ethernet0/0
tasks:
- name: Read boot statements and staged files
cisco.ios.ios_command:
commands:
- show running-config | include ^boot system
- "dir {{ upgrade_fs }}"
register: before
- name: Remove boot statements we added
cisco.ios.ios_config:
lines:
- no boot system
save_when: modified
when: before.stdout[0] | length > 0
- name: Delete staged images
cisco.ios.ios_command:
commands:
- "delete /force {{ upgrade_fs }}{{ item.value.file }}"
loop: "{{ upgrade_catalog | dict2items }}"
loop_control:
label: "{{ item.value.file }}"
when: item.value.file in before.stdout[1]
- name: Check HTTP client source interface
cisco.ios.ios_command:
commands:
- show running-config | include ^ip http client source-interface
register: httpc
- name: Remove HTTP client source interface if present
cisco.ios.ios_config:
lines:
- "no {{ httpc.stdout[0] }}"
save_when: modified
when: httpc.stdout[0] | length > 0
- name: Read back
cisco.ios.ios_command:
commands:
- show running-config | include ^boot system|^ip http client
- "dir {{ upgrade_fs }}"
register: after
- name: Show result
ansible.builtin.debug:
msg: "{{ after.stdout_lines }}"
On br2-edge1 the first run removed the boot lines, deleted the staged image and removed the HTTP client source interface (changed=2, the delete runs through ios_command so it reports ok). The second run changed nothing. The read-back showed no boot or HTTP client lines, and the filesystem back to iol.bin, the config files and pre-upgrade.cfg:
2643122 -rw- 3034 Nov 1 2026 04:15:33 +00:00 pre-upgrade.cfg
794566 -rwx 293729288 Oct 4 2026 06:17:55 +00:00 iol.bin
br2-edge1 : ok=4 changed=0 unreachable=0 failed=0 skipped=3
I kept pre-upgrade.cfg on all six devices. It is the documented rollback config, and it costs 3 KB.
For the second level, on real hardware the plan is: console in, check show boot and show version, and if the new image is the problem, boot system the old image, reload, then configure replace from pre-upgrade.cfg if the config was also converted. None of that could be run here, so I won't pretend to show output for it.
What the lab could and could not prove
Proved, with real output:
- Platform and catalog lookup from
show version. - The pre-check snapshot and a post-check diff against it on six devices.
- The confirm gate and the "already at target" exit.
- An HTTP image pull to a device, including the VRF fix.
- On-device MD5 passing on a good hash and failing on a bad one.
- Boot order configured and saved, idempotent on the second run.
- A clean, idempotent rollback of the staging.
Not proved:
- A version change. There is only one build.
- That the device boots the image in
boot system. The lab image ignores it. - The free-space gate. The lab image reports free equal to total.
- A reload from Ansible. The lab image's config register makes it refuse from a vty.
If you run this on real hardware, the first list carries over directly. For the second, try it on one spare device first, with a console session open.
Part 22 covers the nightly audit report: read-only facts and compliance from every device, a dated HTML and Markdown report, and a systemd timer. The rest of the series is on the series page.