Ansible Compliance Checks on IOS XE: Find Drift, Fix It, Prove It

This is part 11 of the Ansible for network engineers series. Part 10 built a campus from one data model. This one asks the opposite question for the six routers and switches from the earlier parts: does what's running still match what the inventory says it should be? One playbook checks five rules on every device and writes a report. A second playbook fixes what the first one found. Then the check runs again and has to come back clean.

Built in my home lab with ansible-core 2.21.5, cisco.ios 11.6.0, ansible.netcommon 8.7.1 and Python 3.14, against Cisco IOS XE 17.18.2 (router and L2 lab images).

The rules

Intent lives where it already lived in this series, in group_vars and host_vars. Nothing new to maintain:

HQ uses NTP 10.66.0.123 and syslog 10.66.0.51. The branches use NTP 10.66.20.123, branch 1 sends syslog to its own collector at 10.66.21.51, and the edge role adds a second collector at 10.66.0.52.

"Exactly" matters for the first two rules. A check that only asks "is my server configured?" passes a router that also points at a rogue NTP server. Comparing the sorted lists catches extras as well as missing entries.

Setting up the drift

To have something to find, I broke four devices on purpose:

• ip http server on br1-edge1, br2-edge1 and br1-acc1, the kind of thing that comes back after someone pastes a vendor config.
• On hq-core2, an extra ntp server 10.66.0.200 and logging host 10.66.0.99, next to the correct ones.

hq-core1 and hq-edge1 were left alone, so they should pass.

The check playbook

NTP and logging come from the resource modules in gathered state, which return parsed data instead of text. The other three rules are simple enough to pull out of one show running-config with a regex.

# playbooks/90-compliance.yml
---
- name: Compliance check against inventory intent
  hosts: sites
  gather_facts: false
  tasks:
    - name: Read NTP config as structured data
      cisco.ios.ios_ntp_global:
        state: gathered
      register: ntp

    - name: Read logging config as structured data
      cisco.ios.ios_logging_global:
        state: gathered
      register: logg

    - name: Read the running config once for line rules
      cisco.ios.ios_command:
        commands: show running-config
      register: run

    - name: Evaluate the rules
      ansible.builtin.set_fact:
        results:
          - rule: ntp-servers
            want: "{{ [ntp_server] }}"
            have: "{{ ntp.gathered.servers | default([]) | map(attribute='server') | list }}"
          - rule: syslog-hosts
            want: "{{ [syslog_server] + role_syslog_servers }}"
            have: "{{ logg.gathered.hosts | default([]) | map(attribute='host') | list }}"
          - rule: snmp-location
            want: "{{ [site_location] }}"
            have: "{{ cfg | regex_findall('(?m)^snmp-server location (.*)$') }}"
          - rule: http-server-off
            want: []
            have: "{{ cfg | regex_findall('(?m)^ip http server$') }}"
          - rule: vty-ssh-only
            want: ["ssh"]
            have: "{{ cfg | regex_findall('(?m)^ transport input (.*)$') | unique }}"
      vars:
        cfg: "{{ run.stdout[0] }}"

    - name: Flag each rule
      ansible.builtin.set_fact:
        report: "{{ report | default([]) + [item | combine({'ok': (item.want | sort) == (item.have | sort)})] }}"
      loop: "{{ results }}"
      loop_control:
        label: "{{ item.rule }}"

    - name: Show failures for this device
      ansible.builtin.debug:
        msg: "{{ report | rejectattr('ok') | map(attribute='rule') | list }}"

    - name: Write one report for the run
      ansible.builtin.template:
        src: ../templates/compliance.md.j2
        dest: ../reports/compliance.md
        mode: "0644"
      delegate_to: localhost
      run_once: true

    - name: Fail the device if anything drifted
      ansible.builtin.assert:
        that: report | rejectattr('ok') | list | length == 0
        fail_msg: "{{ report | rejectattr('ok') | list | length }} rule(s) out of compliance"
        success_msg: all rules pass
        quiet: true

Each rule is a want list and a have list. That keeps the comparison the same for every rule and makes the report readable, because it shows both sides. The assert at the end marks a drifted device as failed, so the play recap alone tells you who is out of compliance, and a pipeline running this gets a non zero exit code.

The report template:

{# templates/compliance.md.j2 #}
# Compliance report

| Device | Rule | Result | Want | Have |
|---|---|---|---|---|
{% for h in ansible_play_hosts | sort %}
{% for r in hostvars[h].report %}
| {{ h }} | {{ r.rule }} | {{ "pass" if r.ok else "FAIL" }} | {{ r.want | join(", ") }} | {{ r.have | join(", ") }} |
{% endfor %}
{% endfor %}

run_once with delegate_to: localhost writes one file for the whole run, looping over ansible_play_hosts, instead of six devices racing to write the same file.

First run: the drift shows up

ansible-playbook -i site-inv/hosts.yml playbooks/90-compliance.yml

PLAY RECAP *********************************************************************
br1-acc1                   : ok=6    changed=0    unreachable=0    failed=1
br1-edge1                  : ok=6    changed=0    unreachable=0    failed=1
br2-edge1                  : ok=6    changed=0    unreachable=0    failed=1
hq-core1                   : ok=8    changed=1    unreachable=0    failed=0
hq-core2                   : ok=6    changed=0    unreachable=0    failed=1
hq-edge1                   : ok=7    changed=0    unreachable=0    failed=0

The changed=1 on hq-core1 is the report file being written on the control node. run_once picks the first host, so the change is booked against it. Nothing touched the router.

The failing rows from reports/compliance.md, plus hq-core1 for comparison:

All four broken devices were found, with the extra servers on hq-core2 listed by address. The 25 rows also show every rule passing on hq-core1 and hq-edge1, which is the other half of the proof: the check isn't just failing everything.

The remediation playbook

# playbooks/91-remediate.yml
---
- name: Fix what the compliance check flagged
  hosts: sites
  gather_facts: false
  tasks:
    - name: Read NTP config as structured data
      cisco.ios.ios_ntp_global:
        state: gathered
      register: ntp

    - name: Read logging config as structured data
      cisco.ios.ios_logging_global:
        state: gathered
      register: logg

    - name: Build the fix lines from the difference
      ansible.builtin.set_fact:
        fix_lines: "{{ ntp_add + ntp_del + log_add + log_del + http }}"
      vars:
        ntp_have: "{{ ntp.gathered.servers | default([]) | map(attribute='server') | list }}"
        log_have: "{{ logg.gathered.hosts | default([]) | map(attribute='host') | list }}"
        log_want: "{{ [syslog_server] + role_syslog_servers }}"
        ntp_add: "{{ [ntp_server] | difference(ntp_have) | map('regex_replace', '^', 'ntp server ') | list }}"
        ntp_del: "{{ ntp_have | difference([ntp_server]) | map('regex_replace', '^', 'no ntp server ') | list }}"
        log_add: "{{ log_want | difference(log_have) | map('regex_replace', '^', 'logging host ') | list }}"
        log_del: "{{ log_have | difference(log_want) | map('regex_replace', '^', 'no logging host ') | list }}"
        http: ["no ip http server"]

    - name: Show the plan
      ansible.builtin.debug:
        var: fix_lines

    - name: Push the fix lines
      cisco.ios.ios_config:
        lines: "{{ fix_lines }}"

The fix lines come from the same gathered data as the check, using difference both ways: add what's wanted and missing, remove what's there and not wanted. no ip http server goes in every plan. ios_config compares it with the running config and only sends it where ip http server is present.

Check mode first:

ansible-playbook -i site-inv/hosts.yml playbooks/91-remediate.yml --check --diff

ok: [hq-core2] => {
    "fix_lines": [
        "no ntp server 10.66.0.200",
        "no logging host 10.66.0.99",
        "no ip http server"
    ]
}

br1-acc1                   : ok=5    changed=1    unreachable=0    failed=0
br1-edge1                  : ok=5    changed=1    unreachable=0    failed=0
br2-edge1                  : ok=5    changed=1    unreachable=0    failed=0
hq-core1                   : ok=5    changed=0    unreachable=0    failed=0
hq-core2                   : ok=5    changed=1    unreachable=0    failed=0
hq-edge1                   : ok=5    changed=0    unreachable=0    failed=0

Four devices would change, the two clean ones wouldn't. Then the real run:

TASK [Push the fix lines] ******************************************************
ok: [hq-core1]
ok: [hq-edge1]
changed: [br1-acc1]
changed: [hq-core2]
changed: [br1-edge1]
changed: [br2-edge1]

You'll also see a warning that the input lines should look like they do in the running config. That's ios_config noticing no ... lines, which never appear in a running config. It's expected for removals.

Recheck

PLAY RECAP *********************************************************************
br1-acc1                   : ok=7    changed=0    unreachable=0    failed=0
br1-edge1                  : ok=7    changed=0    unreachable=0    failed=0
br2-edge1                  : ok=7    changed=0    unreachable=0    failed=0
hq-core1                   : ok=8    changed=1    unreachable=0    failed=0
hq-core2                   : ok=7    changed=0    unreachable=0    failed=0
hq-edge1                   : ok=7    changed=0    unreachable=0    failed=0

No failures on any of the six devices, and the new report has 30 rows, all pass. hq-core1's changed=1 is the report file again.

Reliability notes

My lab host was under load during these runs. I kept ansible_command_timeout: 120 because a full running config took about a minute to come back at the worst point, and this playbook reads it on every device. I also switched ansible_network_cli_ssh_type to paramiko after libssh sessions kept stalling mid play, which I covered in part 10. Even then, the first check run lost br1-acc1 to a connect timeout. A failed connection isn't compliance data, so I reran the whole check rather than trusting a report with a device missing. If you schedule this, treat unreachable or a timeout as its own alert, separate from drift.

What I'd do in production

Run the check on a schedule and keep the reports in Git so you can see when drift started. Keep the fix playbook separate and run it with --check first, by hand or behind an approval. Some drift is a person fixing an outage at 2 a.m., and you want to read it before you undo it. Add rules one at a time, and make each one compare exact lists so extras fail too.

Part 12 turns show command output into data with cli_parse. The rest of the series is on the series page.