Standardize and Harden IOS XE Services with Ansible: NTP, Syslog, SNMPv3, AAA

Every router in a network should carry the same baseline: the same NTP and syslog targets, the same SNMP access, the same banner, the same AAA method lists. In practice each box drifts a little, and the drift only shows up when something breaks. This post writes that baseline down as Ansible vars, pushes it to six IOS XE devices with the dry-run habit from the last part, and then checks it with a read-only playbook.

I built this in my home lab. Some of it the lab can't prove. There is no NTP server for the routers to sync with and no real TACACS+ server, and I'll show exactly what that looks like instead of pretending it worked.

This builds on part 11's compliance checks and the check mode and diff habit from part 19.

Versions: ansible-core 2.21, cisco.ios 11.6, IOS XE 17.18.2 on five routers and one L2 switch image (br1-acc1).

What the devices looked like before

A quick read of the six devices showed the usual mix:

• SNMP agent not enabled anywhere. Only snmp-server location and contact were set.
• NTP configured but unsynced on every device (stratum 16).
• ip http secure-server on everywhere. Nothing in this series uses RESTCONF, so it can go.
• vty lines with exec-timeout 3 0 and login local.
• HQ already had a banner motd, and part 19 had left logging buffered 64000 informational and ip ssh time-out 60 on the HQ routers only.

Rather than undo part 19's lines, I folded them into the standard so all six carry them.

The standard as data

All the values live in one vars file. Nothing device-specific goes here; ntp_server, syslog_server and role_syslog_servers already come from inventory group vars.

# vars/services.yml
svc_logging_buffer: 64000
svc_ssh_timeout: 60
svc_vty_exec_timeout: 5

svc_hardening_lines:
  - service password-encryption
  - service timestamps log datetime msec localtime
  - "ip ssh time-out {{ svc_ssh_timeout }}"
  - login block-for 120 attempts 5 within 60
  - login on-failure log
  - no ip http secure-server

svc_banner_login: |
  Authorized access only. Activity on this device is logged.
  Disconnect now if you are not authorized.

svc_snmp_acl: SNMP-MGR
svc_snmp_managers:
  - sequence: 10
    grant: permit
    source:
      address: 10.66.10.0
      wildcard_bits: 0.0.0.255
svc_snmp_view: PL-VIEW
svc_snmp_group: PL-RO
svc_snmp_user: plmon

svc_tacacs_name: PL-TAC1
svc_tacacs_server: 10.66.0.49
svc_aaa_group: PL-TACACS

The secrets go in a separate vault file. I generated random values straight into it and encrypted it, so they never appeared on screen or in the post. The playbook only ever references the variable names:

# vars/services-vault.yml (ansible-vault encrypted)
vault_snmp_auth: <snmp-auth-password>
vault_snmp_priv: <snmp-priv-password>
vault_tacacs_key: <tacacs-shared-key>

The playbook

Resource modules where they fit well (NTP, logging, ACLs, banner), ios_config for the rest. This is the final version. The sections after it explain why several tasks look the way they do.

- name: Services standard and hardening
  hosts: sites
  gather_facts: false
  vars_files:
    - ../vars/services.yml
    - ../vars/services-vault.yml
  vars:
    # authPriv everywhere unless a host_vars file says otherwise
    snmp_level: "{{ svc_snmp_level | default('priv') }}"
  tasks:
    - name: NTP servers from inventory
      cisco.ios.ios_ntp_global:
        config:
          servers:
            - server: "{{ ntp_server }}"
          logging: true
        state: merged

    - name: Syslog host list in the shape the module wants
      ansible.builtin.set_fact:
        syslog_hosts: "{{ syslog_hosts | default([]) + [{'host': item}] }}"
      loop: "{{ [syslog_server] + role_syslog_servers }}"

    - name: Syslog hosts and buffer
      cisco.ios.ios_logging_global:
        config:
          buffered:
            size: "{{ svc_logging_buffer }}"
            severity: informational
          trap: informational
          hosts: "{{ syslog_hosts }}"
        state: merged

    - name: Global hardening lines
      cisco.ios.ios_config:
        lines: "{{ svc_hardening_lines }}"

    - name: VTY idle timeout
      cisco.ios.ios_config:
        parents: line vty 0 4
        lines:
          - "exec-timeout {{ svc_vty_exec_timeout }} 0"

    - name: Login banner
      cisco.ios.ios_banner:
        banner: login
        text: "{{ svc_banner_login }}"
        state: present

    - name: SNMP manager ACL through the resource module
      cisco.ios.ios_acls:
        config:
          - afi: ipv4
            acls:
              - name: "{{ svc_snmp_acl }}"
                acl_type: standard
                aces: "{{ svc_snmp_managers }}"
        state: merged

    - name: SNMPv3 view and group
      cisco.ios.ios_config:
        lines:
          - "snmp-server view {{ svc_snmp_view }} iso included"
          - "snmp-server group {{ svc_snmp_group }} v3 {{ snmp_level }} read {{ svc_snmp_view }} access {{ svc_snmp_acl }}"

    - name: Does the SNMPv3 user exist
      cisco.ios.ios_command:
        commands: show snmp user
      register: snmp_users
      # with no SNMP config yet, IOS answers "%SNMP agent not enabled" and the module fails
      failed_when: false

    - name: SNMPv3 user, only when missing
      cisco.ios.ios_config:
        lines:
          - "snmp-server user {{ svc_snmp_user }} {{ svc_snmp_group }} v3 auth sha {{ vault_snmp_auth }}{{ (' priv aes 128 ' ~ vault_snmp_priv) if snmp_level == 'priv' else '' }}"
      when: (snmp_users.stdout | default(['']))[0] is not search('User name. ' ~ svc_snmp_user ~ '$', multiline=True)
      no_log: true

    - name: Turn on AAA first, or the tacacs commands do not exist
      cisco.ios.ios_config:
        lines:
          - aaa new-model

    - name: Is the TACACS+ server already defined
      cisco.ios.ios_command:
        commands: "show running-config | include ^tacacs server"
      register: tac_run

    - name: TACACS+ server definition, only when missing
      when: tac_run.stdout[0] is not search('^tacacs server ' ~ svc_tacacs_name ~ '$', multiline=True)
      cisco.ios.ios_config:
        parents: "tacacs server {{ svc_tacacs_name }}"
        lines:
          - "address ipv4 {{ svc_tacacs_server }}"
          - "key {{ vault_tacacs_key }}"
          - timeout 3
      no_log: true

    - name: TACACS+ server group
      cisco.ios.ios_config:
        parents: "aaa group server tacacs+ {{ svc_aaa_group }}"
        lines:
          - "server name {{ svc_tacacs_name }}"

    - name: AAA method lists with local fallback
      cisco.ios.ios_config:
        lines:
          - "aaa authentication login default group {{ svc_aaa_group }} local"
          - "aaa authentication enable default group {{ svc_aaa_group }} enable"
          - "aaa authorization exec default group {{ svc_aaa_group }} local if-authenticated"

    - name: Save
      cisco.ios.ios_command:
        commands: write memory
      when: not ansible_check_mode

The syslog set_fact loop is there because the tidy one-liner needs community.general, which isn't installed here, and a folded JSON-looking string didn't turn into a list under ansible-core 2.21 ("dictionary requested"). A loop that appends dicts works with only builtins.

Dry run first, and the first thing it caught

The first --check --diff -v failed on all six devices before it touched anything:

[ERROR]: Task failed: Action failed: b'show snmp user\r\n%SNMP agent not enabled\r\nbr1-acc1#'

With no SNMP config at all, show snmp user is an error, not an empty list. That's why the task has failed_when: false and the when uses a default. The second check run went through and reported 11 changes on each branch device and 10 on HQ, with the commands visible for everything except the two no_log tasks:

changed: [hq-core1] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}

That's what you want from no_log. You lose the diff for those lines, which is the price of not printing the secret.

One router for real, and what check mode missed

Following part 19, the first real run went to one device with --limit br2-edge1. It failed:

[ERROR]: Task failed: b"tacacs server PL-TAC1\r\ntacacs server PL-TAC1\r\n      ^\r\n% Invalid input detected at '^' marker.\r\n\r\nbr2-edge1(config)#"

In the first version, aaa new-model came after the TACACS definition. Until AAA is on, the tacacs server command doesn't exist in the parser. Check mode can't see that because it never sends the line. Moving aaa new-model into its own task ahead of everything AAA fixed it.

Notice what the error shows. The task had no_log: true and the task result was censored, but the [ERROR] line still echoes the failing command text. Here that was only the tacacs server PL-TAC1 header, since the parser stopped there. If the key line had been the one rejected, it would have been on screen. Keep that in mind when you paste failure output into a ticket.

Making it idempotent

The next real run on br2-edge1 succeeded, and a recheck straight after still said changed=7. A standard that reports changes on every run is useless for drift detection, so each one had to go:

• ip ssh version 2 is the default on 17.18 and never shows in the running config, so ios_config pushes it every time. I dropped it from the list. SSHv1 isn't on offer here anyway.
• exec-timeout 10 0 was my first value. It's the default, so IOS hides it, and the line never matches. I moved the standard to 5 minutes, which shows up and is tighter anyway.
• The SNMP ACL written as an ios_config line didn't match the form IOS stores. Switching to ios_acls with structured ACEs fixed it.
• The TACACS key is stored encrypted, so key <plain> never matches what's in the config. The definition now only runs when the server is missing. The catch is that changing the key in the vault won't push a new key. Key rotation needs its own playbook that removes and re-adds the server on purpose.
• The SNMP user never appears in the running config at all. The show snmp user guard handles it the same way.

After those changes, the real run on br2-edge1 changed one thing (the new exec-timeout) and the following check run reported changed=0.

The rollout, and a platform limit

Check run across the rest, then the real run. Five devices went through. The L2 switch image did not:

fatal: [br1-acc1]: FAILED! => {"changed": false, "msg": "Task failed: b\"snmp-server group PL-RO v3 priv read PL-VIEW access SNMP-MGR\\r\\nsnmp-server group PL-RO v3 priv read PL-VIEW access SNMP-MGR\\r\\n                           ^\\r\\n% Invalid input detected at '^' marker.\\r\\n\\r\\nbr1-acc1(config)#\""}

This image has no SNMP privacy support, so v3 priv doesn't parse. v3 auth does. Check mode had called this line changed too, the same blind spot part 19 showed. I gave that one host a documented exception rather than lowering the standard for everyone:

# host_vars/br1-acc1.yml
# The L2 IOL image rejects "v3 priv" (no SNMP privacy support), so this switch gets authNoPriv
svc_snmp_level: auth

That's weaker. SNMP from that switch is authenticated but not encrypted. On a production switch I'd want a software image that supports privacy rather than living with the exception, but writing it down in host_vars at least makes it visible and searchable.

The rerun on br1-acc1 changed 6, and a recheck of all six came back changed=0 everywhere.

Testing SNMPv3 for real

A config line isn't proof the service works, so I queried the devices from the management host with net-snmp. Credentials here are placeholders:

snmpget -v3 -l authPriv -u plmon -a SHA -A '<snmp-auth-password>' \
  -x AES -X '<snmp-priv-password>' 10.66.10.31 sysName.0 sysUpTime.0

.1.3.6.1.2.1.1.5.0 = STRING: "br2-edge1.lab.example.net"
.1.3.6.1.2.1.1.3.0 = Timeticks: (3512027) 9:45:20.27

And the negative cases, which matter as much:

# wrong auth password
snmpget: Authentication failure (incorrect password, community or key)

# authNoPriv against a group that requires priv
Error in packet
Reason: authorizationError (access denied to that object)

# v2c with community public
Timeout: No Response from 10.66.10.31.

Five devices answered authPriv. br1-acc1 timed out on authPriv, as expected, and answered authNoPriv with "br1-acc1.lab.example.net".

IOS also logged this after the user was added:

%SYS-4-INSECURE_CONFIG: Module: SNMP - Command: snmp-server user plmon PL-RO v3 auth sha * priv aes 128 * - Reason: SNMP user without ACL - Remediation: Use SNMP user with AccessList

The ACL is on the group, so the manager restriction is in place, but 17.18 would like it on the user as well. That's a reasonable follow-up for the standard.

What the lab can't prove: AAA and NTP

Here's the AAA state on hq-core1 after the rollout:

aaa new-model
aaa group server tacacs+ PL-TACACS
aaa authentication login default group PL-TACACS local
aaa authentication enable default group PL-TACACS enable
aaa authorization exec default group PL-TACACS local if-authenticated

Server name: PL-TAC1
            Server address: 10.66.0.49
   Failed Connect Attempts:          4
        Total Packets Sent:          0
             Server Status: Alive

There is no TACACS+ server at 10.66.0.49 in my lab. Every login since has failed to connect to it and fallen back to the local method, which is why Ansible kept working and the log shows %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: netauto] [Source: 10.66.10.1]. "Alive" in that output doesn't mean the server answered. Zero packets sent says it never did. Logins took about 6 seconds both before and after, so the 3-second timeout wasn't adding visible delay here, but I can't say what it would look like against a real server that's slow.

Two honest limits on that:

• I haven't tested a real TACACS+ authentication, authorization of commands, or accounting. This post configures the client side only.
• Fallback to local only happens when the server is unreachable or errors. If a real server answers and rejects a user, IOS doesn't try local. Make sure your automation account exists on the server before you point devices at it.

Once AAA is on, the vty lines also changed shape. login local disappeared because the default login method list now covers the vty lines:

line vty 0 4
 exec-timeout 5 0
 activation-character 13
 transport input ssh

IOS also warned about the config it was given: the TACACS key is type 0 in the command (%SYS-4-INSECURE_WARNING ... Command: key *), and %TAC-4-NON_TLS_SERVER_CONFIGURED: TACACS+ server PL-TAC1 is configured without tls. Both are real points for a production rollout.

NTP is the other one. The servers are configured, but nothing serves time on that address:

Clock is unsynchronized, stratum 16, no reference clock

address         ref clock       st   when   poll reach  delay  offset   disp
 ~10.66.0.123     .TIME.          16      -     64     0  0.000   0.000 15937.

Reach 0. The config is correct and the device is still not synced, which is exactly why the check below tests sync separately from config.

Checking the standard read-only

Pushing config and checking config are different jobs. The check playbook only reads show running-config, show snmp user and show ntp status, and evaluates one rule per item in the standard:

    - name: Evaluate the rules
      ansible.builtin.set_fact:
        report:
          - rule: ntp-server
            ok: "{{ cfg is search('^ntp server ' ~ ntp_server ~ '$', multiline=True) }}"
          - rule: syslog-hosts
            ok: "{{ ([syslog_server] + role_syslog_servers) | map('regex_replace', '^', 'logging host ') | reject('in', cfg_lines) | list | length == 0 }}"
          - rule: http-off
            ok: "{{ cfg is not search('^ip http (secure-)?server$', multiline=True) }}"
          - rule: vty-timeout
            ok: "{{ cfg is search('^line vty 0 4\\n(?: .*\\n)*? exec-timeout ' ~ svc_vty_exec_timeout ~ ' 0$', multiline=True) }}"
          - rule: no-communities
            ok: "{{ cfg is not search('^snmp-server community ', multiline=True) }}"
          - rule: snmpv3-user
            ok: "{{ out.stdout[1] is search('User name: ' ~ svc_snmp_user ~ '$', multiline=True) }}"
          - rule: aaa-local-fallback
            ok: "{{ cfg is search('^aaa authentication login default group \\S+ local$', multiline=True) }}"
          - rule: ntp-synced
            ok: "{{ out.stdout[2] is search('Clock is synchronized') }}"
          # plus log-buffer, timestamps-localtime, login-banner and snmpv3-group
      vars:
        cfg: "{{ out.stdout[0] }}"
        cfg_lines: "{{ out.stdout[0].splitlines() }}"

    - name: Failures for this device
      ansible.builtin.debug:
        msg: "{{ report | rejectattr('ok') | map(attribute='rule') | list }}"

aaa-local-fallback is there on purpose. A device with TACACS+ and no local fallback is one server outage away from locking out everyone, including your automation.

On all six devices the only failure is the one the lab can't fix:

ok: [hq-core2] => {
    "msg": [
        "ntp-synced"
    ]
}

To make sure the rules actually bite, I broke hq-core2 by hand: turned ip http secure-server back on and removed one syslog host. The check caught both:

ok: [hq-core2] => {
    "msg": [
        "syslog-hosts",
        "http-off",
        "ntp-synced"
    ]
}

The services playbook in check mode showed exactly the two fixes, logging host 10.66.0.51 and no ip http secure-server. The real run changed 2, and the check went back to ntp-synced only.

Where to take it

• Write a separate key rotation playbook for TACACS+ and SNMP users. The "only when missing" guards mean this one will never rotate them.
• Add the SNMP ACL to the user line as well as the group, as 17.18 suggests.
• Point the devices at a real TACACS+ server and test a rejected login, command authorization and accounting before you trust the fallback behaviour.
• Run the read-only check on a schedule and alert on anything other than an empty list.

Part 21 Next: an IOS XE upgrade workflow rehearsal with pre-checks, staging, MD5 verification, post-checks and rollback. The rest of the series is on the series page.