Ansible Network Troubleshooting: 9 Real Failures on IOS XE and How I Fixed Them

This is part 6 of the Ansible for network engineers series. Parts 1 to 5 mostly showed runs that worked. This one is the opposite. I broke a working setup nine ways against one IOS XE router, saved the exact error each time, and then fixed it. If you landed here from a search with an error message in your clipboard, use the headings.

Built in my home lab with ansible-core 2.21.5, cisco.ios 11.6.0, ansible.netcommon 8.7.1, ansible-pylibssh 1.4.0, paramiko 5.1.0 and Python 3.14, against IOS XE 17.18.2. The inventory and vaulted connection vars are the ones from part 2:

# group_vars/all/connection.yml
ansible_connection: ansible.netcommon.network_cli
ansible_network_os: cisco.ios.ios
ansible_user: netauto
ansible_password: "{{ vault_ios_password }}"
ansible_become: true
ansible_become_method: enable
ansible_become_password: "{{ vault_enable_password }}"

The test playbook runs one command on hq-core1 and prints the first lines:

---
- name: Run one show command
  hosts: hq-core1
  gather_facts: false
  tasks:
    - name: Show
      cisco.ios.ios_command:
        commands: "{{ cmd | default(\"show clock\") }}"
      register: out

    - name: First lines
      ansible.builtin.debug:
        msg: "{{ out.stdout_lines[0][:3] }}"

Every case below overrides one variable with -e so you can repeat it on your own lab.

1. Connection type ssh is not valid for this module

ansible-playbook playbooks/50-show.yml -e ansible_connection=ssh

fatal: [hq-core1]: FAILED! => {"changed": false, "msg": "Connection type ssh is not valid for this module"}

The ssh connection plugin is for servers. It logs in, copies a Python module over and runs it, which a router can't do. The cisco.ios modules expect a persistent CLI session, and they check for it before doing anything.

Fix: ansible_connection: ansible.netcommon.network_cli for CLI modules. If you see this on a host that used to work, look for a stray ansible_connection in host_vars or in an inventory line that overrides the group value. ansible-inventory --host hq-core1 prints the merged vars and shows which one won.

2. command timeout triggered

ansible-playbook playbooks/50-show.yml -e cmd="show tech-support" -e ansible_command_timeout=5

fatal: [hq-core1]: FAILED! => {"changed": false, "msg": "command timeout triggered, timeout value is 5 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."}

command_timeout is how long network_cli waits for the prompt to come back after sending one command. My ansible.cfg sets it to 30 under [persistent_connection]. Here I forced 5 seconds on a command that takes longer.

Fix: raise it for the slow task only, instead of everywhere:

    - name: Show
      cisco.ios.ios_command:
        commands: show tech-support
      vars:
        ansible_command_timeout: 120

connect_timeout is a different setting. It covers the SSH login and the persistent socket, not individual commands, so raising it does nothing for this error.

3. show tech still fails with a long timeout

This one surprised me. With the timeout at 120 seconds the timeout error went away, but the task still failed. The message was the raw output in bytes:

fatal: [hq-core1]: FAILED! => {"changed": false, "msg": "b'show tech-support\\r\\n\\r\\n------------------ show clock ------------------\\r\\n\\r\\n\\r\\n*18:58:49.484 UTC Sat Oct 31 2026 ..."}

When the message is the device output, Ansible matched one of the terminal plugin's error patterns somewhere in it. The cisco.ios terminal plugin keeps a list called terminal_stderr_re. Two entries in cisco.ios 11.6.0 are [^\r\n]+ not found and a case insensitive ERROR:. show tech contains counter lines like these:

    AAA attr apply failed    : 0          VRF name not found       : 0
CLI-Config Error: Total-Update:0, Tracking-Index:0, Time=[-]

Both match, so a counter that reads zero gets reported as a failure.

Fix: for that one task, replace the pattern list with the errors you care about:

ansible-playbook playbooks/50-show.yml -e cmd="show tech-support" -e ansible_command_timeout=120 \
  -e '{"ansible_terminal_stderr_re": [{"pattern": "% ?Invalid input"}, {"pattern": "% ?Incomplete command"}]}'

ok: [hq-core1] => {
    "msg": [
        "------------------ show clock ------------------",

Don't set this group wide. Those patterns are what make a mistyped config line fail the task instead of passing silently. For collecting show tech, a narrower set of show commands is usually a better job for Ansible anyway.

4. clear counters hangs until the timeout

ansible-playbook playbooks/52-clear.yml

fatal: [hq-core1]: FAILED! => {"changed": false, "msg": "command timeout triggered, timeout value is 30 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."}

clear counters asks Clear "show interface" counters on all interfaces [confirm]. Ansible waits for a router prompt that never comes, because the router is waiting for you.

My first fix still timed out. I'd used the full prompt text as the regex, quotes included, and it never matched. A short pattern on the part that matters worked:

    - name: Clear counters (answer the confirm prompt)
      cisco.ios.ios_command:
        commands:
          - command: clear counters
            prompt: '\[confirm\]'
            answer: "y"

ok: [hq-core1]

show interfaces Ethernet0/1 | include clearing afterwards showed Last clearing of "show interface" counters 00:09:52, about when the run finished. prompt is a regex, so the square brackets need escaping. Single quotes in YAML keep the backslashes intact.

5. failed to elevate privilege to enable mode

ansible-playbook playbooks/51-config.yml -e ansible_become_password=NotTheEnablePass

fatal: [hq-core1]: FAILED! => {"changed": false, "msg": "failed to elevate privilege to enable mode, at prompt [None] with error: For matched prompt 'b'\\nPassword: '', answer is not valid"}

The login worked. The router asked for the enable password and rejected the answer. The playbook is a one line ios_config change (ip ssh time-out 60) so this case needs config mode.

Fix: check vault_enable_password against the router's enable secret. If your user lands at privilege 15 on login (username netauto privilege 15), you can drop ansible_become entirely and this whole class of failure goes away.

6. operation requires privilege escalation

ansible-playbook playbooks/51-config.yml -e ansible_become=false

fatal: [hq-core1]: FAILED! => {"changed": false, "msg": "operation requires privilege escalation"}

Same playbook, opposite mistake. netauto logs in at the > prompt, and ios_config needs #. Show commands that work at user level will still run, which is why this often appears only when the first config task arrives.

Fix: ansible_become: true and ansible_become_method: enable. With both set, the same run returned changed=1. (I removed the line again afterwards.)

7. Vault: undefined variable, missing file, wrong password

A typo in a vaulted variable name looks like any undefined variable:

ansible-playbook playbooks/50-show.yml -e ansible_password="{{ vault_ios_pasword }}"

fatal: [hq-core1]: FAILED! => {"changed": false, "msg": "Task failed: Error processing keyword 'password': 'vault_ios_pasword' is undefined"}

The vault decrypted fine. Nothing defines vault_ios_pasword. ansible-vault view group_vars/all/vault.yml lists the names you actually have.

A password file that isn't where ansible.cfg says:

ANSIBLE_VAULT_PASSWORD_FILE=/nonexistent ansible-playbook playbooks/50-show.yml

[ERROR]: The vault password file /nonexistent was not found

A file with the wrong password:

ANSIBLE_VAULT_PASSWORD_FILE=/tmp/wv ansible-playbook playbooks/50-show.yml

[ERROR]: Decryption failed (no vault secrets were found that could decrypt).

The second surprise: passing a wrong file on the command line with --vault-password-file did not fail. The run returned ok. Ansible adds the command line file to the one in ansible.cfg and tries every secret, so the good one from the config still decrypted the vault. If you're testing whether a new password file works, override ANSIBLE_VAULT_PASSWORD_FILE instead, or you'll get a false pass.

8. libssh rejects a new host key, paramiko doesn't

network_cli picks the SSH library itself. -vvvv with ANSIBLE_LOG_PATH set shows the choice:

<10.66.10.11> autodetecting ssh_type
<10.66.10.11> ssh type is now set to libssh
<10.66.10.11> USING PYLIBSSH VERSION 1.4.0

With ansible-pylibssh installed you get libssh. I removed hq-core1 from ~/.ssh/known_hosts with ssh-keygen -R and ran the same playbook with each library. host_key_checking is True in my ansible.cfg.

fatal: [hq-core1]: FAILED! => {"changed": false, "msg": "libssh: The authenticity of host '10.66.10.11' can't be established due to 'Host is unknown: 2b:94:e1:c4:58:12:27:2c:12:98:89:d6:be:f0:fa:e6:10:82:3b:45'.\nThe ssh-rsa key fingerprint is SHA1:K5ThxFgSJywSmInWvvD65hCCO0U."}

ansible-playbook playbooks/50-show.yml -e ansible_network_cli_ssh_type=paramiko

ok: [hq-core1]

libssh refused the unknown key. paramiko connected. So switching libraries can quietly change your host key behaviour. That matters after an RMA or a regenerated key, and in a CI runner with an empty known_hosts. Fix it properly by adding the key (ssh hq-core1 once, or ssh-keyscan into known_hosts after you've verified the fingerprint), not by turning checking off.

A side note on testing this: libssh reads ~/.ssh for the real user account, not from $HOME. Pointing HOME at an empty folder didn't give me a clean known_hosts. It just broke the collection lookup.

9. Getting the persistent connection log

For anything not covered above, the useful output is in the persistent connection log, not the play output:

ANSIBLE_LOG_PATH=/tmp/ansible.log ansible-playbook playbooks/50-show.yml -vvvv

That file shows the SSH library choice, every command sent, each prompt matched and the bytes that came back. It's how I found case 3. It also contains whatever the router printed, so delete it when you're done and don't attach it to a public issue without reading it first.

What to check first

When a network playbook fails, read the msg and sort it into one of three buckets before changing anything. Connection and login (cases 1 and 8): look at ansible_connection, the SSH library and known_hosts. Prompts and timing (cases 2, 3 and 4): it's either a slow command, a prompt Ansible doesn't expect, or output that matched an error pattern. Credentials and privilege (cases 5, 6 and 7): run ansible-inventory --host to see what Ansible will actually send, and remember that extra vault password files are additive.

Next in the series: Ansible for Cisco ASA, with object groups, ACLs and NAT.