Everything in this series so far has been run from a terminal or a Git hook. That's fine for the people who wrote the playbooks. It isn't fine for the person on the NOC shift who needs to rerun the SNMP standard on one site at 3 AM without SSH access to the control node.
That's what AWX and Semaphore UI are for: a web front end with stored credentials, job templates, run history and schedules. AWX is the bigger option, but current releases need Kubernetes. Semaphore UI is one Go binary with a SQLite database, so that's what I tried.
I ran it on the control node in my home lab against the same six IOS XE devices and the Git repo from part 24. Every run below is real, including three failed runs. One caveat up front: I drove Semaphore through its REST API, the same API its web pages call, because that gave me clean text logs to copy. I don't have screenshots of the UI in this post; the task logs are the same text the UI shows on a task's page.
Versions: Semaphore UI 2.19.17, ansible-core 2.21, cisco.ios 11.6, IOS XE 17.18.2.
Install: one binary
mkdir -p ~/semaphore && cd ~/semaphore
curl -sSL -o s.tgz https://github.com/semaphoreui/semaphore/releases/download/v2.19.17/semaphore_2.19.17_linux_amd64.tar.gz
tar xzf s.tgz semaphore && rm s.tgz
./semaphore version
2.19.17-f8cefe6-1792391818
The config file, with the three random keys replaced by placeholders:
{
"dialect": "sqlite",
"sqlite": {"host": "~/semaphore/db/semaphore.sqlite"},
"tmp_path": "~/semaphore/tmp",
"address": "127.0.0.1",
"port": ":3000",
"cookie_hash": "<32 random bytes, base64>",
"cookie_encryption": "<32 random bytes, base64>",
"access_key_encryption": "<32 random bytes, base64>",
"max_parallel_tasks": 2
}
My first attempt used "dialect": "bolt", which older guides still show. 2.19 refuses to start with it:
panic: value of field 'Dialect' is not valid: bolt (Must match regex: '^mysql|postgres|sqlite$')
access_key_encryption matters: it's the key Semaphore uses to encrypt the stored credentials in the database. Lose it and every saved secret is unreadable.
Then create the database, an admin user and start the server. The admin password comes from a file so it never appears on a command line or in this post:
./semaphore migrate --config config.json
./semaphore user add --admin --login admin --name Admin \
--email admin@example.net --password "$(cat .admin-pass)" --config config.json
. ~/ansible-venv/bin/activate
nohup env PATH="$PATH" ./semaphore server --config config.json > server.log 2>&1 &
Starting it from the activated venv is the whole Ansible integration: Semaphore runs whatever ansible-playbook is first on its PATH. It listens on 127.0.0.1 only; for anyone else to use it you'd put a reverse proxy with TLS in front.
The project: repo, inventory, key, environment, template
Semaphore's model maps cleanly onto what we already have. A repository is the Git repo. An inventory is a file inside it. A key is a stored credential. An environment is a set of environment variables. A template ties them to one playbook.
These are the calls I made, from a small Python helper that logs in with the admin password file and keeps the session cookie:
P = call("POST", "/projects", {"name": "netops"})["id"]
call("POST", f"/project/{P}/keys", {"name": "none", "type": "none", "project_id": P})
call("POST", f"/project/{P}/keys", {"name": "ansible-vault", "type": "login_password",
"project_id": P, "login_password": {"login": "", "password": "wrong-on-purpose"}})
call("POST", f"/project/{P}/repositories", {"name": "netops", "project_id": P,
"git_url": "file://~/git/netops.git", "git_branch": "main", "ssh_key_id": none_id})
call("POST", f"/project/{P}/inventory", {"name": "sites", "project_id": P, "type": "file",
"inventory": "inventory/hosts.yml", "ssh_key_id": none_id, "repository_id": rid})
call("POST", f"/project/{P}/environment", {"name": "lab", "project_id": P, "json": "{}",
"env": json.dumps({"ANSIBLE_FORCE_COLOR": "0"})})
call("POST", f"/project/{P}/templates", {"name": "SNMP location and contact", "project_id": P,
"app": "ansible", "playbook": "playbooks/snmp-location.yml", "inventory_id": iid,
"repository_id": rid, "environment_id": eid,
"vaults": [{"vault_key_id": vid, "type": "password"}]})
In the web UI each of these is its own page with a single form; the API calls are the same thing without the clicking.
The device login is not a Semaphore key. The SSH key is type none because network_cli logs in with the username and password from the inventory's vaulted group_vars. The only secret Semaphore holds is the vault password. I started it with a deliberately wrong value to see what a failure looks like.
Failed run 1: the collections weren't there
Task SNMP location and contact added to queue
Started task #1 of template 'SNMP location and contact'
Preparing: 1
Cloning Repository file://~/git/netops.git
Cloning into 'repository_1_template_1'...
Get current commit hash
Get current commit message
cloning inventory repository
Cloning Repository file://~/git/netops.git
Cloning into 'repository_1_template_1_inventory_1'...
[WARNING]: Error getting vault password file (default): The vault password file provided '~/semaphore/tmp/project_1/repository_1_template_1' can not be a directory
[ERROR]: couldn't resolve module/action 'cisco.ios.ios_command'. This often indicates a misspelling, missing collection, or incorrect module path.
Origin: ~/semaphore/tmp/project_1/repository_1_template_1/playbooks/snmp-location.yml:20:7
18
19 handlers:
20 - name: Save config
^ column 7
Failed to run task: exit status 4
Two problems in one log. The empty ANSIBLE_VAULT_PASSWORD_FILE I'd set in an early version of the environment turned into a directory path; I removed it. The real error is the second one: Semaphore's task didn't find cisco.ios, even though the same ansible-playbook finds it from my shell. The fix was to set it in the environment explicitly:
{"ANSIBLE_COLLECTIONS_PATH": "~/.ansible/collections", "ANSIBLE_FORCE_COLOR": "0"}
A cleaner long-term fix is a collections/requirements.yml in the repo. Semaphore looks for one before every run (the log lines I trimmed above are it checking four locations) and installs it if found.
The run that shouldn't have worked
The next run succeeded, all six devices ok. That was wrong: the vault key in Semaphore still held wrong-on-purpose.
The reason is the repo's ansible.cfg from part 24, which has vault_password_file pointing at a file on the control node. Semaphore runs the playbook inside its clone of the repo, so Ansible read that config, found the real password file and ignored the wrong one Semaphore passed in. Ansible tries every vault secret it's given, so one right answer is enough.
That matters for a real setup. If Semaphore runs on the same box as your CLI work, it can quietly use credentials you never put into it, and taking the key out of Semaphore wouldn't stop anything. I pointed Semaphore at its own config without the vault line, by adding ANSIBLE_CONFIG to the environment:
[defaults]
inventory = inventory/hosts.yml
host_key_checking = True
gathering = explicit
interpreter_python = auto_silent
[persistent_connection]
command_timeout = 30
connect_timeout = 30
Failed run 2: the vault key
Now the wrong key fails the way it should:
Task SNMP location and contact added to queue
Started task #3 of template 'SNMP location and contact'
Preparing: 3
Updating Repository file://~/git/netops.git
From file://~/git/netops
* branch main -> FETCH_HEAD
Already up to date.
Get current commit hash
Get current commit message
cloning inventory repository
Updating Repository file://~/git/netops.git
From file://~/git/netops
* branch main -> FETCH_HEAD
Already up to date.
PLAY [SNMP location and contact from Git] **************************************
[ERROR]: Decryption failed (no vault secrets were found that could decrypt).
Failed to run task: exit status 4
This is the failed-run view a NOC user would see: the task goes red in the history, and the log ends with the Ansible error and the exit code. Nothing reached the devices, because decryption happens before the first task.
Then I put the real vault password into the key. Updating a key through the API needs "override_secret": true; without it the call returns success and keeps the old secret, which cost me one more failed run (task 4, same error).
A good run
Task SNMP location and contact added to queue
Started task #5 of template 'SNMP location and contact'
Preparing: 5
Updating Repository file://~/git/netops.git
From file://~/git/netops
* branch main -> FETCH_HEAD
Already up to date.
Get current commit hash
Get current commit message
cloning inventory repository
Updating Repository file://~/git/netops.git
From file://~/git/netops
* branch main -> FETCH_HEAD
Already up to date.
PLAY [SNMP location and contact from Git] **************************************
TASK [Set location and contact] ************************************************
...
ok: [hq-core2]
ok: [br1-acc1]
...
TASK [Show the lines that would be sent] ***************************************
...
skipping: [hq-core2]
...
skipping: [br1-acc1]
...
PLAY RECAP *********************************************************************
br1-acc1 : ok=1 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
...
hq-core2 : ok=1 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
...
Twelve seconds, every device ok, nothing changed, because the devices already match the repo. Semaphore records the commit it ran (8c11ea1, the last commit from part 24) next to each task, so the history answers "which version of the playbook ran at 3 AM?".
A schedule
I added a second template with --check --diff as its arguments, a dry-run version a NOC user can run any time without risk, and gave it a schedule with cron format * * * * *:
call("POST", "/project/1/templates", {"name": "SNMP check (dry run)", ...,
"arguments": json.dumps(["--check", "--diff"])})
call("POST", "/project/1/schedules", {"project_id": 1, "template_id": 2,
"cron_format": "* * * * *", "name": "every minute (test)", "active": True})
It fired twice before I deleted it. The second scheduled run:
Task SNMP check (dry run) added to queue
Started task #7 of template 'SNMP check (dry run)'
Preparing: 7
Updating Repository file://~/git/netops.git
From file://~/git/netops
* branch main -> FETCH_HEAD
Already up to date.
Get current commit hash
Get current commit message
cloning inventory repository
Updating Repository file://~/git/netops.git
From file://~/git/netops
* branch main -> FETCH_HEAD
Already up to date.
PLAY [SNMP location and contact from Git] **************************************
TASK [Set location and contact] ************************************************
...
ok: [hq-core2]
ok: [br1-acc1]
...
TASK [Show the lines that would be sent] ***************************************
...
skipping: [hq-core2]
...
skipping: [br1-acc1]
...
PLAY RECAP *********************************************************************
br1-acc1 : ok=1 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
...
hq-core2 : ok=1 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
...
The task history for the project at the end, summarised from the API's task list (times converted to PDT; the API leaves the user empty for scheduled runs):
task template status start commit user
1 SNMP location error 9:49:42 8c11ea1 admin
2 SNMP location success 9:50:04 8c11ea1 admin (wrong key ignored)
3 SNMP location error 9:50:31 8c11ea1 admin
4 SNMP location error 9:50:50 8c11ea1 admin
5 SNMP location success 9:51:08 8c11ea1 admin
6 SNMP check (dry run) success 9:52:01 8c11ea1 (schedule)
7 SNMP check (dry run) success 9:53:01 8c11ea1 (schedule)
For a real deployment you'd schedule the dry-run template nightly and alert on a non-ok result, which is part 22's audit with a UI on top.
Semaphore or the Git hook?
They do different jobs. The part 24 hook decides what is allowed into main. Semaphore decides who can run what's in main, when, and keeps the log. In practice you want both: Semaphore's repository points at the branch the CI gate protects, so the UI can only ever run reviewed code.
What I didn't test: users and roles (I only used the admin), LDAP or OIDC login, and running Semaphore under systemd with TLS. Those are the next steps before anyone else uses it.
Cleanup
This was a trial, so I removed it. I stopped the server, deleted ~/semaphore (binary, SQLite database with the encrypted key, temp clones) and checked that nothing is listening on port 3000 and no cron entry exists. The schedule was deleted before that. The Git repo from part 24 is untouched.
That's the end of the planned topics for this series. The full list is on the series page.