> ## Content Index
> Fetch the complete content index at: https://www.pinglabz.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Oxidized: Automatic Cisco IOS XE Config Backups to Git, with Real Diffs
- URL: https://www.pinglabz.com/config-oxidized-git-backups/
- Published: 2026-11-03T15:59:16.000Z
- Updated: 2026-11-03T16:28:26.000Z
- Description: Oxidized 0.37 pulls IOS XE router and switch configs into Git on a schedule. A real change, the diff that caught a mistake in it, what remove_secret hides, and the limits worth knowing.
- Author: Jaime
- Tags: Grafana, Monitoring, SNMP, Cisco IOS XE, Linux, Tools

Every outage review I've sat through has the same awkward moment: someone asks "what changed?" and nobody can answer with certainty. The router config from last Tuesday is gone, the change ticket says one thing, and the box says another. Oxidized fixes that cheaply. It logs into your routers and switches on a schedule, pulls the running config, and commits it to Git. A config change turns into a diff with a timestamp.

This is the first post in a new series on open-source config and change control. Everything here was built in my home lab: two Cisco IOS XE WAN routers (wan-rtr1, wan-rtr2), one IOS XE access switch (acc-sw1), and a Linux host running Docker.

[![Lab diagram: two IOS XE routers, wan-rtr1 and wan-rtr2, and an IOS XE switch, acc-sw1, are pulled over SSH with show run by Oxidized 0.37.0, which reads a device inventory, removes secrets and polls every hour. Each change becomes a commit in a Git repo, which oxidized-web 0.18.1 shows as a node list and diffs; pushing the repo to a remote or a webhook hook is optional. An engineer can trigger an on-demand fetch with REST GET /node/next/wan-rtr1. A callout shows the v1 to v2 diff on wan-rtr1: a new EDGE-IN ACL, logging buffered 128000, and a wrong Et0/2 description that was fixed in v3. A footnote says Oxidized backs up the running config only and masks secrets as <secret hidden>.](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/diagram-topology-39.png)](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/diagram-topology-39.png?ref=pinglabz.com)

Oxidized pulls the running config over SSH every hour, hides secrets and commits each change to Git. oxidized-web shows the diffs.

## Versions and licence

- Oxidized 0.37.0 (released May 2026), container image `oxidized/oxidized:0.37.0`. Apache-2.0 licence.
- oxidized-web 0.18.1, bundled in the same image.
- Cisco IOS XE 17.18 on all three devices.

Oxidized is fully open source, with no paid tier needed for anything in this post.

## The device side: a dedicated backup user

Don't reuse your admin account. Give Oxidized its own local user so its logins stand out in AAA logs and you can pull its access without touching anyone else. I used privilege 15 because `show running-config` needs it on IOS XE unless you build a custom privilege level. In production, put it behind TACACS+ with a command-authorization set that only allows `show` commands.

```
username oxidized privilege 15 secret <OXIDIZED_PASSWORD>
```

In the lab I added this to the running config only and didn't save it. Do save it in production, or the backups stop after the next reload.

## Oxidized config

Oxidized reads a YAML config and a device list. The simplest source is a CSV-style file, one device per line: name, IP, model.

```
# router.db
wan-rtr1:10.66.20.11:ios
wan-rtr2:10.66.20.12:ios
acc-sw1:10.66.20.21:ios
```

```
---
username: oxidized
password: <OXIDIZED_PASSWORD>
model: ios
interval: 3600
timeout: 30
retries: 1
rest: 0.0.0.0:8888
vars:
  remove_secret: true
input:
  default: ssh
  ssh:
    secure: false
output:
  default: git
  git:
    user: Oxidized
    email: oxidized@example.net
    repo: /home/oxidized/.config/oxidized/configs.git
source:
  default: csv
  csv:
    file: /home/oxidized/.config/oxidized/router.db
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      ip: 1
      model: 2
```

A few of those lines matter more than they look:

- **`remove_secret: true`** masks passwords, secrets and SNMP communities in the stored config. Without it, your Git repo is a password file. More on what it does and doesn't catch below.
- **`interval: 3600`** polls every hour. You can trigger a node on demand through the REST API, which is how I got the diffs below without waiting.
- **`secure: false`** skips SSH host-key checking. That's fine for a lab and a bad idea in production; turn it on and pre-load the known hosts.
- The config file holds the device password, so lock it down: mine is in a directory with mode 700, and the password file is mode 600.

Then run the container:

```
docker run -d --name pl-oxidized --restart unless-stopped \
  -v ~/labs/oxidized/cfg:/home/oxidized/.config/oxidized \
  -p 8888:8888 \
  oxidized/oxidized:0.37.0
```

Within about a minute, all three devices showed a green status in the web UI.

![Oxidized web UI node list showing wan-rtr1, wan-rtr2 and acc-sw1 with green status](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/ox-nodes.png)

oxidized-web 0.18.1: three IOS XE devices backed up, last status green.

## Making a real change

I gave the backup something to catch. Under a made-up change ticket, CHG-1042, I made three changes on wan-rtr1:

```
ip access-list extended EDGE-IN
 10 permit tcp 10.0.0.0 0.255.255.255 any eq 22
 20 permit icmp any any echo-reply
 30 deny tcp any any eq telnet log
 40 permit ip any any
!
interface Ethernet0/2
 description WAN-B to wan-rtr2 | change CHG-1042
!
logging buffered 128000
```

Then I asked Oxidized to fetch wan-rtr1 now, instead of waiting for the hour:

```
curl -s http://<oxidized-host>:8888/node/next/wan-rtr1
```

A few seconds later there was a new commit. The diff view in oxidized-web shows exactly what moved:

![Oxidized side-by-side diff showing the new EDGE-IN ACL, the logging buffered change and a changed interface description](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/ox-diff-acl.png)

Version 1 vs version 2 of wan-rtr1: the new ACL, the logging change and the description edit.

That diff caught a real mistake. Et0/2 on wan-rtr1 faces acc-sw1, not wan-rtr2\. I'd written the wrong description. That is exactly the kind of small, wrong change that misleads someone at 2 a.m. I fixed it, triggered another fetch, and Git recorded the fix as its own version:

```
$ git --git-dir=configs.git log --format='%h %an %ad %s' --date=iso -- wan-rtr1
a40b796 Oxidized 2026-11-03 15:53:45 +0000 update /wan-rtr1
082fdcf Oxidized 2026-11-03 15:52:58 +0000 update /wan-rtr1
4fc7875 Oxidized 2026-11-03 15:51:20 +0000 update /wan-rtr1

$ git --git-dir=configs.git show a40b796 -- wan-rtr1
@@ -12,7 +12,7 @@
-! Last configuration change at 15:52:51 UTC Tue Nov 3 2026 by admin
+! Last configuration change at 15:53:37 UTC Tue Nov 3 2026 by admin
@@ -201,7 +201,7 @@ interface Ethernet0/1
 interface Ethernet0/2
- description WAN-B to wan-rtr2 | change CHG-1042
+ description to acc-sw1 | CHG-1042
  ip flow monitor AKV-MON input
  ip address 10.0.10.1 255.255.255.0
```

The repo is plain Git, so you can clone it, push it to your own Git server, or use any diff tool you like. Oxidized also has hooks to push to a remote or post to a webhook on each change.

## What remove\_secret hides, and what it doesn't

I grepped the stored config for the mask:

```
$ git --git-dir=configs.git show HEAD:wan-rtr1 | grep -n "secret hidden"
164:username admin privilege 15 secret 9 <secret hidden>
165:username oxidized privilege 15 secret 9 <secret hidden>
350:snmp-server host 10.66.20.5 vrf <MGMT-VRF> version 2c <secret hidden> udp-port 1162
```

Local user secrets and the SNMP host community got masked. But `remove_secret` is a list of regexes in the IOS model, not a guarantee. Anything it doesn't know about, like a pre-shared key in an unusual command or a free-text description with a password in it, goes into Git as written. Treat the repo as sensitive anyway: restrict access, and don't push it anywhere public.

## The noise problem

Look at the first hunk of that last diff. IOS XE writes a `! Last configuration change at ...` comment, and that line changes on every edit. That's fine here, because it only changes when the config does. The model already strips the uptime and NVRAM lines that would otherwise change on every poll, so in my lab acc-sw1 and wan-rtr2 were fetched again with no new commits. If you see a device committing every hour with no real change, look for a counter or timestamp in its output and add it to a model override.

## Limits worth knowing

- Oxidized backs up the **running** config. It won't tell you that running and startup differ, which was literally true for wan-rtr1 here because I didn't save. Pair it with a check like Zabbix's config-change trigger from the [syslog and traps post](https://www.pinglabz.com/zabbix-syslog-snmp-traps/).
- A diff tells you *what* changed and *when Oxidized saw it*, not *who* changed it. The IOS comment line names the user, but for a real audit trail use AAA accounting or syslog into something like [Loki](https://www.pinglabz.com/grafana-loki-syslog-ios-xe/).
- Hourly polling can miss a change that's made and reverted inside the hour. If that worries you, trigger a fetch from a syslog event (the `%SYS-5-CONFIG_I` message) through the REST API.

## What's next

A backup tells you what changed after the fact. The next post uses Batfish to check a change *before* it goes live, feeding it the configs Oxidized just collected.

Now live: [Batfish: catch a bad ACL and a null route before they go live](https://www.pinglabz.com/config-batfish-offline-validation/).