> ## Content Index > Fetch the complete content index at: https://www.pinglabz.com/llms.txt > Use this file to discover other available public pages before exploring further. # Cisco IOS XE Syslog into Loki with Grafana Alloy: LogQL for Flaps, Changes and Routing Events - URL: https://www.pinglabz.com/grafana-loki-syslog-ios-xe/ - Published: 2026-11-03T03:15:03.000Z - Updated: 2026-11-03T03:57:59.000Z - Description: IOS XE syslog into Loki via Grafana Alloy: the RFC 3164 parsing gotcha and the RFC 5424 fix, LogQL that finds real link flaps, config changes and OSPF/BGP events, and a logs plus metrics board. - Author: Jaime - Tags: Grafana, Monitoring, Cisco IOS XE, Linux, Tools, SNMP The [last post](https://www.pinglabz.com/grafana-prometheus-snmp-exporter-ios-xe/) polled the routers' counters with Prometheus. Counters show you that something changed. The syslog says what changed, and often who did it. This post sends IOS XE syslog into Loki through Grafana Alloy, uses LogQL to find real link flaps, config changes and OSPF/BGP events, and puts logs and metrics on the same Grafana board. Versions: Loki 3.7.9, Alloy 1.21.3, Grafana OSS 13.0.2, IOS XE 17.18.2. [![Lab diagram: wan-rtr1, wan-rtr2 and acc-sw1 send RFC 5424 syslog over UDP 1514 to Alloy 1.21.3 at 10.66.20.43, with hostname as the origin-id, archive log config and notify syslog turned on. Alloy parses the messages, labels host, severity, source, facility and mnemonic, and pushes them to Loki 3.7.9 at 10.66.20.42:3100 (TSDB schema v13, filesystem storage, 14-day retention). Grafana 13.0.2 shows the logs with LogQL next to Prometheus snmp_exporter metrics, and adds link-down annotations. A strip traces one 50-second flap of wan-rtr1 Et0/1. A callout notes that the RFC 3164 format failed Alloy's parser, so the devices were switched to RFC 5424.](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/diagram-topology-25.png)](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/diagram-topology-25.png?ref=pinglabz.com) IOS XE sends RFC 5424 syslog to Alloy, Alloy labels it and pushes to Loki, and Grafana shows the logs next to the snmp\_exporter metrics. ## Alloy, not Promtail Promtail is deprecated in favour of Alloy, and Alloy's `loki.source.syslog` listens for syslog directly, so I used Alloy. Note that it doesn't support RFC 3164 well enough for Cisco, which caught me out (see below). ## Loki: single binary, filesystem storage ``` auth_enabled: false server: http_listen_port: 3100 common: path_prefix: /loki replication_factor: 1 ring: {kvstore: {store: inmemory}} storage: filesystem: {chunks_directory: /loki/chunks, rules_directory: /loki/rules} schema_config: configs: - from: "2026-01-01" store: tsdb object_store: filesystem schema: v13 index: {prefix: index_, period: 24h} limits_config: retention_period: 336h compactor: working_directory: /loki/compactor retention_enabled: true delete_request_store: filesystem ``` That gives 14 days of retention, enforced by the compactor. For a handful of network devices this is plenty. ## The gotcha: Cisco's RFC 3164 format My first Alloy config used `syslog_format = "rfc3164"`. Nothing reached Loki, and Alloy logged this for every message: ``` level=warn msg="error parsing syslog stream" err="expecting a sequence number (from 1 to max 255 digits) [col 5]" ``` A packet capture showed why. IOS sends `<190>139: *Nov 3 02:59:57.707: %AMDP2_FE-6-...`. That has a message counter after the priority and a leading `*` on the timestamp, and it isn't strict RFC 3164\. Removing the counter (`no logging message-counter syslog`) still left `<190>: *Nov 3...`, and the error didn't go away. Alloy does have a Cisco-aware parser block, `rfc3164_cisco_components`, but in 1.21.3 Alloy refuses to load it unless you run at the experimental stability level. The clean fix is on the device. IOS XE can send RFC 5424: ``` logging source-interface Ethernet0/0 vrf logging host 10.66.20.43 vrf transport udp port 1514 logging trap informational logging trap syslog-format rfc5424 logging origin-id hostname no logging message-counter syslog ! archive log config logging enable notify syslog contenttype plaintext hidekeys ``` Three of these lines matter here: - `**logging origin-id hostname**` fills the RFC 5424 HOSTNAME field. Without it the field is `-`, and you lose the host label. - **`archive log config` with `notify syslog`** logs every configuration command as `%PARSER-5-CFGLOG_LOGGEDCMD`, with the user's name. - **`hidekeys`** keeps passwords out of those logs. The format is set per device, not per logging host, so it also applies to any other syslog server the device sends to. Check that your other receivers handle RFC 5424 before you switch. ## The Alloy pipeline ``` loki.source.syslog "network" { listener { address = "0.0.0.0:1514" protocol = "udp" syslog_format = "rfc5424" use_incoming_timestamp = false labels = { job = "network-syslog" } } relabel_rules = loki.relabel.syslog.rules forward_to = [loki.process.cisco.receiver] } loki.relabel "syslog" { forward_to = [] rule { source_labels = ["__syslog_message_hostname"] target_label = "host" } rule { source_labels = ["__syslog_message_severity"] target_label = "severity" } rule { source_labels = ["__syslog_connection_ip_address"] target_label = "source_ip" } } loki.process "cisco" { stage.replace { expression = "^(BOM)" replace = "" } stage.regex { expression = "%(?P[A-Z0-9_]+)-(?P[0-7])-(?P[A-Z0-9_]+):" } stage.labels { values = { facility = "", mnemonic = "" } } forward_to = [loki.write.local.receiver] } loki.write "local" { endpoint { url = "http://10.66.20.42:3100/loki/api/v1/push" } } ``` Notes: - **The BOM line.** IOS XE puts the literal text `BOM` at the start of each RFC 5424 message body; the capture showed `- - - - - BOM%SSH-5-SSH2_SESSION`. The replace stage strips it. - **Labels.** `facility` and `mnemonic` become labels because they have few distinct values (a few dozen on these devices), and they make volume queries cheap. Never make a label out of an interface name, a neighbour IP or a username. Extract those at query time instead. - **Timestamps.** `use_incoming_timestamp = false` stamps logs on arrival. These devices have no NTP, so their clocks aren't reliable. If yours sync to NTP, set it to true. ## Cause some real events With some ping load running across the WAN link, I made three kinds of change: - I shut wan-rtr1 Ethernet0/1, the link to wan-rtr2, waited 50 seconds and brought it back up. Then I did it again. - I shut and re-enabled Ethernet0/3 on the access switch. - Every one of those changes was a config change, so the change log filled up as well. Here's what the two routers logged for one flap, in time order: ``` wan-rtr1 %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:shutdown wan-rtr1 %BGP-5-NBR_RESET: Neighbor 10.0.12.2 reset (Interface flap) wan-rtr1 %OSPF-5-ADJCHG: Process 1, Nbr 10.255.0.2 on Ethernet0/1 from FULL to DOWN, Neighbor Down: Interface down or detached wan-rtr1 %BGP-5-ADJCHANGE: neighbor 10.0.12.2 Down Interface flap wan-rtr1 %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to down wan-rtr2 %BGP-3-NOTIFICATION: sent to neighbor 10.0.12.1 4/0 (hold time expired) 0 bytes wan-rtr2 %BGP-5-ADJCHANGE: neighbor 10.0.12.1 Down BGP Notification sent wan-rtr2 %OSPF-5-ADJCHG: Process 1, Nbr 10.255.0.1 on Ethernet0/1 from FULL to DOWN, Neighbor Down: Dead timer expired wan-rtr1 %LINK-3-UPDOWN: Interface Ethernet0/1, changed state to up wan-rtr1 %BGP-5-ADJCHANGE: neighbor 10.0.12.2 Up wan-rtr2 %OSPF-5-ADJCHG: Process 1, Nbr 10.255.0.1 on Ethernet0/1 from LOADING to FULL, Loading Done ``` The two ends of the link tell different stories. wan-rtr1 owned the shutdown, so it dropped OSPF and BGP immediately ("Interface flap"). wan-rtr2's port never went down, so it only noticed when timers ran out: the OSPF dead timer, then BGP hold time expired. If you only collect logs from one end, you get half the picture. ## LogQL searches that find real events ``` # every link state change {job="network-syslog"} |~ "LINK-3-UPDOWN|LINEPROTO-5-UPDOWN" # OSPF and BGP neighbour changes, with peer and state pulled out {job="network-syslog"} |~ "BGP-5-ADJCHANGE|OSPF-5-ADJCHG.*(DOWN|FULL)," | regexp "neighbor (?P\\S+) (?PUp|Down)" # who changed what {job="network-syslog"} |= "CFGLOG_LOGGEDCMD" | regexp "User:(?P\\S+)\\s+logged command:(?P.*)" | line_format "{{.host}} {{.user}} > {{.cmd}}" # message rate per mnemonic: spots a log storm sum by (mnemonic) (count_over_time({job="network-syslog"}[1m])) ``` ![Grafana Explore with a LogQL query showing BGP ADJCHANGE and OSPF ADJCHG events from two IOS XE routers](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/g2-explore.png) Explore: neighbour changes from both routers, with peer and state extracted as fields. The rate query found something I wasn't looking for: almost 2,000 `%AMDP2_FE-6-EXCESSCOLL` messages in 15 minutes from the ping load. That one mnemonic was the biggest source of log volume, and nobody would have spotted it reading raw logs. ## Logs and metrics on one board ![Grafana dashboard with syslog rate by mnemonic, WAN throughput, BGP sessions, link up/down logs, routing events and config changes, with red link-down annotations](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/11/g2-board.png) Log volume, WAN throughput and BGP state on top; link, routing and config-change log panels below. The red lines are link-down annotations from Loki. The board combines both data sources: - **From Loki:** stacked bars of syslog rate by mnemonic, plus three log panels (link up/down, routing events, and config changes as "host user > command"). - **From Prometheus:** [the snmp\_exporter metrics](https://www.pinglabz.com/grafana-prometheus-snmp-exporter-ios-xe/), showing WAN throughput and established BGP sessions. - **A dashboard annotation** driven by `{job="network-syslog"} |~ "LINEPROTO-5-UPDOWN.*down"`, which draws a red line on every time-series panel at each link-down. The gaps in the BGP panel line up with the annotations, and the throughput dips line up too. That's the point of the board: when a graph dips, the reason is in the log panel next to it. ## Compared with Zabbix In [the Zabbix version](https://www.pinglabz.com/zabbix-syslog-snmp-traps/), syslog went to rsyslog, and Zabbix triggers matched specific lines. That's great for "alert me on this message". Loki is better for "show me everything around 19:09 on wan-rtr1" and for volume queries across all devices. Many shops run both. Next: streaming telemetry instead of polling, and what to do when the image doesn't support it. Now live: [Telegraf, gNMI and fast SNMP polling on IOS XE](https://www.pinglabz.com/grafana-telegraf-gnmi-ios-xe/).