C9800 Debug Commands Reference: What to Run and When

C9800 Debug Commands Reference: What to Run and When

Debugging on the Catalyst 9800 is fundamentally different from older AireOS controllers. The C9800 runs IOS-XE, which means you're working with a more granular, powerful, and sometimes more verbose debugging ecosystem. This article is your reference guide for deciding which debug approach to use, how to enable it, how to interpret the output, and when to pull the plug (literally) because you're drowning in logs.

The Three Debug Approaches: Know Your Tool

Before you run anything, understand what you're about to activate. The C9800 gives you three distinct debugging methodologies, each with different use cases, overhead, and output characteristics.

The key insight: traditional debug floods your CPU and console; conditional debug lets you narrow the aperture to one MAC; radioactive traces run silently in the background and capture to a file you decode later. Choose based on your pain point, not habit.

Traditional Debug Commands: The Blunt Instrument

These are the commands you likely already know. They're powerful for lab validation and simple "is this thing working?" checks. In production, use them sparingly and in maintenance windows.

AP and Association Debugging

C9800# debug wireless client mac 
C9800# debug wireless ap mac 
C9800# debug wireless association

The debug wireless client mac variant is essential when a specific client can't join. It shows the 802.11 state machine, authentication attempts, DHCP, and key exchange steps in real time. Similarly, debug wireless ap mac traces AP-side events—link quality changes, radio state transitions, and beacon transmission anomalies.

debug wireless association enables debugging for all association attempts across all APs. This is coarser but useful for identifying patterns when multiple clients fail simultaneously.

RADIUS and Authentication

C9800# debug aaa authentication
C9800# debug aaa authorization
C9800# debug aaa accounting
C9800# debug radius

These are your bread and butter for authentication failures. debug aaa authentication shows every EAP packet, RADIUS request, and response. Look for RADIUS request timeouts, rejected credentials, or RADIUS server unreachability. debug radius is lower-level—it shows RADIUS packet-by-packet transmission, retries, and TCP/UDP socket behavior. If your RADIUS server is responding but your C9800 isn't receiving, you'll see it here.

Mobility and Roaming

C9800# debug wireless mobility mac 
C9800# debug wireless mobility roam
C9800# debug wnm

debug wireless mobility roam shows when clients roam between APs, which controller they're communicating with, and whether handoff latency is acceptable. debug wnm (Wireless Network Management, part of 802.11k/v/w) traces the client's receipt of BSS transition requests, neighbors lists, and roam trigger events.

CAPWAP and Controller-AP Communication

C9800# debug capwap events
C9800# debug capwap detail
C9800# debug capwap packet

debug capwap events is the high-level view—AP join, leave, config changes, and reboot requests. debug capwap detail adds packet loss, timer expirations, and state transitions. debug capwap packet is a firehose; it shows every CAPWAP PDU. Only use this in isolated lab scenarios with one or two APs.

DTLS and Encryption

C9800# debug dtls events
C9800# debug dtls detail

CAPWAP traffic between AP and controller is encrypted via DTLS (Datagram TLS). If APs join but then drop sporadically, or if CAPWAP packets are being silently discarded, enable debug dtls events. This shows cipher suite negotiation, certificate validation, and handshake failures. If you see "DTLS handshake timeout," your AP and controller disagree on encryption parameters.

Packet Tracing and Spanning

C9800# debug ip packet detail
C9800# debug platform packet-trace

debug platform packet-trace is a specialized tool that shows you the switching path of individual packets through the forwarding engine. It's invaluable when traffic is being dropped silently (no ACL denies, no error messages). Use it with a BPF (Berkeley Packet Filter) to isolate a single client's traffic.

Conditional Debugging: Precision in Production

This is where the C9800 shines over legacy controllers. Conditional debugging lets you say: "I only care about this one MAC address" or "only log when traffic fails to forward to this IP." The system filters in hardware or in the fast path, meaning minimal overhead on other traffic.

Setting Conditions by MAC Address

C9800# debug platform condition mac 00:11:22:33:44:55
C9800# debug wireless client mac 00:11:22:33:44:55

Once you've set the condition, subsequent debug wireless commands apply only to that MAC. This is critical in production: while you're debugging one user's connectivity, the other 2,000 users on your wireless network see no performance impact. The condition persists until you clear it:

C9800# debug platform condition reset

Conditional Debugging by IP Address

C9800# debug platform condition ip 192.168.1.50
C9800# debug ip packet detail

Use this when you know the client's IP but not its MAC, or when you're troubleshooting a specific data flow. The condition applies to debugging at the IP layer and above, including forwarding, routing, and QoS decisions.

Real Example: One Client's Failed Roam

C9800# debug platform condition mac aa:bb:cc:dd:ee:ff
C9800# debug wireless mobility mac aa:bb:cc:dd:ee:ff
C9800# debug wireless client mac aa:bb:cc:dd:ee:ff
C9800# debug aaa authentication

Now, whenever that client attempts to roam, you'll see:

1. Which AP it was connected to and why it decided to roam
2. The target AP it chose
3. 802.11 authentication exchange
4. Any RADIUS callbacks
5. Final success or failure

Everything else on the network is unaffected.

Radioactive Tracing: Capture Everything, Decode Later

Radioactive tracing (sometimes called "always-on tracing") is debug output that runs continuously in a ring buffer on the controller, automatically wrapped and recycled. You never see it in real time (it's not sent to the console), but you can collect it all at once when an issue occurs, then decode it offline with Cisco tools.

Enabling Radioactive Traces

C9800# debug wireless mac aa:bb:cc:dd:ee:ff trace brief
C9800# debug wireless mac aa:bb:cc:dd:ee:ff trace verbose
C9800# debug wireless ip 10.0.1.100 trace verbose

The brief option logs state transitions and major events. The verbose option includes every function call, every timer, every state check—useful when you need to see exactly where a process diverged. The trace runs silently; to see it, you must capture and export the trace file.

Collecting Trace Outputs

Traces are stored in the controller's internal memory. To extract them, use the Debug Bundle feature in the GUI or the CLI:

C9800# debug trace save 

Or via the Troubleshooting Dashboard in the C9800 GUI: navigate to Troubleshooting > Debug Bundle, select "Radioactive Trace Log," give it a name, and click "Create Debug Bundle." The resulting tar.gz file contains the trace in a decoded-friendly format.

Decoding Traces Offline

Once downloaded, traces are typically in binary or semi-binary format. Cisco provides the Wireless Debug Analyzer tool (available on DevNet) to visualize them. You upload the trace file and it renders the state machine, showing:

• Timeline of events for the target MAC
• State transitions with timestamps
• Message exchanges (RADIUS, DHCP, 802.11)
• Anomalies flagged in red

This is invaluable for intermittent issues: reproduce the problem at 2 AM, leave the trace running, and analyze it the next morning without waking up your network.

Trace Levels Explained

The key advantage: brief traces can run indefinitely; verbose traces will fill the ring buffer quickly. Size your choice based on how long you need to capture before the problem manifests.

Embedded Packet Capture (EPC): The Wireshark Integration

If radioactive tracing logs behavior, Embedded Packet Capture logs actual frames. EPC lets you capture layer 2 and layer 3 traffic directly from the C9800's forwarding engine, then download a standard .pcap file for analysis in Wireshark.

Starting an EPC

C9800# monitor capture buffer MYPCAP interface internal0/0/rp:0
C9800# monitor capture point ip process MYPCAP input
C9800# monitor capture point ip process MYPCAP output
C9800# monitor capture filter MYPCAP "host 192.168.1.100"
C9800# monitor capture start MYPCAP

This captures all traffic to/from 192.168.1.100 on the internal forwarding fabric. The filter syntax is standard BPF; you can filter by MAC, IP, port, protocol, etc. After a few seconds (don't let it run too long—a 1 GB pcap is unwieldy), stop the capture:

C9800# monitor capture stop MYPCAP
C9800# show monitor capture buffer MYPCAP dump

Export to a file:

C9800# show monitor capture buffer MYPCAP dump file bootflash:capture.pcap

Then download via SCP or the file manager and open in Wireshark. You'll see every frame the C9800 forwarded for that client—useful for spotting dropped DHCP packets, malformed DNS queries, or retransmissions that point to RF quality issues.

EPC Best Practices

• Always use a filter. An unfiltered capture on a live network fills memory instantly.
• Capture the moment the problem manifests, not the entire day.
• If capturing control traffic (RADIUS, CAPWAP), be aware that it's encrypted; you'll see packet lengths and timing but not contents.
• For 802.11 frames, capture on the "air" by collecting data from an AP's radio interface, not the controller's backhaul (different tool; requires special AP configuration).

Debug Commands by Category: Quick Reference

AP Bringup and CAPWAP Join

C9800# debug capwap events
C9800# debug capwap detail
C9800# debug dtls events
C9800# show ap summary
C9800# show ap config general

When an AP won't join: enable CAPWAP and DTLS debugging, then reload the AP. Watch for "DTLS handshake timeout," "AP join timeout," or "Config incompatibility." If the AP joins but drops after minutes, likely cause is DTLS renegotiation failure.

Client Connectivity (802.11 Association)

C9800# debug wireless client mac 
C9800# debug wireless association
C9800# debug aaa authentication
C9800# debug radius

Order of operations: the client sends an association request, the AP forwards it to the controller via CAPWAP, the controller validates the SSID and security policy, the client performs 802.1X, RADIUS validates the user, and finally the client gets an IP. Narrow down which step fails by running all four debugs and watching the sequence.

Roaming and Mobility

C9800# debug wireless mobility roam
C9800# debug wireless mobility mac 
C9800# debug wnm
C9800# debug ip cef drops

Add debug ip cef drops if the client roams successfully but then can't reach the gateway; it points to forwarding exceptions. Roaming issues often stem from asymmetric (stateful) firewalls on the backhaul, not the controller itself.

Data Path and Throughput Issues

C9800# debug platform packet-trace
C9800# debug ip cef
C9800# debug wireless client data 

Data path issues are tricky because the control plane (debug output) doesn't always reflect what the fast path is doing. debug platform packet-trace and EPC (packet capture) are your best bets. Look for "dropped," "no interface," or "CEF exception."

RADIUS and AAA Issues

C9800# debug aaa authentication
C9800# debug aaa authorization
C9800# debug radius
C9800# show aaa servers
C9800# show aaa method-lists

Always check show aaa servers first to confirm the RADIUS server is listed and marked "UP." If it's marked "DOWN" but the server is actually alive, your C9800 can't reach it (firewall, routing, DNS). If "UP" but still failing, use debug radius to confirm packets are arriving and check for shared-secret mismatches.

Interpreting Debug Output: What to Look For

Debug output can be cryptic. Here's a guide to common patterns and what they mean.

Example 1: Failed Association

*Jan 29 21:51:31.736: %CAPWAP5_WARN_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 RO/0: wcd: AP Event: AP Name: 9130 Mac: 04e8.409f.c320 Session-ID: 10.6.1.1[15256] 192.168.1.5[246] Disjoined Tag Modified
*Jan 29 21:52:23.634: %WAPME_TRACE_MESSAGE-2-WLC_APPMR_CRIT_MSG: Chassis 1 RO/0: Configured policy-tag wrong-tag not defined, pick up default policy-tag

Translation: An AP tagged with a policy tag that doesn't exist on the controller. The controller fell back to a default tag, but this is a misconfiguration warning. Check the AP's config vs. the controller's tag list.

Example 2: RADIUS Authentication Timeout

[aaa] [auth] req_timeout: No response from RADIUS server 10.0.1.50, retrying...
[aaa] [auth] req_timeout: No response from RADIUS server 10.0.1.50 after 3 retries

Translation: The RADIUS server at 10.0.1.50 is not responding. Check the server's status, verify the C9800 can ping it, and confirm the shared secret matches.

Example 3: Roaming Latency

*Jan 29 21:55:12.456: %WIRELESS_MOBILITY_ROAM-5-AP_ROAM: Client [aa:bb:cc:dd:ee:ff] roamed from AP [9130] to [9150] in 245 ms

Translation: Roam took 245 ms, which is acceptable (under 300 ms is good). If you see consistently > 500 ms, investigate QoS on the backhaul, RADIUS latency, or AP processing delays.

Performance Impact and Safety Warnings

Debug commands consume CPU, memory, and I/O. Running them on production controllers requires care.

Golden Rules:

1. Always use conditions (by MAC or IP) in production. Unconditional debugging affects all clients.
2. Never run debug capwap packet on a live network. Use in the lab only, with 1-2 APs.
3. Monitor your CPU and memory while debugging. If CPU exceeds 80%, stop immediately.
4. Set a mental timer. Debug commands should be short bursts, not left running overnight (unless using trace or EPC with filters).
5. Disable debugging with undebug all or no debug platform condition when done. Forgot debugs are a common cause of unexplained performance degradation.

The Debug Bundle and Offline Analysis

The C9800's GUI Troubleshooting Dashboard allows you to bundle multiple debug outputs into a single .tar.gz file for Cisco TAC or offline analysis. To create one:

1. Navigate to Troubleshooting > Debug Bundle on the C9800 GUI.
2. Name your bundle (e.g., "debug_client_9130_20260129").
3. Select which outputs to include:
   • Show tech wireless
   • Web server log
   • Core file (if available)
   • Radioactive trace log
4. Click "Create Debug Bundle."
5. Download the resulting file.

This bundle contains: all CLI output (show commands), system logs, core dumps (if a process crashed), and any active radioactive traces. It's a complete snapshot of the controller's state and is invaluable for TAC analysis.

Advanced: Conditional Debug Chains

You can layer conditions to isolate complex scenarios:

C9800# debug platform condition mac aa:bb:cc:dd:ee:ff
C9800# debug wireless client mac aa:bb:cc:dd:ee:ff
C9800# debug wireless mobility mac aa:bb:cc:dd:ee:ff
C9800# debug ip packet detail

Now you're watching a single client's 802.11 association, roaming attempts, and IP packet forwarding. Add debug radius if AAA is involved. This level of detail, filtered to one device, is typically safe even on large production networks.

Key Takeaways

• Choose the right tool: Traditional debug for lab work and quick checks; conditional debug for production; radioactive traces for intermittent issues; EPC for data path analysis.
• Always use filters in production: Narrow by MAC or IP to avoid impacting other users. Conditional debugging is your secret weapon for troubleshooting live networks without causing pain.
• Understand the overhead: debug capwap packet is a CPU killer; debug wireless client mac with a condition is safe. debug platform packet-trace is specialized but powerful for forwarding issues.
• Leverage the GUI: The Troubleshooting Dashboard, Debug Bundle feature, and file manager make debug collection and export painless. Use them to organize your logs and share with TAC.
• Decode offline: Radioactive traces and pcaps are designed for offline analysis. Capture during business hours, analyze at your leisure with the Wireless Debug Analyzer or Wireshark.
• Know the sequence: Association → RADIUS → DHCP → roaming → data forwarding. Enable debug commands in that order to isolate which step is failing.
• Clean up after yourself: Running debug commands is common; forgetting to disable them is worse. Make disabling debugging your last step, always.

This reference should be bookmarked. Print it, save it, share it with your team. The C9800 gives you incredibly granular troubleshooting capability—use these tools to stay ahead of wireless issues before they become customer outages.