There is a particular kind of fear that stops engineers from using the most powerful troubleshooting tool on a Cisco router. You type debug ip packet, the console erupts, the CPU pins at 100 percent, the box stops forwarding, and someone asks why the site went down. After that happens once, "never debug on a production router" becomes received wisdom.
It is the wrong lesson. The problem was never debugging; it was unfiltered debugging. Conditional debug lets you ask the router a narrow question and get a narrow answer, and this article proves it works by running two pings and showing you that only one of them appears in the output. It extends the IP Services cluster guide.
Why Debug Melts the Box
Understand the mechanism and the fix becomes obvious.
Normal forwarding happens in CEF, in the fast path, and never troubles the CPU. When you enable debug ip packet, the router must inspect and report on packets, which means punting them to the CPU for process switching. Two things then compound:
- Every packet now costs CPU cycles instead of being switched in hardware or optimised code.
- Every packet generates a log message, which itself costs CPU, and if console logging is on, that message is written to a 9600-baud console port. The console becomes the bottleneck, and the router blocks on it.
On a router forwarding 50,000 packets per second, that is 50,000 punts and 50,000 log messages per second. The CPU has no chance. This is a self-inflicted denial of service, and the router will not stop to warn you.
The fix is not to avoid debug. The fix is to make the router only punt and report on the handful of packets you actually care about.
First: The Safety Net
Before you enable any debug on a box that matters, three things:
R1(config)# no logging console
R1(config)# logging buffered 64000 debugging
R1(config)# service timestamps debug datetime msecno logging console is the single most important line here. It sends debug output to a memory buffer instead of the console port, which removes the 9600-baud bottleneck entirely. You then read the buffer at your leisure with show logging. Many production outages attributed to "debug" were really attributed to console logging.
service timestamps debug datetime msec gives millisecond timestamps, which you will want the moment you are trying to work out the order of events.
And know your escape hatch before you need it: undebug all (or u all, which you can type in under a second while panicking).
The Main Technique: Condition on an ACL
This is the one to reach for first, and the one most people never learn. debug ip packet accepts an access list, and only packets permitted by that ACL are debugged.
R1(config)# access-list 99 permit 10.0.10.100
R1# debug ip packet 99
IP packet debugging is on for access list 99The router now punts and reports on packets matching ACL 99, and leaves everything else in the fast path. On a busy router that is the difference between a hundred messages and a hundred thousand.
Proving it actually filters
Configuration is a claim; here is the evidence. With the debug above running, the lab sends two pings from the same interface, at the same time, to two different hosts on the same subnet. Only one of them matches ACL 99.
R1#ping 10.0.10.100 source Ethernet0/2 repeat 2
Sending 2, 100-byte ICMP Echos to 10.0.10.100, timeout is 2 seconds:
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 2/2/3 ms
R1#ping 10.0.10.10 source Ethernet0/2 repeat 2
Sending 2, 100-byte ICMP Echos to 10.0.10.10, timeout is 2 seconds:
.!
Success rate is 50 percent (1/2), round-trip min/avg/max = 3/3/3 msBoth pings ran. Both generated traffic on the same interface. Now the debug output:
R1#show logging | include IP:
Jul 12 00:57:56.702: IP: s=10.0.10.100 (Ethernet0/2), d=10.0.10.1 (nil), len 100, input feature, Input-Flexible-NetFlow(20), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Jul 12 00:57:56.702: IP: s=10.0.10.100 (Ethernet0/2), d=10.0.10.1 (nil), len 100, input feature, MCI Check(110), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Jul 12 00:57:56.702: IP: s=10.0.10.100 (Ethernet0/2), d=10.0.10.1 (nil), len 100, rcvd 2
Jul 12 00:57:56.702: IP: s=10.0.10.100 (Ethernet0/2), d=10.0.10.1 (nil), len 100, stop process pak for forus packet
Jul 12 00:57:56.702: IP: tableid=0, s=10.0.10.100 (Ethernet0/2), d=10.0.10.1 (Ethernet0/2) nexthop=10.0.10.1, routed via RIB
Jul 12 00:57:56.705: IP: s=10.0.10.100 (Ethernet0/2), d=10.0.10.1 (nil), len 100, rcvd 2
Jul 12 00:57:56.705: IP: tableid=0, s=10.0.10.100 (Ethernet0/2), d=10.0.10.1 (Ethernet0/2) nexthop=10.0.10.1, routed via RIBEvery single line is s=10.0.10.100. There is not one line about 10.0.10.10, despite that traffic crossing the identical interface at the identical moment. The ACL filtered it out before it ever became a debug message, which means those packets were never punted and never cost the CPU anything.
That is the whole technique, demonstrated. The router answered a narrow question with a narrow answer.
Reading the output
The line itself is dense but rewards attention:
s= / d=Source and destination, with the interface the packet arrived on in parentheses.rcvd 2Receive code. 2 means "for us, and we are going to process it."routed via RIBThe forwarding decision, and which table made it (tableid=0 is the global table; a VRF shows a different id).input feature, ...The feature chain each packet traverses, in order.That feature chain is quietly one of the most useful things in the output. Look at what the router volunteered: Input-Flexible-NetFlow(20). The debug is telling us this interface has a NetFlow monitor applied, and showing us exactly where in the packet's journey it sits. When you are debugging why a packet is being dropped or mangled, the feature chain tells you which feature to blame, and in what order they ran.
The One Thing That Will Confuse You
You will sometimes enable debug ip packet with a perfectly good ACL, send traffic you know is crossing the router, and see nothing at all. This is not a broken debug. It is CEF.
debug ip packet only sees process-switched packets. Traffic that is CEF-switched in the fast path never touches the CPU and therefore never generates a debug message. Packets destined to the router (like the pings above, which are addressed to the router's own interface) are process-switched and show up. Packets merely transiting the router are CEF-switched and do not.
Notice the lab output says forus FALSE ... stop process pak for forus packet, which is the router narrating this very distinction.
The classic workaround is no ip route-cache on the interface to force process switching. Do not do this on a production router. You have just disabled hardware forwarding, and you will cause the exact outage you were trying to avoid. If you need to see transit traffic, use a packet capture instead: SPAN or ERSPAN, or the Embedded Packet Capture feature below.
Conditional Debug for Protocol Debugs
The ACL trick works for debug ip packet. For protocol debugs (BGP, OSPF, DHCP, and so on), the tool is debug condition, which applies a global filter to any debug that is condition-aware.
R1#debug condition interface Ethernet0/2
Condition 1 set
R1#show debug condition
Condition 1: interface Et0/2 (1 flags triggered)
Flags: Et0/2With that condition set, condition-aware debugs will only report events on Ethernet0/2. The available conditions include:
debug condition interface <intf>debug condition ip <address>debug condition vrf <name>(invaluable on an MPLS L3VPN PE with hundreds of customers)debug condition username <user>debug condition standby <intf> <group>
Clear them with no debug condition all, and always check show debug condition before you conclude a debug is broken. A leftover condition from last week is a superb way to convince yourself a protocol is dead when it is merely being filtered.
An honest caveat: not every debug honours conditions. Condition support is per-debug, and the coverage is inconsistent. If you set a condition and still get a firehose, that debug is not condition-aware and you need a different approach.
The Modern Answer: Embedded Packet Capture
On current IOS XE, there is a better tool for most "what is happening to this packet" questions, and it does not punt anything to the CPU:
R1# monitor capture CAP interface Ethernet0/2 both
R1# monitor capture CAP match ipv4 host 10.0.10.100 any
R1# monitor capture CAP buffer size 10
R1# monitor capture CAP start
...
R1# monitor capture CAP stop
R1# show monitor capture CAP buffer brief
R1# monitor capture CAP export flash:cap.pcapEPC captures into a memory buffer, sees CEF-switched traffic (unlike debug ip packet), and exports a real pcap you can open in Wireshark. For packet-level questions on a modern box, reach for this first. Debug remains the right tool when you want to see the router's decision-making (why it chose that next hop, why it rejected that BGP update) rather than the packets themselves.
The Discipline
no logging console + logging buffered. Non-negotiable.show processes cpu sorted | ex 0.00 in another window. If it climbs, u all.show debug should be empty when you leave.FAQ
Why does my debug show nothing even though traffic is flowing?
CEF. debug ip packet only sees process-switched packets, and transit traffic is CEF-switched. Use Embedded Packet Capture or SPAN instead.
Is it ever safe to run debug on a production router?
Yes, with an ACL or condition, console logging off, and a finger on u all. The blanket "never debug in production" rule costs more outages than it prevents, because it pushes people toward guessing.
How do I stop everything immediately?
undebug all, abbreviated u all. Learn to type it without thinking.
Does the ACL in debug ip packet 99 also drop traffic?
No. It is used purely as a match filter for the debug. It is not applied to any interface and does not affect forwarding.
Why did I get output for one interface when my condition said another?
That debug is probably not condition-aware. Condition support varies by debug. Verify with show debug condition and, if in doubt, use the ACL method.
Key Takeaways
- Debug melts routers because it punts packets to the CPU and writes to a slow console, not because debugging is inherently dangerous.
no logging console+logging bufferedremoves most of the risk before you type anything else.debug ip packet <acl>is the workhorse. In the lab, two simultaneous pings produced debug output for only the ACL-matched source. Zero lines for the other.- The debug output names the feature chain each packet traverses (
Input-Flexible-NetFlow,MCI Check), which tells you which feature to blame and in what order they ran. debug ip packetcannot see CEF-switched transit traffic. Do not disable route-cache in production to work around this.debug conditionfilters protocol debugs by interface, IP, VRF, or user. Not every debug honours it.- On modern IOS XE, Embedded Packet Capture is usually the better tool: it sees CEF traffic, does not punt, and exports a pcap.
- A forgotten debug is a scheduled outage.
show debugbefore you log off.
Next: the IP Services cluster guide.
