Cisco ACL Packet Drop: Trace It with Linux Packet Captures

The client sends four echo requests. A capture at the Cisco router's ingress sees those same four requests. The server sees none, and one ACL deny counter increases by four. That combination tells you considerably more than a failed ping.

We'll create a working Linux-to-Cisco path, apply a narrowly scoped outbound ACL, and collect evidence on both sides. Then we'll remove the ACL attachment and verify recovery. This is a packet-location exercise, separate from the VLAN mismatch lab, where ARP failed before an echo request could be sent.

A small routed topology

The client and server are Linux namespaces on one VM. Containerlab connects them to a single Cisco IOL router:

• pld-client: 192.0.2.10/24, gateway 192.0.2.1.
• R1 Ethernet0/1: 192.0.2.1/24, facing the client.
• R1 Ethernet0/2: 198.51.100.1/24, facing the server.
• pld-server: 198.51.100.10/24, gateway 198.51.100.1.

Tested with: Containerlab 0.79.0, Docker 29.8.2, local image vrnetlab/cisco_iol:17.18.02 (IOS XE 17.18.2 per show version), Ubuntu 26.04.1 LTS, iproute2 6.19.0, tcpdump 4.99.6. Supply your own locally available Cisco image; device software is not part of the article.

Create the router and links

Save this topology as cisco-loss.clab.yml in a new lab directory:

name: cisco-loss
topology:
  nodes:
    r1:
      kind: cisco_iol
      image: vrnetlab/cisco_iol:17.18.02
      startup-config: r1.partial.cfg
  links:
    - endpoints: ["r1:e0/1", "host:plcsrc0"]
    - endpoints: ["r1:e0/2", "host:plcdst0"]

Save the starting config as r1.partial.cfg. The ACL is defined but initially unattached, so it cannot block this baseline:

hostname R1
!
interface Ethernet0/1
 description TO-LAB-CLIENT
 ip address 192.0.2.1 255.255.255.0
 no shutdown
!
interface Ethernet0/2
 description TO-LAB-SERVER
 ip address 198.51.100.1 255.255.255.0
 no shutdown
!
ip access-list extended LAB-DROP
 10 deny icmp host 192.0.2.10 host 198.51.100.10 echo
 20 permit ip any any

Check the image, validate the topology, and deploy it from the lab directory:

docker image inspect vrnetlab/cisco_iol:17.18.02 --format '{{.RepoTags}}'
containerlab validate -t cisco-loss.clab.yml
containerlab deploy -t cisco-loss.clab.yml
containerlab inspect interfaces -t cisco-loss.clab.yml

Containerlab creates two new host-side veth interfaces. Move those interfaces into fresh client/server namespaces, then configure them:

sudo ip netns add pld-client
sudo ip netns add pld-server
sudo ip link set plcsrc0 netns pld-client
sudo ip link set plcdst0 netns pld-server
sudo ip -n pld-client link set plcsrc0 name eth0
sudo ip -n pld-server link set plcdst0 name eth0

for ns in pld-client pld-server; do
  sudo ip -n "$ns" link set lo up
  sudo ip -n "$ns" link set eth0 up
done
sudo ip -n pld-client addr add 192.0.2.10/24 dev eth0
sudo ip -n pld-client route add default via 192.0.2.1
sudo ip -n pld-server addr add 198.51.100.10/24 dev eth0
sudo ip -n pld-server route add default via 198.51.100.1

Use only these newly created test interfaces. Moving the VM's management NIC would interrupt its normal network access. The Linux endpoints have no competing firewall policy in this fresh setup.

Prove the baseline

In another terminal, connect to the router's CLI over SSH:

ssh admin@clab-cisco-loss-r1

Log in with your lab's credentials (Containerlab's cisco_iol default is admin/admin), enter privileged mode with enable if necessary, and disable pagination:

terminal length 0
show ip interface brief
show ip route connected
show access-lists LAB-DROP

Both data interfaces should be up/up with the planned addresses, and both subnets should have connected routes. The ACL's existence is not evidence that it is applied. Check the actual interface attachment when diagnosing a policy.

From the Linux VM, warm ARP with a short ping, then run the baseline probe:

sudo ip netns exec pld-client ping -n -c 2 -W 2 198.51.100.10
sudo ip netns exec pld-client ping -n -c 4 -i 0.2 -W 2 198.51.100.10

The measured four-probe baseline returned all replies, with an average RTT of 1.304 ms. Starting with a working path avoids confusing a pre-existing route or interface problem with the fault we introduce next.

Capture at three labeled points

Open three capture terminals on the VM. Capture A is the client; B is the router's Linux-side ingress link; C is the server:

# Terminal A: client
sudo ip netns exec pld-client tcpdump --immediate-mode -ni eth0 -s 0 -w /tmp/drop-client.pcap 'icmp'

# Terminal B: Cisco container's client-facing link
sudo ip netns exec clab-cisco-loss-r1 tcpdump --immediate-mode -ni eth1 -s 0 -w /tmp/drop-router-ingress.pcap 'icmp'

# Terminal C: server
sudo ip netns exec pld-server tcpdump --immediate-mode -ni eth0 -s 0 -w /tmp/drop-server.pcap 'icmp'

For this Containerlab/IOL combination, inspect interfaces showed eth1 with alias e0/1. The Linux eth1 capture observes traffic arriving at the IOL container boundary; it is not an IOS Embedded Packet Capture. Confirm that mapping on your own node rather than assuming a container interface name equals an IOS name.

Containerlab registered the running container's network namespace as clab-cisco-loss-r1. If your version uses a different access method, inspect it before copying the middle command. The tcpdump reference covers capture filters and output files.

Attach the deliberate drop

In the R1 SSH session, apply the ACL outbound on the server-facing interface:

configure terminal
interface Ethernet0/2
 ip access-group LAB-DROP out
end
show ip interface Ethernet0/2 | include access list

The deny matches only client-to-server ICMP echo requests. The final permit prevents an unrelated implicit deny from becoming the explanation for other traffic.

Run the same four-probe ping, then read the ACL counter:

sudo ip netns exec pld-client ping -n -c 4 -i 0.2 -W 2 198.51.100.10

R1# show access-lists LAB-DROP
Extended IP access list LAB-DROP
    10 deny icmp host 192.0.2.10 host 198.51.100.10 echo (4 matches)
    20 permit ip any any

No echo replies returned. Ping also printed two Packet filtered errors from 192.0.2.1; the router sent administrative-prohibition responses for those probes. IOS rate-limits ICMP unreachables by default (ip icmp rate-limit unreachable, one per 500 ms), so with probes 200 ms apart it did not emit one error for every denied request. Count the matching requests and ACL delta rather than expecting a one-to-one error message.

Stop each capture with Ctrl+C so tcpdump flushes the files and prints its capture statistics. Filter for the particular echo requests when counting; administrative errors are ICMP too:

tshark -r - \
  -Y 'icmp.type == 8 && !(icmp.type == 3) && ip.src == 192.0.2.10 && ip.dst == 198.51.100.10' \
  -T fields -e icmp.ident -e icmp.seq < /tmp/drop-client.pcap

Repeat that command against the ingress and server files. In the measured fault case, A contained four matching requests, B contained the same four, and C contained zero. Each capture reported zero packets dropped by the kernel.

The missing server packets alone would only narrow the fault to somewhere after B. The matching deny counter, verified outbound attachment, and known single change identify the ACL as the cause in this lab. In an unknown network, also check capture filters, capture drops, return traffic, and counter deltas before drawing that conclusion. Captures on separate hosts also need timestamp alignment.

Remove the attachment and verify recovery

In the R1 SSH session, remove the interface attachment while keeping the ACL definition:

configure terminal
interface Ethernet0/2
 no ip access-group LAB-DROP out
end
show ip interface Ethernet0/2 | include access list

Repeat the same Linux ping:

sudo ip netns exec pld-client ping -n -c 4 -i 0.2 -W 2 198.51.100.10

All four replies returned, averaging 1.385 ms. Recovery captures again contained the requests at the server. Close the SSH session with exit when finished.

Apply the method to a less cooperative fault

Choose one identifiable flow, prove it leaves the sender, find the furthest capture point that sees it, and look for a matching decision or counter before the next point. Then change that one condition and repeat the original probe.

This exercise tested a deliberate ACL deny, not congestion loss or a bad cable. For controlled loss and ordering faults, use the netem lab. For a TCP transfer that retransmits, continue with retransmission, reordering, and capture artifacts. The packet-analysis hub brings together the capture tools used across those investigations. If the drop is on a Linux router instead, nftables tracing shows which chain and rule made the decision, and the conntrack and SNAT lab covers flows that pass the filter but break on translation state.