tc netem: Reproduce Delay, Loss, Jitter, and Reordering

A ping that takes 50 ms longer, a packet that never arrives, and a packet that arrives late are different problems. Linux tc netem lets you produce each one deliberately, then compare the application result with the queue counters and capture.

We'll use the namespace network lab, apply one impairment at a time, and verify recovery between tests. The point is to create a known fault for troubleshooting, not to assemble a random collection of tc flags.

Put the impairment on the intended direction

The client is 192.0.2.10 in pl-client. The server is 198.51.100.10 in pl-server. Between them, pl-router bridges the client LAN and routes packets out wan0 toward the server.

Every netem command below runs inside pl-router and attaches to egress on wan0. It affects packets headed toward the server. Echo replies leave the router through its LAN side, so they do not traverse that same egress queue.

Tested with: Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, iproute2 6.19.0, tcpdump 4.99.6. Run the foundation lab first, then check the existing queue state:

sudo ip netns exec pl-router tc qdisc show dev wan0
sudo ip netns exec pl-client ping -n -c 20 -i 0.1 198.51.100.10

The baseline had 20 replies from 20 probes and an average RTT of 0.067 ms. That tiny delay is a same-VM lab measurement. The probes run as root through sudo ip netns exec; older iputils releases reject intervals below 0.2 s for unprivileged users, so keep sudo if you adapt the faster pings below.

A queueing discipline, or qdisc, controls packets waiting to leave an interface. On these veth links the starting root qdisc was noqueue. If you already attached a shaper from another experiment, restore this lab interface before using add; don't overwrite an unrelated queue. The tc-netem manual documents the parameters and limitations.

Add 50 ms of one-way delay

Attach netem to the router's server-facing interface:

sudo ip netns exec pl-router tc qdisc add dev wan0 root handle 10: netem delay 50ms
sudo ip netns exec pl-client ping -n -c 20 -i 0.1 198.51.100.10
sudo ip netns exec pl-router tc -s qdisc show dev wan0

The measured ping summary was:

20 packets transmitted, 20 received, 0% packet loss
rtt min/avg/max/mdev = 50.177/50.202/50.261/0.016 ms

The RTT increased by approximately 50 ms, not 100 ms. Only the request crossed the delayed egress queue. Delaying both directions by 50 ms would add roughly 100 ms to the round trip.

The qdisc reported delay 50ms and zero dropped packets. A latency symptom alone would not tell you which direction was delayed; the placement and queue state establish that here.

Remove this lab qdisc before the next case:

sudo ip netns exec pl-router tc qdisc del dev wan0 root

This is the reset for a deliberately changed test interface. Keep each experiment's qdisc output before resetting its counters.

Add variation around the delay

Use a 50 ms base with a 10 ms jitter parameter and normal distribution:

sudo ip netns exec pl-router tc qdisc add dev wan0 root handle 10: netem delay 50ms 10ms distribution normal
sudo ip netns exec pl-client ping -n -c 40 -i 0.05 198.51.100.10
sudo ip netns exec pl-router tc -s qdisc show dev wan0

All 40 probes returned. Their RTTs ranged from 19.308 to 72.002 ms, with an average of 47.855 ms and mdev of 9.208 ms.

The 10ms argument is not a promise that every result will fall between 40 and 60 ms. Read the sampled distribution rather than treating the configuration as a measured bound. This case uses random delay variation; another run will produce different samples. Timer resolution and VM scheduling also affect the observation.

Large delay variation can change arrival order when a later packet gets a shorter delay than an earlier one. If you need an experiment that isolates order, use the explicit reordering case below rather than assuming jitter preserves sequence.

sudo ip netns exec pl-router tc qdisc del dev wan0 root

Inject loss and read the counter

This run uses independent random loss with a fixed seed supported by the installed iproute2 version:

sudo ip netns exec pl-router tc qdisc add dev wan0 root handle 10: netem loss random 10% seed 42
sudo ip netns exec pl-client ping -n -c 100 -i 0.02 -W 2 198.51.100.10
sudo ip netns exec pl-router tc -s qdisc show dev wan0

The actual sample contained 12 lost probes:

100 packets transmitted, 88 received, 12% packet loss

qdisc netem 10: root ... limit 1000 loss 10% seed 42
 Sent 8694 bytes 89 pkt (dropped 12, overlimits 0 requeues 0)

Ten percent is the configured probability, not a quota forcing exactly ten drops in one hundred packets. The seed makes the pseudo-random sequence repeatable under equivalent inputs, but changes in background ARP or packet order can change which probe consumes which draw. The queue counts packets passing through it, which can include ARP as well as the ICMP probes.

An application loss figure and a queue drop figure answer different questions. Here we know where the fault was introduced, and the counter supports that diagnosis. On an unknown path, missing replies could also come from return-path loss or an endpoint that does not answer. The Linux-to-Cisco drop investigation shows how to narrow that location with additional evidence.

sudo ip netns exec pl-router tc qdisc del dev wan0 root

Make later packets overtake earlier packets

Open a second terminal on the VM and capture arriving echo requests at the server:

sudo ip netns exec pl-server tcpdump --immediate-mode -ni eth0 -s 0 -w /tmp/netem-reorder.pcap 'icmp'

In the first terminal, delay most packets by 50 ms but let every fifth packet bypass that delay:

sudo ip netns exec pl-router tc qdisc add dev wan0 root handle 10: netem delay 50ms reorder 100% gap 5
sudo ip netns exec pl-client ping -n -c 40 -i 0.01 -W 2 198.51.100.10
sudo ip netns exec pl-router tc -s qdisc show dev wan0

Stop the capture with Ctrl+C. With probes about 10 ms apart, a bypassed packet can reach the server before preceding delayed packets. All 40 probes received replies, and netem recorded zero drops. The ping RTTs ranged from 0.057 to 50.244 ms: bypassed requests were fast, while most requests waited.

Inspect the server's arrival order with TShark. On Ubuntu, install it first with sudo apt-get install tshark if it is missing:

tshark -r - -Y 'icmp.type == 8' -T fields \
  -e frame.time_relative -e icmp.seq < /tmp/netem-reorder.pcap

The redirection feeds the capture through standard input. This also worked on the tested Ubuntu system when its TShark AppArmor profile restricted direct reads of the lab directory. There is no need to disable that profile.

Use the icmp.seq values in capture order to prove overtaking. A repeat capture began 5, 1, 2, 3, 4: request 5 reached the server before requests 1 through 4. A mixed list of fast and slow ping replies is a clue, but a capture at the receiver gives you the arrival evidence.

Restore the baseline before another test

Remove netem and repeat the original probe:

sudo ip netns exec pl-router tc qdisc del dev wan0 root
sudo ip netns exec pl-router tc qdisc show dev wan0
sudo ip netns exec pl-client ping -n -c 20 -i 0.1 198.51.100.10

The recovery run returned all 20 replies with an average RTT of 0.065 ms. That verifies the deliberately added impairment is gone.

Netem is useful because the cause is known. Keep the baseline, changed state, queue statistics, capture point, and recovery result together. For a controlled TCP comparison on this same path, continue with retransmission, reordering, and capture artifacts. For capture syntax, use the existing tcpdump reference. If small probes pass but larger transfers stall, suspect path MTU rather than random loss; IPv6 Packet Too Big: why TCP uploads stall walks through that case.