> ## Content Index
> Fetch the complete content index at: https://www.pinglabz.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# tc netem: Reproduce Delay, Loss, Jitter, and Reordering
- URL: https://www.pinglabz.com/tc-netem-delay-loss-reordering/
- Published: 2026-10-13T06:25:13.000Z
- Updated: 2026-10-13T06:25:13.000Z
- Description: Add one impairment at a time to an isolated Linux path. Compare ping results, queue counters, and packet arrival order, then verify recovery.
- Author: Jaime
- Tags: Linux, Troubleshooting, Packet Analysis

A ping that takes 50 ms longer, a packet that never arrives, and a packet that arrives late are different problems. Linux `tc netem` lets you produce each one deliberately, then compare the application result with the queue counters and capture.

We'll use the [namespace network lab](https://www.pinglabz.com/linux-namespace-network-lab/), apply one impairment at a time, and verify recovery between tests. The point is to create a known fault for troubleshooting, not to assemble a random collection of `tc` flags.

## Put the impairment on the intended direction

The client is `192.0.2.10` in `pl-client`. The server is `198.51.100.10` in `pl-server`. Between them, `pl-router` bridges the client LAN and routes packets out `wan0` toward the server.

Every netem command below runs inside `pl-router` and attaches to **egress on** `wan0`. It affects packets headed toward the server. Echo replies leave the router through its LAN side, so they do not traverse that same egress queue.

Tested with: Ubuntu 26.04.1 LTS, kernel `7.0.0-38-generic`, iproute2 `6.19.0`, tcpdump `4.99.6`. Run the foundation lab first, then check the existing queue state:

```bash
sudo ip netns exec pl-router tc qdisc show dev wan0
sudo ip netns exec pl-client ping -n -c 20 -i 0.1 198.51.100.10
```

The baseline had 20 replies from 20 probes and an average RTT of `0.067 ms`. That tiny delay is a same-VM lab measurement. The probes run as root through `sudo ip netns exec`; older iputils releases reject intervals below 0.2 s for unprivileged users, so keep `sudo` if you adapt the faster pings below.

A queueing discipline, or qdisc, controls packets waiting to leave an interface. On these veth links the starting root qdisc was `noqueue`. If you already attached a shaper from another experiment, restore this lab interface before using `add`; don't overwrite an unrelated queue. The [tc-netem manual](https://man7.org/linux/man-pages/man8/tc-netem.8.html?ref=pinglabz.com) documents the parameters and limitations.

## Add 50 ms of one-way delay

Attach netem to the router's server-facing interface:

```bash
sudo ip netns exec pl-router tc qdisc add dev wan0 root handle 10: netem delay 50ms
sudo ip netns exec pl-client ping -n -c 20 -i 0.1 198.51.100.10
sudo ip netns exec pl-router tc -s qdisc show dev wan0
```

The measured ping summary was:

```text
20 packets transmitted, 20 received, 0% packet loss
rtt min/avg/max/mdev = 50.177/50.202/50.261/0.016 ms
```

The RTT increased by approximately 50 ms, not 100 ms. Only the request crossed the delayed egress queue. Delaying both directions by 50 ms would add roughly 100 ms to the round trip.

The qdisc reported `delay 50ms` and zero dropped packets. A latency symptom alone would not tell you which direction was delayed; the placement and queue state establish that here.

Remove this lab qdisc before the next case:

```bash
sudo ip netns exec pl-router tc qdisc del dev wan0 root
```

This is the reset for a deliberately changed test interface. Keep each experiment's qdisc output before resetting its counters.

## Add variation around the delay

Use a 50 ms base with a 10 ms jitter parameter and normal distribution:

```bash
sudo ip netns exec pl-router tc qdisc add dev wan0 root handle 10: netem delay 50ms 10ms distribution normal
sudo ip netns exec pl-client ping -n -c 40 -i 0.05 198.51.100.10
sudo ip netns exec pl-router tc -s qdisc show dev wan0
```

All 40 probes returned. Their RTTs ranged from `19.308` to `72.002 ms`, with an average of `47.855 ms` and mdev of `9.208 ms`.

The `10ms` argument is not a promise that every result will fall between 40 and 60 ms. Read the sampled distribution rather than treating the configuration as a measured bound. This case uses random delay variation; another run will produce different samples. Timer resolution and VM scheduling also affect the observation.

Large delay variation can change arrival order when a later packet gets a shorter delay than an earlier one. If you need an experiment that isolates order, use the explicit reordering case below rather than assuming jitter preserves sequence.

```bash
sudo ip netns exec pl-router tc qdisc del dev wan0 root
```

## Inject loss and read the counter

This run uses independent random loss with a fixed seed supported by the installed iproute2 version:

```bash
sudo ip netns exec pl-router tc qdisc add dev wan0 root handle 10: netem loss random 10% seed 42
sudo ip netns exec pl-client ping -n -c 100 -i 0.02 -W 2 198.51.100.10
sudo ip netns exec pl-router tc -s qdisc show dev wan0
```

The actual sample contained 12 lost probes:

```text
100 packets transmitted, 88 received, 12% packet loss

qdisc netem 10: root ... limit 1000 loss 10% seed 42
 Sent 8694 bytes 89 pkt (dropped 12, overlimits 0 requeues 0)
```

Ten percent is the configured probability, not a quota forcing exactly ten drops in one hundred packets. The seed makes the pseudo-random sequence repeatable under equivalent inputs, but changes in background ARP or packet order can change which probe consumes which draw. The queue counts packets passing through it, which can include ARP as well as the ICMP probes.

An application loss figure and a queue drop figure answer different questions. Here we know where the fault was introduced, and the counter supports that diagnosis. On an unknown path, missing replies could also come from return-path loss or an endpoint that does not answer. The [Linux-to-Cisco drop investigation](https://www.pinglabz.com/linux-cisco-packet-drop-lab/) shows how to narrow that location with additional evidence.

```bash
sudo ip netns exec pl-router tc qdisc del dev wan0 root
```

## Make later packets overtake earlier packets

Open a second terminal on the VM and capture arriving echo requests at the server:

```bash
sudo ip netns exec pl-server tcpdump --immediate-mode -ni eth0 -s 0 -w /tmp/netem-reorder.pcap 'icmp'
```

In the first terminal, delay most packets by 50 ms but let every fifth packet bypass that delay:

```bash
sudo ip netns exec pl-router tc qdisc add dev wan0 root handle 10: netem delay 50ms reorder 100% gap 5
sudo ip netns exec pl-client ping -n -c 40 -i 0.01 -W 2 198.51.100.10
sudo ip netns exec pl-router tc -s qdisc show dev wan0
```

Stop the capture with Ctrl+C. With probes about 10 ms apart, a bypassed packet can reach the server before preceding delayed packets. All 40 probes received replies, and netem recorded zero drops. The ping RTTs ranged from `0.057` to `50.244 ms`: bypassed requests were fast, while most requests waited.

Inspect the server's arrival order with TShark. On Ubuntu, install it first with `sudo apt-get install tshark` if it is missing:

```bash
tshark -r - -Y 'icmp.type == 8' -T fields \
  -e frame.time_relative -e icmp.seq < /tmp/netem-reorder.pcap
```

The redirection feeds the capture through standard input. This also worked on the tested Ubuntu system when its TShark AppArmor profile restricted direct reads of the lab directory. There is no need to disable that profile.

Use the `icmp.seq` values in capture order to prove overtaking. A repeat capture began `5, 1, 2, 3, 4`: request 5 reached the server before requests 1 through 4\. A mixed list of fast and slow ping replies is a clue, but a capture at the receiver gives you the arrival evidence.

## Restore the baseline before another test

Remove netem and repeat the original probe:

```bash
sudo ip netns exec pl-router tc qdisc del dev wan0 root
sudo ip netns exec pl-router tc qdisc show dev wan0
sudo ip netns exec pl-client ping -n -c 20 -i 0.1 198.51.100.10
```

The recovery run returned all 20 replies with an average RTT of `0.065 ms`. That verifies the deliberately added impairment is gone.

Netem is useful because the cause is known. Keep the baseline, changed state, queue statistics, capture point, and recovery result together. For a controlled TCP comparison on this same path, continue with [retransmission, reordering, and capture artifacts](https://www.pinglabz.com/tcp-retransmission-reordering-capture-artifact/). For capture syntax, use the existing [tcpdump reference](https://www.pinglabz.com/tcpdump-for-network-engineers/). If small probes pass but larger transfers stall, suspect path MTU rather than random loss; [IPv6 Packet Too Big: why TCP uploads stall](https://www.pinglabz.com/ipv6-packet-too-big-tcp-stall-lab/) walks through that case.