iPerf3 Latency Under Load: Measure Bufferbloat with ping

The iPerf3 receiver reported 9.56 Mbit/s through a 10 Mbit/s bottleneck. That looks healthy for a bulk transfer. During the same run, the median loaded ping RTT was 325 ms; before and after the transfer, it was 0.071 ms.

The queue explains how both observations can be true. A transfer can keep the link busy while smaller packets wait behind its traffic. This lab measures the delay and backlog together so we can identify the queue involved rather than blame the whole path.

Start with a known bottleneck

Use the Linux namespace lab. The client at 192.0.2.10 reaches server 198.51.100.10 through pl-router. We attach a Token Bucket Filter, or TBF, to the router's wan0 egress. No netem delay or loss is active in this experiment.

Tested with: Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, iproute2 6.19.0, iPerf3 3.20. Install iperf3 on the VM if needed. All endpoints share that installation because namespaces share the filesystem.

On this fresh lab interface, add the shaper:

sudo ip netns exec pl-router tc qdisc show dev wan0
sudo ip netns exec pl-router tc qdisc add dev wan0 root handle 10: tbf rate 10mbit burst 32kb latency 400ms
sudo ip netns exec pl-router tc -s qdisc show dev wan0

If the previous netem experiments left a root qdisc attached, remove that experiment's qdisc first. Do not replace a queue on a production interface to follow this tutorial.

TBF releases traffic according to available tokens and queues packets when the allowance runs out. Here, rate 10mbit creates the bottleneck, burst 32kb sets the bucket size, and latency 400ms is used to calculate the queue limit. The TBF manual explains those controls. We leave its default FIFO queue in place; attaching a different child qdisc changes the queue-limit behavior.

The impairment is directional: client-to-server data and echo requests use wan0 egress. Replies return through the LAN side. This isolates a forward-path queue rather than adding delay in both directions.

Measure the idle path first

Run an idle probe while the shaper exists but no bulk transfer is active:

sudo ip netns exec pl-client ping -n -c 30 -i 0.1 198.51.100.10

All 30 probes returned. The measured average was 0.073 ms, with a maximum of 0.136 ms. A configured shaper does not automatically add a large delay to sparse traffic: there are enough tokens and little queue backlog.

In terminal A, start a foreground server and leave it running for the test:

sudo ip netns exec pl-server iperf3 -s

For iPerf3's client/server operation and the output fields, see How to Use iPerf3. We use one TCP stream, one direction, and a fixed 15-second requested duration.

Record one continuous idle/load/recovery timeline

In terminal B, start a continuous probe before generating load:

sudo ip netns exec pl-client ping -n -D -i 0.1 -c 260 -W 2 198.51.100.10 > ping-timeline.txt

The -D timestamp is written when ping reports a received reply. It is useful for aligning observations, but it is not the request's original transmit timestamp. Allow a few seconds of idle samples before the transfer starts.

In terminal C, start the client in the background and sample the queue while it runs:

sudo -v   # cache sudo credentials so the background loop can't stall at a password prompt
sleep 3
date +%s.%N > load-start.txt
sudo ip netns exec pl-client iperf3 -c 198.51.100.10 -t 15 -J > load.json &
load_pid=$!
while kill -0 "$load_pid" 2>/dev/null; do
  date +%s.%N
  sudo ip netns exec pl-router tc -s qdisc show dev wan0
  sleep 0.5
done > queue-samples.txt
wait "$load_pid"
date +%s.%N > load-end.txt

Let the ping finish so its file includes recovery samples. Stop the foreground server with Ctrl+C when the test is complete. The measured run used these commands, except that each queue sample was saved as JSON (tc -s -j qdisc show) rather than text. All timestamps came from the same VM clock.

Read latency, throughput, and queue backlog together

Open the chart at full size.

The shaded interval starts at client launch and ends when the client process completes. The middle panel shows the client's per-interval sender rate, not per-interval delivered throughput. The receiver's whole-test average is annotated separately. Queue samples were collected while the client was running.

The continuous ping captured 260 replies from 260 probes. For phase comparison, the loaded summary excludes the first and last second of the client window; recovery samples start one second after client completion:

• Before load: 29 replies; median 0.071 ms, maximum 0.109 ms.
• During load: 134 replies in the trimmed window; median 325 ms, maximum approximately 425 ms in the displayed samples.
• After load: 65 replies after the recovery margin; median 0.071 ms, maximum 0.174 ms.

The full ping summary retained higher precision and reported a maximum RTT of 425.385 ms. The application didn't lose a probe in this run, but it waited substantially longer during the transfer.

The largest sampled queue backlog was 531,876 bytes. At the configured rate, a half-megabyte of queued data represents a substantial wait. That backlog and its rise alongside RTT are direct evidence for this particular egress queue. ICMP was not given a separate priority class.

The final qdisc excerpt was:

qdisc tbf 10: root ... rate 10Mbit burst 32Kb lat 400ms
 Sent 19677057 bytes 13289 pkt (dropped 24, overlimits 39374 requeues 0)
 backlog 0b 0p requeues 0

overlimits counts occasions when the shaper had insufficient tokens and deferred sending. It is not another packet-loss total. dropped 24 is the qdisc drop count; by the final observation, the queue had drained to zero.

Use the receiver result for delivered throughput

The whole-test iPerf3 values were 10.48 Mbit/s at the sender and 9.56 Mbit/s at the receiver, with 24 reported TCP retransmissions. The sender handed data to its socket during its measurement window; the receiver measured delivery over its own window. Those rates and windows can differ, especially when a queue holds data near the end of a short test.

Read both results from the saved JSON instead of quoting one interval:

python3 - <<'PY'
import json
with open('load.json') as f:
    result = json.load(f)
for side in ('sum_sent', 'sum_received'):
    total = result['end'][side]
    print(side, round(total['bits_per_second'] / 1e6, 2), 'Mbit/s',
          'over', round(total['seconds'], 3), 'seconds')
PY

Don't subtract sender bytes from receiver bytes and call the difference the network's packet-loss count. Correlate TCP evidence and queue counters. The retransmission comparison shows why retransmission counts alone also need context.

Verify recovery and interpret the limit

Repeat a 30-probe ping after load stops, with the shaper still attached. In the measured run, all probes returned and average RTT was 0.095 ms. That check matters: the large delay disappeared when the queue drained, before the shaper was removed.

Then restore this lab interface and check again:

sudo ip netns exec pl-router tc qdisc del dev wan0 root
sudo ip netns exec pl-client ping -n -c 20 -i 0.1 198.51.100.10

All 20 probes returned, averaging 0.039 ms. The test therefore has a healthy baseline, a measured loaded condition, and a verified return to an unloaded path.

This run demonstrates delay from a deliberately sized Linux FIFO behind a shaper. It does not establish the queue depth of your ISP, a Cisco hardware platform, or another VM. On an unfamiliar path, concurrent latency is a useful symptom; interface and queue evidence are what help locate it. Continue with Troubleshooting Slow Network Throughput for the wider diagnosis, or the QoS guide for classification and scheduling choices. If latency stays normal but large uploads stall outright, check path MTU instead with IPv6 Packet Too Big: why TCP uploads stall.