Two iPerf3 flags separate people who run throughput tests from people who understand them: -P (parallel streams) and -w (TCP window size). Together they explain the most common head-scratcher in network testing - a fast link that benchmarks slow the moment latency enters the picture. We reproduced that exact failure on a live Cisco Modeling Labs topology with 100 ms of injected round-trip time, so every number below is real. This article is part of our complete iPerf guide.
The Bandwidth-Delay Product, Quickly
TCP can only keep a window's worth of unacknowledged data in flight. Once the window is full, the sender stops and waits for ACKs, and ACKs take a round trip. So the ceiling on any single TCP flow is:
max throughput = window size / round-trip timeFlip it around and you get the bandwidth-delay product (BDP), the window you need to fill a given path:
BDP = bandwidth x RTT
35 Mbit/sec x 0.104 sec = ~3.6 Mbit = ~455 KBOur lab path carries about 35 Mbit/sec with 104 ms RTT, so a single flow needs roughly 455 KB of window to fill it. Hold that number; you are about to watch what happens when TCP has far less.
The Experiment: Same Path, Three Windows
First, the path with negligible latency (RTT around 3.6 ms). A default iPerf3 run lands at 34.6 Mbit/sec - the path's natural ceiling. Then we added 50 ms of one-way delay on the WAN link (104 ms RTT, confirmed by ping) and ran three tests.
Default window, 104 ms RTT
client:~$ iperf3 -c 10.0.20.10
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 5.00 MBytes 41.9 Mbits/sec 5 539 KBytes
[ 5] 3.00-4.00 sec 2.75 MBytes 23.1 Mbits/sec 0 318 KBytes
[ 5] 6.00-7.00 sec 2.75 MBytes 23.0 Mbits/sec 0 321 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 0.00-10.00 sec 31.4 MBytes 26.3 Mbits/sec 39 sender
[ 5] 0.00-10.11 sec 27.8 MBytes 23.0 Mbits/sec receiver
Linux autotuning grows the window past 300 KB and recovers most of the throughput: 26.3 Mbit/sec. Not bad. Watch the Cwnd column doing exactly what the BDP math predicts it must.
Forced 64 KB window, same path
client:~$ iperf3 -c 10.0.20.10 -w 64K
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 640 KBytes 5.24 Mbits/sec 0 161 KBytes
[ 5] 4.00-5.00 sec 768 KBytes 6.29 Mbits/sec 0 112 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 0.00-10.00 sec 6.88 MBytes 5.77 Mbits/sec 11 sender
[ 5] 0.00-10.10 sec 6.88 MBytes 5.71 Mbits/sec receiver
Same link, same routers, same everything - 5.77 Mbit/sec. The math checks out almost exactly: 64 KB per 104 ms round trip is about 5 Mbit/sec. Nothing is broken. TCP is simply spending most of its time waiting for ACKs. This is what legacy applications with small fixed socket buffers experience on every long-haul path, and it is why "the link is 1 Gbps but the transfer runs at 50 Mbps" is usually not a network problem at all.
Same tiny window, four parallel streams
client:~$ iperf3 -c 10.0.20.10 -w 64K -P 4
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 7.00 MBytes 5.87 Mbits/sec 0 sender
[ 7] 0.00-10.00 sec 4.50 MBytes 3.77 Mbits/sec 4 sender
[ 9] 0.00-10.00 sec 6.88 MBytes 5.77 Mbits/sec 1 sender
[ 11] 0.00-10.00 sec 6.88 MBytes 5.77 Mbits/sec 1 sender
[SUM] 0.00-10.00 sec 25.2 MBytes 21.2 Mbits/sec 6 sender
[SUM] 0.00-10.11 sec 25.2 MBytes 21.0 Mbits/sec receiver
Each stream is still window-starved at roughly 5 Mbit/sec, but four of them together deliver 21.2 Mbit/sec. Parallel streams multiply the effective in-flight data - four windows instead of one. This is exactly why browsers open multiple connections and why backup tools have a "streams" setting.
What the Comparison Tells You
26.3 Mbit/sec. Modern autotuning handles moderate BDPs on its own. If your OS is current and buffers are not capped, you rarely need -w.
5.77 Mbit/sec. A 78% collapse from a setting, not a fault. Window / RTT is a hard ceiling no amount of bandwidth fixes.
21.2 Mbit/sec. Parallelism buys back what small windows lose. If -P helps a lot, suspect per-flow limits: windows, per-flow policers, or single-flow load-balancing hashes.
How to Use -P and -w Diagnostically
Run a single stream, then -P 4, and compare:
- Single stream is slow, parallel sum is much higher: a per-flow limit. Check RTT and window sizes (BDP math above), per-flow QoS policers, or an ECMP/port-channel hash pinning one flow to one member link.
- Single and parallel both hit the same ceiling: a shared limit - the link itself, a shaper on the path, or CPU on either endpoint. Confirm the interface rate on the router while the test runs (
show interfacesand watch the output rate counters climb). - Parallel is worse than single: usually endpoint CPU. iPerf3 runs all streams in one thread until recent versions, so a saturated core caps the sum. Check with
topduring the test; iPerf2 multithreads if you need to rule this out (see iPerf2 vs iPerf3).
Two cautions with -w. First, the OS may silently cap what you request (check the "socket buffer size" line iPerf3 prints). Second, what you set with -w is a buffer size request on both ends; setting it lower than autotune would reach makes things worse, so use it to reproduce problems, not as a routine "optimization."
Key Takeaways
A single TCP flow can never exceed window divided by RTT, so know your path's BDP before judging a test. Modern autotuning usually gets close to the ceiling; small fixed windows crater throughput in exact proportion to the math, and parallel streams recover it by flying multiple windows at once. Use -P as a diagnostic fork in the road: it separates per-flow limits from shared limits in under a minute. When a path still disappoints after this analysis, packet loss is the usual suspect - pick up the trail in troubleshooting slow throughput with iPerf3, or go back to the iPerf complete guide for the full command reference.
