Everybody runs traceroute. Far fewer people can say what it actually sent. That gap matters, because the three probe types Linux traceroute can use (UDP, ICMP and TCP) get treated very differently by the firewalls and routers between you and the destination, and picking the wrong one is the difference between a clean four hop path and a screen full of asterisks that tells you nothing.
This article walks the same four hop path three ways, shows the packets on the wire so you can see exactly what each mode emits, and explains the two things that confuse people most: why every hop reports a different IP address than you expected, and why the final hop sometimes flickers in and out on an otherwise perfect path. Every capture below is real output from a Debian 13 host bridged into a Cisco Modeling Labs topology, and it is part of the Linux networking commands cluster.
The path under test
One Linux host, three IOS XE routers, one container at the far end. The Linux box sits on the edge subnet at 10.77.0.100 and everything past its gateway is simulated:
Note what the intermediate hops report. Hop 2 is 10.77.12.2, which is R2's ingress interface on the link from R1. It is not R2's loopback, and it is not the interface R2 forwarded the packet out of. A router builds its ICMP time exceeded message with the source address of the interface the expired packet arrived on. That single rule explains most of the "why is this IP in my traceroute" confusion in the field, and it is why a traceroute reveals the incoming side of every hop and nothing about the outgoing side.
How traceroute works, in one paragraph
Traceroute sends a probe with the IP TTL set to 1. The first router decrements the TTL to zero, drops the packet and returns an ICMP time exceeded message, which reveals its address. Traceroute then sends TTL 2, TTL 3 and so on, walking the path one router at a time. What differs between the three modes is only the probe: the TTL trick and the ICMP time exceeded replies are identical in all of them. What also differs is how traceroute knows it has arrived, and that turns out to be the interesting part.
UDP mode: the default, and the one that lies to you
With no flags, Linux traceroute sends UDP datagrams to a high, deliberately unused port, incrementing the destination port with every probe:
j@llmbits:~$ traceroute -n 10.77.3.10
traceroute to 10.77.3.10 (10.77.3.10), 30 hops max, 60 byte packets
1 10.77.0.1 4.230 ms 4.490 ms 4.591 ms
2 10.77.12.2 4.601 ms 5.309 ms 5.394 ms
3 10.77.23.3 6.210 ms 6.826 ms 7.809 ms
4 10.77.3.10 7.871 ms 7.935 ms 8.178 msHere is the same run with tcpdump watching the interface. The port ladder is the point:
j@llmbits:~$ sudo tcpdump -i ens224 -n -c 12 'udp and dst 10.77.3.10'
22:14:10.628204 IP 10.77.0.100.37088 > 10.77.3.10.33434: UDP, length 32
22:14:10.628316 IP 10.77.0.100.49215 > 10.77.3.10.33435: UDP, length 32
22:14:10.628414 IP 10.77.0.100.48440 > 10.77.3.10.33436: UDP, length 32
22:14:10.628506 IP 10.77.0.100.38569 > 10.77.3.10.33437: UDP, length 32Four probes, four destination ports starting at 33434 and counting up. The port is how traceroute matches a reply to a probe, because the ICMP error quotes the original packet header back at you. The source port varies too, which is why --sport exists if you need it pinned for a firewall rule.
And the replies:
j@llmbits:~$ sudo tcpdump -i ens224 -n -c 8 'icmp'
22:14:22.987504 IP 10.77.0.1 > 10.77.0.100: ICMP time exceeded in-transit, length 36
22:14:22.989600 IP 10.77.12.2 > 10.77.0.100: ICMP time exceeded in-transit, length 36
22:14:22.989937 IP 10.77.23.3 > 10.77.0.100: ICMP time exceeded in-transit, length 36
22:14:22.990712 IP 10.77.3.10 > 10.77.0.100: ICMP 10.77.3.10 udp port 33437 unreachable, length 68Three time exceeded messages from the routers, then something different from the destination: ICMP port unreachable. That is how UDP traceroute knows it arrived. It is not a reply to the probe in any useful sense, it is an error. The destination received a UDP datagram for a port nothing is listening on and complained, and traceroute reads the complaint as success.
The router side confirms who is generating what. R2 has sent 562 time exceeded messages and only 52 unreachables over the life of this lab:
R2#show ip traffic | section ICMP
ICMP statistics:
Sent: 0 redirects, 52 unreachable, 0 echo, 0 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
0 info reply, 562 time exceeded, 0 parameter problemWhy the last hop flickers
Run the default traceroute with name resolution on and the path suddenly looks broken:
j@llmbits:~$ traceroute 10.77.3.10
traceroute to 10.77.3.10 (10.77.3.10), 30 hops max, 60 byte packets
1 10.77.0.1 (10.77.0.1) 3.874 ms 3.876 ms 4.486 ms
2 10.77.12.2 (10.77.12.2) 5.132 ms 5.417 ms 5.444 ms
3 10.77.23.3 (10.77.23.3) 4.987 ms 4.937 ms 4.915 ms
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * 10.77.3.10 (10.77.3.10) 7.597 ms 7.452 msNothing is broken. The destination is a Linux host, and Linux rate limits the ICMP errors it generates:
/ # sysctl net.ipv4.icmp_ratelimit net.ipv4.icmp_ratemask
net.ipv4.icmp_ratelimit = 1000
net.ipv4.icmp_ratemask = 6168icmp_ratelimit is 1000 milliseconds, and icmp_ratemask is a bitmask of which ICMP types are subject to it. 6168 decimal is 0x1818, which sets bits 3, 4, 11 and 12: destination unreachable, source quench, time exceeded and parameter problem. Type 3 is destination unreachable, and code 3 of that type is port unreachable, which is exactly the message UDP traceroute needs.
So the destination will emit roughly one port unreachable per second. Traceroute fires three probes per hop back to back and expects three answers. It gets one, at best, and prints asterisks for the rest until enough time has passed. The stars at hops 4 through 8 in that run are hop 4 answering slowly, not eight dead routers.
Two ways to make it stop. Send one probe per hop instead of three, which gives the rate limiter time to recover:
j@llmbits:~$ traceroute -n -q 1 10.77.3.10
traceroute to 10.77.3.10 (10.77.3.10), 30 hops max, 60 byte packets
1 10.77.0.1 3.337 ms
2 10.77.12.2 4.940 ms
3 10.77.23.3 5.175 ms
4 10.77.3.10 5.763 msOr stop relying on ICMP errors at all, which is what the other two modes do.
ICMP mode: -I
With -I, traceroute sends ICMP echo requests instead of UDP datagrams, exactly like ping does, with the TTL walked up. It needs root or the cap_net_raw capability, because it is building raw ICMP packets:
j@llmbits:~$ sudo traceroute -I -n 10.77.3.10
traceroute to 10.77.3.10 (10.77.3.10), 30 hops max, 60 byte packets
1 10.77.0.1 2.970 ms 4.490 ms 4.530 ms
2 10.77.12.2 4.871 ms 5.332 ms 5.355 ms
3 10.77.23.3 7.332 ms 9.259 ms 9.536 ms
4 10.77.3.10 7.569 ms 7.595 ms 8.022 msThe intermediate hops behave identically. The final hop is different in kind: the destination replies with an ICMP echo reply, which is a normal answer, not an error, and echo replies are not in that rate limit mask. All three probes get answered and the flicker disappears.
ICMP mode is what Windows tracert has always done, which is why a Windows box and a Linux box tracing the same path can disagree about whether the last hop responds. They are not sending the same thing.
TCP mode: -T, the one that gets through firewalls
TCP mode sends a TCP SYN to a real service port. It is the mode to reach for when UDP and ICMP both come back as asterisks, because a firewall that drops everything else will still let a SYN through to a port it is supposed to allow:
j@llmbits:~$ sudo traceroute -T -p 80 -n 10.77.2.10
traceroute to 10.77.2.10 (10.77.2.10), 30 hops max, 60 byte packets
1 10.77.0.1 4.086 ms 3.945 ms 4.087 ms
2 10.77.12.2 6.233 ms 6.148 ms 6.134 ms
3 10.77.2.10 6.268 ms 6.400 ms 6.827 msThat is a trace to the nginx server on R2's LAN, three hops away rather than four. The destination answers with a SYN/ACK, and traceroute treats a SYN/ACK as arrival. It never completes the handshake, so nothing is logged as a connection on most servers.
The same thing to a different service, the iperf3 daemon on port 5201 at the far end of the path:
j@llmbits:~$ sudo traceroute -T -p 5201 -n 10.77.3.10
traceroute to 10.77.3.10 (10.77.3.10), 30 hops max, 60 byte packets
1 10.77.0.1 2.737 ms 3.197 ms 5.052 ms
2 10.77.12.2 6.258 ms 6.142 ms 5.965 ms
3 10.77.23.3 6.001 ms 7.447 ms 7.443 ms
4 10.77.3.10 8.085 ms 8.074 ms 8.043 msThe practical value of TCP mode is that it tests the path the application actually uses. If HTTPS is broken but ping is fine, traceroute -T -p 443 tells you whether the SYN is getting there, and a policy device that drops it will usually show up as the last hop that answers.
Choosing a mode
The flags worth memorizing
-n skips reverse DNS. Use it always unless you specifically want the PTR names, because DNS lookups add latency to the display and can make a fast path look slow.
-q sets probes per hop. The default is 3. -q 1 is faster and dodges ICMP rate limiting, at the cost of losing the min and max spread on each hop.
-f and -m set the first and last TTL. Handy when you already know the near end and only care about a segment:
j@llmbits:~$ traceroute -n -f 3 -m 5 10.77.3.10
traceroute to 10.77.3.10 (10.77.3.10), 5 hops max, 60 byte packets
3 10.77.23.3 5.050 ms 4.997 ms 5.267 ms
4 * * *
5 * * *That run also shows the rate limiter again, from the other side. Hop 4 is the destination and it had already spent its one error per second answering the previous trace, so it printed nothing.
-w sets the per probe wait in seconds. The default is 5, which is why a trace into a black hole takes forever. Drop it to 1 when you are just checking whether anything answers.
-p sets the base destination port for UDP mode, or the fixed port for TCP mode. --sport pins the source port, which is what you want when a firewall rule is keyed on it.
Reading asterisks correctly
An address on a live subnet with nothing answering to it produces a wall of stars:
j@llmbits:~$ traceroute -n -w 1 -q 1 10.77.23.99
traceroute to 10.77.23.99 (10.77.23.99), 30 hops max, 60 byte packets
1 10.77.0.1 5.366 ms
2 10.77.12.2 5.269 ms
3 *
4 *
5 *
[...]
30 *Hops 1 and 2 answered, then silence. That address sits on a subnet R2 is directly connected to, so R2 has nothing to forward the packet to. It ARPs for a host that does not exist, gets nothing, and drops the packet quietly. Nobody is at fault, and nothing further along the path is broken.
That is the general rule for asterisks. Stars that continue to the end of a trace mean traffic stopped or replies are being suppressed. Stars in the middle of a trace that resolves at the end mean one router declined to generate ICMP, which is common, deliberate and harmless. Many providers rate limit or disable ICMP generation on their core boxes, so a hop that never answers while the hops on both sides of it do is normal.
Here is a real internet path from the same host for contrast, where hops 7 through 10 show load balancing across multiple next hops within a single TTL:
j@llmbits:~$ traceroute -n 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
1 192.168.88.1 0.484 ms 0.330 ms 0.271 ms
2 142.254.183.113 9.669 ms 17.255 ms 17.105 ms
3 76.167.27.25 17.021 ms 16.956 ms 17.124 ms
4 72.129.5.22 16.262 ms 16.214 ms 16.142 ms
5 72.129.5.30 16.526 ms 16.461 ms 16.387 ms
6 72.129.13.0 16.840 ms 16.609 ms 16.504 ms
7 * 24.27.236.44 16.389 ms 66.109.3.230 17.108 ms
8 107.14.19.41 16.240 ms 66.109.5.135 16.187 ms 107.14.19.41 15.796 ms
9 141.101.72.52 22.686 ms * *
10 141.101.72.19 15.060 ms 141.101.72.100 15.628 ms 141.101.72.98 14.559 ms
11 1.1.1.1 14.586 ms 14.442 ms 14.326 msThree different addresses at hop 10 is equal cost multipath doing its job, not an error. Each of the three probes took a different physical route through the same provider. This is also why hop latencies sometimes go down as you move along a path: you are not measuring one route, you are measuring three, and the reply from a distant router can be deprioritized relative to the traffic it forwards.
Latency in a traceroute is not what you think
Each number is the round trip time for that probe to reach that hop and for the ICMP error to come back. It is not the time to traverse that link, and it is not a measure of that router's health. Generating an ICMP error is a control plane job on most hardware, handled by the CPU, while forwarding is done in silicon. A busy router will happily forward at line rate while taking 200 ms to tell you it dropped your TTL zero packet. A single high hop in the middle of an otherwise clean trace is nearly always this, and not congestion.
The rule that does hold: latency that rises at a hop and stays high for every hop after it reflects real delay on the path. That is the pattern to look for, and watching it over time rather than in one snapshot is what mtr is for.
FAQ
Why does traceroute need root for some modes but not others?
UDP mode uses an ordinary datagram socket that any user can open, and it reads the ICMP errors through that socket's error queue. ICMP and TCP modes have to construct raw packets with a chosen TTL and protocol, which requires CAP_NET_RAW. On most distributions the binary carries that capability or is used through sudo.
What is the difference between traceroute and tracepath?
tracepath is a smaller tool that needs no privileges and additionally discovers the path MTU as it goes. It has fewer knobs and no TCP mode. If you want to know where a path narrows rather than who is on it, that is the tool. See tracepath and path MTU discovery.
Why does the reverse traceroute show different addresses?
Because routing is not symmetric and because every hop reports its ingress interface. Tracing A to B reveals the incoming interfaces along the forward path; tracing B to A reveals the incoming interfaces along the return path, which may be entirely different routers. Any conclusion about a path drawn from one direction only is half a conclusion.
Does traceroute show switches and firewalls?
No. It shows devices that decrement the IP TTL, which means routers and routed firewall interfaces. Layer 2 switches, transparent firewalls, taps and most load balancers in one arm mode are invisible. In the lab path above there is a switch between the Linux host and R1, and it never appears.
The last hop never answers, but the service works. Why?
Almost always because the destination or its firewall drops UDP to unused high ports, or suppresses ICMP unreachables entirely. Switch to -T against the port the service actually listens on. If the SYN/ACK comes back, the path is fine and only the probe was wrong.
Key takeaways
- All three modes use the same TTL trick and read the same ICMP time exceeded replies. Only the probe and the arrival signal differ.
- Every hop reports the address of the interface the probe arrived on, which is why traceroute never shows you loopbacks or egress interfaces.
- UDP mode counts destination ports up from 33434 and detects arrival with an ICMP port unreachable, which Linux rate limits to one per second by default. That is why the last hop flickers.
-q 1is the cheapest fix for a flickering final hop;-Ior-Tis the real fix, because echo replies and SYN/ACKs are not ICMP errors.-T -p <port>is the mode that gets through firewalls, and the only one that tests the path your application actually uses.- Stars in the middle of a completed trace are a router declining to generate ICMP. Stars that run to the end mean traffic stopped.
- A single slow hop is usually control plane scheduling, not congestion. Only latency that rises and stays risen means anything.
- Multiple addresses at one hop is equal cost multipath, not a fault.
For a moving picture of the same path rather than one snapshot, go to mtr for continuous path monitoring, and for the MTU dimension of a path see tracepath and path MTU discovery on Linux. If the reachability question is simpler than this, the ping cluster covers it, and every tool in this series is indexed on the Linux networking commands guide.