Every tool in this cluster is, underneath, a formatter. ifconfig reads a file. netstat -s reads a file. nload, ifstat and bmon read the same file as each other. When those tools disagree, or when the one you want is not installed on the box in front of you, knowing where the numbers actually live turns a dead end into a one-line answer.
This article is part of the Linux networking commands guide. All output is from a Debian 13 host at 10.77.0.100 bridged into a CML lab, captured while it moved real traffic to servers at 10.77.2.10 and 10.77.3.10.
What is in there
j@llmbits:~$ ls /proc/net | wc -l
57
j@llmbits:~$ ls /proc/net | head -12
anycast6
arp
connector
dev
dev_mcast
dev_snmp6
fib_trie
fib_triestat
hci
icmp
icmp6
if_inet6Two things to know before you read any of it.
First, these are not files. They are kernel data structures rendered as text at the moment you read them. There is no disk involved and no caching, which is why cat /proc/net/dev twice in a row gives different numbers.
Second, and this catches people constantly: /proc/net is per network namespace. Read it inside a container and you get that container's counters, not the host's. This is exactly why netstat run inside a pod shows an almost empty socket table while the host is busy. It is not broken; it is looking at a different namespace.
/proc/net/dev: the file behind half the tooling
j@llmbits:~$ cat /proc/net/dev
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
lo: 3808252 64631 0 0 0 0 0 0 3808252 64631 0 0 0 0 0 0
ens224: 75028297 803129 0 0 0 0 0 28 2185022768 527116 0 0 0 0 0 0Sixteen counters per interface. Now the same numbers through the modern command:
j@llmbits:~$ ip -s link show ens224
RX: bytes packets errors dropped missed mcast
75028297 803129 0 0 0 28
TX: bytes packets errors dropped carrier collsns
2185022768 527116 0 0 0 0Identical, to the byte. ifconfig, ip -s link, nload, ifstat and bmon all read this and divide by elapsed time. That is the whole mechanism behind the live bandwidth monitors, and it explains why the counter-based ones agree with each other so closely while the packet-capture-based ones drift slightly.
The practical use is that you can build a rate meter with no tools at all:
j@llmbits:~$ A=$(awk '/ens224:/{print $10}' /proc/net/dev); sleep 5
j@llmbits:~$ B=$(awk '/ens224:/{print $10}' /proc/net/dev); echo $(( (B-A)*8/5/1000000 )) Mbit/sField 10 is transmitted bytes. That is the entire algorithm every one of those monitors implements.
/proc/net/snmp: the counters netstat -s formats
j@llmbits:~$ grep -A1 '^Tcp:' /proc/net/snmp
Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts InCsumErrors
Tcp: 1 200 120000 -1 11296 10614 490 30 1 976175 1662308 9438 0 57 0The format is a header line naming the fields and a value line in the same order. It is genuinely awkward to read by eye and trivial to parse in awk, which is the point: this file is an interface, not a report. Here, 1,662,308 segments out against 9,438 retransmitted is a retransmit rate of about 0.57 percent for the life of the host.
The counters are live. Snapshot them around 20 pings:
before: InMsgs=2338 InEchoReps=522 OutMsgs=1894
after : InMsgs=2358 InEchoReps=542 OutMsgs=1914Exactly 20 on each counter. That precision is what makes these files worth reaching for: you can prove a packet was received by the stack, not merely that a tool claims it was.
The Forwarding=2 trap
This one has cost people real time. The first field of the Ip line looks like a boolean and is not:
j@llmbits:~$ sysctl -n net.ipv4.ip_forward
0
j@llmbits:~$ grep '^Ip: [0-9]' /proc/net/snmp | awk '{print "snmp Ip Forwarding field =", $2}'
snmp Ip Forwarding field = 2The sysctl says forwarding is off. The file says 2. Both are correct, because /proc/net/snmp reports the SNMP MIB value of ipForwarding, where 1 means forwarding and 2 means not-forwarding. Any script that treats that field as true or false gets the answer backwards. Read /proc/sys/net/ipv4/ip_forward if you want a boolean.
/proc/net/netstat: where the interesting TCP counters hide
The extended counters, the ones that actually diagnose things, are in a separate file under the TcpExt prefix:
j@llmbits:~$ awk 'NR==1{for(i=1;i<=NF;i++)h[i]=$i} NR==2{for(i=1;i<=NF;i++) if(h[i]~/Retrans/) print h[i]"="$i}' /proc/net/netstat
TCPLostRetransmit=111
TCPFastRetrans=9348
TCPSlowStartRetrans=3
TCPRetransFail=0
TCPSynRetrans=60Read that as a diagnosis. 9,348 fast retransmits against 3 slow-start retransmits means loss is being detected by duplicate ACKs, not by timeout, which is the good failure mode: the sender noticed quickly and recovered without stalling. A host with those numbers reversed is timing out, and users are feeling multi-second pauses. TCPSynRetrans=60 counts connection attempts that had to retry the SYN, which is the counter you want when someone reports intermittently slow connection setup.
ListenDrops and ListenOverflows live in the same file, and they are the definitive answer to "is my accept backlog too small." That question connects directly to net.core.somaxconn further down this page.
The hex files, and how to read them
/proc/net/tcp is a socket table written in hexadecimal, little endian, which looks hostile until you decode it once:
j@llmbits:~$ head -3 /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7039
1: 0100007F:0277 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 400884
2: 64004D0A:1F90 00000000:0000 0A 00000000:00000000 00:00000000 00000000 1000 0 525870The address is four bytes in host order, so on x86 they are reversed. Port is plain hex:
j@llmbits:~$ python3 -c "
import socket,struct
h='00000000:0016'; ip,port=h.split(':')
print(socket.inet_ntoa(struct.pack('<L',int(ip,16))), int(port,16))"
0.0.0.0 22Which matches what ss reports for the same socket:
j@llmbits:~$ ss -ltn
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 4096 127.0.0.1:631 0.0.0.0:*
LISTEN 0 5 10.77.0.100:8080 0.0.0.0:*Decode the third entry yourself: 64004D0A reverses to 0A 4D 00 64, which is 10.77.0.100, and 1F90 is 8080. The st column is the TCP state as a hex enum, where 0A is LISTEN and 01 is ESTABLISHED. Everything ss shows you starts here.
/proc/net/route works the same way:
j@llmbits:~$ head -4 /proc/net/route
Iface Destination Gateway Flags RefCnt Use Metric Mask
ens192 00000000 0158A8C0 0003 0 0 101 00000000
ens224 00004D0A 00000000 0001 0 0 0 00FFFFFF
ens224 00004D0A 01004D0A 0003 0 0 0 0000FFFFDecoded:
iface destination gateway mask
ens192 0.0.0.0 192.168.88.1 0.0.0.0
ens224 10.77.0.0 0.0.0.0 255.255.255.0
ens224 10.77.0.0 10.77.0.1 255.255.0.0
ens192 192.168.88.0 0.0.0.0 255.255.255.0Two routes to 10.77.0.0 with different masks: the connected /24 on the wire and a /16 static pointing at the lab gateway. That is the longest-prefix-match setup described in ip route, visible in raw form.
Note what this file cannot show you. It is IPv4 only, it has no concept of routing tables beyond the main one, and it cannot express anything policy routing does. ip route is not a nicer front end for this file; it is a different, more capable interface to the kernel. Same story for /proc/net/arp:
j@llmbits:~$ cat /proc/net/arp
IP address HW type Flags HW address Mask Device
10.77.0.1 0x1 0x2 aa:bb:cc:00:04:00 * ens224
j@llmbits:~$ ip neigh show dev ens224
10.77.0.1 lladdr aa:bb:cc:00:04:00 REACHABLEThe file gives you a flags bitmask, 0x2, which means the entry is complete and tells you nothing else. ip neigh gives you the actual neighbor state, REACHABLE here, which is the difference between "there is an entry" and "the kernel has confirmed it recently." Watch the same entry a minute later and it will read STALE. The full state machine is in ARP on Linux.
/proc/net/sockstat: memory pressure at a glance
j@llmbits:~$ cat /proc/net/sockstat
sockets: used 253
TCP: inuse 4 orphan 0 tw 11 alloc 6 mem 0
UDP: inuse 3 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0Small file, high value. orphan counts sockets with no file descriptor still holding kernel memory, tw counts TIME_WAIT, and mem is in pages, not bytes. When a box starts refusing connections and the socket table looks fine, this file and net.ipv4.tcp_mem usually explain it.
The sysctls worth knowing
/proc/sys/net is the writable half. sysctl is a thin wrapper: sysctl -n net.ipv4.ip_forward and cat /proc/sys/net/ipv4/ip_forward return the same value from the same place.
j@llmbits:~$ for k in net.ipv4.ip_forward net.ipv4.conf.all.rp_filter \
net.ipv4.tcp_congestion_control net.ipv4.ip_local_port_range net.core.somaxconn \
net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.ipv4.icmp_ratelimit \
net.core.netdev_max_backlog net.ipv4.tcp_syncookies; do
printf '%-42s %s\n' "$k" "$(sysctl -n $k)"; done
net.ipv4.ip_forward 0
net.ipv4.conf.all.rp_filter 0
net.ipv4.tcp_congestion_control cubic
net.ipv4.ip_local_port_range 32768 60999
net.core.somaxconn 4096
net.ipv4.tcp_rmem 4096 131072 6291456
net.ipv4.tcp_wmem 4096 16384 4194304
net.ipv4.icmp_ratelimit 1000
net.core.netdev_max_backlog 1000
net.ipv4.tcp_syncookies 1ListenDrops in /proc/net/netstat climbs.Changing one and watching the wire change
Reading is half of it. Here are two sysctls changed live on the lab host, with the observable result.
Congestion control
j@llmbits:~$ sysctl -n net.ipv4.tcp_available_congestion_control
reno cubicOnly two are loaded here; others (bbr, vegas) ship as modules and appear once you modprobe them. Running iperf3 to the same server under each, three ten-second runs each, alternating:
cubic : [ 5] 0.00-10.00 sec 38.5 MBytes 32.3 Mbits/sec 122 sender
reno : [ 5] 0.00-10.00 sec 36.8 MBytes 30.8 Mbits/sec 155 sender
cubic : [ 5] 0.00-10.00 sec 38.8 MBytes 32.5 Mbits/sec 144 sender
reno : [ 5] 0.00-10.00 sec 38.0 MBytes 31.9 Mbits/sec 171 sender
cubic : [ 5] 0.00-10.00 sec 38.0 MBytes 31.9 Mbits/sec 148 sender
reno : [ 5] 0.00-10.00 sec 36.0 MBytes 30.2 Mbits/sec 165 sendercubic averaged 32.2 Mbit/s with 138 retransmits; reno averaged 31.0 with 164. A small, consistent win for cubic on this path, and consistently fewer retransmits, which is what you would expect from cubic's gentler response to loss.
The methodology note matters more than the result. The first time I ran this, one run each, reno came out at 44.7 Mbit/s against cubic's 33.5, which would have made a dramatic and completely wrong claim. Six runs later the ordering had reversed and stabilized. A single ten-second sample on a virtual path with real loss is noise. If you are going to tune a sysctl on the strength of a measurement, measure it more than once.
You can confirm the setting took effect per socket, mid-flow:
j@llmbits:~$ ss -tin dst 10.77.3.10
ESTAB 0 273672 10.77.0.100:45098 10.77.3.10:5201
reno wscale:11,7 rto:208 rtt:5.016/0.803 mss:1448 pmtu:1500 rcvmss:536
advmss:1448 cwnd:17 ssthresh:11 bytes_sent:16305965 bytes_retrans:123080
bytes_acked:16158270 segs_out:11264 segs_in:5999 data_segs_out:11262
send 39259968bps lastrcv:3000 pacing_rate 47111960bps delivery_rate 28029032bps
delivered:11161 busy:3000ms unacked:17 retrans:0/85 rcv_space:14480
rcv_ssthresh:64088 notsent:249056 minrtt:1.568 snd_wnd:352256 rcv_wnd:64256(That is one long line in the terminal, wrapped here for readability.)
The algorithm name is the first token. Note also ssthresh:11 below cwnd:17, which means this connection has already hit loss and is in congestion avoidance rather than slow start.
Ephemeral port range
Squeeze the pool down to eleven ports and watch source ports land inside it:
j@llmbits:~$ sysctl -w net.ipv4.ip_local_port_range="45000 45010"
net.ipv4.ip_local_port_range = 45000 45010
j@llmbits:~$ sudo tshark -i ens224 -c 4 -f "tcp port 80 and tcp[tcpflags] & tcp-syn != 0" \
-T fields -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport
10.77.0.100 45008 10.77.2.10 80
10.77.2.10 80 10.77.0.100 45008
10.77.0.100 45010 10.77.2.10 80
10.77.2.10 80 10.77.0.100 45010Both connections drew from the eleven-port window. This is the mechanism behind port exhaustion: a NAT gateway or a proxy making thousands of short-lived outbound connections per second, with each one holding its port through TIME_WAIT, can genuinely run out. When it does, the application reports connection failures and the network looks perfectly healthy, because it is.
Making it stick
sysctl -w writes to a kernel data structure and nothing else. Reboot and it is gone. For persistence, drop a file in /etc/sysctl.d/:
j@llmbits:~$ ls /etc/sysctl.d/
README.sysctl
j@llmbits:~$ echo 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/99-routing.conf
j@llmbits:~$ sudo sysctl --systemFiles are read in lexical order, so a 99- prefix wins against distribution defaults. Two things that bite here: settings applied to an interface that does not exist yet at boot are silently skipped, so per-interface tunables for a bond or VLAN often need to be reapplied after the interface comes up; and a container gets its own namespace, so a sysctl set on the host does not necessarily apply inside it.
Key takeaways
/proc/netfiles are kernel structures rendered on read, not files, and they are per network namespace. That single fact explains most container networking confusion./proc/net/devis what ifconfig,ip -s link, nload, ifstat and bmon all read. A five-line shell loop over field 10 is a bandwidth monitor.- In
/proc/net/snmp,Ip: Forwardinguses SNMP semantics where 1 means on and 2 means off. It is not a boolean. - The useful TCP counters are in
/proc/net/netstatunder TcpExt. ManyTCPFastRetranswith fewTCPSlowStartRetransis fast recovery working; the reverse means timeouts and user-visible stalls. /proc/net/tcpand/proc/net/routeare hex and little endian. Reverse the four address bytes and the port is plain hex.ssandipare not just prettier versions of these files. They use netlink and can express states, families and tables the legacy files cannot.- Changing
tcp_congestion_controlaffects only new sockets, andss -tishows the algorithm per connection. - Measure a sysctl change more than once. A single run gave the opposite result to six runs on the same host.
sysctl -wdoes not survive a reboot. Use/etc/sysctl.d/99-name.confandsysctl --system.
These files are where every other tool in the cluster gets its numbers, which makes them the place to go when the tools disagree or are missing. The rest of the toolkit, from ss to tshark to the bandwidth monitors, is indexed in the complete guide to Linux networking commands.