Packet loss explained:
causes, detection & how to fix it

Back to Blog

What Packet Loss Actually Is at the Kernel Level

A packet is "lost" whenever it is transmitted but never received by the destination — or whenever it is received but discarded before being delivered to the application. These are very different events with very different causes, and the kernel tracks them separately.

When a packet arrives at the NIC, the hardware DMA-copies it into a ring buffer in kernel memory. If the ring buffer is full (because the kernel's softirq processing can't drain it fast enough), the NIC drops the packet before it ever reaches the IP stack. This is an rx_dropped event, visible in /proc/net/dev. The packet never made it to the socket. On the transmit side, if the NIC's TX queue is full, the kernel increments tx_dropped and the packet is silently discarded on the way out.

Above the NIC, the kernel may drop packets in the IP layer (TTL expired, checksum failure, no route, martian source), at the netfilter/iptables layer (policy drops), or at the socket receive buffer level (the application is reading too slowly and SO_RCVBUF fills up). Each of these shows up in different counters.

Types of Packet Loss

Random loss

Individual packets lost independently of each other, typically caused by physical layer errors: a marginal fiber connector, RF interference on a wireless link, or an aging copper cable. In the ICMP ping output, you see occasional non-consecutive sequence numbers missing. The Gilbert-Elliott model describes this well — random loss is bursty at the bit error level but appears uniformly distributed at the packet level when the error rate is low.

Burst loss

Consecutive packets lost in clusters, typically caused by momentary link saturation, a spanning tree topology change, or a routing protocol reconvergence. With ping you see runs of consecutive losses (seq=142, 143, 144 all missing) followed by clean reception. Burst loss is much more damaging to TCP than random loss of equivalent average rate, because TCP interprets a burst as congestion and cuts its window aggressively.

Tail-drop

The most common queue management strategy: when a FIFO queue reaches its maximum depth, all incoming packets are dropped until the queue drains. This is the default behavior on most Linux interfaces. Tail-drop under sustained congestion creates synchronization between TCP flows (all flows back off simultaneously when the queue fills), which is why more sophisticated algorithms like FQ-CoDel or CAKE were developed.

Head-drop

The opposite — the oldest packet at the head of the queue is dropped to make room for the newest arrival. This is used in real-time applications (VoIP, video conferencing) where a fresh packet is more useful than a stale one. Head-drop causes lower latency at the cost of higher apparent packet loss to legacy protocols that do not implement it.

Measuring Packet Loss Accurately

Reading /proc/net/dev

Start at the source: the kernel counters. The columns in /proc/net/dev are: rx_bytes rx_packets rx_errs rx_dropped rx_fifo rx_frame rx_compressed rx_multicast | tx_bytes tx_packets tx_errs tx_dropped. The fields to watch are rx_errs (packets with errors, usually CRC failures), rx_dropped (NIC or kernel dropped — no buffer space), and rx_fifo (FIFO overrun at the driver level, indicating the ring buffer overflowed):

# Watch drop counters update in real time
watch -n1 'cat /proc/net/dev | grep -E "^(\s+eth0|\s+ens)"'

# Delta calculation — drops per second
IFACE=eth0
prev_drop=$(awk "/$IFACE/{print \$5}" /proc/net/dev)
sleep 10
curr_drop=$(awk "/$IFACE/{print \$5}" /proc/net/dev)
echo "rx_dropped delta: $(( curr_drop - prev_drop )) in 10s"

# ethtool gives more granular NIC-level counters
ethtool -S eth0 | grep -i drop
ethtool -S eth0 | grep -i miss

mtr for path-level loss

When the loss is not at the host NIC level, you need to find where in the network path it is occurring. mtr (Matt's Traceroute) combines traceroute and ping into a continuous probe that shows per-hop loss statistics:

# Run 100 probes, report mode (no interactive UI), show AS numbers
mtr --report --report-cycles 100 --aslookup 8.8.8.8

# UDP mode (some routers don't reply to ICMP but will show up with UDP)
mtr --report --report-cycles 100 --udp --port 53 8.8.8.8

# Increase packet size to test for MTU-related loss
mtr --report --report-cycles 50 --psize 1400 target.example.com

When reading mtr output, loss at an intermediate hop that does not persist to subsequent hops is almost always ICMP rate-limiting by that router — not real packet loss on the path. Real loss appears at a hop and remains at that percentage or higher for all subsequent hops.

iperf3 for throughput-level loss measurement

For UDP loss measurement at a known bitrate (useful for establishing a baseline and for testing after fixes), iperf3 is the right tool. It sends UDP at a specified rate and reports the receive rate and loss percentage:

# Server side (run on the remote host)
iperf3 -s -p 5201

# Client side: send 100 Mbps UDP for 30 seconds, report every second
iperf3 -c target.example.com -u -b 100M -t 30 -i 1 -p 5201

# Test with large datagrams to stress buffer handling
iperf3 -c target.example.com -u -b 1G -t 10 -l 1400 -p 5201

Compare the sender's 0.0% loss line to the receiver's reported loss. More than 0.1% loss at 100 Mbps on a LAN or datacenter link indicates a genuine problem. On the public internet, 0.5-1% is concerning; above 2% is a clear issue requiring investigation.

DDoS-Induced Packet Loss: What It Looks Like

DDoS attacks cause packet loss in two distinct ways. First, volumetric attacks saturate the inbound link at the provider edge or transit router, causing tail-drop before traffic reaches your server at all. In this scenario, your /proc/net/dev counters show no increase in rx_dropped because the drops are happening upstream — you just see lower-than-expected inbound packet rates. Second, high-PPS attacks that do reach the host overwhelm the kernel's softirq processing capacity, causing the NIC ring buffer to overflow. This shows up as rapidly climbing rx_dropped and rx_fifo counters.

The pattern is distinctive. DDoS-induced loss appears suddenly, correlates with a PPS spike, and often shows a skewed source IP distribution (many unique IPs or suspiciously few IPs on a specific port). Legitimate congestion-induced loss appears gradually as traffic increases and shows a normal distribution of source IPs across a range of ports.

What the PCAP shows during a flood

When Flowtriq captures packets during an active attack, the PCAP tells you exactly what is causing the loss. For a UDP amplification attack, you will see oversized UDP datagrams (frequently 1400-1480 bytes, fragmented to fill the MTU) arriving from dozens of source ports. For a SYN flood, you will see a near-zero ratio of ACK to SYN packets — the attacker's sources never complete the handshake, so your connection table fills with half-open sockets. Download the PCAP from your Flowtriq dashboard and open it in Wireshark; the Statistics > I/O Graph and Statistics > Endpoints views will immediately show you the attack shape.

Common Fixes by Cause

• NIC ring buffer overflow: Increase the RX ring buffer with ethtool -G eth0 rx 4096 and enable multi-queue RSS if the NIC supports it (ethtool -L eth0 combined 8 for 8 queues). Also tune net.core.netdev_max_backlog and net.core.rmem_max in /etc/sysctl.conf.
• Socket buffer overflow: Increase socket buffers: net.core.rmem_max = 134217728 and net.core.wmem_max = 134217728. Applications using UDP should also call setsockopt(SO_RCVBUF) to request a larger buffer.
• DDoS volumetric flooding: Host-level fixes are insufficient once the inbound link is saturated. Engage your upstream provider for null-routing or scrubbing, or implement BGP blackhole routing for the targeted IP.
• Physical layer errors: Replace suspect cables or SFPs. Clean fiber connectors. Check for duplex mismatch with ethtool eth0 | grep -i duplex.
• Upstream congestion: If mtr shows consistent loss at a specific transit hop, the problem is outside your control. Report it to your provider with the hop's IP and your mtr output.

Back to Blog

Related Articles