Conntrack Table Full: Fix Dropped Packets on Linux

Your server is idle, CPU is bored, the NIC is happy, and yet new connections randomly vanish. Welcome to the Linux conntrack table full problem: the kernel keeps a ledger of every flow, the ledger has a maximum size, and when it fills up the kernel starts silently dropping packets. This guide shows how to confirm it in thirty seconds, how to size the table properly, and when the smartest fix is to stop tracking altogether.

What connection tracking actually is

Netfilter's connection tracking system (conntrack) remembers the state of each flow passing through the box: NEW, ESTABLISHED, RELATED and so on. NAT depends on it, stateful nftables or iptables rules such as 'ct state established accept' depend on it, and so do container networks, since Docker and Kubernetes lean on NAT. Each tracked flow occupies an entry in a hash table, and each flow is stored twice, once per direction.

The kernel documentation for the nf_conntrack sysctls spells out the consequence: nf_conntrack_max is the maximum number of allowed entries, and because every flow is stored under two keys, a full table at default settings has an average hash chain length of 2, not 1. When the count reaches the max, new flows cannot be tracked, and the kernel drops the packet and logs 'nf_conntrack: table full, dropping packet'.

Confirm the diagnosis in 30 seconds

Do not guess. Compare the live entry count with the ceiling, check the kernel log for the drop message, and look at the per-CPU failure counters. The conntrack tool comes from the conntrack-tools package; the same counters are also exposed in /proc/net/stat/nf_conntrack.

# Current entries vs limit
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_buckets

# The smoking gun in the kernel log
dmesg -T | grep -i 'nf_conntrack: table full'
journalctl -k --since '1 hour ago' | grep -i conntrack

# Per-CPU stats: look at insert_failed and drop
sudo conntrack -S

# Which protocols and states dominate? (needs conntrack-tools)
sudo conntrack -L 2>/dev/null | awk '{print $1, $4}' | sort | uniq -c | sort -rn | head
sudo conntrack -L 2>/dev/null | grep -o 'src=[0-9.]*' | sort | uniq -c | sort -rn | head

If nf_conntrack_count sits at or near nf_conntrack_max and the log carries the table full message, you have your answer. The last two commands tell you whether one noisy client, one scanner, or one chatty service is burning the budget, which matters more than the number you eventually pick.

Why the default is what it is

According to the kernel docs, nf_conntrack_buckets, the hash table size, defaults when no module parameter is given to total memory divided by 16384, clamped between 1024 and 262144 buckets. nf_conntrack_max defaults to the bucket count. In other words, the default scales with RAM but is capped, so a big machine with plenty of memory can still sit on a tiny table. A busy NAT gateway, a load balancer, or a Kubernetes node running many short connections can exhaust it without ever being short of memory.

Sizing it properly

Two numbers matter: how many flows you need to hold, and how long chains get. Raising nf_conntrack_max alone lets more entries in but lengthens hash chains, which costs CPU on every packet lookup. For a sane setup, raise buckets and max together, keeping max at a small multiple of buckets. Each entry costs kernel memory, so check with slabtop before you go wild.

# How much memory does one entry cost on this kernel?
sudo slabtop -o | grep -i nf_conntrack

# Runtime change (max is a sysctl; buckets is writable in the initial netns only)
sudo sysctl -w net.netfilter.nf_conntrack_max=1048576
echo 262144 | sudo tee /sys/module/nf_conntrack/parameters/hashsize

# Persist across reboots
cat <<'CONF' | sudo tee /etc/sysctl.d/90-conntrack.conf
net.netfilter.nf_conntrack_max = 1048576
CONF
echo 'options nf_conntrack hashsize=262144' | sudo tee /etc/modprobe.d/nf_conntrack.conf
sudo sysctl --system

Timeouts: the cheaper fix

Often the table is not too small, it is too forgetful-averse. The docs list the default for nf_conntrack_tcp_timeout_established as 432000 seconds, which is five days. A client that vanishes without a FIN, such as a phone leaving Wi-Fi or a crashed peer, keeps its entry for days. Meanwhile UDP defaults are short: nf_conntrack_udp_timeout is 30 seconds and nf_conntrack_udp_timeout_stream is 120 seconds, applied once a UDP stream is detected. Unknown protocols fall under nf_conntrack_generic_timeout at 600 seconds, and nf_conntrack_tcp_timeout_time_wait is 120.

Pulling the established timeout down to something like a day, provided your applications use keepalives well under that, frees entries from dead peers much sooner. Keep it comfortably longer than your longest legitimate idle connection, or you will break long-lived idle sessions such as SSH tunnels and database connections without keepalives.

sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=86400
sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=60

# Keepalives must beat the timeout, or idle-but-alive connections get cut
sudo sysctl -w net.ipv4.tcp_keepalive_time=600

Best fix of all: do not track what you do not need

A DNS resolver answering millions of UDP queries, a stateless load balancer front end, or a scrubbing box has no use for per-flow state. nftables lets you exempt traffic from tracking in the raw hook, which runs before conntrack. The notrack statement is documented in the nftables wiki; untracked packets then also cannot match 'ct state established', so you need explicit accept rules for them.

sudo nft -f - <<'NFT'
table inet raw_notrack {
  chain prerouting {
    type filter hook prerouting priority raw; policy accept;
    udp dport 53 notrack
    tcp dport 53 notrack
  }
  chain output {
    type filter hook output priority raw; policy accept;
    udp sport 53 notrack
    tcp sport 53 notrack
  }
}
NFT

sudo nft list table inet raw_notrack

One more gotcha: containers. Each network namespace has its own conntrack state, but the bucket count is only writable in the initial network namespace, as the docs note. On a Kubernetes or Docker host, size the table on the host, and remember that every pod doing NAT through that host shares the same budget. A single misbehaving workload opening thousands of short connections can starve its neighbours, so pair the sizing with per-client counts from conntrack -L to find the culprit.

Watching it in production

Alert on utilisation, not on the log line, because by the time the kernel logs it you are already dropping. A simple ratio of count to max, alerting at 80 percent, gives you hours of warning on a growing service. The one-liner below works with any agent that can run a script, such as a node_exporter textfile collector.

awk -v c="$(cat /proc/sys/net/netfilter/nf_conntrack_count)" \
    -v m="$(cat /proc/sys/net/netfilter/nf_conntrack_max)" \
    'BEGIN { printf "conntrack_utilisation_percent %.1f\n", 100*c/m }'

Takeaway

• Symptoms: new connections fail, existing ones work, and the log shows 'nf_conntrack: table full, dropping packet'.
• Diagnose with nf_conntrack_count against nf_conntrack_max, then conntrack -S for insert_failed and drop.
• Defaults scale with RAM but are clamped, so a large box can still have a small table.
• Fix in order: stop tracking what you do not need, shorten dead-peer timeouts, then raise buckets and max together.
• Alert at 80 percent utilisation so the next outage is a ticket, not a page.