BBR vs CUBIC: Pick a TCP Congestion Control

Every few months someone on the internet declares that one sysctl line will make your server twice as fast, and the line is almost always about BBR. Sometimes it is true. Sometimes it makes things worse. Instead of arguing, this post builds a small lab with network namespaces, iperf3 and tc netem so you can measure BBR versus CUBIC on a link that looks like yours, and learn why the answer changes with loss and buffer depth.

Two philosophies of congestion control

CUBIC is the loss-based default on most Linux systems. The kernel source describes it as a variant that, unless CUBIC is enabled and the congestion window is large, behaves the same as the original Reno. In practice it grows the window until a packet is lost, treats the loss as a congestion signal, backs off, and grows again. Its defaults in the source are a back-off factor beta of 717 on a 1024 scale (about 0.7), plus fast convergence and the HyStart slow-start exit enabled.

BBR, Bottleneck Bandwidth and RTT, works from a different idea. The header comment in the kernel's tcp_bbr.c gives the model in three lines: the bottleneck bandwidth is the windowed maximum of delivered over elapsed across 10 round trips, the minimum RTT is a windowed minimum over 10 seconds, and the pacing rate is the pacing gain times the bottleneck bandwidth. BBR does not wait for a drop to learn the pipe is full. It estimates how fast the path can deliver and how short the round trip can be, then paces to match.

Why loss changes the winner

A loss-based algorithm cannot tell the difference between a full buffer and a flaky radio link. Both look like a dropped packet, so CUBIC shrinks its window either way. On a Wi-Fi or mobile path with random non-congestive loss, that means throughput collapses long before the link is full. BBR sets its rate from delivery measurements rather than from each loss, which is why it can hold up better there.

The reverse failure also exists. Where the bottleneck buffer is shallow and shared with other flows, a rate-based sender can push in more than the queue holds and cause retransmits for everyone. The BBR project also keeps iterating: the v3 branch of the google/bbr repository is described by its maintainers as a set of small deltas on mainline Linux, released to enable research collaboration and wider testing. Treat it as something to evaluate, not something to assume is in your distribution's kernel.

Build the lab

Everything below runs on one machine using two network namespaces joined by a veth pair, the same pattern as our network namespaces post. You need iproute2, iperf3, jq, and the tcp_bbr kernel module. We shape the sending side with netem: a base delay, optional random loss, a rate cap that acts as the bottleneck, and a queue limit in packets that decides how deep the buffer is.

#!/usr/bin/env bash
set -euo pipefail

sudo modprobe tcp_bbr
sysctl net.ipv4.tcp_available_congestion_control

sudo ip netns add a
sudo ip netns add b
sudo ip link add va type veth peer name vb
sudo ip link set va netns a
sudo ip link set vb netns b
sudo ip -n a addr add 10.7.0.1/24 dev va
sudo ip -n b addr add 10.7.0.2/24 dev vb
sudo ip -n a link set va up
sudo ip -n b link set vb up
sudo ip -n a link set lo up
sudo ip -n b link set lo up

# the iperf3 server lives in namespace b
sudo ip netns exec b iperf3 -s -D

The available list should include cubic, reno and, after the modprobe, bbr. The kernel's ip-sysctl documentation describes tcp_available_congestion_control as showing the congestion control choices that are registered, and notes that modules may exist that are not loaded yet. If bbr is missing, your kernel was built without it.

Run the experiment

The helper below applies one netem profile, then for each algorithm runs a 15 second iperf3 transfer with the -C option, which selects the congestion control for that test only. At the same time it pings from the sender to measure latency under load, which exposes how much queue each algorithm builds.

#!/usr/bin/env bash
# usage: ./run.sh 0.5% 100      (loss, queue limit in packets)
set -euo pipefail
LOSS=${1:-0%}
LIMIT=${2:-1000}

sudo ip netns exec a tc qdisc replace dev va root netem \
  delay 40ms 2ms loss "$LOSS" rate 50mbit limit "$LIMIT"

for cc in cubic bbr; do
  sudo ip netns exec a ping -c 12 -i 1 -q 10.7.0.2 > /tmp/ping.$cc &
  sudo ip netns exec a iperf3 -c 10.7.0.2 -t 15 -C "$cc" -J > /tmp/iperf.$cc.json
  wait
  mbps=$(jq '.end.sum_sent.bits_per_second / 1e6 | floor' /tmp/iperf.$cc.json)
  retr=$(jq '.end.sum_sent.retransmits' /tmp/iperf.$cc.json)
  rtt=$(grep -o 'rtt min/avg/max/mdev = [^ ]*' /tmp/ping.$cc)
  echo "$cc: ${mbps} Mbit/s, ${retr} retransmits, $rtt"
done

Run a matrix rather than a single sample. A good starter set is: clean link with a deep queue (0% 1000), lossy link with a deep queue (0.5% 1000), clean link with a shallow queue (0% 50), and lossy link with a shallow queue (1% 50). Repeat each three times and compare medians. I am deliberately not printing sample results here: they depend on your CPU, kernel version and the exact netem settings, and numbers from a virtual lab should never be copied into a production decision.

Switching it on for real

Once your own measurements justify it, set the default for new connections. The kernel docs describe tcp_congestion_control as the algorithm used for new connections, with reno always available. Existing connections keep what they started with. Persist the setting with a sysctl drop-in and load the module at boot.

echo tcp_bbr | sudo tee /etc/modules-load.d/bbr.conf
echo 'net.ipv4.tcp_congestion_control = bbr' | sudo tee /etc/sysctl.d/90-bbr.conf
sudo sysctl --system

sysctl net.ipv4.tcp_congestion_control

# inspect a live connection: look for "bbr" and bw / mrtt fields
ss -ti dst 10.7.0.2

A note on pacing. The tcp_bbr.c comment says BBR may be used with the fq qdisc with pacing enabled; otherwise the TCP stack falls back to internal pacing. So fq is not mandatory, but if you already run fq or fq_codel for bufferbloat reasons, check how your qdisc setup interacts with it by testing, not by assuming.

A decision checklist

• Mostly clean datacenter or LAN traffic with short RTTs: CUBIC is fine, so leave the default alone.
• Long-haul or lossy paths where you control the sender (a download server, a media origin, a VPN endpoint): BBR is worth a measured trial.
• Shared shallow-buffer bottlenecks: test retransmit counts, not just throughput.
• Anything you change gets a before and after on real traffic, with retransmits and tail latency tracked, not only average throughput.
• Roll out on a subset of hosts first. The setting is per host and reversible with one sysctl.

Takeaway

BBR is not a free speed-up and CUBIC is not obsolete. They make different assumptions about what a signal of trouble looks like, and your link decides which assumption fits. Build the lab, vary loss and buffer depth, record throughput, retransmits and latency under load, and change the default only when your own data says so.