Why Your GPU Is Slow: PCIe, ReBAR and NUMA

You bought a flagship GPU, and it behaves like a hand-me-down. Nine times in ten the silicon is fine and the plumbing is not: the card negotiated half the lanes it should have, the CPU can only peek at VRAM through a tiny window, or your job is running on the wrong CPU socket. This guide shows how to check PCIe link speed on Linux, verify Resizable BAR and pin work to the NUMA node that owns the GPU, using nothing but lspci, sysfs and numactl.

Step one: read the PCIe link, not the box

Every PCIe device and its upstream port negotiate a link at boot, and the result is the smaller of what both ends support. The signalling rate per lane doubled with each generation: 8 GT/s for PCIe 3.0, 16 GT/s for 4.0 and 32 GT/s for 5.0. From 3.0 onwards the line code is 128b/130b, which costs about 1.5% instead of the 20% of the old 8b/10b scheme. So usable bandwidth per lane and direction is the rate times 128/130 divided by eight.

• Gen3 x16: 8 GT/s x 16 lanes x 128/130 / 8 is roughly 15.75 GB/s each way.
• Gen4 x16: double that, roughly 31.5 GB/s each way.
• Gen4 x4: roughly 7.9 GB/s each way, which is why a GPU in a chipset slot feels quarter-speed for transfers.
• Gen1 x16 (2.5 GT/s with 8b/10b): only about 4 GB/s each way.

The lspci tool prints both what the link can do (LnkCap) and what it is doing right now (LnkSta). Find your GPU first, then ask for the verbose capabilities, which needs root to show everything.

lspci | grep -Ei 'vga|3d|display'
# e.g. 01:00.0 VGA compatible controller: ...

sudo lspci -s 01:00.0 -vv | grep -E 'LnkCap:|LnkSta:'
# LnkCap: Port #0, Speed 16GT/s, Width x16 ...
# LnkSta: Speed 16GT/s, Width x16 ...

If LnkSta shows a lower speed or width than LnkCap, you have a downtrain. The same data is in sysfs as plain files, which is handy for scripts and monitoring, and the NVIDIA driver can report it too.

D=/sys/bus/pci/devices/0000:01:00.0
cat $D/current_link_speed $D/max_link_speed
cat $D/current_link_width $D/max_link_width

nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max,pcie.link.width.current,pcie.link.width.max --format=csv

Real downtrains usually have boring causes. The card sits in a slot that is wired x4 or x8 (check the motherboard manual, many boards share lanes with M.2 drives), a riser or extension cable cannot carry the faster signalling rate, or the slot is dusty or not fully seated. Forcing a lower generation in firmware is a quick way to test a flaky riser.

Step two: Resizable BAR and the 256 MiB peephole

A GPU exposes its memory to the CPU through a PCI base address register (BAR). Historically the window onto VRAM was only 256 MiB, so the CPU had to slide it around to reach the rest. Resizable BAR is a PCIe capability that lets the window grow to cover the whole of VRAM. It is not a magic speed-up for every program, but it can help workloads that stream a lot of data to the card. Phoronix's Linux testing of NVIDIA Resizable BAR found a big boost for some games and little for others.

Check the capability and the current size with lspci. Look for a line mentioning Physical Resizable BAR and the BAR's current size next to its supported sizes.

sudo lspci -s 01:00.0 -vv | grep -iA4 'resizable bar'
# Capabilities: [bb0 v1] Physical Resizable BAR
#     BAR 1: current size: 16GB, supported: 256MB 512MB ... 16GB

# NVIDIA's view of the BAR1 aperture (a 256 MiB total usually means it is off)
nvidia-smi -q | grep -i 'bar1' -A 3

If the card supports it but the size is stuck at 256 MiB, enable Above 4G Decoding and Resizable BAR (sometimes called Smart Access Memory) in the firmware setup, and make sure the machine boots in UEFI mode. Be aware that tools can disagree: one forum user reported lspci showing a 16 GB BAR1 while nvidia-smi still printed 256 MiB, so trust lspci for the hardware state and test your driver version separately.

Step three: stay on the GPU's NUMA node

On a dual-socket server, or on a large single-socket chip with several memory domains, each PCIe slot hangs off one particular CPU. If your process runs on the other socket and allocates memory there, every transfer to the GPU crosses the inter-socket link first. The fix costs nothing: find the GPU's node and pin the CPU threads and the memory to it.

D=/sys/bus/pci/devices/0000:01:00.0
cat $D/numa_node          # e.g. 1  (-1 means the firmware did not say)
cat $D/local_cpulist      # CPUs closest to this device

numactl --hardware        # nodes, CPUs and memory per node
nvidia-smi topo -m        # shows CPU Affinity and NUMA Affinity per GPU

numactl --cpunodebind=1 --membind=1 python3 train.py

A numa_node of -1 means the system did not report a node, which happens on single-node machines and on firmware that lacks the ACPI proximity data. On a multi-socket box that is worth a firmware update, because the placement information is only as good as what the BIOS provides.

A five-minute triage checklist

• Run a load, then compare LnkSta with LnkCap for the GPU.
• If width is low, reseat the card, drop any riser and read the board's lane-sharing table.
• If speed is low under load, test in a different slot and check firmware for a forced generation limit.
• Confirm Resizable BAR in lspci, and Above 4G Decoding in firmware if it is missing.
• On multi-socket hardware, read numa_node and pin with numactl.
• Only then start tuning drivers and kernels.

Takeaway

A GPU is only as fast as the path that feeds it. Three one-line checks, the PCIe link state, the BAR size and the NUMA node, catch the majority of invisible bottlenecks before you spend a cent on new hardware or a day on driver tuning.