Linux CPU Isolation: isolcpus, nohz_full, rcu_nocbs

You pinned your latency-critical service to a core, and it still hiccups. Rude. The reason is that Linux CPU isolation is not one switch but a stack of them: the scheduler, the timer tick, RCU callbacks, interrupts and kernel workqueues all want a turn on your precious core. This guide walks through isolcpus, nohz_full, rcu_nocbs, IRQ affinity and cgroup v2 cpuset partitions, so you can give one workload a quiet CPU and then prove it with a measurement.

What kernel noise actually is

The kernel's own CPU isolation document defines the goal as leaving a CPU exclusive to a given workload without any undesired code interference from the kernel. It calls the interference noise, and splits it into asynchronous events (interrupts, timers, scheduler preemption by workqueues and kthreads) and synchronous ones (syscalls and page faults). Most of the time nobody notices. For a trading loop, a packet processor, an audio engine or a control loop, a few microseconds of jitter is the whole bug.

The key concept is housekeeping: the non-isolated CPUs where the kernel noise is allowed to go. Isolation is really a migration exercise. You do not delete the work, you push it onto a few sacrificial cores and keep the rest clean.

Layer 1: the scheduler (isolcpus domain)

The isolcpus boot parameter takes an optional flag list and a CPU list. Per the kernel parameter docs, the default flag is domain, which isolates the CPUs from the general SMP balancing and scheduling algorithms. The docs warn that doing this at boot is irreversible, that is, you cannot bring the CPU back into the scheduler domains without a reboot, and point to cpusets for dynamic configuration. You can still place a process on an isolated CPU through the affinity syscalls or cpuset.

Layer 2: the tick (nohz_full)

By default the kernel interrupts every CPU on a periodic scheduling-clock tick. The no_hz documentation explains that CONFIG_NO_HZ_FULL=y lets the kernel avoid sending those interrupts to a CPU that has only one runnable task. By default no CPU is an adaptive-ticks CPU; you select them with nohz_full=. The rules matter:

• The system needs at least two CPUs, and you cannot mark them all as adaptive-tick. The parameter docs say the boot CPU is forced outside the list to maintain timekeeping.
• A CPU only quiets down when it has a single runnable task. Run two busy threads on it and the tick comes back.
• A residual 1 Hz tick remains. The isolcpus docs say it is offloaded to workqueues, which you should affine to housekeeping CPUs through /sys/devices/virtual/workqueue/cpumask, and that by default the global workqueue runs on all CPUs, so this file has to be configured manually after boot.
• CPUs in nohz_full also get their RCU callbacks offloaded, just as if they were listed in rcu_nocbs=.

Layer 3: RCU callbacks (rcu_nocbs)

RCU callbacks are deferred cleanup work. If they pile up on your isolated core, they interrupt your thread. With rcu_nocbs=, the callbacks run in rcuo kernel threads instead. The no_hz document is blunt that it is up to userspace to pin those rcuo kthreads to specific CPUs if desired, so offloading alone moves the work off the callback path but not necessarily off your core unless you pin the threads to housekeeping CPUs.

Layer 4: interrupts (IRQ affinity)

Device interrupts are the loudest neighbour. The kernel's SMP IRQ affinity document explains that /proc/irq/N/smp_affinity and smp_affinity_list specify which CPUs are permitted for an interrupt source, that you cannot turn off all CPUs, and that controllers without affinity support stay at the default of all CPUs. /proc/irq/default_smp_affinity applies to IRQs not yet activated. The irqaffinity= boot parameter sets that default mask.

There is a catch for NVMe and modern multi-queue NICs: managed interrupts. The kernel assigns their affinity itself and it cannot be changed via /proc/irq. The isolcpus managed_irq flag steers them to housekeeping CPUs on a best-effort basis, but only if the queue's mask contains both isolated and housekeeping CPUs. If a queue's mask contains only isolated CPUs, the flag has no effect on routing.

Hands-on: isolating CPUs 2-5 on an 8-CPU box

Add the parameters to your bootloader. This example leaves CPUs 0, 1, 6 and 7 as housekeeping. Treat it as a lab recipe: CONFIG_NO_HZ_FULL must be enabled in your kernel (check your distribution's config).

# Is the running kernel built with adaptive ticks?
zgrep -E 'CONFIG_NO_HZ_FULL|CONFIG_RCU_NOCB_CPU' /boot/config-$(uname -r) 2>/dev/null \
  || zgrep -E 'CONFIG_NO_HZ_FULL|CONFIG_RCU_NOCB_CPU' /proc/config.gz

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet isolcpus=domain,managed_irq,2-5 nohz_full=2-5 rcu_nocbs=2-5 irqaffinity=0,1,6,7"

sudo update-grub && sudo reboot

After the reboot, verify what the kernel actually accepted, then move the remaining noise to housekeeping CPUs.

cat /proc/cmdline
cat /sys/devices/system/cpu/isolated
cat /sys/devices/system/cpu/nohz_full

# Move the unbound workqueues to housekeeping CPUs (hex mask for CPUs 0,1,6,7 = c3)
echo c3 | sudo tee /sys/devices/virtual/workqueue/cpumask

# Pin every movable IRQ to housekeeping CPUs (some will refuse; that is normal)
for d in /proc/irq/[0-9]*; do
  echo 0,1,6,7 | sudo tee "$d/smp_affinity_list" >/dev/null 2>&1
done

# Pin RCU offload threads (rcuo*) to housekeeping CPUs
for pid in $(pgrep -f '^rcuo'); do sudo taskset -pc 0,1,6,7 "$pid" >/dev/null; done

# Run your workload on an isolated core with a real-time-ish priority
sudo taskset -c 3 chrt -f 50 ./latency-critical-service

The flexible way: cgroup v2 cpuset partitions

Because boot-time domain isolation cannot be undone, the docs recommend cpusets for runtime configuration. In cgroup v2, cpuset.cpus.partition accepts member, root and isolated. Setting a partition to isolated puts its CPUs in an isolated state without scheduler load balancing and excludes them from unbound workqueues. The docs add that tasks placed in an isolated partition with multiple CPUs should be carefully distributed and bound to individual CPUs for optimal performance, because nobody will balance them for you.

cd /sys/fs/cgroup
echo +cpuset | sudo tee cgroup.subtree_control
sudo mkdir -p rt-iso
echo 2-5       | sudo tee rt-iso/cpuset.cpus
echo isolated  | sudo tee rt-iso/cpuset.cpus.partition
cat rt-iso/cpuset.cpus.partition     # should print: isolated (or "isolated invalid (<reason>)")

# Move a shell into it and start the workload from there
echo $$ | sudo tee rt-iso/cgroup.procs
taskset -c 3 ./latency-critical-service

Measure it: before and after

Isolation you did not measure is a rumour. cyclictest from the rt-tests package is the usual yardstick: it repeatedly sleeps and measures how late it wakes up. Run it once on a normal core and once on an isolated one under the same background load, and compare the maximum latency rather than the average, because jitter lives in the tail. Your numbers depend on the hardware, firmware (SMIs) and kernel config, so do not trust anyone else's, including ours.

sudo apt install rt-tests stress-ng   # package names on Debian/Ubuntu

# Background noise on the housekeeping CPUs only
taskset -c 0,1,6,7 stress-ng --cpu 4 --io 2 --timeout 70s &

# Baseline: a housekeeping CPU
sudo cyclictest --smp -a 0 -t1 -p 90 -i 200 -D 60s -q
# Isolated CPU
sudo cyclictest --smp -a 3 -t1 -p 90 -i 200 -D 60s -q

Takeaway

• Decide your housekeeping CPUs first; everything else follows from that list.
• Layer the fixes: scheduler domain, tick (nohz_full), RCU (rcu_nocbs plus pinned rcuo threads), interrupts, workqueues.
• Prefer cpuset partitions when you need to change your mind without rebooting.
• One runnable task per isolated CPU, or the tick returns.
• Verify with /sys/devices/system/cpu/isolated, /proc/interrupts and cyclictest tails, not vibes.