Chapter 8: Resource Isolation Strategies

Updated

October 31, 2026

Chapter 8: Resource Isolation Strategies

The primary objective of running multiple kernels on a single physical host is to eliminate cross-workload interference (the “noisy neighbour” effect). On high-performance hardware, interference rarely arises from raw CPU cycle exhaustion; rather, it manifests in shared resource contention: CPU cache thrashing, interrupt routing disruptions, NUMA interconnect saturation, and storage queue locking. This chapter details kernel tuning and isolation primitives to achieve deterministic performance.

CPU Isolation: isolcpus, nohz_full, rcu_nocbs

Linux kernels provide specialized boot parameters to shield specific CPU cores from general operating system interference:

  1. isolcpus=domain,managed_irq,<cpulist>:
    • Removes the specified cores from the kernel’s Completely Fair Scheduler (CFS/EEVDF) load-balancing domains.
    • The kernel will never automatically place or migrate any process onto these cores. Only tasks explicitly bound via taskset, sched_setaffinity(), or cpusets will execute here.
  2. nohz_full=<cpulist>:
    • Puts specified cores into adaptive-ticks mode.
    • When exactly one task is running on an isolated core, the periodic scheduling timer tick (\(1000\text{ Hz}\) or \(250\text{ Hz}\)) is completely turned off, eliminating timer interrupt jitter.
  3. rcu_nocbs=<cpulist>:
    • Offloads Read-Copy-Update (RCU) callback invocations to housekeeping cores. RCU processing threads will never execute on isolated cores, preserving strict CPU execution budgets.

Example host kernel boot line:

isolcpus=domain,managed_irq,16-31 nohz_full=16-31 rcu_nocbs=16-31

Cpusets and Cgroup-Based CPU Management

Cgroups v2 cpuset controllers provide dynamic hierarchical partitioning of CPU cores and memory nodes:

# 1. Create a dedicated cpuset for the primary management stack
sudo mkdir -p /sys/fs/cgroup/management
echo "0-15" | sudo tee /sys/fs/cgroup/management/cpuset.cpus
echo "0" | sudo tee /sys/fs/cgroup/management/cpuset.mems

# 2. Create an isolated cpuset partition for secondary kernel worker instances
sudo mkdir -p /sys/fs/cgroup/isolated-workers
echo "root" | sudo tee /sys/fs/cgroup/isolated-workers/cpuset.cpus.partition
echo "16-31" | sudo tee /sys/fs/cgroup/isolated-workers/cpuset.cpus
echo "1" | sudo tee /sys/fs/cgroup/isolated-workers/cpuset.mems

Setting cpuset.cpus.partition to root marks these cores as an isolated partition, preventing any child or parent groups from sharing those CPU resources.

Memory Isolation and NUMA Policies

Cross-socket memory accesses incur severe interconnect delays and bandwidth throttling:

  • Strict NUMA Allocation: Bind memory strictly to Node 1 via numactl --membind=1. If memory is exhausted on Node 1, allocations fail immediately rather than allocating slow remote memory from Node 0.
  • Hugepages Isolation: Pre-allocate 1GB hugepages at boot time. Hugepages bypass the kernel’s transparent hugepage (THP) daemon (khugepaged), eliminating memory compaction and page migration overhead.

Interrupt Routing and IRQ Affinity

By default, the irqbalance daemon distributes hardware interrupts across all available CPU cores:

  • The Problem: A high-volume network interface or storage controller can fire 100,000+ interrupts per second, repeatedly interrupting tasks executing on isolated cores.

  • The Solution: Mask isolated cores out of the irqbalance banned list, and pin device interrupts explicitly via /proc/irq/<irq>/smp_affinity:

    # Exclude cores 16-31 from irqbalance in /etc/default/irqbalance
IRQBALANCE_BANNED_CPUS="00000000,ffff0000"

# Manually route host management NIC interrupts to Core 0
echo 1 | sudo tee /proc/irq/42/smp_affinity

I/O Scheduling and Storage Contention

When storage devices are shared (rather than passed through via dedicated NVMe controllers):

  • I/O Scheduler Selection: Use none (passthrough) for high-speed NVMe drives to eliminate kernel I/O scheduling latency. Use bfq or mq-deadline on slower SATA SSDs to enforce fair queuing.
  • I/O Weight and Throttling: Use cgroup v2 io.weight and io.max to prevent batch background processes in one kernel environment from starving low-latency transactions in another.

What Can and Cannot Be Safely Shared

Hardware Resource Can Be Shared? Mechanism / Risk
CPU Execution Pipeline No Core must be dedicated. SMT sibling threads must also be isolated.
Physical Memory (RAM) No Static physical reservation or EPT isolation required. No overcommit.
L3 Cache (LLC) Conditionally Shared by default on a die. Requires Intel RDT / AMD CAT to partition.
Memory Bus Bandwidth Conditionally Heavy memory streaming in one instance can saturate physical bus bandwidth.
PCIe Bus / Root Port Yes Multiple devices share PCIe bandwidth if root complex throughput is sufficient.
Power Budget (TDP) Yes (Shared) Heavy AVX-512 loads on one socket can cause thermal downclocking across cores.