Chapter 19: CPU and Memory Tuning for Production

Updated

October 31, 2026

Chapter 19: CPU and Memory Tuning for Production

Production performance engineering requires translating profiling discoveries into stable, deterministic host configurations. Leaving server hardware and memory management in their default out-of-the-box states introduces dynamic frequency scaling jitter, memory allocation stalls, and cross-NUMA interconnect thrashing.

This chapter provides battle-tested production recipes for CPU governors, IRQ affinity, task isolation, Huge Pages, and memory sysctl parameters, accompanied by validation tests.

CPU Governor Selection and Configuration

Default distribution power-saving governors (powersave, ondemand) introduce 10\(\mu\)s–100\(\mu\)s latency penalties whenever CPU cores ramp up clock frequency from idle P-states:

# Lock all cores to performance governor
sudo cpupower frequency-set -g performance

# Set Energy Performance Preference (EPP) to maximum performance on Intel/AMD
for f in /sys/devices/system/cpu/cpu*/power/energy_performance_preference; do
  echo "performance" | sudo tee "$f" > /dev/null
done

# Restrict deep C-states to prevent sleep exit latency penalties
sudo cpupower idle-set -D 2   # Disables C-states deeper than C1

IRQ Affinity and NUMA-Optimal Placement

Hardware devices (NICs, NVMe controllers, RAID adapters) fire interrupts to CPU cores. If irqbalance routes network interrupts to Core 0 while the network worker thread runs on Core 32 on a separate NUMA socket, every received packet traverses the inter-socket interconnect:

# Disable generic irqbalance service on dedicated low-latency hosts
sudo systemctl stop irqbalance
sudo systemctl disable irqbalance

# Manually route IRQ 84 (eth0 RX queue 0) to CPU Core 4 (bitmask: 1 << 4 = 0x10)
echo "10" | sudo tee /proc/irq/84/smp_affinity

Always align IRQ affinities to cores within the same NUMA node as the PCIe device:

# Discover NUMA node of PCIe device
cat /sys/bus/pci/devices/0000:01:00.0/numa_node

CPU Isolation for Critical Workloads

For deterministic microsecond execution (trading algorithms, packet processing, dedicated database engines), isolate dedicated physical cores via /etc/default/grub:

# Append isolation parameters to GRUB_CMDLINE_LINUX_DEFAULT
# Example: 64-core system, isolate cores 32-63 for dedicated worker threads
GRUB_CMDLINE_LINUX_DEFAULT="... isolcpus=32-63 nohz_full=32-63 rcu_nocbs=32-63"

Update GRUB and reboot:

sudo update-grub

isolcpus: Excludes cores from CFS/EEVDF automatic load balancing.
nohz_full: Disables the scheduler timer interrupt tick whenever a single thread runs on the core.
rcu_nocbs: Moves RCU callback processing off isolated cores to housekeeping cores (cores 0–31).

Huge Pages: Configuring and Validating

Huge Pages eliminate TLB cache thrashing for memory-intensive workloads:

Allocating 1GB Static Huge Pages at Boot

In /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="... default_hugepagesz=1G hugepagesz=1G hugepages=32"

This reserves 32GB of contiguous physical memory at boot, mapped as 32 1GB pages.

Runtime Allocation of 2MB Huge Pages

# Allocate 4096 2MB huge pages (8 GB total) across NUMA nodes
echo 4096 | sudo tee /proc/sys/vm/nr_hugepages

# Verify allocated and free huge pages
grep -i huge /proc/meminfo

Mount hugetlbfs for applications:

sudo mkdir -p /mnt/huge
sudo mount -t hugetlbfs nodev /mnt/huge

Memory Sysctl Parameters

Tune kernel virtual memory behavior for production stability:

# /etc/sysctl.d/99-memory-tuning.conf

# Reduce aggressiveness of swapping anonymous memory (default 60)
vm.swappiness = 10

# Keep 5% of memory reserved for kernel atomic allocations to prevent OOM livelock
vm.min_free_kbytes = 1048576

# Regulate dirty page writeback to avoid massive disk stalls on large RAM systems
vm.dirty_background_bytes = 268435456
vm.dirty_bytes = 1073741824

# Maintain balanced inode and dentry cache reclamation
vm.vfs_cache_pressure = 100

# Prevent overcommit memory panics for predictable database sizing
vm.overcommit_memory = 0
vm.overcommit_ratio = 50

Apply immediately:

sudo sysctl --system

Validating CPU and Memory Tuning

Verify that your tuning eliminated latency spikes and stabilized throughput:

# 1. Verify CPU frequencies are locked at maximum clock speed
watch -n 1 "grep 'cpu MHz' /proc/cpuinfo"

# 2. Check for timer tick interrupts on nohz_full cores (should be near zero)
cat /proc/interrupts | grep "Local timer interrupts"

# 3. Benchmark memory allocation and access latency before and after
sysbench memory --memory-block-size=2M --memory-total-size=50G --threads=8 run

# 4. Measure tail latency jitter with cyclictest
sudo cyclictest -m -p 90 -t 4 -n -i 1000 -l 100000

Under tuned conditions, cyclictest should report maximum latency jitter under 15\(\mu\)s, compared to 150\(\mu\)s+ on stock out-of-the-box installations.