Chapter 12: Performance Tuning and Optimization
Chapter 12: Performance Tuning and Optimization
Achieving maximum performance from modern server silicon requires aligning the operating system’s software settings with the underlying CPU microarchitecture, memory channels, and I/O bus topology.
Ubuntu 26.04 LTS introduces transformative hardware optimizations: optimized amd64-v3 architecture builds, high-frequency low-latency kernel packages, the BBRv3 TCP congestion control algorithm, and kernel Profile-Guided Optimization (PGO) via Sysprof.
This chapter details the performance tuning parameters, sysctl controls, and profiling techniques required to extract optimal throughput and minimal tail latency from production systems.
Understanding System Performance Bottlenecks
Performance bottlenecks manifest across four foundational system resources:
| Subsystem | Primary Metrics to Monitor | Saturation Indicators | Primary Bottleneck Causes |
|---|---|---|---|
| CPU | %usr, %sys, Run Queue Length |
High load average, context switching | Inefficient algorithms, lock contention |
| Memory | PSI (/proc/pressure/memory), Page Faults |
allocstall, high anonymous paging |
Working set exceeds physical RAM, leaks |
| Storage I/O | r/s, w/s, %util, await (ms) |
High queue depth, I/O wait CPU | Unaligned I/O, synchronous fsync thrashing |
| Network | Packet drops, TCP retransmits | High socket buffer backlog | Saturated link, small TCP windows, bufferbloat |
CPU Scheduling and Low-Latency Kernels
For workloads demanding microsecond determinism—such as audio workstations, high-frequency trading engines, and edge robotics—standard generic kernels favor throughput at the expense of scheduling latency.
Ubuntu provides specialized low-latency kernels:
# Install the official low-latency kernel
sudo apt install -y linux-lowlatency
# Inspect scheduling preemption model
uname -v | grep PREEMPTCPU Frequency Governor Tuning
Ensure cores do not dynamically throttle during latency-critical processing:
# Set CPU scaling governor to 'performance' across all cores
sudo apt install -y cpufrequtils
sudo cpupower frequency-set -g performance
# Verify clock speeds
cpupower frequency-infoMemory Management and Swap Tuning
Tune kernel virtual memory behavior via /etc/sysctl.d/99-performance-memory.conf:
# /etc/sysctl.d/99-performance-memory.conf
# Bias against swapping anonymous memory (default is 60; set to 10 for servers)
vm.swappiness = 10
# Percentage of RAM allocated to dirty page caching before background writeback begins
vm.dirty_background_ratio = 5
# Maximum percentage of RAM that can be dirty before allocating threads stall
vm.dirty_ratio = 10
# Increase maximum memory map regions for high-thread databases
vm.max_map_count = 1048576
# Transparent Huge Pages: enable madvise to let runtimes manage hugepages explicitly
# (echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled)Apply settings:
sudo sysctl --systemDisk I/O Optimization and Storage Schedulers
Modern NVMe SSDs utilize thousands of hardware submission and completion queues, rendering legacy multi-path elevators obsolete:
# Verify active I/O scheduler for NVMe drive
cat /sys/block/nvme0n1/queue/scheduler
# [none] mq-deadline kyber bfqnone: Standard for fast NVMe SSDs. Delivers maximum parallel IOPS directly to hardware queues without kernel scheduling overhead.mq-deadline: Recommended for enterprise SATA SSDs and cloud block volumes.bfq: Recommended for rotating mechanical HDDs to prevent desktop stuttering during background transfers.
Set scheduler persistently via udev rule /etc/udev/rules.d/60-io-schedulers.rules:
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"Network Performance Tuning with sysctl and BBRv3
Ubuntu 26.04 LTS includes BBRv3 (Bottleneck Bandwidth and RTT)—Google’s state-of-the-art model-based congestion control algorithm designed to maximize throughput over lossy and high-latency links:
Harden /etc/sysctl.d/99-performance-network.conf:
# Enable BBRv3 congestion control and fair queueing
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
# Increase maximum socket receive and transmit buffers (64 MB for 40/100GbE)
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
# Configure TCP autotuning buffer windows (min, default, max bytes)
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
# Increase listen backlog queue for high-concurrency web services
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 32768
# Enable TCP Fast Open (client + server)
net.ipv4.tcp_fastopen = 3
# Disable slow start restart to prevent window collapse on idle sockets
net.ipv4.tcp_slow_start_after_idle = 0Apply:
sudo sysctl --systemsystemd Performance Optimization
Measure and shave seconds from server startup times:
# Analyze total boot time breakdown (firmware, loader, kernel, userspace)
systemd-analyze
# Startup finished in 1.412s (firmware) + 0.980s (loader) + 1.204s (kernel) + 2.812s (userspace) = 6.408s
# Identify the slowest services delaying boot
systemd-analyze blame | head -n 10
# Generate an SVG critical-chain timeline waterfall diagram
systemd-analyze plot > boot_timeline.svgDisable unneeded services (e.g., modemmanager or bluetooth on headless cloud instances):
sudo systemctl disable --now ModemManager.service bluetooth.serviceamd64-v3 Architecture Variants
Standard Linux packages are built against the generic x86-64-v1 or x86-64-v2 baselines to maintain compatibility with 15-year-old CPUs.
Ubuntu 26.04 LTS provides optimized package repositories compiled specifically for the x86-64-v3 microarchitecture level: - Requires AVX, AVX2, BMI1, BMI2, FMA, and MOVBE instruction sets (present on Intel Haswell / AMD Zen 1 and newer). - Recompiled libraries (glibc, OpenSSL, Python, Go) execute vector instructions directly without runtime CPU dispatch branches. - Delivers 10% to 25% throughput gains across cryptographic operations, mathematical processing, and database lookups.
Verify if your hardware supports x86-64-v3:
/lib64/ld-linux-x86-64.so.2 --help | grep -A 5 "Subdirectories of glibc-hwcaps"Profile-Guided Optimization with Sysprof
Profile your production binaries under real-world traffic to discover hot code paths:
# Record system-wide performance profile using Sysprof CLI
sudo sysprof-cli -c -d 15 -o production.syscap
# Inspect symbol CPU consumption
sysprof-cli -d production.syscapChapter Summary
- Performance engineering requires identifying saturation across CPU, memory, storage, and networking.
- Low-latency kernels and static performance governors eliminate scheduling delays for real-time services.
vm.swappinessand dirty writeback ratios balance page cache I/O against memory exhaustion.- NVMe drives achieve maximum IOPS when configured with the
nonemulti-queue scheduler. - BBRv3 congestion control and enlarged TCP socket buffers deliver superior network throughput.
- Modern
amd64-v3repositories deliver 10%–25% performance boosts by harnessing AVX2 vector extensions.