Chapter 15: Performance Optimization, Profiling, and Debugging
Chapter 15: Performance Optimization, Profiling, and Debugging
In production environments, memory issues rarely announce themselves with a clear error code. Instead, they manifest as creeping latency degradation, unpredictable tail latency, intermittent CPU spikes caused by background compaction, or sudden out-of-memory crashes that tear down critical microservices.
Taming these behaviors requires an engineering methodology that moves from high-level system metrics down to low-level kernel event tracing and post-mortem crash dump analysis.
This chapter details the tools, kernel instrumentation, and step-by-step diagnostic workflows required to profile, debug, and optimize the Linux virtual memory subsystem.
Core Memory Performance Metrics
When diagnosing memory bottlenecks, systems engineers must evaluate five core telemetry dimensions:
| Metric Dimension | Key Kernel Counters | Diagnostic Tool | Symptom of Distress |
|---|---|---|---|
| Allocation Latency | pgalloc_normal, allocstall |
perf, bpftrace |
Slow thread startup, tail latency |
| Reclaim Pressure | pgscan_direct, pgsteal_kswapd |
/proc/vmstat |
CPU saturated by kernel reclaimers |
| TLB Miss Penalty | dtlb_load_misses.walk_duration |
perf stat |
High cycles per instruction (CPI) |
| Page Fault Rate | minor_faults, major_faults |
pidstat -r, sar -B |
Excessive disk reads during faults |
| Stall Time | some, full memory pressure |
/proc/pressure/memory |
Direct stalls waiting on RAM/swap |
Deep /proc and /sys Subsystem Inspection
Before launching dynamic tracers, start by querying the kernel’s built-in accounting files.
1. Granular System Memory: /proc/meminfo
cat /proc/meminfo | grep -E "(MemTotal|MemFree|MemAvailable|Buffers|Cached|Dirty|Writeback|AnonPages|Shmem|Slab|SReclaimable|SUnreclaim|KernelStack|PageTables)"MemAvailable: The kernel’s estimated quantity of RAM available for starting new applications without swapping.DirtyvsWriteback: High dirty pages indicate pending disk writeback; if writeback climbs steadily, disk storage bandwidth is saturated.SReclaimablevsSUnreclaim: Identifies whether SLUB cache footprint can be shrunk under memory pressure.
2. Microscopic VM Counters: /proc/vmstat
# Monitor page allocation stalls and direct reclaim rate in real time
watch -n 1 "grep -E '(allocstall|pgsteal|pgscan|compact)' /proc/vmstat"allocstall_normal 4891 # Direct allocations stalled waiting on reclaim
pgscan_kswapd 18920114 # Pages scanned by background kswapd
pgscan_direct 2849102 # Pages scanned synchronously by allocating threads
pgsteal_direct 2610481 # Pages successfully reclaimed by direct reclaim
compact_stall 1204 # Threads stalled waiting for page compaction
compact_fail 319 # Compaction attempts that failedIf allocstall or pgscan_direct are increasing rapidly, the system’s memory pressure has escalated past background kswapd into application-blocking direct reclaim.
3. Detailed Per-Process Mapping: /proc/[pid]/smaps_rollup
# Obtain aggregate memory consumption of a process in O(1) time
cat /proc/$(pgrep -o postgres)/smaps_rollupOutput:
Rss: 4194304 kB
Pss: 1842100 kB
Shared_Clean: 819200 kB
Shared_Dirty: 1536000 kB
Private_Clean: 0 kB
Private_Dirty: 1839104 kB
Referenced: 3900000 kB
Anonymous: 1839104 kB
LazyFree: 0 kB
AnonHugePages: 409600 kB
ShmemPmdMapped: 0 kB
FilePmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kBPss(Proportional Set Size): Divides shared memory evenly among all processes sharing the pages. Summing PSS across all processes gives the true total memory footprint without double-counting shared libraries.
Profiling with perf and Flame Graphs
Hardware Performance Counters (PMUs) allow you to measure CPU execution stalls caused by memory hierarchy bottlenecks.
Measuring TLB Bottlenecks
# Measure TLB miss overhead during workload execution
sudo perf stat -e dTLB-loads,dTLB-load-misses,iTLB-loads,iTLB-load-misses -p $(pgrep my_app) -- sleep 10If dTLB-load-misses exceeds 3% of total dTLB-loads, the application’s working set spans too many 4 KiB pages; enabling Transparent Huge Pages (THP) or explicit HugeTLB will recover lost CPU cycles.
Generating Memory Allocation Flame Graphs
# Record kernel stack traces during page allocation events
sudo perf record -a -g -e kmem:mm_page_alloc -- sleep 10
# Generate interactive flame graph using FlameGraph tools
sudo perf script | stackcollapse-perf.pl | flamegraph.pl > page_alloc_flame.svgIn the resulting flame graph, wide plateaus indicate functions that trigger disproportionate numbers of physical page allocations.
┌────────────────────────────────────────────────────────────────────────┐
│ ALLOCATION FLAME GRAPH INTERPRETATION │
│ │
│ │
│ ┌───────────────────────────────────┐ │
│ │ do_anonymous_page (PTE Fault) │ │
│ ├───────────────────────────────────┤ │
│ │ handle_pte_fault │ │
│ ├───────────────────────────────────┤ │
│ │ __handle_mm_fault │ │
│ ├───────────────────────────────────┤ │
│ │ handle_mm_fault │ │
│ ┌──────────────────┴───────────────────────────────────┴─────────────┐ │
│ │ do_user_addr_fault │ │
│ └────────────────────────────────────────────────────────────────────┘ │
│ Wide column = Dominant source of physical memory allocations │
└────────────────────────────────────────────────────────────────────────┘eBPF and BCC Tracing
eBPF (Extended Berkeley Packet Filter) allows dynamic tracing of kernel functions without modifying code or restarting the machine.
Tracing Allocation Latency with bpftrace
The following one-liner measures the exact latency distribution (in microseconds) of physical page allocations in the buddy allocator:
sudo bpftrace -e '
kprobe:__alloc_pages {
@start[tid] = nsecs;
}
kretprobe:__alloc_pages /@start[tid]/ {
$duration_us = (nsecs - @start[tid]) / 1000;
@alloc_latency_us = hist($duration_us);
delete(@start[tid]);
}
'Output:
@alloc_latency_us:
[1, 2) 142095 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2, 4) 38210 |@@@@@@@@@@@@@ |
[4, 8) 5402 |@ |
[8, 16) 891 | |
[16, 32) 120 | |
[32, 64) 14 | |
[64, 128) 2 | |
[128, 256) 0 | |
[256, 512) 1 | |A long tail extending beyond 100 microseconds reveals that threads are getting blocked in direct compaction or synchronous page writeback.
Detecting Slab Leaks with memleak
The BCC memleak tool intercepts user-space (malloc/free) or kernel-space (kmalloc/kfree) allocations to uncover leaks:
# Trace outstanding kernel slab allocations every 5 seconds
sudo /usr/share/bcc/tools/memleak -K --min-age=10000[15:30:12] Top 5 outstanding kernel allocations:
14820 bytes allocated at:
__kmalloc+0x1a8
ext4_alloc_inode+0x24
alloc_inode+0x5c
new_inode+0x18Kernel Function Tracing with ftrace
When debugging lock contention or compaction behavior without installing compiler toolchains, the kernel’s built-in ftrace subsystem provides zero-dependency tracing via debugfs.
# Navigate to tracing directory
cd /sys/kernel/tracing
# Reset filters
echo 0 > tracing_on
echo > trace
# Enable function graph tracing on page compaction
echo function_graph > current_tracer
echo compact_zone > set_ftrace_filter
# Start trace, execute test workload, and stop
echo 1 > tracing_on
sleep 1
echo 0 > tracing_on
# Inspect execution call hierarchy with precise function timestamps
head -n 25 traceExperimental Kernel Module: Memory Fault Injector
To observe how the kernel handles severe page allocation pressure and analyze locking paths, engineers can write diagnostic load modules:
// mem_inspector.c - Kernel module inspecting page tables and buddy state
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/mm.h>
#include <linux/sched.h>
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Systems Engineer");
MODULE_DESCRIPTION("Kernel Memory Hierarchy Inspector");
static int __init mem_inspector_init(void) {
struct task_struct *task = current;
struct mm_struct *mm = task->mm;
struct sysinfo si;
si_meminfo(&si);
pr_info("=== MEMORY INSPECTOR: %s [PID: %d] ===\n", task->comm, task->pid);
pr_info("Total RAM: %lu MB, Free RAM: %lu MB, HighOrder Free: %lu MB\n",
(si.totalram * si.mem_unit) >> 20,
(si.freeram * si.mem_unit) >> 20,
(si.sharedram * si.mem_unit) >> 20);
if (mm) {
down_read(&mm->mmap_lock);
pr_info("Process Total VM: %lu KiB, RSS: %lu KiB, Data: %lu KiB\n",
mm->total_vm << (PAGE_SHIFT - 10),
get_mm_rss(mm) << (PAGE_SHIFT - 10),
mm->data_vm << (PAGE_SHIFT - 10));
up_read(&mm->mmap_lock);
}
return 0;
}
static void __exit mem_inspector_exit(void) {
pr_info("Memory inspector unloaded.\n");
}
module_init(mem_inspector_init);
module_exit(mem_inspector_exit);Build and load the module using a standard kernel Makefile:
make -C /lib/modules/$(uname -r)/build M=$PWD modules
sudo insmod mem_inspector.ko
dmesg | tail -n 10
sudo rmmod mem_inspectorPost-Mortem Crash Dump Analysis with kdump and crash
When an unrecoverable memory corruption, double-free, or kernel panic occurs, kdump utilizes kexec to boot a secondary “capture kernel” that writes the physical RAM of the crashed system into /var/crash/vmcore.
Analyzing a Memory Panic with crash
# Launch interactive crash analyzer against kernel binary and dump
sudo crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/2026-10-30/vmcoreWithin the crash interactive shell:
crash> bt
# Shows backtrace of the crashing CPU.
# E.g., BUG: unable to handle page fault for address: 0000000000000028
# RIP: 0010:kmem_cache_alloc+0x54/0x2f0
crash> kmem -i
# Displays global kernel memory inventory at the exact instant of crash
PAGES BYTES PERCENTAGE
TOTAL MEM 8388608 32.0 GB ----
FREE 104201 407.0 MB 1%
USED 8284407 31.6 GB 98%
SHARED 2097152 8.0 GB 25%
BUFFERS 12401 48.4 MB 0%
CACHED 3145728 12.0 GB 37%
SLAB 2621440 10.0 GB 31%
crash> kmem -s
# Displays slab allocation breakdown, identifying leaking kmem_cache
CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
ffff888100032000 kmalloc-512 512 8491024 10000000 625000 8k
ffff888100033000 ext4_inode_cache 1184 41209 50000 1562 32kIf a single cache (like kmalloc-512) consumes 80% of total slabs, examine the active callers using the bt -a command to pinpoint the leaking subsystem.
The Complete Debugging Workflow
When addressing memory performance regressions in production, adhere to this diagnostic escalation path:
┌────────────────────────────────────────────────────────────────────────┐
│ PRODUCTION VM DEBUGGING ESCALATION │
│ │
│ Level 1: System-Wide Triage │
│ ├─ Check /proc/pressure/memory (Is PSI stall time increasing?) │
│ ├─ Check /proc/meminfo (Is RAM exhausted? High dirty pages?) │
│ └─ Check vmstat 1 (Are allocstall or direct reclaim counters firing?) │
│ │
│ Level 2: Per-Process Profiling │
│ ├─ pidstat -r (Identify fast-growing RSS) │
│ ├─ smaps_rollup (Determine PSS vs Anonymous vs File mapping) │
│ └─ perf stat -e dTLB-load-misses (Detect TLB thrashing) │
│ │
│ Level 3: Dynamic Tracing │
│ ├─ bpftrace / BCC memleak (Pinpoint user-space or kernel leaks) │
│ ├─ perf record -e kmem:mm_page_alloc (Flame graph allocation sources) │
│ └─ ftrace (Analyze compact_zone and kswapd execution chains) │
│ │
│ Level 4: Post-Mortem Forensics │
│ └─ crash / kdump (Extract physical slab states, backtraces, registers)│
└────────────────────────────────────────────────────────────────────────┘Following this structured workflow prevents premature optimization and guides the engineer directly to the root architectural cause.