Chapter 10: Memory Compaction, Defragmentation, and Huge Pages

Updated

October 31, 2026

Chapter 10: Memory Compaction, Defragmentation, and Huge Pages

In modern computing, translation overhead is a significant bottleneck. With standard \(4\text{ KiB}\) pages, a database server with 256 GiB of RAM requires \(67\text{ million}\) distinct page table entries. The CPU’s on-chip Translation Lookaside Buffer (TLB)—holding only a few thousand entries—suffers continuous TLB misses, spending up to 20–30% of execution cycles stalling on hardware page table walks.

To maximize TLB reach, modern processors and Linux support Huge Pages: mapping memory in chunks of \(2\text{ MiB}\) (Order 9) or \(1\text{ GiB}\) (Order 18).

However, allocating a 2 MiB huge page requires finding 512 physically contiguous \(4\text{ KiB}\) page frames. In long-running servers, memory fragmentation scatters allocations, breaking up contiguous chunks.

This chapter explores how Linux solves this dual challenge: memory compaction algorithms, kcompactd daemons, Transparent Huge Pages (THP), static hugetlbfs pools, and measuring the performance impact of huge pages.

Fragmentation in Long-Running Systems

External memory fragmentation occurs when total free memory is abundant, but no single contiguous chunk of physical frames is large enough to satisfy a high-order allocation request:

┌────────────────────────────────────────────────────────────────────────┐
│                        PHYSICAL FRAGMENTATION DILEMMA                  │
│                                                                        │
│   Zone Memory (32 KiB total free space):                               │
│   [4K Free] [4K Pinned] [4K Free] [4K Pinned] [4K Free] [4K Free]      │
│                                                                        │
│   Request: Allocate Order 2 Block (16 KiB contiguous):                │
│   - Total Free RAM = 16 KiB                                            │
│   - BUT maximum contiguous span is only 8 KiB!                         │
│   - Result: ALLOCATION FAILS unless memory is compacted!               │
└────────────────────────────────────────────────────────────────────────┘

The Compaction Algorithm: Migration and Scanning

To restore contiguous blocks without evicting data to disk, Linux employs Memory Compaction:

┌────────────────────────────────────────────────────────────────────────┐
│                        TWO-POINTER COMPACTION SCAN                     │
│                                                                        │
│   Zone Start                                                  Zone End │
│   ┌──────────────────────────────────────────────────────────────┐     │
│   │ [Used] [Free] [Used] [Used] ...... [Free] [Used] [Free] [Used│     │
│   └──────────────────────────────────────────────────────────────┘     │
│       ▲                                                ▲               │
│       │                                                │               │
│   Migration Scanner (moves right)              Free Scanner (moves left│
│   Finds movable pages to relocate              Finds free target slots │
│                                                                        │
│   Action: Move allocated page from start into free slot at end.        │
│   Result: Coalesces contiguous free pages at the start of the zone!    │
└────────────────────────────────────────────────────────────────────────┘

The algorithm operates via two converging pointers within each memory zone: 1. Migration Scanner: Starts at the beginning of the zone, scanning forward to identify allocated, movable anonymous or page cache pages. 2. Free Scanner: Starts at the end of the zone, scanning backward to identify empty, unallocated page slots. 3. Page Relocation: The kernel copies page data from the migration scanner’s location to the free scanner’s slot, updates the process’s page table entries to point to the new physical frame address, and flushes local TLBs. 4. Coalescence: The vacated slots coalesce under the Buddy Allocator, synthesizing clean high-order blocks.

Direct Reclaim vs kswapd vs Compact Daemons

Compaction is executed in three operational modes:

  • Background Compaction (kcompactd): Autonomous kernel threads wake up when the number of high-order free blocks falls below the zone low watermark, performing proactive, low-overhead compaction in the background.
  • Direct Compaction: If an allocation request with __GFP_DIRECT_RECLAIM fails and background compaction has not caught up, the calling thread is paused and forced to run the compaction scanner synchronously.
  • Proactive Compaction (vm.compaction_proactiveness): Introduced in Linux 5.8, periodically scans and defragments memory during idle CPU windows to maintain a high baseline of contiguous blocks.

Huge Pages and TLB Efficiency

A single TLB entry for a standard page covers \(4\text{ KiB}\). A single TLB entry for a Huge Page covers:

\[2\text{ MiB} = 2,097,152\text{ bytes} \quad (512 \times \text{larger TLB coverage!})\]

┌────────────────────────────────────────────────────────────────────────┐
│                        TLB REACH COMPARISON (1024 ENTRIES)             │
│                                                                        │
│   Standard 4 KiB Pages:  1024 entries x 4 KiB   = 4 MiB TLB Reach      │
│   Huge 2 MiB Pages:      1024 entries x 2 MiB   = 2 GiB TLB Reach      │
│   Giant 1 GiB Pages:     1024 entries x 1 GiB   = 1 TiB TLB Reach      │
└────────────────────────────────────────────────────────────────────────┘

For workloads with massive active working sets (in-memory analytics, PostgreSQL, JVM applications), huge pages reduce TLB misses from millions per second to near zero, boosting instructions per cycle (IPC) by 15–30%.

Transparent Huge Pages: Automatic Promotion and Demotion

Traditional huge pages required modifying application source code to invoke special APIs. Transparent Huge Pages (THP) automates this entirely within the kernel:

  • Transparent Allocation: When an application allocates a large anonymous buffer via mmap() or malloc(), the kernel speculatively backs the allocation with a single \(2\text{ MiB}\) compound physical page mapped directly at the Page Middle Directory (PMD) level.
  • Khugepaged Background Promotion: The khugepaged daemon continuously scans process memory. If it discovers 512 contiguous \(4\text{ KiB}\) pages that can be unified, it allocates a 2 MiB huge page, copies the data, updates the PMD entry to point to the huge page, and frees the 512 small pages.
  • Page Splitting: If an application subsequently calls munmap() or mprotect() on a 4 KiB slice of a huge page, the kernel transparently splits the 2 MiB compound page back into 512 individual 4 KiB page entries.

hugetlbfs and Preallocated Huge Page Pools

While THP is transparent, it cannot guarantee availability: if memory is heavily fragmented, the kernel falls back to standard 4 KiB pages.

For mission-critical workloads (e.g., KVM hypervisors, DPDK packet processing, Oracle databases), administrators allocate dedicated, non-swappable huge page pools at boot using hugetlbfs:

# Preallocate 2048 x 2 MiB huge pages (4 GiB total)
sudo sysctl -w vm.nr_hugepages=2048

# Mount hugetlbfs filesystem
sudo mount -t hugetlbfs none /dev/hugepages

Applications open files under /dev/hugepages or pass MAP_HUGETLB to mmap(), guaranteeing 100% huge page backing without fallback.

Practical Experiment: Enabling and Measuring THP Effects

THP modes are configured via /sys/kernel/mm/transparent_hugepage/enabled:

# Modes: [always] madvise never
# 'always': Automatically back all eligible memory with THP
# 'madvise': Back with THP only if application requests it via madvise(..., MADV_HUGEPAGE)
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

# Measure active THP usage across the system
grep -i AnonHugePages /proc/meminfo