Chapter 11: NUMA, Memory Policy, and Scalability

Updated

October 31, 2026

Chapter 11: NUMA, Memory Policy, and Scalability

In early symmetric multiprocessing (SMP) systems, all CPU cores connected to physical RAM through a single, shared system bus. As server architectures scaled to dozens and hundreds of CPU cores, the shared memory bus became an insurmountable bottleneck: cores spent the majority of their time stalled, arbitrating for bus access.

Modern multi-socket enterprise servers solve this by adopting Non-Uniform Memory Access (NUMA). In NUMA architectures, physical memory is partitioned: each CPU socket owns a local memory controller and local DRAM channels, connecting to adjacent sockets across high-speed point-to-point coherent interconnects (AMD Infinity Fabric, Intel Ultra Path Interconnect - UPI).

While NUMA provides immense total memory bandwidth, access times are non-uniform: reading local memory takes \(60\text{ to } 80\text{ ns}\), while reading remote memory across an interconnect link takes \(120\text{ to } 160\text{ ns}\)—a 2x latency penalty accompanied by interconnect bus saturation.

This chapter explores how Linux manages NUMA topologies: the pg_data_t node hierarchy, NUMA memory allocation policies, automatic NUMA balancing, task placement coordination, and NUMA-aware systems programming.

The NUMA Problem: Distance Matters

In a dual-socket server with 64 cores per socket:

┌────────────────────────────────────────────────────────────────────────┐
│                        DUAL-SOCKET NUMA TOPOLOGY                       │
│                                                                        │
│   ┌──────────────────────────────┐    Interconnect    ┌──────────────────────────────┐ │
│   │ Socket 0 (NUMA Node 0)       │◄── (UPI/Infinity) ─►│ Socket 1 (NUMA Node 1)       │ │
│   │ Cores 0-63                   │                    │ Cores 64-127                 │ │
│   │ Local Memory Controller      │                    │ Local Memory Controller      │ │
│   └──────────────┬───────────────┘                    └──────────────┬───────────────┘ │
│                  │ (Local DRAM: ~70 ns)                              │ (Local DRAM: ~70 ns)
│                  ▼                                                   ▼                 │
│   ┌──────────────────────────────┐                    ┌──────────────────────────────┐ │
│   │ Local DRAM (256 GiB)         │                    │ Local DRAM (256 GiB)         │ │
│   └──────────────────────────────┘                    └──────────────────────────────┘ │
│                  ▲                                                   ▲                 │
│                  └────────── Remote Read (~140 ns) ──────────────────┘                 │
└────────────────────────────────────────────────────────────────────────┘

The performance impact is governed by the NUMA Distance Ratio: - Local Memory Access: Distance \(10\) (Baseline latency). - Remote Socket Access: Distance \(20\) to \(32\) (\(2\times \text{ to } 3.2\times \text{ higher latency}\)).

If an application thread running on Core 0 on Socket 0 allocates memory that is placed in Node 1’s DRAM, every memory access must traverse the physical inter-socket interconnect. If dozens of threads do this concurrently, the interconnect links saturate, degrading application throughput by up to 50%.

Node, Zone, and Memory Hierarchy in Linux

Inside the kernel, each physical NUMA node is represented by a pg_data_t structure (defined in include/linux/mmzone.h):

typedef struct pglist_data {
    struct zone node_zones[MAX_NR_ZONES]; /* Memory zones inside this node */
    struct zonelist node_zonelists[MAX_ZONELISTS]; /* Fallback search order */
    int nr_zones;
    int node_id;                          /* NUMA Node ID (0, 1, ...) */
    unsigned long node_start_pfn;         /* Lowest PFN in this node */
    unsigned long node_present_pages;
    unsigned long node_spanned_pages;
    struct task_struct *kswapd;           /* Per-node background reclaim thread */
    wait_queue_head_t kswapd_wait;
} pg_data_t;

Each NUMA node operates as a self-contained memory subsystem: - Each node maintains its own independent Buddy Allocator free lists. - Each node has its own dedicated kswapd background reclaim daemon. - Allocations default to searching the local node first; if the local node is saturated, the kernel falls back across the node_zonelists array to allocate from remote nodes.

Memory Policies: Default, Interleave, Bind, and Prefer

Linux provides fine-grained control over NUMA allocation policies via set_mempolicy(2) and the numactl command:

  • MPOL_DEFAULT: Memory is allocated on the local node where the allocating CPU core is currently executing.
  • MPOL_BIND: Memory must strictly be allocated from a specified set of nodes. If those nodes run out of memory, the allocation fails (-ENOMEM) rather than spilling over to other nodes. Essential for latency-critical database instances pinned to specific cores.
  • MPOL_INTERLEAVE: Memory allocations round-robin page-by-page across multiple nodes (\(4\text{ KiB}\) on Node 0, \(4\text{ KiB}\) on Node 1). Ideal for shared read-heavy caches accessed equally by threads on both sockets, distributing bandwidth across memory controllers.
  • MPOL_PREFERRED: Strongly prefers a specific node, but permits falling back to remote nodes if the preferred node is exhausted.
# Pin an application to execute only on Socket 0 cores using Socket 0 memory
numactl --cpunodebind=0 --membind=0 ./high_performance_db

# Interleave memory across all available NUMA nodes
numactl --interleave=all ./shared_cache_service

Task Migration and Memory Placement Coordination

A common failure mode in multi-socket systems is Thread Migration: 1. A thread starts executing on Socket 0 and allocates 30 GiB of local memory in Node 0. 2. The Linux CFS/EEVDF CPU scheduler detects that Socket 1 is idle and migrates the thread to Socket 1. 3. Suddenly, 100% of the thread’s memory accesses become expensive remote accesses across the interconnect link!

Automatic NUMA Balancing (kernel.numa_balancing)

To correct this, modern Linux kernels include Auto-NUMA Balancing: - The kernel periodically marks a subset of the process’s page table entries as inaccessible (PROT_NONE). - When the thread accesses the page, a soft NUMA Hinting Page Fault triggers. - The fault handler inspects the CPU core executing the fault: - If the thread is executing on Socket 1, but the page resides on Node 0, the kernel tracks migration hints. - If access frequency remains high, the kernel either migrates the page across the bus to Node 1’s DRAM, or signals the CPU scheduler to migrate the thread back to Socket 0.

Performance Measurement and NUMA-Aware Programming

Inspect NUMA topology and memory allocation distribution:

# Display hardware NUMA node distances and core mapping
numactl --hardware

# View real-time per-node memory allocation statistics
numastat -m
Per-node memory (MB):
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
Active                   24102.4         18201.1         42303.5
Inactive                 12091.2         14012.8         26104.0
MemFree                   8102.5         14201.9         22304.4
Node Distances:
node   0   1 
  0:  10  21 
  1:  21  10

numastat tracks two vital counters: - numa_hit: Successful allocations from the local NUMA node. - numa_miss: Allocations intended for local node that spilled to remote node due to low memory. High numa_miss indicates severe local memory imbalance.