Chapter 6: NUMA Architecture and Memory Placement
Chapter 6: NUMA Architecture and Memory Placement
On modern multi-socket and high-core-count servers, uniform memory access is physically impossible. Memory controllers are integrated directly into processor sockets and multi-chip modules (CCDs). This creates a Non-Uniform Memory Access (NUMA) architecture where the latency and bandwidth of accessing physical memory depend heavily on which CPU core issues the load instruction and which physical memory channel holds the data.
This chapter explores the performance implications of NUMA topologies, diagnostics, allocation policies, and optimization techniques.
What NUMA Means for Performance
In a NUMA system, the machine is partitioned into multiple NUMA nodes. Each node consists of a group of CPU cores and their physically adjacent memory channels:
┌────────────────────────────────────────────────────────────────────────┐
│ DUAL-SOCKET NUMA TOPOLOGY │
│ │
│ NUMA Node 0 NUMA Node 1 │
│ ┌───────────────────────────┐ ┌────────────────────────┐ │
│ │ Cores 0 - 31 │ │ Cores 32 - 63 │ │
│ │ Local Memory Controller │ │ Local Memory Controller│ │
│ │ 256 GB Local DDR5 │ │ 256 GB Local DDR5 │ │
│ └─────────────┬─────────────┘ └────────────┬───────────┘ │
│ │ │ │
│ └─────────── Interconnect Link ──────────┘ │
│ (Intel UPI / AMD Infinity Fabric) │
│ Latency: ~140ns (vs ~75ns Local) │
│ Bandwidth Constrained │
└────────────────────────────────────────────────────────────────────────┘Accessing memory on the local node incurs minimal latency (~75ns). Accessing memory attached to a remote socket requires traversing the inter-socket interconnect (Intel Ultra Path Interconnect - UPI, or AMD Infinity Fabric), adding 60ns–100ns of latency and consuming inter-socket interconnect bandwidth.
NUMA Nodes, Local vs Remote Memory Access
The kernel discovers NUMA distances from the ACPI System Locality Distance Information Table (SLIT):
# Display NUMA hardware topology, core mapping, and distance matrix
numactl --hardwareExample SLIT distance matrix:
node distances:
node 0 1
0: 10 21
1: 21 10 A value of 10 represents normalized local access latency. A distance of 21 indicates that remote accesses incur more than double the access cost.
NUMA-Aware Allocation and Policies
Linux provides memory allocation policies that control where new pages are physically allocated:
- Local Allocation (Default): The kernel attempts to allocate memory on the same NUMA node as the thread executing the allocation (
MPOL_DEFAULT). - Interleaved (
MPOL_INTERLEAVE): Allocations alternate across specified NUMA nodes in round-robin fashion (page by page). Ideal for large multi-threaded shared caches that exceed single-node memory capacity. - Preferred (
MPOL_PREFERRED): Strongly prefers allocating on a specific node, falling back to other nodes if memory is exhausted. - Bind (
MPOL_BIND): Strictly allocates memory from specified nodes. Fails if memory on those nodes is exhausted.
Diagnosing NUMA Imbalance
When memory is allocated unevenly, or threads run on Node 0 while accessing memory on Node 1, severe performance degradation occurs.
# Check memory allocation and hit/miss counters per NUMA node
numastat -cKey counters to observe: - numa_hit: Pages allocated successfully on the local node. - numa_miss: Pages allocated on a remote node because the intended local node was full. - numa_foreign: Pages intended for another node that ended up allocated here. - local_node vs other_node: High ratios of other_node indicate severe cross-NUMA traffic.
Profiling Cache-to-Cache Traffic with perf c2c
The perf c2c (cache-to-cache) tool samples memory load/store events to detect cross-socket cache line bouncing:
# Record memory access events to detect false sharing and remote NUMA contention
sudo perf c2c record -F 60000 -- ./application-workload
sudo perf c2c report --stdioOptimizing for NUMA: Placement and Binding
Binding with numactl
To eliminate cross-NUMA latency, launch critical workloads bound entirely to a single NUMA node (both CPU cores and memory):
# Execute pinned to Node 0 cores and Node 0 memory
numactl --cpunodebind=0 --membind=0 ./redis-server /etc/redis/node0.conf
# Interleave memory across nodes for large in-memory databases (e.g., PostgreSQL)
numactl --interleave=all ./postgres -D /data/postgresAutomatic NUMA Balancing (kernel.numa_balancing)
Linux includes an autonomous balancing daemon that periodically unmaps memory pages to generate minor page faults. By inspecting which CPU core touched the page, the kernel dynamically migrates pages to the accessing node.
[!TIP] While helpful for generic batch workloads, Automatic NUMA Balancing introduces periodic TLB shootdown and migration overhead. For predictable low latency in databases or trading systems, disable automatic balancing (
sysctl kernel.numa_balancing=0) and enforce static binding usingnumactlor systemd cpusets.
NUMA in Virtualized and Cloud Environments
In virtualized environments (KVM, VMware, AWS/GCP bare-metal and large instances):
- vNUMA Topology: Hypervisors must expose a virtual NUMA topology to the guest that accurately mirrors the physical NUMA layout of the underlying bare metal.
- Cross-NUMA Overcommit: If a virtual machine is assigned 32 vCPUs spanning two physical NUMA nodes without vNUMA awareness, the guest operating system treats memory as uniform, leading to pathological cross-socket interconnect thrashing.
- Cloud Instance Selection: Large cloud instances (e.g., AWS c6i.32xlarge, m6a.48xlarge) span multiple NUMA sockets. Infrastructure engineers must verify socket layout using
lscpuand configure application worker thread pools per NUMA socket.