Chapter 7: The Page Cache and Memory-Mapped Files
Chapter 7: The Page Cache and Memory-Mapped Files
The Linux Page Cache is the central caching layer between persistent block storage and application address spaces. By keeping frequently accessed file data in unused physical memory, the Page Cache accelerates read and write performance by orders of magnitude compared to direct disk access.
This chapter examines the internal architecture of the Page Cache, read-ahead algorithms, memory-mapped files (mmap), cache pressure eviction, and dirty page writeback tuning.
Page Cache: Architecture and Behavior
Whenever an application issues a standard read() or write() system call, the kernel operates on the Page Cache rather than talking directly to the disk:
┌────────────────────────────────────────────────────────────────────────┐
│ PAGE CACHE DATA FLOW ARCHITECTURE │
│ │
│ Userspace Process │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ Application Buffer (read() / write()) │ │
│ └───────────────────────────────┬────────────────────────────────┘ │
│ │ copy_to_user / copy_from_user │
│ Kernel Page Cache ▼ │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ Inode Address Space (struct address_space) │ │
│ │ Indexed via XArray / Radix Tree of Page Folios │ │
│ │ [ Folio 0 ] [ Folio 1 (Dirty) ] [ Folio 2 (Clean) ] │ │
│ └───────────────────────────────┬────────────────────────────────┘ │
│ │ Async Writeback (wb_workfn) │
│ Block Layer & Storage ▼ │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ NVMe / SSD Storage Media │ │
│ └────────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────┘- Address Space Indexing: Every cached file is represented by a
struct address_spaceattached to its filesystem inode. Cached pages/folios are indexed by their file offset using an efficient in-memory XArray (formerly radix tree). - Read Path: The kernel searches the XArray for the requested file offset. If found (Page Cache Hit), data is copied directly to the user buffer in nanoseconds without generating disk I/O. If missing (Page Cache Miss), the kernel triggers a synchronous read from block storage.
- Write Path: Writes update the in-memory page folio and mark it Dirty. The
write()system call returns immediately to userspace. Disk writes are deferred and flushed asynchronously by background kernel threads.
Read-Ahead and Write-Behind
Adaptive Read-Ahead
When the kernel detects sequential read patterns, it speculatively loads upcoming file blocks into the Page Cache before the application requests them:
- Initial Window: Starts small (e.g., 16KB).
- Expansion Window: Scales dynamically up to
read_ahead_kb(typically 128KB–512KB) as long as sequential hits continue. - Application Hints (
posix_fadvise): Applications can explicitly instruct the kernel on expected access patterns:POSIX_FADV_SEQUENTIAL: Doubles the read-ahead window.POSIX_FADV_RANDOM: Disables read-ahead entirely to avoid wasting I/O bandwidth.POSIX_FADV_WILLNEED: Triggers non-blocking background population of the Page Cache.
# Query and adjust device read-ahead window (in 512-byte sectors)
sudo blockdev --getra /dev/nvme0n1
sudo blockdev --setra 1024 /dev/nvme0n1 # Set to 512 KBMemory-Mapped Files and mmap
Instead of copying data between kernel buffers and userspace buffers via read() and write(), applications can map files directly into their virtual address spaces using mmap():
// Map 1GB file into userspace address space
int fd = open("large_data.bin", O_RDWR);
void *addr = mmap(NULL, 1024*1024*1024, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);mmap vs read() / write() Trade-offs
- Zero-Copy Access: Memory-mapped reads avoid copying bytes through
copy_to_user(). Userspace pointer dereferences directly touch the Page Cache frame. - TLB Shootdown and Page Fault Cost: Initial access triggers minor page faults to populate the process’s page table entries. On multi-threaded systems with frequent unmapping, TLB shootdown interrupts across cores can negate zero-copy savings.
- Modern high-performance storage engines (e.g., LMDB, RocksDB, Kafka) select
mmapor standard I/O based on working-set-to-RAM ratios.
Cache Pressure and Eviction Policies
Under memory pressure, the kernel must reclaim memory pages. Linux categorizes pages using Least Recently Used (LRU) lists:
- Active List: Pages touched recently. Protected from immediate eviction.
- Inactive List: Eviction candidates. Clean inactive pages are dropped immediately; dirty pages are queued for writeback.
The balance between reclaiming file-backed Page Cache pages versus anonymous heap/stack pages is controlled by vm.vfs_cache_pressure and vm.swappiness:
# Adjust inode/dentry cache reclamation priority (default 100)
# Values > 100 reclaim dentries/inodes more aggressively
sudo sysctl vm.vfs_cache_pressure=100Dirty Page Management and Writeback
Dirty pages accumulate in RAM until flushed by background writeback workers (kworker/flush). Misconfigured dirty ratios cause massive “write stalls” where interactive latency blows up because userspace processes are forced to freeze and write dirty pages to disk synchronously.
Critical Writeback Knobs
# Percentage of available memory where background writeback starts
cat /proc/sys/vm/dirty_background_ratio # Typical default: 10%
# Hard percentage where userspace processes block and perform synchronous writes
cat /proc/sys/vm/dirty_ratio # Typical default: 20%
# Maximum age of dirty data before writeback is forced (in hundredths of a second)
cat /proc/sys/vm/dirty_expire_centisecs # Typical default: 3000 (30 seconds)[!IMPORTANT] On modern enterprise servers with 512GB+ RAM, default ratios are disastrous:
dirty_ratio=20means the kernel permits 100GB of dirty data to accumulate in RAM before blocking applications. When flushing starts, storage queues choke and latency spikes to tens of seconds. Production Fix: Use byte-based limits on large RAM hosts:sudo sysctl -w vm.dirty_background_bytes=268435456 # 256 MB sudo sysctl -w vm.dirty_bytes=1073741824 # 1 GB