Chapter 23: Databases and High-Performance Applications

Updated

October 31, 2026

Chapter 23: Databases and High-Performance Applications

Databases and high-performance applications push the Linux operating system to its absolute limits. Relational databases (PostgreSQL, MySQL), LSM-tree key-value engines (RocksDB, Cassandra), and modern application runtimes (JVM, Go, Python) interact with memory, disk, and CPU scheduling in specialized ways. Configuring the host operating system without understanding these runtime models guarantees severe performance degradation.

This chapter details database I/O patterns, shared memory architectures, lock contention, NUMA binding, runtime optimization, and tail latency mitigation.

How Databases Use Linux: I/O and Memory Patterns

Databases typically maintain two distinct I/O paths:

┌────────────────────────────────────────────────────────────────────────┐
│                        DATABASE I/O ENGINE DUALITY                     │
│                                                                        │
│   Write-Ahead Log (WAL / Redo Log)             Data Buffer Pool        │
│   ├─ Sequential append-only writes             ├─ Large in-memory cache│
│   ├─ fsync() on every transaction commit       ├─ Random reads & writes│
│   ├─ Latency-critical (governs client p99)     ├─ Checkpoint flushes   │
│   └─ Separate disk/filesystem recommended      └─ O_DIRECT or PageCache│
└────────────────────────────────────────────────────────────────────────┘
  1. Write-Ahead Logging (WAL): Ensures ACID durability. Writes are strictly sequential appends, followed immediately by an fsync(). The round-trip commit latency of the WAL disk determines the maximum possible single-thread transaction throughput of the database.
  2. Buffer Pool Management: Databases manage their own internal page caches (e.g. PostgreSQL shared_buffers, MySQL InnoDB Buffer Pool). If the database uses standard buffered I/O, data is cached twice: once in the database buffer pool and once in the Linux Page Cache (“double buffering”), wasting physical RAM.

Shared Memory and IPC for Databases

PostgreSQL and enterprise databases rely on shared memory to communicate across worker processes:

  • POSIX Shared Memory (shm_open) vs System V IPC (shmget): Modern applications use POSIX shared memory mapped via /dev/shm (tmpfs).
  • Huge Pages for Shared Buffers: Configuring PostgreSQL shared_buffers with 2MB or 1GB Huge Pages is mandatory for multi-gigabyte buffer pools. Huge Pages prevent page table thrashing and protect the database cache from being accidentally swapped out by the kernel.
# postgresql.conf
shared_buffers = 32GB
huge_pages = on              # Fails to start if hugetlbfs is not available

Locking and Contention in Database Workloads

High-concurrency database transactions trigger lock contention in both userspace and kernel space:

  • Userspace Spinlocks vs Futexes: Databases use lightweight spinlocks for brief buffer header updates and futex sleep locks for long-running row or page locks. Under extreme concurrency, threads spin on shared memory cache lines, causing CPU saturation without useful work.
  • Kernel Lock Contention: Traced with perf lock record or bpftrace. High contention on filesystem inode mutexes or mm page table locks during concurrent checkpoints indicates need for partitioning or switching to XFS allocation groups.

NUMA and Database Placement

For databases larger than the memory capacity of a single NUMA socket:

  • Cross-NUMA Thrashing: If PostgreSQL processes run on Node 0 while shared memory buffers reside on Node 1, every query pays an interconnect penalty.
  • Interleaving vs Node Binding:
    • For single database instances spanning the whole machine: launch with numactl --interleave=all to evenly stripe buffer pages across all memory channels, maximizing aggregate memory bandwidth.
    • For multi-instance architectures: run multiple independent PostgreSQL instances, each strictly pinned to a dedicated NUMA node (numactl --cpunodebind=0 --membind=0).

Application Runtimes: JVM, Go, Python Considerations

Runtime Execution Model Performance Bottlenecks & Tuning
JVM (Java) JIT compiled, managed heap GC Pauses: Stop-the-world pauses destroy p99 latency. Use modern low-pause collectors (ZGC or Shenandoah). Set -XX:+AlwaysPreTouch to allocate all pages at startup and avoid runtime page faults.
Go Native machine code, M:N scheduler GOMAXPROCS: By default equals physical host cores. In containers with CPU quotas, explicitly set GOMAXPROCS to match the integer CPU limit (e.g. using uber-go/automaxprocs) to avoid goroutine scheduling stalls.
Python Bytecode interpreter, GIL GIL Contention: Multi-threaded Python cannot scale across multiple CPU cores for CPU-bound tasks. Use multi-process architectures (e.g. gunicorn with worker processes) or Python 3.13+ free-threaded builds (--disable-gil).

Tail Latency Optimization

Tail latency (p99 and p99.9) dictates user experience in distributed systems. Eliminating tail latency spikes requires removing all sources of non-deterministic jitter:

  1. Pre-Fault Application Memory: Call mlockall(MCL_CURRENT | MCL_FUTURE) or runtime equivalents to pin all code and data into physical RAM, eliminating major page faults during execution.
  2. Use Epoll / io_uring Asynchronous Architectures: Eliminate synchronous blocking I/O calls on request-handling worker threads.
  3. Decouple Checkpoints and Compactions: Ensure background LSM-tree compactions (RocksDB) or database checkpoints execute on isolated background threads with lowered I/O priority (ionice -c 3).