Chapter 20: I/O and Filesystem Tuning for Production

Updated

October 31, 2026

Chapter 20: I/O and Filesystem Tuning for Production

Storage bottlenecks are among the most frequent causes of tail latency degradation in stateful systems. Without deliberate device and filesystem configuration, fast hardware SSDs spend substantial time waiting on software locks, synchronous write stalls, excessive journal commits, and uncoordinated background discards.

This chapter provides a production guide to configuring I/O schedulers, filesystem mount flags, block queue depths, dirty page thresholds, and NVMe driver tuning.

I/O Scheduler Selection by Workload

Configure the optimal block I/O scheduler per device type via udev rules:

# /etc/udev/rules.d/60-io-schedulers.rules

# NVMe drives: use 'none' (direct hardware submission)
ACTION=="add|change", KERNEL=="nvme[0-9]*n[0-9]*", ATTR{queue/scheduler}="none"

# SATA SSDs and fast cloud virtual disks: use 'mq-deadline'
ACTION=="add|change", KERNEL=="sd[a-z]|vd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"

# Mechanical hard drives (rotational): use 'bfq'
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"

Apply immediately without rebooting:

sudo udevadm trigger --subsystem-match=block

Filesystem Mount Options for Performance

When mounting production filesystems for databases, message queues, and high-throughput logging:

Writeback and Dirty Page Tuning

On systems with 64GB to 512GB+ RAM, default percentage-based dirty ratios cause catastrophic multi-second write stalls:

# /etc/sysctl.d/99-storage-writeback.conf

# Start background asynchronous writeback at 256 MB of dirty data
vm.dirty_background_bytes = 268435456

# Force synchronous blocking writeback if dirty pages reach 1 GB
vm.dirty_bytes = 1073741824

# Expire dirty pages after 15 seconds
vm.dirty_expire_centisecs = 1500

# Wakeup writeback flusher threads every 2 seconds
vm.dirty_writeback_centisecs = 200

This enforces smooth, continuous background streaming of dirty pages to persistent storage, preventing massive multi-gigabyte flush bursts that overwhelm disk controller queues.

Device-Level Parameters: Queue Depth and Read-Ahead

Tune block layer device queues:

# Increase block layer request queue depth to prevent request starvation
echo 1024 | sudo tee /sys/block/nvme0n1/queue/nr_requests

# Tune read-ahead based on workload:
# For random I/O (OLTP databases): set read-ahead to 0 to avoid wasted reads
echo 0 | sudo tee /sys/block/nvme0n1/queue/read_ahead_kb

# For sequential streaming (Kafka, Hadoop, video): set read-ahead to 512KB
echo 512 | sudo tee /sys/block/nvme0n1/queue/read_ahead_kb

SSD and NVMe-Specific Tuning

  1. Disable Continuous Discard (nodiscard): Real-time TRIM on every file deletion incurs severe synchronous latency penalties. Mount with nodiscard and execute scheduled periodic trims via fstrim.timer:

    sudo systemctl enable --now fstrim.timer

  2. NVMe Interrupt Polling (io_poll): For sub-10\(\mu\)s database latency, enable polling queues on high-end NVMe drives:

    echo 1 | sudo tee /sys/block/nvme0n1/queue/io_poll

  3. Hardware Sector Size: Ensure the NVMe drive is formatted with native 4096-byte LBA sectors (4Kn) rather than emulated 512-byte sectors (512e) to eliminate Read-Modify-Write penalties inside the flash controller:

    sudo nvme id-ns -H /dev/nvme0n1 | grep -i "lba format"

Validating I/O Performance Improvements

Execute before-and-after tests using fio and biolatency to verify that queue times and p99 completion latency dropped:

# 1. Measure disk I/O latency distribution in real time using eBPF
sudo biolatency-bpfcc -m 5 1

# 2. Execute standardized 4KB random write benchmark
fio --name=randwrite --filename=/data/test.bin --size=10G --bs=4k \
    --rw=randwrite --ioengine=io_uring --direct=1 --iodepth=32 \
    --numjobs=4 --runtime=60 --group_reporting --time_based

A properly tuned storage stack should show: - 0% multi-second latency outliers in biolatency. - Tightly clustered p99 latency in fio (< 200\(\mu\)s on NVMe). - Predictable steady-state write throughput without periodic dips.