Chapter 8: I/O Subsystems: From Block Layer to Disk

Updated

October 31, 2026

Chapter 8: I/O Subsystems: From Block Layer to Disk

The Linux storage I/O subsystem bridges high-level POSIX filesystem operations with physical storage controllers. Over the last decade, storage bandwidth evolved from hundreds of IOPS on spinning rust (HDDs) to millions of IOPS on multi-channel NVMe SSDs. This transition forced a total architectural overhaul of the Linux block layer from the single-lock legacy request queue to the multi-queue (blk-mq) architecture.

This chapter traces an I/O request from the initial system call down to the physical disk controller, examining request queues, I/O schedulers, bio splitting, and completion interrupts.

The I/O Path: From open() to the Disk

Every storage I/O operation traverses distinct kernel layers:

┌────────────────────────────────────────────────────────────────────────┐
│                         STORAGE I/O DATA PATH                          │
│                                                                        │
│   Userspace (open / read / write / fsync / io_uring)                   │
│                                │                                       │
│                                ▼                                       │
│   Virtual Filesystem (VFS) Layer (sys_write / vfs_write)               │
│                                │                                       │
│                                ▼                                       │
│   Concrete Filesystem (ext4, XFS, btrfs)                               │
│   - Translates file offset to logical block addresses (LBAs)           │
│   - Allocates extents and builds bio structures                        │
│                                │                                       │
│                                ▼                                       │
│   Multi-Queue Block Layer (blk-mq)                                     │
│   - Software Staging Queues (per-CPU) ──► I/O Scheduler (Optional)    │
│   - Hardware Dispatch Queues (per-channel)                             │
│                                │                                       │
│                                ▼                                       │
│   Device Driver (nvme, mpt3sas, virtio-blk)                            │
│   - Maps DMA scatter-gather lists to controller ring buffers           │
│                                │                                       │
│                                ▼                                       │
│   Physical Storage Controller & Flash/Disk Media                       │
└────────────────────────────────────────────────────────────────────────┘

The Block Layer and Request Queues (blk-mq)

In the legacy Linux block layer, all CPU cores submitted requests to a single global request queue protected by a single spinlock. On modern NVMe devices capable of 1M+ IOPS, that single lock became an immediate bottleneck.

The modern Multi-Queue Block Layer (blk-mq) eliminates this contention through a two-tiered queue design:

  1. Software Staging Queues: Per-CPU or per-core software queues. Tasks submit requests locklessly to their local CPU queue.
  2. Hardware Dispatch Queues: Mapped directly to the physical submission queues of the storage hardware controller (e.g., NVMe controllers support up to 64,000 independent hardware queues).

I/O Schedulers: noop, deadline, cfq, mq-deadline, kyber, bfq

I/O schedulers decide the order and timing of requests dispatched to storage devices:

Scheduler Architecture Primary Strength Recommended Use Case
none Direct pass-through Zero software queue overhead, lowest CPU utilization High-speed NVMe drives with native hardware queues
mq-deadline Deadline queues (read/write) Prevents read starvation by prioritizing reads over writes SATA SSDs, SAS SSDs, fast cloud block devices
bfq (Budget Fair Queue) Proportional weight sharing Interactive responsiveness and fairness under heavy write load Desktop workstations, SATA spinning disks (HDDs)
kyber Latency-target feedback loop Regulates queue depths to maintain strict read latency limits Shared cloud nodes, fast flash with high write concurrency 
# Query active and available schedulers for a device
cat /sys/block/nvme0n1/queue/scheduler

# Switch scheduler dynamically (e.g., set to 'none' on NVMe)
echo "none" | sudo tee /sys/block/nvme0n1/queue/scheduler

Block Devices and Queueing Models

A storage device’s ability to absorb parallel requests is governed by queue depth:

  • Queue Depth (nr_requests): The maximum number of requests the block layer permits to be concurrently staged.
  • Little’s Law for Storage: \[\text{Throughput} = \frac{\text{Queue Depth}}{\text{Latency}}\] To achieve 1,000,000 IOPS on a device with 100\(\mu\)s latency, the host must sustain an active queue depth of: \[QD = 1,000,000 \times 0.0001 = 100\] If the application sends synchronous single-threaded requests with \(QD=1\), the maximum possible throughput is capped at only 10,000 IOPS regardless of how fast the NVMe drive is.

I/O Merge, Splitting and Bio Structures

The fundamental unit of I/O in the Linux block layer is struct bio:

  • A bio represents an in-flight I/O request composed of one or more memory vectors (scatter-gather list of physical memory folios).
  • Bio Merging: If an application submits adjacent small block requests, the block layer merges them into a single contiguous struct request before sending it to the controller, amortizing DMA setup overhead.
  • Bio Splitting: If a bio exceeds the device’s maximum transfer size (max_sectors_kb), the block layer splits it into multiple smaller requests.
# Inspect maximum request and sector limits
cat /sys/block/nvme0n1/queue/max_sectors_kb
cat /sys/block/nvme0n1/queue/hw_sector_size

I/O Completion and Interrupt Handling

When the storage controller completes an I/O operation:

  1. Hardware Interrupt: The controller fires an MSI-X interrupt targeted to the CPU core that submitted the request.
  2. Interrupt Service Routine (ISR): The driver acknowledges the controller’s completion queue.
  3. Softirq Callback: BLOCK_SOFTIRQ executes, completing the bio, invoking filesystem end-I/O callbacks, and waking up sleeping threads or completing io_uring ring entries.

Hybrid Polling Mode for Extreme Latency

On modern ultra-low-latency NVMe and Optane devices, the overhead of interrupt handling (~2–5\(\mu\)s) is larger than the media access latency. Linux supports polling mode (io_uring with IORING_SETUP_IOPOLL or /sys/block/<dev>/queue/io_poll), where the application thread directly polls the NVMe completion queue, achieving sub-10\(\mu\)s deterministic latency.