Chapter 9: Storage Technologies and Performance Characteristics

Updated

October 31, 2026

Chapter 9: Storage Technologies and Performance Characteristics

Storage media physical characteristics determine the fundamental limits of latency, throughput, and parallelism. An application architect cannot optimize storage I/O without knowing whether the underlying device is a mechanical spinning platter, a multi-layer 3D TLC NAND flash drive, or a virtualized cloud network block volume with provisioned burst credits.

This chapter compares the physical constraints and architectural behaviors of HDDs, SATA/SAS SSDs, NVMe devices, RAID controllers, and cloud storage volumes.

HDDs: Seek Latency, Rotational Latency and Throughput

Mechanical Hard Disk Drives (HDDs) are governed by physical mechanics:

  • Seek Latency: Moving the mechanical actuator arm across concentric tracks (~3–10ms).
  • Rotational Latency: Waiting for the requested sector to rotate under the drive head (at 7200 RPM, average rotation takes 4.16ms).
  • Total Random Access Latency: Typically 8ms to 15ms per random operation. This limits a single HDD to approximately 75 to 150 random IOPS.
  • Sequential Throughput: When the drive head reads continuous sectors along a track, bandwidth reaches 150MB/s to 260MB/s.

[!NOTE] On HDDs, random I/O is catastrophically slower than sequential I/O (a 1000x difference in effective throughput). Filesystems and database engines designed for HDDs go to great lengths to convert random writes into sequential appends (e.g., WAL logs, LSM-trees).

SSDs: NAND Characteristics, Wear Leveling and GC

Solid-State Drives (SSDs) use non-volatile NAND flash memory:

  • Page Reads vs Block Erases: NAND flash reads and programs data in Pages (typically 4KB to 16KB). However, flash cells cannot be overwritten in place; they must first be erased in large Erase Blocks (typically 4MB to 16MB).
  • Garbage Collection (GC): As files are updated, existing pages become invalid. The drive controller periodically copies surviving valid pages to a new erase block and erases the old block in the background. Heavy GC activity causes severe latency spikes and write amplification.
  • Write Amplification Factor (WAF): The ratio of bytes written to NAND flash versus bytes written by the operating system: \[\text{WAF} = \frac{\text{Bytes Written to NAND}}{\text{Bytes Written by Host}}\]
  • TRIM / Discard: Informs the SSD controller which logical blocks no longer hold valid data, enabling efficient background GC without copying dead pages.

NVMe: Protocol Advantages and Queue Architecture

Non-Volatile Memory Express (NVMe) was designed from scratch for solid-state storage, replacing legacy SATA/AHCI protocols that were engineered around spinning disks:

┌────────────────────────────────────────────────────────────────────────┐
│                        AHCI (SATA) vs NVMe PROTOCOL                    │
│                                                                        │
│   Legacy AHCI (SATA)                   Modern NVMe                     │
│   ├─ 1 Command Queue                   ├─ Up to 64,000 Command Queues  │
│   ├─ Max Queue Depth: 32 commands      ├─ Max Queue Depth: 64,000 cmds │
│   ├─ Requires single register lock     ├─ Lockless per-CPU queue pairs │
│   └─ High CPU interrupt overhead       └─ Direct PCIe Gen4/Gen5 fabric │
│                                                                        │
│   Latency: ~500µs                      Latency: ~10µs - 50µs           │
│   Max Throughput: ~550 MB/s            Max Throughput: 7,000 - 14,000+ │
└────────────────────────────────────────────────────────────────────────┘

By placing submission and completion queue pairs on individual CPU cores, NVMe eliminates software lock contention, achieving millions of IOPS per server.

RAID and Hardware Controllers

RAID arrays combine multiple physical drives to balance fault tolerance, capacity, and performance:

RAID Level Min Disks Read Performance Write Performance Failure Tolerance
RAID 0 2 \(N \times\) throughput \(N \times\) throughput 0 drives (loss of any drive destroys array)
RAID 1 2 \(N \times\) throughput Single drive throughput \(N - 1\) drives
RAID 10 4 \(N \times\) throughput \(\frac{N}{2} \times\) throughput 1 drive per mirror set (up to \(\frac{N}{2}\))
RAID 5 3 \((N-1) \times\) throughput Penalized by parity write (\(\approx \frac{1}{4}\) speed) 1 drive
RAID 6 4 \((N-2) \times\) throughput Severe double parity write penalty 2 drives

Hardware Controller Cache Modes

Hardware RAID controllers feature onboard DRAM caches backed by batteries or supercapacitors (BBU / Flash-Backed Write Cache):

  • Write-Back (WB): The controller acknowledges writes as soon as data hits the battery-backed DRAM, delivering sub-millisecond write latency and absorbing write bursts.
  • Write-Through (WT): The controller waits until data is physically written to the underlying disks before acknowledging the write. Safe, but degrades write latency to raw disk speeds.

[!CAUTION] If a RAID controller’s battery fails, it typically falls back from Write-Back to Write-Through automatically, resulting in an unannounced 90% drop in database write throughput. Always monitor controller battery health via megacli or storcli.

Cloud Block Storage and Shared Storage

Cloud virtual disks (AWS EBS, GCP Persistent Disk, Azure Managed Disks, Ceph/SAN) are not local disks; they are network-attached storage services accessed over virtualized Ethernet fabrics:

  • Baseline vs Burst IOPS: Cloud volumes provide baseline IOPS tied to allocated volume size, with burst credits that deplete during prolonged write operations.
  • EC2 / Instance Throttling: Even if an EBS volume supports 64,000 IOPS, the host EC2 instance has hard limits on EBS bandwidth and network throughput (EBS-optimized bandwidth limits).
  • Network Latency Floor: Because every request travels over an internal data center network, the latency floor is rarely lower than 500\(\mu\)s–1ms, compared to 15\(\mu\)s on local bare-metal NVMe.

Choosing Storage for the Workload

Workload Type Ideal Storage Architecture Key Considerations
High-Throughput Databases (OLTP) Local NVMe in RAID 10 or raw NVMe with software replication Prioritize low p99 write latency and high random IOPS; configure write-back or battery-backed storage.
Time-Series / Append Logs (Kafka) Striped NVMe or SAS SSDs (RAID 10 / filesystem extents) High sequential write throughput; maximize Page Cache dirty writeback bandwidth.
Warm Data & Object Stores High-density SATA enterprise HDDs in RAID 6 or erasure-coded ZFS Prioritize lowest $/TB cost; sequential reads dominate; random reads buffered by NVMe cache tiers.
Stateless Cloud Applications Provisioned Cloud SSDs with predictable IOPS Ensure VM network bandwidth does not throttle disk queue depth; monitor EBS burst exhaustion.