Chapter 18: Benchmarking Tools and Suites

Updated

October 31, 2026

Chapter 18: Benchmarking Tools and Suites

An engineer is only as effective as their instrumentation and benchmarking toolchain. Using the wrong tool or running a utility with flawed arguments produces distorted data that misleads engineering decisions.

This chapter provides a practical, deep-dive reference for the industry-standard benchmarking tools used to stress and profile storage, CPU, memory, networking, and application runtimes on Linux servers.

fio: Storage Benchmarking Deep Dive

Flexible I/O Tester (fio) is the gold standard for storage benchmarking. It simulates any arbitrary block or file access pattern, supporting multiple I/O engines (libaio, io_uring, posix_aio), thread counts, queue depths, and mixed read/write ratios.

Standard NVMe Random Read/Write Job File

Create a declarative job specification nvme-benchmark.fio:

[global]
ioengine=io_uring
direct=1
runtime=60
time_based=1
group_reporting=1
filename=/dev/nvme0n1
size=100%

[random-read-4k]
bs=4k
rw=randread
iodepth=64
numjobs=4
stonewall

[random-write-4k]
bs=4k
rw=randwrite
iodepth=64
numjobs=4
stonewall

[sequential-read-128k]
bs=128k
rw=read
iodepth=16
numjobs=2
stonewall

Execute the job:

sudo fio nvme-benchmark.fio --output=fio-results.json --output-format=json

Key output metrics to evaluate: - iops: Total operations per second. - clat (Completion Latency): Device turnaround time. Inspect the 95th, 99th, and 99.9th percentiles (p99). - lat (Total Latency): Time from submission to completion, including block layer queueing delays.

stress-ng: Comprehensive System Stress Testing

stress-ng stresses Linux systems across hundreds of specialized hardware and kernel stressor modules (virtual memory, schedulers, IPC, file operations, CPU vectors, matrix math):

# Stress CPU ALU, matrix math, and floating point on 8 cores for 60 seconds
stress-ng --cpu 8 --cpu-method matrixprod --timeout 60s --metrics-brief

# Stress virtual memory subsystem (allocating, dirtying, and unmapping 16GB RAM)
stress-ng --vm 4 --vm-bytes 4G --vm-method all --verify --timeout 60s

# Stress context switching and scheduler runqueues
stress-ng --switch 16 --timeout 30s

stress-ng reports bogo-ops/sec (operations per second), providing an immediate measure of system throughput under intense hardware and kernel saturation.

sysbench: CPU, Memory, File I/O, Database Tests

sysbench is a modular multi-threaded benchmark tool:

# 1. CPU Prime Number Benchmark (tests integer execution pipelines)
sysbench cpu --cpu-max-prime=20000 --threads=8 run

# 2. Memory Sequential and Random Throughput
sysbench memory --memory-block-size=1M --memory-total-size=100G --threads=4 run

# 3. Thread Mutex Contention Benchmark
sysbench mutex --mutex-num=1024 --mutex-locks=1000000 --threads=16 run

# 4. OLTP Database Benchmark (PostgreSQL / MySQL)
sysbench /usr/share/sysbench/oltp_read_write.lua \
  --db-driver=pgsql --pgsql-host=127.0.0.1 --pgsql-user=test --pgsql-db=test \
  --tables=10 --table-size=1000000 --threads=32 --time=120 run

iperf3: Network Throughput and Latency

iperf3 measures point-to-point TCP, UDP, and SCTP bandwidth and packet loss:

# High-throughput multi-stream TCP test (saturating 10G/40G links)
# Server node:
iperf3 -s -p 5201

# Client node (8 parallel streams, 60-second duration, 1-second interval reports):
iperf3 -c 10.0.0.1 -p 5201 -P 8 -t 60 -i 1

# UDP test measuring jitter, latency, and packet loss at 1 Gbps rate
iperf3 -c 10.0.0.1 -u -b 1G -t 30

lmbench: Latency and Bandwidth Microbenchmarks

lmbench is the historic suite for measuring low-level OS primitives with microsecond precision: - Null System Call Overhead: Time to enter Ring 0 and return (getpid()). - Context Switch Latency: Time to switch between 2 to 100 competing processes. - Memory Read/Write Bandwidth: Bandwidth across L1, L2, L3 caches, and main DRAM. - Pipe and IPC Latency: Round-trip time of passing tokens through UNIX pipes.

Application-Level: wrk, ApacheBench, curl

wrk (High-Concurrency HTTP Benchmarking)

wrk uses epoll and event loops to generate massive HTTP concurrency from a single client node:

# Run HTTP benchmark with 12 threads and 400 concurrent connections for 30s
wrk -t12 -c400 -d30s --latency http://10.0.0.1:8080/api/v1/health

Using wrk2 adds constant-throughput rate limiting (-R <req/sec>), eliminating coordinated omission and providing accurate p99/p99.9 tail latency percentiles.

curl (Detailed Connection Timing)

For individual request profiling, format curl to print exact phase durations:

curl -w "\nLookup: %{time_namelookup}s\nConnect: %{time_connect}s\nTLS: %{time_appconnect}s\nTTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n" \
  -o /dev/null -s https://example.com/api

Phoronix Test Suite and Comparative Testing

The Phoronix Test Suite (PTS) is an automated open-source benchmarking and validation framework: - Automates download, compilation, execution, and metric parsing for hundreds of standard test profiles (Linux kernel compilation, computational fluid dynamics, OpenSSL encryption, PostgreSQL, 7-zip). - Generates reproducible, standardized XML/HTML reports and supports upload to OpenBenchmarking.org for cross-hardware performance comparisons.