Chapter 16: System Monitoring and Observability

Updated

October 31, 2026

Chapter 16: System Monitoring and Observability

Long before an incident prompts deep-dive kernel tracing, continuous system monitoring must establish normal baseline behavior, identify emerging resource saturation, and fire early warning alerts. Linux provides both time-tested classic monitoring utilities and cutting-edge metrics like Pressure Stall Information (PSI).

This chapter teaches the correct interpretation of core monitoring tools, exposes misleading counters, and provides patterns for building production dashboards and alerts.

Classic Tools: sar, vmstat, iostat, mpstat, pidstat

The sysstat suite and core Linux utilities provide the foundation of command-line performance triage:

vmstat 1 (System Overview)

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 412030  12400 892010    0    0    12    45  820 1450 45 12 40  3  0
  • r (Runqueue): Runnable threads waiting for CPU time. If r > number of CPU cores, the system is CPU-saturated.
  • b (Blocked): Threads sleeping in uninterruptible disk I/O (TASK_UNINTERRUPTIBLE / D-state).
  • si / so: Swap-in and swap-out rates in KB/s. Any continuous non-zero values indicate critical memory exhaustion.
  • wa (iowait): Time CPUs spent idle while at least one I/O request was in flight. High wa means the workload is I/O-bound.

iostat -xz 1 (Extended Storage Metrics)

# Monitor disk utilization and queue latency, hiding idle devices (-z)
iostat -xz 1
  • r/s and w/s: Read and write requests completed per second (IOPS).
  • r_await and w_await: Average time in milliseconds that read and write requests took from creation to device completion (includes block queue time + disk service time).
  • %util: Percentage of CPU time during which requests were in flight to the device. Note: on modern multi-queue SSDs, %util = 100% does not mean the drive is saturated, because multi-queue NVMe devices can handle 64 parallel queues simultaneously. Use r_await and queue depth instead.

mpstat -P ALL 1 (Per-Core Breakdown)

Identifies single-core CPU saturation: - %usr: Time spent in userspace code. - %sys: Time spent executing kernel code and system calls. - %si: Time spent handling software interrupts (softirqs). If a single core shows 90% %si, network packet processing is bottlenecked on that core.

pidstat 1 (Per-Process Attribution)

Identifies which specific process is driving system resource consumption: - pidstat -u 1: CPU usage by process. - pidstat -d 1: Disk I/O read/write rates in KB/s per process. - pidstat -w 1: Voluntary vs involuntary context switches per process.

Modern Monitoring: top, htop, numastat

  • top / htop: Real-time process viewers. In top, press 1 to toggle per-core display; press H to toggle thread-level visibility. Pay close attention to process state: R (running), S (interruptible sleep), D (uninterruptible disk sleep), and Z (zombie).
  • numastat: Shows per-NUMA node memory allocation hits and misses. High miss ratios indicate memory is being allocated on remote sockets, causing interconnect latency penalties.

Network Monitoring: ss, ethtool, nstat

# Inspect socket queue backlogs (Send-Q and Recv-Q)
ss -tulpn

# Monitor hardware NIC error counters and dropped frames
ethtool -S eth0 | grep -E "drop|miss|error|fifo"

# View Linux IP/TCP protocol engine error counters
nstat -az
  • In ss -tln: For listening sockets, Recv-Q indicates the number of connections waiting in the accept queue (somaxconn), while Send-Q indicates the maximum queue size. If Recv-Q == Send-Q, the application is failing to accept connections fast enough, causing incoming client connections to time out.

Pressure Stall Information (PSI) and Modern Metrics

Traditional load averages (/proc/loadavg) are flawed: they blend CPU and disk D-state tasks together, make no distinction between a 4-core machine and a 128-core machine, and react with slow exponential decay.

Pressure Stall Information (PSI), introduced in modern Linux, measures the exact percentage of time tasks are stalled waiting for CPU, Memory, or I/O:

# View CPU pressure
cat /proc/pressure/cpu

# View Memory pressure
cat /proc/pressure/memory

# View I/O pressure
cat /proc/pressure/io

Example PSI output:

some avg10=2.45 avg60=1.12 avg300=0.50 total=4820194
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
  • some: The percentage of wall-clock time during which at least one task was stalled waiting for the resource while other tasks could still make progress.
  • full: The percentage of wall-clock time during which all runnable tasks were stalled waiting for the resource (total system freeze / livelock).

PSI provides an unambiguous, objective saturation indicator that powers modern container resource managers and Kubernetes autoscalers.

Building Production Dashboards and Alerting

When engineering production dashboards (e.g. Prometheus + Grafana):

  1. Dashboard Structure:
    • Level 1 (Top Level): Red/Amber/Green service SLOs (p95/p99 latency, error rate, request throughput).
    • Level 2 (Host Saturation): PSI metrics (avg10 for CPU, memory, and I/O pressure), network saturation (ss backlog, NIC drops).
    • Level 3 (Subsystem Detail): Per-core %si, memory PSS, block r_await/w_await, filesystem dirty page count.
  2. Alerting Rules:
    • Alert on symptoms affecting SLOs (e.g. p99 latency > target for 3 minutes).
    • Alert on leading indicators of failure (e.g. Memory PSI some > 20% for 2 minutes, indicating imminent swapping and OOM-killer invocation).
    • Avoid alerting on raw CPU utilization alone; a system at 95% CPU with 0% CPU PSI pressure and stable p99 latency is simply running efficiently.