Chapter 15: Performance Analysis Tools and Flame Graphs

Updated

October 31, 2026

Chapter 15: Performance Analysis Tools and Flame Graphs

When profiling a complex application running millions of lines of code across thousands of threads, raw text profiler outputs (such as flat function tables) become overwhelming. Flame Graphs, invented by Brendan Gregg, convert deep call stacks into an intuitive, interactive visual representation that allows engineers to identify CPU bottlenecks and off-CPU blocking latency in seconds.

This chapter covers the construction, interpretation, and automation of on-CPU and off-CPU Flame Graphs using perf, eBPF, and continuous profiling pipelines.

Flame Graphs: Construction and Interpretation

A Flame Graph visualizes sampled stack traces:

┌────────────────────────────────────────────────────────────────────────┐
│                        FLAME GRAPH ANATOMY                             │
│                                                                        │
│   Y-Axis: Stack Depth (Bottom = Root / Entry, Top = Leaf / Active)     │
│   X-Axis: Alphabetical Sorting of Samples (Width = % of Total Samples) │
│                                                                        │
│   ┌────────────────────────────────────────────────────────────────┐   │
│   │               do_checksum() [25% width]                        │   │
│   ├────────────────────────────────┬───────────────────────────────┤   │
│   │    parse_json_payload() [40%]  │  serialize_protobuf() [30%]   │   │
│   ├────────────────────────────────┴───────────────────────────────┤   │
│   │                process_http_request() [75% width]              │   │
│   ├────────────────────────────────────────────────────────────────┤   │
│   │                        worker_loop() [90% width]               │   │
│   ├────────────────────────────────────────────────────────────────┤   │
│   │                         main() [100% width]                    │   │
│   └────────────────────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────────────────────┘

Reading Rules

  1. Width Represents Sample Proportion: The wider a frame appears on the horizontal axis, the more times that function was present in the sampled stack traces.
  2. Horizontal Axis Is Not Time: The x-axis does not represent the passage of time; stacks are sorted alphabetically to merge identical code paths.
  3. Plateaus Are Hotspots: A wide frame with a flat top (“plateau”) indicates that the function itself was actively executing on the CPU (leaf frame), burning cycles directly rather than calling sub-functions.

perf Record and Report

Generating an On-CPU Flame Graph begins with sampling stack traces using perf:

# 1. Sample stack traces at 99 Hertz across all CPU cores for 30 seconds
# Using 99Hz instead of 100Hz avoids sampling in lockstep with periodic timers
sudo perf record -F 99 -a -g -- sleep 30

# 2. Inspect results interactively in terminal
sudo perf report -n --stdio

Unwinding Call Stacks: Frame Pointers vs DWARF

To reconstruct full stack traces, perf must unwind the stack:

  • Frame Pointers (-fno-omit-frame-pointer): Keeps the RBP register dedicated to stack frames. Fast, low overhead, and works seamlessly with eBPF and perf. Modern enterprise distros (Ubuntu 24.04+, Fedora 38+) compile packages with frame pointers enabled by default.
  • DWARF Debug Info (--call-graph dwarf): Unwinds using .debug_frame sections. High overhead; can record gigabytes of stack data in seconds.
  • LBR (Last Branch Record): Uses hardware CPU buffers for leaf-level call stacks with near-zero overhead.

Generating the Interactive SVG

# Script perf output and pipe to FlameGraph scripts
sudo perf script | stackcollapse-perf.pl | flamegraph.pl > cpu-profile.svg

Hotspot Analysis and Call Path Tracing

On-CPU vs Off-CPU Profiling

Not all performance bottlenecks burn CPU cycles:

  • On-CPU Flame Graphs: Show where threads spend time while actively executing instructions on a core. Essential for compute-bound algorithms, JSON parsing, compression, and crypto.
  • Off-CPU Flame Graphs: Show where threads spend time blocked and descheduled (waiting on disk I/O, network sockets, mutexes, condition variables, or kernel locks).
# Record off-CPU blocking time using eBPF (BCC offcputime tool)
sudo offcputime-bpfcc -df -p <PID> 10 | flamegraph.pl --color=io > offcpu.svg

If an application is unresponsive but CPU utilization is only 5%, an Off-CPU Flame Graph will immediately reveal the exact lock, socket, or file read stalling the worker threads.

Combining Tools: Integrated Workflows

A mature performance drill-down combines high-level monitoring with stack profiling:

┌────────────────────────────────────────────────────────────────────────┐
│                      INTEGRATED INVESTIGATION FLOW                     │
│                                                                        │
│   1. System Overview (USE Method)                                      │
│      sar / mpstat / vmstat ──► Detects CPU Saturation                  │
│                                │                                       │
│                                ▼                                       │
│   2. Process Identification                                            │
│      pidstat / top ──────────► Pinpoints Problematic PID               │
│                                │                                       │
│                                ▼                                       │
│   3. Stack Sampling & Flame Graphs                                     │
│      perf record -p <PID> ───► Identifies Hot Function Path            │
│                                │                                       │
│                                ▼                                       │
│   4. Kernel/Hardware Drill-Down                                        │
│      perf stat -e ... ───────► Pinpoints PMU Cache Miss / IPC Stall    │
│      bpftrace ───────────────► Measures Function Argument Outliers     │
└────────────────────────────────────────────────────────────────────────┘

Sampling vs Tracing: Trade-offs

Dimension Sampling (perf record) Event Tracing (ftrace / kprobes)
Mechanism Samples stack at periodic intervals (e.g., 99Hz) Captures 100% of event entries and exits
Overhead Fixed, deterministic (<1–2% CPU load) Scales linearly with event frequency (can exceed 100% overhead)
Precision Statistical approximation; can miss rare micro-bursts Exact counts and nanosecond durations for every call
Best Used For Continuous profiling, CPU hotspot discovery Rare latency outliers, lock contention, exact error paths

Automated Analysis Pipelines

In modern cloud-native fleets, manually logging into individual nodes to run perf is impractical. Production infrastructure deploys Continuous Profiling Agents:

  • Parca & Pyroscope: Open-source, eBPF-based profiling daemons that run on every node, continuously sample CPU and memory profiles at <1% overhead, and push pprof data to central query stores.
  • Diff Flame Graphs: Highlight architectural regressions between releases by overlaying two Flame Graphs: code paths that grew wider are colored red (regressions), while paths that shrank are colored blue (optimizations).