Chapter 13: System Call Tracing and Process Analysis

Updated

October 31, 2026

Chapter 13: System Call Tracing and Process Analysis

When an application misbehaves, degrades in latency, or hangs, performance engineers must observe the boundaries where userspace requests kernel services. Tracing reveals the exact arguments, execution durations, return codes, and call stacks of every system call and internal kernel function.

This chapter details the mechanics, operational overhead, and best practices of system call tracing, dynamic probes, ftrace, perf trace, and traditional tracing frameworks.

strace and ltrace: Mechanics and Use Cases

strace is the ubiquitous first tool engineers reach for when debugging process behavior. However, using strace in production without understanding its internal mechanics is dangerous:

  • Mechanics of ptrace(): strace relies on the ptrace() system call. For every single system call executed by the target process, ptrace stops the process twice: once upon syscall entry, and once upon syscall exit.
  • The Overhead Trap: Stopping a process twice per syscall forces multiple context switches, TLB flushes, and inter-process signals. On an application executing 100,000 syscalls per second, strace can slow execution down by 10x to 100x, often turning minor latency anomalies into severe production outages.

Safe Production Use of strace

If strace must be used in production, strictly limit its scope:

# Summarize syscall counts and execution time without printing every call (-c)
strace -c -p <PID> -f

# Filter for specific problematic syscalls (-e) and record per-call latency (-T)
strace -e trace=openat,futex,read -T -p <PID>

ltrace operates similarly to intercept dynamic library (libc, libssl) calls, sharing the same ptrace overhead profile.

perf: Architecture and Profiling Modes

The Linux perf tool (part of the kernel source tree) is a high-performance profiler that operates directly via the perf_event_open() system call, sampling CPU hardware counters and kernel tracepoints with negligible overhead (<1–2% CPU load):

# Low-overhead syscall tracing with 'perf trace' (replaces strace safely)
sudo perf trace -p <PID> --duration 5.0   # Only print syscalls taking > 5ms

# Profile top system calls system-wide for 10 seconds
sudo perf top -e raw_syscalls:sys_enter

ftrace: Kernel Tracing Infrastructure

ftrace is the built-in function tracer built directly into the Linux kernel. It operates via tracefs (typically mounted at /sys/kernel/tracing or /sys/kernel/debug/tracing):

  • Zero-Overhead Inactive Probes: Modern Linux kernels use gcc -pg -mfentry to place 5-byte NOP instructions at the prologue of every kernel function. When tracing is inactive, execution cost is zero. When activated, the kernel replaces NOPs with jump instructions to tracing trampolines dynamically.
  • function_graph Tracer: Traces function call entries and exits, measuring the execution duration of kernel functions with nanosecond precision.
# Tracing kernel function graph via tracefs
cd /sys/kernel/tracing
echo 0 > tracing_on
echo function_graph > current_tracer
echo vfs_read > set_graph_function
echo 1 > tracing_on
head -n 30 trace
echo 0 > tracing_on

Tracepoints and Dynamic Probes

Linux provides two distinct mechanisms for instrumenting execution:

┌────────────────────────────────────────────────────────────────────────┐
│                   TRACEPOINTS vs DYNAMIC PROBES                        │
│                                                                        │
│   Static Tracepoints (tracepoint:subsystem:event)                      │
│   ├─ Explicitly placed by kernel developers in C source code           │
│   ├─ Stable API across kernel releases (guaranteed semantics)          │
│   └─ Example: sched:sched_switch, block:block_rq_issue                 │
│                                                                        │
│   Dynamic Probes (kprobes & uprobes)                                   │
│   ├─ Injected dynamically at runtime at ANY instruction address        │
│   ├─ Breakpoint instruction (INT3 / BRK) patched into running code     │
│   ├─ kprobes: Instruments arbitrary kernel functions                   │
│   └─ uprobes: Instruments arbitrary userspace binaries and libraries   │
└────────────────────────────────────────────────────────────────────────┘
# Add a dynamic kprobe to measure do_sys_openat2 latency using perf
sudo perf probe --add 'do_sys_openat2 dfd:s32 filename:string'
sudo perf record -e probe:do_sys_openat2 -a sleep 5
sudo perf script

SystemTap: Overview and Comparison

SystemTap was the pioneering dynamic tracing infrastructure for Linux. It compiles user-written scripts into kernel C modules, loads them into the running kernel, and executes instrumentation handlers.

  • Strengths: Powerful language, deep support for legacy RHEL/CentOS distributions.
  • Weaknesses: Requires kernel development packages (kernel-devel, kernel-debuginfo), compilation takes tens of seconds, and a buggy script can cause kernel crashes.
  • Modern Consensus: In modern production engineering (kernel 5.x/6.x+), SystemTap has been almost completely superseded by eBPF and bpftrace.

Building a Tracing Strategy

To minimize production risk while debugging performance:

  1. Level 1 (Counters & Metrics): Start with non-invasive metrics (vmstat, sar, node_exporter, PSI).
  2. Level 2 (Sampling Profiling): Use perf record -F 99 or perf top to find hot functions without instrumenting individual calls.
  3. Level 3 (Static Tracepoints): Use perf trace or bpftrace on established tracepoints (sched:sched_switch, block:block_rq_issue).
  4. Level 4 (Dynamic Probes): Use kprobes/uprobes only when hunting specific function arguments or rare latency outliers.
  5. Level 5 (strace): Restrict strace to short diagnostic bursts with tight filtering (-e trace=... -c), never on high-throughput production request loops.