Chapter 14: eBPF and BPF-Based Observability

Updated

October 31, 2026

Chapter 14: eBPF and BPF-Based Observability

Extended Berkeley Packet Filter (eBPF) revolutionized Linux systems engineering by turning the operating system kernel into a safely programmable platform. Instead of modifying kernel source code, loading fragile out-of-tree kernel modules, or incurring ptrace context-switch penalties, engineers can load sandboxed bytecode directly into the kernel to measure, filter, and trace events at line rate.

This chapter details the internal eBPF architecture, the in-kernel verifier, BPF map data structures, BCC diagnostic tools, bpftrace one-liners, and production safety considerations.

eBPF Architecture and Safety Model

eBPF programs are event-driven: they attach to specific kernel hooks (kprobes, tracepoints, socket filters, XDP, perf events, or cgroups) and execute whenever the associated event fires.

┌────────────────────────────────────────────────────────────────────────┐
│                        eBPF EXECUTION LIFECYCLE                        │
│                                                                        │
│   Userspace (C / Rust / bpftrace)                                      │
│   ┌────────────────────────────────────────────────────────────────┐   │
│   │ Compile code to eBPF Bytecode via Clang/LLVM                   │   │
│   └───────────────────────────────┬────────────────────────────────┘   │
│                                   │ sys_bpf(BPF_PROG_LOAD)             │
│   Kernelspace                     ▼                                    │
│   ┌────────────────────────────────────────────────────────────────┐   │
│   │ 1. In-Kernel Verifier                                          │   │
│   │    - Proves memory bounds, prevents invalid pointer derefs     │   │
│   │    - Guarantees termination (bounded loops, instruction limits)│   │
│   │    - Validates type safety and register states                 │   │
│   │                                                                │   │
│   │ 2. JIT Compiler                                                │   │
│   │    - Translates verified bytecode into native x86/ARM machine  │   │
│   │                                                                │   │
│   │ 3. Hook Execution ──► Reads/Updates BPF Maps (In-Memory Tables)│   │
│   └───────────────────────────────┬────────────────────────────────┘   │
│                                   │ Shared Memory / Ring Buffer        │
│   Userspace Reader                ▼                                    │
│   ┌────────────────────────────────────────────────────────────────┐   │
│   │ Reads aggregated histograms and metrics without copying events │   │
│   └────────────────────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────────────────────┘
  • The In-Kernel Verifier: Guarantees that the program cannot crash the kernel, dereference uninitialized pointers, or enter infinite loops. If the verifier cannot prove safety mathematically, program loading is rejected.
  • JIT Compilation: Bytecode is compiled directly into native CPU machine instructions, executing at near-native C speed.

BPF Map Types and Communication

BPF programs communicate with userspace and maintain state across events using BPF Maps—key-value storage structures allocated in kernel memory:

Map Type Description Optimization Target
BPF_MAP_TYPE_HASH Generic associative hash table Fast lookup by arbitrary keys (e.g. PID, IP address).
BPF_MAP_TYPE_ARRAY Fixed-size pre-allocated array Fastest possible indexed lookup; no allocation during execution.
BPF_MAP_TYPE_PERCPU_HASH Hash table with private per-CPU storage Lockless execution; eliminates cross-core cache invalidation.
BPF_MAP_TYPE_RINGBUF High-throughput, multi-producer ring buffer Asynchronous streaming of events to userspace with low overhead.

bpftrace: High-Level Tracing

bpftrace is the premier high-level tracing language for Linux, inspired by AWK and DTrace:

# Measure block I/O latency distribution (visualized as a power-of-2 histogram)
sudo bpftrace -e '
  kprobe:blk_account_io_start { @start[arg0] = nsecs; }
  kprobe:blk_account_io_done /@start[arg0]/ {
    @latency_us = hist((nsecs - @start[arg0]) / 1000);
    delete(@start[arg0]);
  }'

# Trace slow filesystem sync calls taking longer than 10 milliseconds
sudo bpftrace -e '
  kprobe:ksys_fsync { @t[tid] = nsecs; }
  kretprobe:ksys_fsync /@t[tid]/ {
    $dur_ms = (nsecs - @t[tid]) / 1000000;
    if ($dur_ms > 10) {
      printf("fsync PID %d (%s) took %d ms\n", pid, comm, $dur_ms);
    }
    delete(@t[tid]);
  }'

BCC Tools: Key Utilities

The BPF Compiler Collection (BCC) ships pre-built, production-tested diagnostic utilities:

# Monitor new process executions in real time with parent PID and command line
sudo execsnoop-bpfcc

# Trace file open operations, including latency and error codes (ENOENT, EACCES)
sudo opensnoop-bpfcc -T -e

# Summarize disk I/O latency distribution every 5 seconds
sudo biolatency-bpfcc 5 1

# Trace TCP connection establishment latency (SYN to SYN-ACK)
sudo tcpconnect-bpfcc -u

# Detect TCP packet retransmissions with source/destination addresses
sudo tcpretrans-bpfcc

Writing Custom BPF Programs (CO-RE and libbpf)

Modern eBPF development uses Compile Once – Run Everywhere (CO-RE) with libbpf:

  • BTF (BPF Type Format): Embedded kernel debug metadata that allows eBPF programs to read kernel structs without needing kernel headers installed on the target machine.
  • Relocation: Clang and libbpf dynamically adjust struct field offsets at load time, ensuring a binary compiled on kernel 6.2 runs unmodified on kernel 6.8.
// Example CO-RE probe reading task_struct->pid safely
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>

SEC("kprobe/sched_switch")
int BPF_KPROBE(trace_sched_switch, bool preempt, struct task_struct *prev, struct task_struct *next) {
    pid_t prev_pid = BPF_CORE_READ(prev, pid);
    bpf_printk("Switched from PID %d\n", prev_pid);
    return 0;
}
char LICENSE[] SEC("license") = "GPL";

eBPF Overhead and Production Safety

While eBPF is vastly safer and faster than kernel modules or ptrace:

  1. High-Frequency Hooks: Attaching probes to functions called millions of times per second (e.g., vfs_read or memory allocators) can introduce 3%–10% CPU overhead purely from probe trampolines. Prefer tracepoints or sample-based perf events.
  2. Map Exhaustion: Hash maps with unbounded growth will reject insertions once full. Always size maps appropriately or use LRU maps (BPF_MAP_TYPE_LRU_HASH).
  3. In-Kernel Aggregation: Never push raw events to userspace one by one. Accumulate counts, sums, and histograms in BPF maps inside kernel space, and emit only summarized results periodically.