pprof and Execution Tracer Internals

Updated

September 8, 2026

pprof and Execution Tracer Internals

Overview

Workflows live in benchmarking & profiling. This chapter is the mechanism: how samples appear, what they mean, and how the execution tracer differs from CPU pprof.

Diagram: Profile vs trace

flow:
  [OnCPU]
       |
       v
  [Tool]

  [Events]
       |
       v
  [Tool2]

CPU Profile

Rough model:

OS signal / interrupt (platform dependent sampling)
  -> record stack of current G/M
  -> aggregate into profile proto
go test -cpuprofile=cpu.out -bench=.
go tool pprof -http=:0 cpu.out

Implications:

  • Sampling — short functions may be undercounted
  • Inlined frames can look surprising
  • CPU profile is not wall-clock wait time

Heap / Alloc Profiles

  • Heap: objects still live at sample time (retained)
  • Alloc: cumulative allocations (even if freed)
go tool pprof http://localhost:6060/debug/pprof/heap

Use alloc profiles to find churn; heap profiles to find leaks/retention.

Block and Mutex Profiles

runtime.SetBlockProfileRate(1)       // careful in prod
runtime.SetMutexProfileFraction(5)

These answer “who waited on locks/channels” — complementary to CPU.

Goroutine Profile

Stacks of all Gs — best leak and deadlock forensic tool.

Execution Tracer (go tool trace)

Records events with timestamps:

  • Goroutine create/block/unblock
  • Syscall enter/exit
  • GC STW / mark assist
  • Network poll
  • Processor start/stop
go test -trace=trace.out -bench=.
go tool trace trace.out

Use when CPU is low but latency is high — classic scheduler / stall story.

Cost of Profiling

Mode Overhead
Occasional CPU 30s Usually acceptable
Always-on mutex rate=1 Can be heavy
Full traces long windows Large files, overhead

Sample in production; continuous profiling products add aggregation layers.

Experiment

go test -bench=BenchmarkWork -cpuprofile=/tmp/cpu.out -benchtime=1s
go tool pprof -top /tmp/cpu.out | head
package work_test

import "testing"

func work(n int) int {
    x := 0
    for i := 0; i < n; i++ {
        x += i
    }
    return x
}

func BenchmarkWork(b *testing.B) {
    for i := 0; i < b.N; i++ {
        _ = work(1000)
    }
}

What to notice: Top frames should show work or inlined sites; if everything is runtime.*, you may be measuring something else (GC, tiny benchmark).

Try next: Generate a 200ms trace around an HTTP handler and find a long “block” on network or channel.