Benchmarking and Profiling Workflow

Updated

July 30, 2026

Benchmarking and Profiling Workflow

Optimization should be a controlled loop, not guesswork. Go’s toolchain makes the loop tight: benchmarks, pprof, and execution traces.

Why / Overview

Without measurement you will:

  • Optimize cold code
  • Add complexity for noise-level gains
  • Miss regressions until customers report them
baseline bench/load -> profile hotspot -> focused change -> remeasure -> benchstat keep/revert

Writing Good Benchmarks

func BenchmarkParse(b *testing.B) {
    in := fixedInput() // stable, realistic
    b.ReportAllocs()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        if _, err := Parse(in); err != nil {
            b.Fatal(err)
        }
    }
}

Rules:

  1. Realistic inputs — tiny synthetic data lies.
  2. Avoid measuring setup — use ResetTimer, or do setup outside the loop.
  3. Prevent dead-code elimination — assign to sink or use results.
  4. Report allocs — b.ReportAllocs() or -benchmem.
  5. Sub-benchmarks for sizes: b.Run("1k", ...).
var sink int

func BenchmarkSum(b *testing.B) {
    for i := 0; i < b.N; i++ {
        sink = Sum(data)
    }
}

Parallel benchmarks

func BenchmarkHandlerParallel(b *testing.B) {
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            // concurrent path
        }
    })
}

Useful for mutex/contention issues; pair with CPU counts.

Running and Comparing

go test -bench=BenchmarkParse -benchmem -count=10 ./... | tee old.txt
# make change
go test -bench=BenchmarkParse -benchmem -count=10 ./... | tee new.txt
benchstat old.txt new.txt

benchstat shows geometric mean differences and statistical confidence. Do not trust single runs.

Flags:

  • -benchtime=3s or -benchtime=1000x for stability
  • -cpu=1,4 for scaling behavior
  • -c for concurrent package tests carefully

Profiling from Benchmarks

go test -bench=BenchmarkParse -cpuprofile=cpu.out -memprofile=mem.out
go tool pprof -http=:8080 cpu.out
go tool pprof -http=:8081 mem.out

Look at:

  • CPU flame graph — widest plateaus
  • alloc_space — bytes allocated (retained vs ephemeral)
  • alloc_objects — allocation count (GC pressure)

Profiling a Running Service

import _ "net/http/pprof"

// admin listener only
go func() {
    log.Println(http.ListenAndServe("127.0.0.1:6060", nil))
}()
go tool pprof -http=:8081 http://127.0.0.1:6060/debug/pprof/profile?seconds=30
curl -o heap.pb.gz http://127.0.0.1:6060/debug/pprof/heap
go tool pprof -http=:8082 heap.pb.gz

Mutex and block profiles

runtime.SetMutexProfileFraction(1)    // sample all mutex events; lower in prod if needed
runtime.SetBlockProfileRate(1)        // 1 ns; too heavy for always-on prod — sample carefully
go tool pprof http://127.0.0.1:6060/debug/pprof/mutex
go tool pprof http://127.0.0.1:6060/debug/pprof/block

Execution Tracer

curl -o trace.out 'http://127.0.0.1:6060/debug/pprof/trace?seconds=5'
go tool trace trace.out

Use when:

  • Latency spikes with low CPU
  • Suspect GC STW or scheduler delays
  • Goroutines stuck in syscall or channel ops

Trace shows timeline, not aggregated hotspots — complementary to pprof.

Load Testing Complements Microbenches

Microbenchmarks miss:

  • Network
  • Real JSON sizes
  • Multi-tenant cache effects
  • GC under steady heap

Use hey, vegeta, k6, or custom Go load clients for end-to-end. Still attach pprof during the load.

Workflow Playbook

  1. Define success: e.g. p99 < 50ms at 2k RPS, or bench ns/op −20%.
  2. Baseline with count≥10 and production-like data.
  3. Profile under that baseline load.
  4. Change one hotspot (algorithm, alloc, lock scope).
  5. Remeasure with benchstat or load test.
  6. Document why the change is correct (not only faster).
  7. Guard with a regression benchmark in CI if critical.

CI Integration

# Example: fail on large regression using benchstat + golden
go test -bench=BenchmarkCritical -count=5 ./internal/hotpath | tee ci.txt
# compare against stored baseline with thresholds

Keep CI benches short and stable; flaky benches train people to ignore them.

Production Checklist

  • Critical paths have benchmarks with -benchmem
  • pprof available on admin interface only
  • Know how to capture CPU, heap, mutex, trace
  • Use benchstat for claims
  • Load test environment approximates prod limits (CPU, GOMAXPROCS)
  • Profiles stored for major incidents (artifact)
  • Optimization PRs include before/after numbers

Common Pitfalls

  1. Benchmarking with -race only — useful but not speed truth.
  2. Tiny inputs that stay on stack and never show heap costs.
  3. Compiler optimizing away work.
  4. Comparing different machines without noting noise.
  5. Always-on block profiling at rate 1 in prod — heavy.
  6. Optimizing before algorithmic rewrite (O(n²) → map).
  7. Ignoring variance — one lucky run.

Exercises

  1. Write a bad benchmark that the compiler optimizes away; fix with sink.
  2. Benchmark two JSON libraries on the same payload; benchstat them.
  3. Capture CPU profile of a deliberately slow function; identify it in the flame graph.
  4. Add ReportAllocs; reduce allocs with a sync.Pool; remeasure.
  5. Run parallel benchmark; show scaling from 1 to 4 CPUs.
  6. Enable mutex profile on a contended lock demo; find the holder.
  7. Capture a 5s trace during GC-heavy load; find STW-ish gaps.
  8. Load-test a HTTP handler; profile mid-test; fix top hotspot.
  9. Introduce a 5% performance regression; see if your CI would catch it.
  10. Write a one-page playbook for your team’s pprof ports and commands.

Interpreting Flame Graphs Quickly

When the CPU profile UI opens:

  1. Sort by cum (cumulative) to see time including callees.
  2. Look for wide plateaus in your package paths — not only runtime.*.
  3. If most time is runtime.mallocgc / scanobject, switch to heap/alloc profiles (next chapter).
  4. If most time is sync.(*Mutex).Lock, switch to mutex profile (contention chapter).
  5. If time is in syscall.Read / network, you may have an IO or dependency problem, not a Go micro-opt.

Always keep a copy of the profile file (cpu.out) next to the PR description for reviewers.

Production Capture Safety

  • Bind pprof to localhost or a private admin network.
  • Prefer short profiles (15–30s) under representative load.
  • Do not leave SetBlockProfileRate(1) on permanently in large fleets.
  • Redact profile upload paths if they include customer identifiers in function args (rare but possible with bad instrumentation).

More examples

Mini benchmark harness (stdlib testing style)

mkdir -p /tmp/go-bench-mini && cd /tmp/go-bench-mini
go mod init example.com/bench-mini

Save as main.go (illustrates the measurement pattern; real benches use testing.B):

package main

import (
    "fmt"
    "strings"
    "time"
)

func concatPlus(n int) string {
    var s string
    for i := 0; i < n; i++ {
        s += "x"
    }
    return s
}

func concatBuilder(n int) string {
    var b strings.Builder
    b.Grow(n)
    for i := 0; i < n; i++ {
        b.WriteByte('x')
    }
    return b.String()
}

func timeIt(name string, n int, fn func(int) string) {
    start := time.Now()
    _ = fn(n)
    fmt.Printf("%s n=%d %s\n", name, n, time.Since(start))
}

func main() {
    timeIt("plus", 5000, concatPlus)
    timeIt("builder", 5000, concatBuilder)
}
go run .

Expected output (durations vary; builder should not be slower by orders of magnitude):

plus n=5000 ...
builder n=5000 ...

pprof HTTP handler registration (httptest)

mkdir -p /tmp/go-pprof-reg && cd /tmp/go-pprof-reg
go mod init example.com/pprof-reg

Save as main.go:

package main

import (
    "fmt"
    "net/http"
    "net/http/httptest"
    "net/http/pprof"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("GET /debug/pprof/", pprof.Index)
    mux.HandleFunc("GET /debug/pprof/cmdline", pprof.Cmdline)
    mux.HandleFunc("GET /debug/pprof/profile", pprof.Profile)
    mux.HandleFunc("GET /debug/pprof/symbol", pprof.Symbol)
    mux.HandleFunc("GET /debug/pprof/trace", pprof.Trace)

    ts := httptest.NewServer(mux)
    defer ts.Close()
    resp, err := http.Get(ts.URL + "/debug/pprof/")
    if err != nil {
        panic(err)
    }
    resp.Body.Close()
    fmt.Println("pprof index:", resp.StatusCode)
}
go run .

Expected output:

pprof index: 200

Runnable example

A real go test benchmark file plus runtime/pprof heap profile written to a buffer—stdlib profiling workflow without a long-lived server.

mkdir -p /tmp/go-bench-pprof && cd /tmp/go-bench-pprof
go mod init example.com/bench-pprof

Save as work.go:

package benchpprof

import "strings"

func BuildReport(n int) string {
    var b strings.Builder
    for i := 0; i < n; i++ {
        b.WriteString("line\n")
    }
    return b.String()
}

Save as work_test.go:

package benchpprof

import "testing"

func BenchmarkBuildReport(b *testing.B) {
    for i := 0; i < b.N; i++ {
        _ = BuildReport(1000)
    }
}

func BenchmarkBuildReportAlloc(b *testing.B) {
    b.ReportAllocs()
    for i := 0; i < b.N; i++ {
        _ = BuildReport(1000)
    }
}

Save as cmd/heapdemo/main.go:

package main

import (
    "bytes"
    "fmt"
    "runtime"
    "runtime/pprof"

    benchpprof "example.com/bench-pprof"
)

func main() {
    // Create some heap traffic
    var hold []string
    for i := 0; i < 100; i++ {
        hold = append(hold, benchpprof.BuildReport(500))
    }
    runtime.GC()

    var buf bytes.Buffer
    if err := pprof.WriteHeapProfile(&buf); err != nil {
        panic(err)
    }
    fmt.Println("heap_profile_bytes:", buf.Len())
    fmt.Println("held_reports:", len(hold))
    fmt.Println("tip: go test -bench=BenchmarkBuildReport -benchmem -count=5")
}
go test -bench=BenchmarkBuildReport -benchmem -count=3 .
go run ./cmd/heapdemo

Expected output (illustrative):

BenchmarkBuildReport-8         xxxx      yyyy ns/op     zzzz B/op      w allocs/op
heap_profile_bytes: <nonzero>
held_reports: 100
tip: go test -bench=BenchmarkBuildReport -benchmem -count=5

What to notice

  • -benchmem reports B/op and allocs/op—the first filter before guessing.
  • pprof.WriteHeapProfile works without binding HTTP; use net/http/pprof on an admin port in services.
  • Run benchmarks multiple -count times before believing a regression.

Try next

  • go test -cpuprofile=cpu.out -bench=. then go tool pprof -http=:8080 cpu.out.
  • Compare two commits with benchstat old.txt new.txt.

Further Reading

  • Go blog: profiling Go programs
  • benchstat documentation
  • Next: Memory, Allocations, and GC