Performance Engineering Overview

Updated

July 30, 2026

Performance Engineering Overview

This part frames performance as an evidence-driven workflow. The primary rule is simple: measure first, optimize second. Guessing hotspots is a reliable way to complicate code without moving p99.

Why / Overview

Go is fast enough for most network services out of the box. Performance work becomes necessary when:

  • Tail latency violates SLOs
  • Allocation rate drives GC pauses under load
  • Lock contention caps throughput below CPU saturation
  • A deploy regresses a critical path
baseline -> profile -> hypothesize -> change one thing -> remeasure -> keep/revert

Learning Path

Chapter Focus Outcome
181 Benchmarking & profiling bench, pprof, trace workflow Find real hotspots
182 Memory, allocs, GC Escape analysis, pooling, GC knobs Control allocation pressure
183 Contention & concurrency Mutex profiles, sharding Raise throughput ceilings

Core Principles

  1. User-visible metrics first — p99 latency, error rate, throughput.
  2. Reproduce with stable benchmarks or load tests before micro-tweaks.
  3. One variable per experiment.
  4. Prefer algorithmic fixes over micro-optimizations.
  5. Complexity costs — pooling and sharding need proof.
  6. Regressions are bugs — keep benchmarks in CI for critical paths.

Go Tooling Map

Tool Question
testing.B How fast is this function?
pprof CPU Where is CPU time?
pprof heap Where are allocations?
pprof mutex/block Who waits?
go tool trace Scheduler, GC, latency gaps
benchstat Are two runs different?
metrics / APM Production truth

Relationship to Other Parts

  • Observability detects symptoms; this part deepens local diagnosis.
  • Concurrency chapters supply patterns you will tune here.
  • Network timeouts interact with performance (slow is not always “needs pprof”).

Literacy Checklist

  • Write a meaningful benchmark with stable inputs
  • Capture and read a CPU profile flame graph
  • Read allocs/op and B/op from benches
  • Identify GC pressure symptoms
  • Diagnose mutex contention with profiles
  • Use benchstat for before/after
  • Know when to stop optimizing

Part Warm-Up Exercises

  1. List your top three latency SLOs and whether you have benchmarks for them.
  2. Enable pprof on a dev binary; open /debug/pprof/.
  3. Run go test -bench=. -benchmem on a module you own.
  4. Recall a past “optimization” that did nothing; what was missing?
  5. Note GOMAXPROCS and container CPU limits in your deploy env.

Performance vs Correctness vs Cost

Optimize in this priority order unless the product says otherwise:

  1. Correctness under concurrency (-race, invariants)
  2. User SLOs (latency, errors)
  3. Efficiency (CPU, memory, $)
  4. Microbenchmark bragging rights

A faster wrong answer is still wrong. A 2% CPU win that adds a subtle race is a net loss.

When Not to Optimize

  • You have no measurement of the problem
  • The code is not on a hot path (profile says <1%)
  • Clarity loss is large and gain is within noise
  • The real bottleneck is a dependency you do not control (fix timeouts/caching instead)

“Premature optimization” is not “never optimize” — it is “optimize without evidence.”

Environment Parity

Benchmarks lie when:

Local Production
8 fast cores 2 throttled vCPU
NVMe disk network block volume
empty cache multi-tenant noisy neighbor
-race off always sometimes on in CI only

Record GOMAXPROCS, CPU quota, and Go version next to every performance claim.

Suggested Lab Sequence

  1. Write a deliberately slow handler (JSON + lock + alloc)
  2. Load test; capture CPU + mutex + heap profiles
  3. Fix the top issue only; remeasure with benchstat / p99
  4. Repeat once more; stop when SLO is met

That sequence is the entire part in miniature.

Regression Guardrails

Protect critical paths the way you protect correctness:

# local loop
go test -bench=BenchmarkHotPath -benchmem -count=10 ./internal/hot > new.txt
benchstat old.txt new.txt

In CI, store a baseline for a small set of benches and fail on large regressions (with noise tolerance). Flaky performance CI is worse than none — invest in stable inputs and warm-up.

Interaction with Timeouts

A “performance” incident is sometimes a timeout budget incident:

  • Dependency p99 is 400ms; you budget 200ms → “Go is slow” tickets
  • Retries multiply latency → p99 explodes without CPU hotspots

Always check SLOs, budgets, and dependency health before rewriting algorithms.

Team Norms Worth Adopting

  1. Performance claims require numbers (benchstat or load p99).
  2. Profiles attached to optimization PRs.
  3. Prefer readability until a profile forces otherwise.
  4. Document GOMAXPROCS / limits when quoting throughput.
  5. Revisit pools and shards annually — complexity expires.

Mapping Symptoms → Chapter

Symptom Start here
“Need numbers / workflow” 181 Benchmarking & profiling
High GC CPU, allocs/op 182 Memory & GC
Low CPU, high latency, lock waits 183 Contention
Dependency timeouts Parts 13 & 15 (budgets, breakers)
Cannot find which service is slow Part 16 tracing

Use this table in incident channels so people stop randomly rewriting JSON libraries.

Definition of Done for a Perf Change

A performance PR is done when:

  1. Before/after numbers exist (benchstat or load p99/RPS).
  2. A profile explains why the change helped.
  3. Correctness tests still pass (go test, race where relevant).
  4. Complexity is justified in the description.
  5. A rollback plan exists if production disagrees with the lab.

If any item is missing, the change is an experiment, not a finish.

Quick Command Card

# microbench + mem
go test -bench=BenchmarkX -benchmem -count=10 ./...

# compare
benchstat old.txt new.txt

# cpu profile from bench
go test -bench=BenchmarkX -cpuprofile=cpu.out
go tool pprof -http=:8080 cpu.out

# live service (admin port)
go tool pprof -http=:8081 http://127.0.0.1:6060/debug/pprof/profile?seconds=20
curl -o trace.out 'http://127.0.0.1:6060/debug/pprof/trace?seconds=5'
go tool trace trace.out

Pin this card next to your runbooks.

Glossary

  • ns/op, B/op, allocs/op — benchmark cost units
  • pprof — sampling profiler suite
  • benchstat — statistical comparison of bench outputs
  • GOGC / GOMEMLIMIT — GC pacing and soft memory limit
  • Contention — waiting on shared resources
  • Flame graph — visualization of aggregated stacks
  • p99 — 99th percentile latency (tail)
  • Escape analysis — compiler decision stack vs heap

More examples

Tour: timed work + allocation awareness

mkdir -p /tmp/go-perf-tour && cd /tmp/go-perf-tour
go mod init example.com/perf-tour

Save as main.go:

package main

import (
    "fmt"
    "time"
)

func sumPrealloc(n int) int {
    s := make([]int, 0, n)
    for i := 0; i < n; i++ {
        s = append(s, i)
    }
    total := 0
    for _, v := range s {
        total += v
    }
    return total
}

func main() {
    start := time.Now()
    got := sumPrealloc(10_000)
    fmt.Println("sum:", got, "dur_ms:", time.Since(start).Milliseconds())
}
go run .

Expected output:

sum: 49995000 dur_ms: 0

(dur_ms may be 0–1 on a fast machine.)

Runnable example

Tour of this part: a microbenchmark-style timing loop comparing two algorithms and a tiny allocation count via testing.AllocsPerRun pattern inlined in main.

mkdir -p /tmp/go-perf-tour && cd /tmp/go-perf-tour
go mod init example.com/perf-tour

Save as main.go:

package main

import (
    "fmt"
    "strings"
    "testing"
    "time"
)

func concatPlus(n int) string {
    s := ""
    for i := 0; i < n; i++ {
        s += "x"
    }
    return s
}

func concatBuilder(n int) string {
    var b strings.Builder
    b.Grow(n)
    for i := 0; i < n; i++ {
        b.WriteByte('x')
    }
    return b.String()
}

func timeIt(name string, n int, fn func(int) string) {
    start := time.Now()
    _ = fn(n)
    fmt.Printf("%s took %s\n", name, time.Since(start))
}

func main() {
    const n = 20000
    timeIt("plus", n, concatPlus)
    timeIt("builder", n, concatBuilder)

    allocPlus := testing.AllocsPerRun(10, func() { _ = concatPlus(1000) })
    allocBld := testing.AllocsPerRun(10, func() { _ = concatBuilder(1000) })
    fmt.Printf("allocs_per_run plus=%.0f builder=%.0f\n", allocPlus, allocBld)
    fmt.Println("tour: measure before optimize")
}
go run .

Expected output (illustrative):

plus took ...
builder took ...
allocs_per_run plus=... builder=1
tour: measure before optimize

What to notice

  • Evidence first: builder wins on allocs for repeated concatenation—the part’s core habit.
  • Chapters 181–183 deepen pprof, GC, and contention.

Try next

  • go test -bench=. -benchmem on extracted functions.
  • Capture a CPU profile once you have a real hotspot.

Next Chapter

Benchmarking and Profiling Workflow — the loop you will reuse forever.