Production Debugging Tooling

Production Debugging Tooling

Overview

When production hurts, you rarely attach a debugger first. You grab profiles, traces, stacks, and core-ish dumps — fast.

Diagram: Debug surfaces

flow:
  [Pprof]
       |
       v
  [Heap]

  [Pprof]
       |
       v
  [Goroutine]

Always-On Endpoints (Carefully)

import _ "net/http/pprof"
// bind to internal listener only
go http.ListenAndServe("127.0.0.1:6060", nil)

Expose via secure admin network / port-forward, never public internet.

Signals

Signal	Common action
SIGQUIT	Stack dump (often)
SIGTERM	Graceful shutdown
panic	GOTRACEBACK controls detail

GOTRACEBACK=all ./app
kill -QUIT $pid   # platform dependent behavior

Delve (Dev / Staging)

dlv attach $pid
dlv dap # editor

Prefer staging for breakpoints; prod attach can freeze worlds.

Core / Crash

# Linux examples — ops dependent
ulimit -c unlimited
GOTRACEBACK=crash

Pair with symbolized binaries (don’t strip staging).

Continuous Profiling

Products sample CPU/heap cheaply over time — use alongside on-demand pprof. Compare versions after deploys.

Checklist

1. Secure pprof listener
2. Mutex profile fraction documented
3. Version/build stamped (-X)
4. Runbooks for “goroutine explosion” and “GC CPU”

Experiment

go mod init example
# tiny server with pprof

package main

import (
    "net/http"
    _ "net/http/pprof"
    "time"
)

func main() {
    go func() {
        for {
            time.Sleep(10 * time.Millisecond)
        }
    }()
    _ = http.ListenAndServe("127.0.0.1:6060", nil)
}

# other terminal
go tool pprof http://127.0.0.1:6060/debug/pprof/goroutine

What to notice: Sleeping G still appears in goroutine profile; counts matter more than one stack.

Try next: Automate capturing 30s CPU profile on SIGUSR1 in a side project.