Production Mistakes Deep Dive

Production Mistakes Deep Dive

Overview

Recurring themes from production Go engineering discussions (including widely shared checklists on X): timeouts, leaks, default clients, lock+I/O, and observability gaps. This chapter maps each mistake to runtime symptoms and fixes covered elsewhere in this part.

Diagram: Incident loop

flow:
  [Prof]
       |
       v
  [Trace]

  [Trace]
       |
       v
  [Fix]

The List (Mapped)

Mistake	Symptom	Deep dive / fix
No context timeouts	Goroutine + FD growth	Context trees, netpoller
Unbounded goroutines	RSS↑, sched latency	Scheduler, leaks
Default http.Client	Hung dials/TLS	Transport pool
Errors as strings	Untestable retries	Error cost
nil vs empty JSON	Client contract breaks	encoding stdlib
Unbounded channels	Memory spikes	Channel internals
Lock held over I/O	Throughput collapse	Mutex runtime
time.After in hot select	Timer/alloc churn	Timers
DB without limits/context	Connection storms	HTTP/DB practices
Logs without IDs/metrics	Blind incidents	slog, observability parts

Incident Playbook

1. NumGoroutine / FD count / RSS chart
2. goroutine pprof (wait sites)
3. mutex/block profile if CPU low
4. trace 200–500ms around incident
5. check client/server timeouts
6. check deploy diff for "go func" and "time.After"

Engineering Questions (From “knowing Go”)

1. Do you need concurrency at all?
2. Goroutine or worker pool?
3. Channel or mutex?
4. How does context propagate?
5. Did you run -race, pprof, -gcflags=-m?

Mini Lab

Take any service and grep:

rg -n 'http\\.Get|http\\.DefaultClient|time\\.After|go func|context\\.Background\\(\\)' --glob '*.go'

Classify each hit as safe or debt.

Try next: Add a CI check that fails on http.Get in non-test packages (project policy).