Goroutine Leaks and Diagnosis

Goroutine Leaks and Diagnosis

Overview

Goroutines are cheap until they never exit. Leaks show up as monotonically rising goroutine counts, memory growth (stacks + closed-over data), and eventually latency death.

This is one of the top production issues called out in Go engineering threads: spawn without stop conditions.

Diagram: Leak patterns

  go worker
       │
       ├── forever block on chan
       ├── missing ctx.Done
       └── lock never released
       │
       v
  NumGoroutine ↑   pprof goroutine ──► fix bounds + cancel

Common Leak Patterns

Pattern	Why it leaks
Blocked on channel send/recv forever	Partner gone
select without ctx.Done	No cancel path
HTTP request without timeout	Stuck on body read
for range ticker.C without Stop/exit	Forever loop
Waiting on mutex held by dead owner	Deadlock-ish stall
Callback registered never removed	Global map grows

Fix Templates

// Always pair spawn with context
go func() {
    select {
    case <-ctx.Done():
        return
    case jobs <- job:
    }
}()

// Bound fan-out
sem := make(chan struct{}, 32)
for _, item := range items {
    sem <- struct{}{}
    go func(it Item) {
        defer func() { <-sem }()
        process(ctx, it)
    }(item)
}

Diagnosis Toolkit

# Live
curl -s localhost:6060/debug/pprof/goroutine?debug=1 | head
go tool pprof http://localhost:6060/debug/pprof/goroutine

# Go 1.27+: goroutines the GC can prove will never wake
go tool pprof http://localhost:6060/debug/pprof/goroutineleak

# Binary
import _ "net/http/pprof"

The goroutineleak profile (experimental in 1.26, generally available in 1.27) reports goroutines blocked on a concurrency primitive that is unreachable from any runnable goroutine. It will not catch leaks parked on globals or on locals of still-runnable Gs. Use it alongside the ordinary goroutine profile, not instead of it.

runtime.NumGoroutine()

In tests:

func TestNoLeak(t *testing.T) {
    before := runtime.NumGoroutine()
    // run code under test with cancel
    // ...
    deadline := time.Now().Add(2 * time.Second)
    for time.Now().Before(deadline) {
        if runtime.NumGoroutine() <= before {
            return
        }
        time.Sleep(10 * time.Millisecond)
    }
    t.Fatalf("goroutines leaked: before=%d now=%d", before, runtime.NumGoroutine())
}

For concurrent unit tests with fake time, see synctest.

Experiment

go mod init example
go run .

package main

import (
    "fmt"
    "runtime"
    "time"
)

func leak() {
    ch := make(chan int)
    go func() { ch <- 1 }() // blocked forever if no receiver
}

func fixed(done <-chan struct{}) {
    ch := make(chan int)
    go func() {
        select {
        case ch <- 1:
        case <-done:
        }
    }()
}

func main() {
    fmt.Println("start", runtime.NumGoroutine())
    leak()
    time.Sleep(20 * time.Millisecond)
    fmt.Println("after leak", runtime.NumGoroutine())

    done := make(chan struct{})
    fixed(done)
    close(done)
    time.Sleep(20 * time.Millisecond)
    fmt.Println("after fixed", runtime.NumGoroutine())
}

What to notice: One blocked send permanently increases the count.

Try next: Dump goroutine?debug=1 in a real service after load; group stacks by chan receive sites.