Contention and Concurrency Tuning

Updated

July 30, 2026

Contention and Concurrency Tuning

Throughput often degrades due to contention long before CPU saturates. Adding goroutines can make the system slower when they fight over locks, shared maps, or a single connection.

Why / Overview

goroutines -> shared resource -> queueing -> latency growth -> throughput collapse

Classic symptoms:

  • Low CPU, high latency
  • Throughput flat while goroutine count climbs
  • Mutex profile shows long waiters
  • block profile hot on channels or locks

Sources of Contention

Resource Example
sync.Mutex / RWMutex Shared cache, metrics map
Channels Single consumer bottleneck
Shared map without sharding Global session table
Single DB connection / conn pool too small Query pileup
log writer lock Sync logging every request
time.Now + heavy formatting under lock Lock held too long

Diagnostic Workflow

  1. Confirm problem with load test: RPS vs latency vs CPU.
  2. If CPU low and latency high → contention or IO wait.
  3. Capture mutex and block profiles under load.
  4. Capture trace for scheduler wait.
  5. Fix by reducing critical sections, sharding, or buffering.
  6. Remeasure.
import _ "net/http/pprof"
import "runtime"

func init() {
    runtime.SetMutexProfileFraction(5) // 1/5 of events; tune
    // runtime.SetBlockProfileRate(1000) // sample; careful in prod
}

Critical Section Hygiene

// BAD: slow work under lock
mu.Lock()
data := fetchFromDB() // network under lock!
cache[key] = data
mu.Unlock()

// GOOD: compute outside
data := fetchFromDB()
mu.Lock()
cache[key] = data
mu.Unlock()

Copy what you need under lock; work on the copy outside.

RWMutex misuse

RWMutex helps when reads dominate and critical sections are short. If writers are frequent, RWMutex can be worse. Measure.

Sharding

const shards = 16

type sharded struct {
    ms [shards]struct {
        mu sync.Mutex
        m  map[string]int
    }
}

func (s *sharded) inc(key string) {
    h := fnv32(key) % shards
    sh := &s.ms[h]
    sh.mu.Lock()
    sh.m[key]++
    sh.mu.Unlock()
}

Sharding reduces collision probability; pick shard count relative to GOMAXPROCS and key distribution.

Atomically Updated Stats

var hits atomic.Int64

func hit() { hits.Add(1) }

Prefer atomics for simple counters over mutexes. For complex metrics, use Prometheus client (already concurrent) rather than home-grown locked maps with high cardinality.

Channel Contension and Pipelines

// Single channel to one worker: max throughput ~ worker speed
jobs := make(chan Job, 1024)

// Multiple workers consuming one channel: good
for i := 0; i < n; i++ {
    go worker(jobs)
}

Pitfalls:

  • Unbuffered channels serialize producers
  • Huge buffers hide deadlocks and delay backpressure
  • Multiple stages with mismatched rates need intentional buffering

Bound Fan-Out

g, ctx := errgroup.WithContext(ctx)
g.SetLimit(32)
for _, u := range urls {
    u := u
    g.Go(func() error {
        return fetch(ctx, u)
    })
}
return g.Wait()

Unbounded go func per item under a hot HTTP handler is a latency and memory bomb.

Connection Pools

Database and HTTP pools are concurrency knobs:

db.SetMaxOpenConns(25)
db.SetMaxIdleConns(25)
db.SetConnMaxLifetime(time.Hour)
transport.MaxConnsPerHost = 32

Too few → wait on pool. Too many → overload dependency. Tune against dependency capacity and latency.

GOMAXPROCS and Containers

Go 1.5+ defaults GOMAXPROCS to CPU count. In cgroups, recent Go versions respect container CPU limits more carefully — still verify:

# inside container
go env GOMAXPROCS
# or log runtime.GOMAXPROCS(0) at start

Mismatch (thinks 64 CPUs, has 2) → oversubscription, latency noise.

False Sharing and Cache Lines (Advanced)

Rare in app Go, more in tight numeric loops with adjacent atomics. If profiling shows multi-core scaling collapse on atomics, pad/shard counters. Don’t start here.

Work Stealing and Fairness

The Go scheduler multiplexes goroutines on GOMAXPROCS threads. CPU-bound tight loops without function calls can delay preemption (improved in recent Go). For fairness, insert runtime.Gosched() only when profiling proves need — rare.

Production Checklist

  • Load test shows CPU vs latency relationship
  • Mutex/block profiles known how-to
  • No IO under locks on hot paths
  • Fan-out bounded (errgroup limits / semaphores)
  • Pools sized for deps
  • Shared maps protected and not global bottlenecks
  • Logging async or sampled under load if contended
  • GOMAXPROCS aligned with container CPUs
  • After changes, remeasure p99 and throughput

Common Pitfalls

  1. More goroutines = more speed belief.
  2. Global mutex around entire request handling.
  3. Map + mutex as the only cache for everything.
  4. Holding lock across http.Client.Do.
  5. Unbuffered event bus used for high QPS metrics.
  6. Ignoring dependency pool wait — looks like “Go is slow.”
  7. Fixing contention by removing locks without other synchronization → data races (-race).

Exercises

  1. Build a counter with mutex vs atomic; parallel bench both.
  2. Create a hot map with one mutex; shard it; benchstat.
  3. Hold a lock during time.Sleep to simulate IO; show latency; move sleep out.
  4. Capture mutex profile; identify the stack.
  5. Unbounded fan-out HTTP handler vs SetLimit; compare memory and p99.
  6. Shrink MaxOpenConns to 1; observe pool wait in traces/profiles.
  7. Use go test -race on a deliberately racy counter; fix.
  8. Measure throughput vs worker count curve; find the knee.
  9. Replace contended log.Printf with async slog or buffered writer; measure.
  10. Document a contention incident postmortem template (signals → profile → fix → result).

Throughput Curve Lab (Do This Once)

Pick a handler that does shared-map updates under a mutex:

workers: 1, 2, 4, 8, 16, 32
measure: RPS and p99
plot: RPS rises then flattens/falls; p99 rises with contention

Then shard the map or switch counters to atomics and replot. The shape of that curve is the entire point of this chapter: more concurrency is not free.

Combining with Timeouts and Breakers

Contention often surfaces as timeouts:

  • Lock wait → handler exceeds budget → client retries → more lock wait

Fixing only retries makes it worse. Profile locks under the load that triggers pages, then reduce critical sections or capacity-limit admission (semaphore) so the system degrades with 503 instead of unlimited queueing on a mutex.

More examples

Hot mutex vs sharded counters

mkdir -p /tmp/go-shard-count && cd /tmp/go-shard-count
go mod init example.com/shard-count

Save as main.go:

package main

import (
    "fmt"
    "sync"
    "sync/atomic"
    "time"
)

func hotMutex(n, workers int) time.Duration {
    var mu sync.Mutex
    var c int
    var wg sync.WaitGroup
    start := time.Now()
    for w := 0; w < workers; w++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for i := 0; i < n; i++ {
                mu.Lock()
                c++
                mu.Unlock()
            }
        }()
    }
    wg.Wait()
    _ = c
    return time.Since(start)
}

func atomicCount(n, workers int) time.Duration {
    var c atomic.Int64
    var wg sync.WaitGroup
    start := time.Now()
    for w := 0; w < workers; w++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for i := 0; i < n; i++ {
                c.Add(1)
            }
        }()
    }
    wg.Wait()
    return time.Since(start)
}

func main() {
    const n, w = 20_000, 4
    fmt.Println("mutex", hotMutex(n, w))
    fmt.Println("atomic", atomicCount(n, w))
}
go run .

Expected output (durations vary; both complete):

mutex ...
atomic ...

Admission semaphore (shed load)

mkdir -p /tmp/go-sem-admit && cd /tmp/go-sem-admit
go mod init example.com/sem-admit

Save as main.go:

package main

import (
    "fmt"
    "net/http"
    "net/http/httptest"
)

func withLimit(n int, next http.Handler) http.Handler {
    sem := make(chan struct{}, n)
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        select {
        case sem <- struct{}{}:
            defer func() { <-sem }()
            next.ServeHTTP(w, r)
        default:
            http.Error(w, "busy", http.StatusServiceUnavailable)
        }
    })
}

func main() {
    entered := make(chan struct{})
    block := make(chan struct{})
    h := withLimit(1, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        close(entered) // signal that the slot is held
        <-block
        fmt.Fprintln(w, "ok")
    }))
    ts := httptest.NewServer(h)
    defer ts.Close()

    go func() {
        resp, _ := http.Get(ts.URL)
        resp.Body.Close()
    }()
    <-entered // wait until first request holds the semaphore
    resp, _ := http.Get(ts.URL)
    fmt.Println("shed:", resp.StatusCode)
    resp.Body.Close()
    close(block)
}
go run .

Expected output:

shed: 503

Runnable example

Contended mutex counter vs sharded counters under many goroutines—measure elapsed time to see the throughput ceiling.

mkdir -p /tmp/go-contention && cd /tmp/go-contention
go mod init example.com/contention

Save as main.go:

package main

import (
    "fmt"
    "sync"
    "sync/atomic"
    "time"
)

func benchMutex(goroutines, iters int) time.Duration {
    var mu sync.Mutex
    var total int64
    var wg sync.WaitGroup
    start := time.Now()
    wg.Add(goroutines)
    for g := 0; g < goroutines; g++ {
        go func() {
            defer wg.Done()
            for i := 0; i < iters; i++ {
                mu.Lock()
                total++
                mu.Unlock()
            }
        }()
    }
    wg.Wait()
    _ = total
    return time.Since(start)
}

func benchAtomic(goroutines, iters int) time.Duration {
    var total atomic.Int64
    var wg sync.WaitGroup
    start := time.Now()
    wg.Add(goroutines)
    for g := 0; g < goroutines; g++ {
        go func() {
            defer wg.Done()
            for i := 0; i < iters; i++ {
                total.Add(1)
            }
        }()
    }
    wg.Wait()
    _ = total.Load()
    return time.Since(start)
}

func benchSharded(goroutines, iters, shards int) time.Duration {
    type pad struct {
        mu sync.Mutex
        n  int64
        _  [56]byte // reduce false sharing a bit
    }
    cs := make([]pad, shards)
    var wg sync.WaitGroup
    start := time.Now()
    wg.Add(goroutines)
    for g := 0; g < goroutines; g++ {
        g := g
        go func() {
            defer wg.Done()
            s := &cs[g%shards]
            for i := 0; i < iters; i++ {
                s.mu.Lock()
                s.n++
                s.mu.Unlock()
            }
        }()
    }
    wg.Wait()
    return time.Since(start)
}

func main() {
    const g, iters = 8, 100000
    fmt.Println("mutex:  ", benchMutex(g, iters))
    fmt.Println("atomic: ", benchAtomic(g, iters))
    fmt.Println("shard8: ", benchSharded(g, iters, 8))
    fmt.Println("more concurrency is not free — profile locks under real load")
}
go run .

Expected output (order of magnitude varies by machine):

mutex:   ...
atomic:  ...
shard8:  ...
more concurrency is not free — profile locks under real load

What to notice

  • A single global mutex serializes work; atomics/sharding raise the ceiling for simple counters.
  • Real contention shows up as flat RPS and rising p99—capture a mutex profile in services.
  • Fix critical sections and admission control before cranking retries.

Try next

  • go test -bench=. -cpu=1,2,4,8 and plot ns/op.
  • Enable mutex profiling: runtime.SetMutexProfileFraction(5) and pull pprof/mutex.

Further Reading

  • Go blog: mutex profile, scheduler
  • Previous: allocations/GC (often co-occur with contention under load)
  • Concurrency part of this book for patterns (worker pools, pipelines)
  • Synthesis parts for consolidating modern Go practices