Rate Limiting and Concurrency Caps

Updated

September 8, 2026

Rate Limiting and Concurrency Caps

Overview

Protect the process from overload: limit request rate (token bucket) and in-flight concurrency (semaphore). Start simple—stdlib + golang.org/x/time/rate or a tiny token bucket.

Concurrency semaphore

func MaxInFlight(n int) func(http.Handler) http.Handler {
    sem := make(chan struct{}, n)
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            select {
            case sem <- struct{}{}:
                defer func() { <-sem }()
                next.ServeHTTP(w, r)
            default:
                http.Error(w, "busy", http.StatusServiceUnavailable)
            }
        })
    }
}

Token bucket (x/time/rate)

go get golang.org/x/time/rate@latest

func RateLimit(r rate.Limit, burst int) func(http.Handler) http.Handler {
    lim := rate.NewLimiter(r, burst)
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            if !lim.Allow() {
                w.Header().Set("Retry-After", "1")
                http.Error(w, "rate limit", http.StatusTooManyRequests)
                return
            }
            next.ServeHTTP(w, r)
        })
    }
}
// e.g. RateLimit(10, 20) → 10 req/s, burst 20

Per-IP limit (minimal)

type ipLimiter struct {
    mu   sync.Mutex
    m    map[string]*rate.Limiter
    r    rate.Limit
    b    int
}

func (l *ipLimiter) get(ip string) *rate.Limiter {
    l.mu.Lock()
    defer l.mu.Unlock()
    if lim, ok := l.m[ip]; ok {
        return lim
    }
    lim := rate.NewLimiter(l.r, l.b)
    l.m[ip] = lim
    return lim
}

Production: bound map size, use X-Forwarded-For only behind trusted proxies.

Client-side (outbound)

lim := rate.NewLimiter(5, 5)
_ = lim.Wait(ctx) // before third-party API call

Rules of thumb

Do	Don’t
503/429 with Retry-After	Hang requests forever in queue without bound
Cap expensive routes tighter	One global limit for static + checkout
Limit outbound fan-out	Unlimited parallel dependency calls

Try next

Burst 5 then 429.
Combine MaxInFlight(50) + RateLimit.
Load test with hey and tune.