Health, Liveness, and Readiness Probes

Health, Liveness, and Readiness Probes

Overview

Orchestrators ask “is this process OK?” in two different ways:

Probe	Question	Bad answer means
Liveness	Should we restart the process?	Kill + restart
Readiness	Should we send traffic?	Remove from LB

Confusing them causes restart loops or black-hole traffic.

Minimal endpoints

var ready atomic.Bool

func init() { ready.Store(true) }

func livez(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("ok"))
}

func readyz(w http.ResponseWriter, r *http.Request) {
    if !ready.Load() {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    // optional: ping DB with short timeout
    w.WriteHeader(http.StatusOK)
}

Dependency checks

Check in readiness	Check in liveness
DB ping (short timeout)	Almost never deep deps
Required config loaded	Process deadlocked (hard)
Warmup finished

Do not make liveness fail when a dependency is down—that restarts everyone during a DB blip.

Drain on shutdown

// on SIGTERM:
ready.Store(false)
time.Sleep(2 * time.Second) // allow LB to observe
_ = srv.Shutdown(ctx)

Startup probe (K8s)

Slow boots: separate startup probe so liveness doesn’t kill during init.

Rules of thumb

Do	Don’t
Cheap liveness	Heavy DB work on livez
Ready=false while draining	Keep ready=true until process death
Bound check timeouts	Hang probe handlers

Try next

1. Add /livez + /readyz to a service.
   
2. Fail readiness when DB ping fails; keep liveness OK.
   
3. Wire drain on SIGTERM; curl readyz during shutdown.