Day 78 — Graceful shutdown & health

Day 78 — Graceful shutdown & health

Stage VIII · ~3h
Goal: Implement signal-aware graceful shutdown for an HTTP service, distinguish liveness vs readiness, and drain in-flight requests within a configurable timeout.

Note

Containers and orchestrators will send SIGTERM. Ignoring it is a production bug, not a style choice.

Why this day exists

Typical stop sequence:

1. SIGTERM
   
2. grace period
   
3. SIGKILL

If you ignore SIGTERM, users see dropped connections on every deploy. Health probes misconfigured cause restart storms. This day is mandatory production literacy—wired on Day 68 images that run your binary as PID 1.

---

Theory 1 — Process lifecycle

start → load config → open deps → serve
                              ↑
SIGTERM → stop readiness → Server.Shutdown → close deps → exit

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

signal.NotifyContext cancels when a signal arrives—clean integration with Shutdown. Prefer it over manual signal.Notify channels unless you need custom multiplexing.

Why both Interrupt and SIGTERM

Signal	Source
SIGINT	Ctrl-C in terminal
SIGTERM	docker stop, Kubernetes, systemd

Handle both so laptop and cluster behave the same.

---

Theory 2 — http.Server.Shutdown

srv := &http.Server{
    Addr:              cfg.Addr,
    Handler:           mux,
    ReadHeaderTimeout: 5 * time.Second,
}

go func() {
    if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
        log.Error("serve", "err", err)
        // coordinate fatal via err channel in real main
    }
}()

<-ctx.Done()
log.Info("shutdown started")

shCtx, cancel := context.WithTimeout(context.Background(), time.Duration(cfg.ShutdownSec)*time.Second)
defer cancel()
if err := srv.Shutdown(shCtx); err != nil {
    log.Error("shutdown", "err", err)
    _ = srv.Close()
}
// close DB pool, flush metrics, otel Shutdown, etc.

Call	Behavior
Shutdown	Stop accept; wait in-flight up to ctx
Close	Immediate; use if Shutdown times out

Ordering of teardown

1. Mark not ready
   
2. Optional short sleep for LB de-register
   
3. Server.Shutdown
   
4. Cancel worker contexts
   
5. Close DB / message consumers
   
6. OTel TracerProvider.Shutdown / flush metrics if needed
   
7. Exit 0

Closing the DB before HTTP drain causes handler errors mid-flight.

---

Theory 3 — Liveness vs readiness

Probe	Question	Should fail when
Liveness /healthz	Is process deadlocked / stuck?	Only if process is beyond recovery
Readiness /readyz	Can it serve traffic?	DB down, warming, shutting down

Classic mistake: liveness fails when DB blips → Kubernetes restarts healthy pods that were temporarily unready → worse outage.

type Health struct {
    ready atomic.Bool
}

func (h *Health) Live(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("ok"))
}

func (h *Health) Ready(w http.ResponseWriter, r *http.Request) {
    if !h.ready.Load() {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    // optional: ping DB with short timeout
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("ready"))
}

h.ready.Store(true)  // after deps open
// on signal:
h.ready.Store(false) // fail readiness first
time.Sleep(1 * time.Second) // optional: let LB de-register
srv.Shutdown(...)

Optional DB ping on ready

func (h *Health) Ready(w http.ResponseWriter, r *http.Request) {
    if !h.ready.Load() {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    ctx, cancel := context.WithTimeout(r.Context(), 300*time.Millisecond)
    defer cancel()
    if err := h.db.PingContext(ctx); err != nil {
        http.Error(w, "db", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("ready"))
}

Keep the timeout tiny so probes do not pile up.

---

Theory 4 — In-flight work and contexts

Handlers should honor r.Context() cancellation when the client goes away. For shutdown, Shutdown waits for handlers to return—long Sleep without select on ctx blocks drain.

select {
case <-r.Context().Done():
    return
case <-time.After(work):
}

Background workers: listen on the same root context; exit on cancel.

go func() {
    for {
        select {
        case <-rootCtx.Done():
            return
        case job := <-jobs:
            process(rootCtx, job)
        }
    }
}()

Long requests vs grace period

If max handler time is 30s, ShutdownSec and Kubernetes terminationGracePeriodSeconds must be greater than that (plus de-register buffer). Otherwise SIGKILL cuts work mid-flight.

---

Theory 5 — Kubernetes sketch (awareness)

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
terminationGracePeriodSeconds: 30
lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]  # if you still have a shell; distroless may skip

With distroless, prefer app-side readiness flip + terminationGracePeriodSeconds ≥ ShutdownSec.

---

Theory 6 — Testing shutdown

Automated tests can drive Shutdown without real signals:

func TestShutdownDrains(t *testing.T) {
    srv := &http.Server{Addr: "127.0.0.1:0", Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        time.Sleep(50 * time.Millisecond)
        w.WriteHeader(200)
    })}
    ln, err := net.Listen("tcp", srv.Addr)
    if err != nil {
        t.Fatal(err)
    }
    go srv.Serve(ln)

    // fire request...
    shCtx, cancel := context.WithTimeout(context.Background(), time.Second)
    defer cancel()
    if err := srv.Shutdown(shCtx); err != nil {
        t.Fatal(err)
    }
}

Manual demo remains required for SIGTERM path.

---

Worked example — compact main

func main() {
    cfg := mustConfig()
    rootCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    db := mustDB(cfg)
    defer db.Close()

    health := &Health{}
    mux := routes(db, health)
    srv := &http.Server{Addr: cfg.Addr, Handler: mux, ReadHeaderTimeout: 5 * time.Second}

    go func() {
        log.Printf("listen %s", cfg.Addr)
        if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
            log.Fatal(err)
        }
    }()
    health.ready.Store(true)

    <-rootCtx.Done()
    health.ready.Store(false)
    log.Printf("draining...")

    shCtx, cancel := context.WithTimeout(context.Background(), time.Duration(cfg.ShutdownSec)*time.Second)
    defer cancel()
    if err := srv.Shutdown(shCtx); err != nil {
        log.Printf("shutdown error: %v", err)
    }
}

Manual test:

go run .
# other terminal
curl -s localhost:8080/readyz
kill -TERM <pid>
# curl readyz should go 503; in-flight finishes; process exits 0

Config keys (Day 77)

Key	Default	Meaning
SHUTDOWN_SEC	15	Drain timeout
ADDR	:8080	Listen address

---

Lab

Suggested workspace: your HTTP service.

1. signal.NotifyContext + Server.Shutdown with timeout from config.
   
2. /healthz and /readyz with readiness false during drain.
   
3. Close DB/tracer after HTTP drain.
   
4. Document probe semantics in README.
   
5. Demo notes: command sequence + observed behavior.
   
6. Prove with kill -TERM (or docker stop if containerized).

Stretch

• Integrate OTel Shutdown and Prometheus gatherer flush if needed.
  
• preStop sleep discussion for mesh/LB timing.
  
• Integration test that hits a slow handler during Shutdown.

---

Common gotchas

Gotcha	Fix
Liveness = DB ping	Move DB to readiness
Shutdown timeout too short	Align with longest request + grace
Ignoring ErrServerClosed	Treat as success
Shell ENTRYPOINT eats signals	Day 68 exec-form binary
Workers leak after HTTP stop	Cancel shared context
Ready before migrations	Run migrate then ready=true
Closing DB before Shutdown	Reverse order
Logging FATAL on ErrServerClosed	Filter it

---

Checkpoint

• SIGTERM triggers drain
  
• Shutdown timeout configurable
  
• Distinct live vs ready endpoints
  
• Readiness false on shutdown
  
• Deps closed after HTTP drain
  
• Demo steps written

Commit

git add .
git commit -m "day78: graceful shutdown health probes"

Write three personal gotchas before continuing.

---

Tomorrow

Day 79 — Feature flags: progressive delivery awareness with one real flag gate in code.