Day 87 — Capstone build 5 (observability & ops)

Updated

July 30, 2026

Day 87 — Capstone build 5 (observability & ops)

Stage VIII · ~3h · Capstone day 5/6
Goal: Wire production hooks into the capstone: structured logs, metrics (and optional traces), pprof guardrails, graceful shutdown, and config completeness—so Days 88–89 are polish and evidence, not first-time ops. Go 1.26.

Why this day exists

Capstones fail demos when:

Nobody can see what the process is doing
Deploy kill drops in-flight work messily
Metrics missing for any load story
Config knobs exist only in the author’s head

You already practiced these skills in Stage VII (Days 65–70, 78)—apply them to the capstone binary. Day 88 packages; Day 89 hardens. Today makes the process operable.

Daily goals by option

Shared (A/B services; C where applicable)

Hook	Target
`slog` JSON	default for service logs
`/metrics`	Prometheus counters/histograms on core path
Graceful shutdown	SIGTERM drain (A/B)
Health	live + ready (A/B)
Config	all knobs documented in README table
pprof	localhost-only debug server or build tag

Option A extras

Metric agent_samples_total
Gauge agent_last_sample_unixtime
Counter agent_collect_errors_total
Log each sample at debug; info on errors

Option B extras

RED metrics on HTTP (rate, errors, duration)
DB pool metrics optional
OTel root span on one write path (optional light, Day 70)

Option C extras

--verbose / log level
Command timing via log fields (dur_ms)
version includes commit via ldflags
Optional: emit JSON logs for scripting

Theory 1 — Minimum metrics set

var (
    requests = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "app_requests_total",
        Help: "HTTP requests partitioned by route template and code",
    }, []string{"route", "code"})

    inFlight = promauto.NewGauge(prometheus.GaugeOpts{
        Name: "app_in_flight_requests",
        Help: "In-flight HTTP requests",
    })

    reqDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "app_request_duration_seconds",
        Help:    "Request latency",
        Buckets: prometheus.DefBuckets,
    }, []string{"route"})
)

CLI substitute: structured log line event=cmd_done dur_ms=... err=....

Cardinality rule

Label with route patterns (/v1/items/:id), not raw URLs with IDs. High-cardinality labels will melt Prometheus—and your demo laptop.

Theory 2 — slog defaults

func newLogger(level string) *slog.Logger {
    var lv slog.Level
    _ = lv.UnmarshalText([]byte(level))
    h := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: lv})
    return slog.New(h)
}

Do	Don’t
Log route, status, dur_ms	Log `Authorization` headers
Log error strings server-side	Dump stack traces to clients
Use `slog.Error` for failures	`fmt.Println` in hot paths

Redact config when printing startup summary (DSN → postgres://***).

Theory 3 — Shutdown order (services)

1. signal received (SIGINT/SIGTERM)
2. readiness = false   // fail k8s ready probe
3. http.Server.Shutdown(ctx)
4. stop background loops (agent ticker)
5. otel provider Shutdown if any
6. db.Close()
7. process exit 0

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

go func() {
    if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
        log.Error("serve", "err", err)
        stop()
    }
}()

<-ctx.Done()
log.Info("shutting down")
ready.Store(false)

shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
    log.Error("shutdown", "err", err)
}
// stop agent, close db

Agent loop

func (a *Agent) Run(ctx context.Context) error {
    t := time.NewTicker(a.Interval)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            a.Log.Info("agent stop")
            return nil
        case <-t.C:
            cctx, cancel := context.WithTimeout(ctx, a.CollectTimeout)
            snap, err := a.Collector.Collect(cctx)
            cancel()
            if err != nil {
                a.Log.Error("collect", "err", err)
                collectErrs.Inc()
                continue
            }
            samples.Inc()
            lastSample.Set(float64(snap.TS.Unix()))
            _ = a.Store.Append(snap)
        }
    }
}

Theory 4 — Health vs readiness

Probe	Meaning	Fail when
Live (`/healthz`)	Process up	Never (or only total deadlock)
Ready (`/readyz`)	Accept traffic	DB down, still migrating, shutting down

func (h *Handler) Ready(w http.ResponseWriter, r *http.Request) {
    if !h.ReadyFlag.Load() {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    if err := h.DB.PingContext(r.Context()); err != nil {
        http.Error(w, "db", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte(`{"status":"ready"}`))
}

Set ready true only after migrations succeed.

Theory 5 — pprof guardrails

// separate server — never mount on public :8080 without auth
go func() {
    addr := "127.0.0.1:6060"
    mux := http.NewServeMux()
    mux.HandleFunc("/debug/pprof/", pprof.Index)
    mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
    mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
    mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
    mux.HandleFunc("/debug/pprof/trace", pprof.Trace)
    log.Info("pprof", "addr", addr)
    _ = http.ListenAndServe(addr, mux)
}()

Or gate with build tag //go:build debug. Document in OPS.md.

Theory 6 — Feature flag hook (optional)

If Day 79 patterns help demo a progressive path, gate one non-critical behavior. Do not introduce flag debt on the primary path unless designed in Day 82.

if cfg.Flags.EnableExperimentalList {
    // optional route
}

Worked example — slog + metrics middleware (A/B)

type statusWriter struct {
    http.ResponseWriter
    code int
}

func (w *statusWriter) WriteHeader(code int) {
    w.code = code
    w.ResponseWriter.WriteHeader(code)
}

func withOps(log *slog.Logger, route string, next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        inFlight.Inc()
        defer inFlight.Dec()
        sw := &statusWriter{ResponseWriter: w, code: http.StatusOK}
        next(sw, r)
        code := strconv.Itoa(sw.code)
        requests.WithLabelValues(route, code).Inc()
        reqDuration.WithLabelValues(route).Observe(time.Since(start).Seconds())
        log.Info("request",
            "route", route,
            "method", r.Method,
            "code", sw.code,
            "dur_ms", time.Since(start).Milliseconds(),
        )
    }
}

mux.Handle("GET /metrics", promhttp.Handler())

CLI structured completion

start := time.Now()
err := runApply(ctx, cfg)
log.Info("cmd_done",
    "cmd", "apply",
    "dur_ms", time.Since(start).Milliseconds(),
    "err", errString(err),
)
if err != nil {
    os.Exit(mapExit(err))
}

ldflags version (C and services)

go build -ldflags "-X example.com/app/internal/version.Commit=$(git rev-parse --short HEAD)" -o bin/app ./cmd/app

Lab 1 — Logging + config completeness

JSON slog as default for services (CLI: text or JSON via flag).
Log level from config/env.
On startup, log non-secret config summary.
Fill README env table: ADDR, DSN, LOG_LEVEL, intervals, etc.

| Variable | Default | Meaning |
|----------|---------|---------|
| ADDR | :8080 | HTTP listen |
| LOG_LEVEL | info | slog level |
| DSN | | database |

Lab 2 — Metrics or CLI telemetry

Services: /metrics with at least request counter + one domain metric.
CLI: structured cmd_done lines on primary commands.
Hit smoke traffic; confirm metrics non-empty.

curl -s localhost:8080/metrics | head
# after traffic:
curl -s localhost:8080/metrics | grep app_requests_total

Lab 3 — Shutdown + probes (services)

Implement SIGINT/SIGTERM shutdown order.
Live + ready endpoints.
Prove ready fails during shutdown or before DB.
kill -TERM $PID → clean exit within timeout.

./bin/service &
PID=$!
curl -sf localhost:8080/healthz
curl -sf localhost:8080/readyz
kill -TERM $PID
wait $PID

Lab 4 — Write `OPS.md` runbook

# Ops runbook — capstone

## Start
...

## Probes
curl -sf localhost:8080/healthz
curl -sf localhost:8080/readyz

## Metrics
curl -s localhost:8080/metrics | head

## pprof (local only)
curl -o cpu.pb.gz http://127.0.0.1:6060/debug/pprof/cpu?seconds=5

## Stop
kill -TERM <pid>

Keep it short—10–30 lines. Day 88 demo script will call these paths.

Lab 5 — CAPSTONE milestone + residual gaps

## Milestone 87 — observability & ops
- [x] slog JSON
- [x] metrics or CLI telemetry
- [x] shutdown + probes (if service)
- [x] OPS.md
- Leftovers for 88/89: ...

List anything still missing (tracing, histograms, rate limit) without starting them unless critical for demo.

Hour-by-hour suggestion

Time	Focus
0:00–0:40	slog handler + config log level
0:40–1:30	metrics or CLI telemetry
1:30–2:20	shutdown + readiness (services)
2:20–2:50	OPS.md + smoke metrics
2:50–3:00	CAPSTONE milestone check

Common gotchas

Gotcha	Fix
Metrics without load	Hit smoke script first
Ready=true before DB migrate	Order startup
Agent goroutine leak	cancel context on shutdown
Logging secrets	redact config
pprof on `:8080` public	separate bind `127.0.0.1:6060`
Cardinality on routes	use patterns not raw URLs
Double-register Prometheus	`promauto` once at package init
`Shutdown` without timeout	`context.WithTimeout`
CLI still using `log.Printf` only	add structured fields for demo
Histogram buckets unconsidered	`DefBuckets` OK for lab

Checkpoint

Commit

git add .
git commit -m "day87: capstone observability ops hooks"

Write three personal gotchas before continuing.

Tomorrow

Day 88 — Capstone build 6: integration polish, Docker/release artifacts, and a repeatable demo script that walks the story without improvisation—using today’s ops hooks as proof points.

Day 87 — Capstone build 5 (observability & ops)

Why this day exists

Daily goals by option

Shared (A/B services; C where applicable)

Option A extras

Option B extras

Option C extras

Theory 1 — Minimum metrics set

Cardinality rule

Theory 2 — slog defaults

Theory 3 — Shutdown order (services)

Agent loop

Theory 4 — Health vs readiness

Theory 5 — pprof guardrails

Theory 6 — Feature flag hook (optional)

Worked example — slog + metrics middleware (A/B)

CLI structured completion

ldflags version (C and services)

Lab 1 — Logging + config completeness

Lab 2 — Metrics or CLI telemetry

Lab 3 — Shutdown + probes (services)

Lab 4 — Write OPS.md runbook

Lab 5 — CAPSTONE milestone + residual gaps

Hour-by-hour suggestion

Common gotchas

Checkpoint

Commit

Tomorrow

Lab 4 — Write `OPS.md` runbook