Day 27 — Config, Graceful Shutdown & Feature Flags

Updated

July 30, 2025

Day 27 — Config, Graceful Shutdown & Feature Flags

Day 77 — Config layering

Stage VIII · ~3h
Goal: Implement a layered configuration loader—defaults → file → environment → flags—with clear precedence, typed config structs, validation, and no secret-in-repo footguns.

Note

Only the composition root (and a dedicated config package) should read env/flags. Lower layers receive values.

Why this day exists

Production processes need configuration. Ad-hoc os.Getenv sprinkled across packages causes:

  • Undocumented knobs
  • Different defaults in tests vs prod
  • Secrets committed to git
  • Flags that fight env vars

12-factor-ish discipline: config from environment (and files for local), validated at startup, injected into the composition root. Capstone days 83–88 will copy this pattern.


Theory 1 — Precedence model

Recommended order (later wins):

1. Code defaults
2. Config file (optional YAML/JSON/TOML)
3. Environment variables
4. CLI flags

Document this in README. Surprises here are outages.

type Config struct {
    Addr        string
    DatabaseURL string
    LogLevel    string
    ShutdownSec int
}

Why flags win last

Local overrides for debugging (--addr :9090) should beat shell env without editing files. Deployments usually set env and pass no flags.


Theory 2 — Stdlib-first loading

func Load(args []string) (Config, error) {
    cfg := Config{
        Addr:        ":8080",
        LogLevel:    "info",
        ShutdownSec: 15,
    }

    // optional file
    if path := os.Getenv("CONFIG_FILE"); path != "" {
        if err := loadFile(path, &cfg); err != nil {
            return Config{}, err
        }
    }

    if v := os.Getenv("ADDR"); v != "" {
        cfg.Addr = v
    }
    if v := os.Getenv("DATABASE_URL"); v != "" {
        cfg.DatabaseURL = v
    }
    if v := os.Getenv("LOG_LEVEL"); v != "" {
        cfg.LogLevel = v
    }
    if v := os.Getenv("SHUTDOWN_SEC"); v != "" {
        n, err := strconv.Atoi(v)
        if err != nil {
            return Config{}, fmt.Errorf("SHUTDOWN_SEC: %w", err)
        }
        cfg.ShutdownSec = n
    }

    fs := flag.NewFlagSet("service", flag.ContinueOnError)
    fs.SetOutput(io.Discard) // or os.Stderr for user CLIs
    addr := fs.String("addr", cfg.Addr, "listen address")
    db := fs.String("database-url", cfg.DatabaseURL, "database URL")
    logLevel := fs.String("log-level", cfg.LogLevel, "log level")
    if err := fs.Parse(args); err != nil {
        return Config{}, err
    }
    cfg.Addr = *addr
    cfg.DatabaseURL = *db
    cfg.LogLevel = *logLevel

    if err := cfg.Validate(); err != nil {
        return Config{}, err
    }
    return cfg, nil
}

func (c Config) Validate() error {
    if c.DatabaseURL == "" {
        return errors.New("DATABASE_URL required")
    }
    if c.ShutdownSec < 1 {
        return errors.New("shutdown sec must be >= 1")
    }
    switch strings.ToLower(c.LogLevel) {
    case "debug", "info", "warn", "error":
    default:
        return fmt.Errorf("invalid log level %q", c.LogLevel)
    }
    return nil
}

File format

JSON is enough without deps:

func loadFile(path string, cfg *Config) error {
    b, err := os.ReadFile(path)
    if err != nil {
        return err
    }
    return json.Unmarshal(b, cfg)
}
{
  "addr": ":8080",
  "log_level": "debug",
  "shutdown_sec": 20
}

YAML needs a library—fine if you already use one. Prefer one format per project.


Theory 3 — Secrets

Do Don’t
Env / secret manager Commit .env with real secrets
Document var names Log full DATABASE_URL
Separate Config vs Secrets if helpful Put tokens in config files in images
// redact when logging
func (c Config) LogValue() slog.Value {
    return slog.GroupValue(
        slog.String("addr", c.Addr),
        slog.String("log_level", c.LogLevel),
        slog.Int("shutdown_sec", c.ShutdownSec),
        slog.String("database_url", redactDSN(c.DatabaseURL)),
    )
}

func redactDSN(dsn string) string {
    // naive: hide password between : and @
    // improve for your DSN shape
    if i := strings.Index(dsn, "://"); i >= 0 {
        rest := dsn[i+3:]
        if at := strings.Index(rest, "@"); at >= 0 {
            return dsn[:i+3] + "***@" + rest[at+1:]
        }
    }
    return "***"
}

.env for local dev is OK if gitignored; provide .env.example with empty values.

# .env.example
DATABASE_URL=
ADDR=:8080
LOG_LEVEL=info
SHUTDOWN_SEC=15
CONFIG_FILE=

Theory 4 — Injection at the root

func main() {
    cfg, err := config.Load(os.Args[1:])
    if err != nil {
        fmt.Fprintf(os.Stderr, "config: %v\n", err)
        os.Exit(2)
    }
    logger := newLogger(cfg.LogLevel)
    logger.Info("starting", "cfg", cfg) // uses LogValue if slog.LogValuer
    db := mustOpen(cfg.DatabaseURL)
    // wire adapters with cfg fields — packages don't call os.Getenv
    _ = db
}

Rule: only config package (and main) reads env/flags. Lower layers receive values.

Why exit code 2

Convention: 2 for misuse/config; 1 for runtime failure. CLIs (Day 71) already care—services should too.


Theory 5 — Testing config

func TestLoadDefaults(t *testing.T) {
    t.Setenv("DATABASE_URL", "sqlite://file.db")
    t.Setenv("ADDR", "") // ensure not leaking from parent env if needed
    cfg, err := Load(nil)
    if err != nil {
        t.Fatal(err)
    }
    if cfg.Addr != ":8080" {
        t.Fatalf("addr %s", cfg.Addr)
    }
}

func TestFlagBeatsEnv(t *testing.T) {
    t.Setenv("DATABASE_URL", "sqlite://file.db")
    t.Setenv("ADDR", ":9090")
    cfg, err := Load([]string{"-addr", ":9091"})
    if err != nil {
        t.Fatal(err)
    }
    if cfg.Addr != ":9091" {
        t.Fatalf("got %s", cfg.Addr)
    }
}

func TestValidateMissingDB(t *testing.T) {
    t.Setenv("DATABASE_URL", "")
    _, err := Load(nil)
    if err == nil {
        t.Fatal("expected error")
    }
}

t.Setenv keeps tests hermetic (Go 1.17+).


Theory 6 — K8s / Docker alignment

Surface Maps to
container env: process environment
ConfigMap file mount CONFIG_FILE
Secret mount env or file; never image layers
CLI debug flags on command:

Document one canonical name per knob (DATABASE_URL not sometimes DB_URL).

GOMEMLIMIT is process env that Go runtime reads—document alongside app config even if you do not parse it in Go.


Worked example — layered demo

export DATABASE_URL=postgres://localhost/app
export ADDR=:9090
go run ./cmd/service --addr :9091
# effective ADDR :9091 (flag wins)

Document matrix in CONFIG.md:

Key Default Env Flag
addr :8080 ADDR -addr
database (none) DATABASE_URL -database-url
log level info LOG_LEVEL -log-level
shutdown 15 SHUTDOWN_SEC -shutdown-sec

Worked example — empty env must not clobber

// wrong: always assign
cfg.Addr = os.Getenv("ADDR") // empty string wipes file/default

// right: only override when set
if v := os.Getenv("ADDR"); v != "" {
    cfg.Addr = v
}

Lab

Suggested workspace: service composition root.

  1. Central internal/config (or config) package.
  2. Precedence: defaults → file (optional) → env → flags.
  3. Validate() fails fast on missing required values.
  4. .env.example + README table.
  5. Unit tests for precedence and validation errors.
  6. Redacted logging of secrets.

Stretch

  • GOMEMLIMIT documented alongside app config.
  • Feature flag file path field prepared for Day 79.
  • String() method that never prints secrets (for fmt debugging).

Common gotchas

Gotcha Fix
Env read deep in store package Inject DSN from main
Empty string overwriting file values Only override when env set
flag.Parse global in library FlagSet in Load
Secrets in logs Redact
Different names in K8s vs local One canonical env name
Panic on bad config Return error; main exits 2
Bool flags defaulting wrong Document true/false strings
Required vs optional confusion Validate() is the source of truth

Checkpoint

  • Typed Config + Load
  • Documented precedence
  • Validation at startup
  • Tests for env/flag behavior
  • Example env file without secrets
  • Redaction on log path

Commit

git add .
git commit -m "day77: config layering flags env file"

Write three personal gotchas before continuing.


Tomorrow

Day 78 — Graceful shutdown & health: production process lifecycle—SIGTERM, drain, liveness vs readiness.


Day 78 — Graceful shutdown & health

Stage VIII · ~3h
Goal: Implement signal-aware graceful shutdown for an HTTP service, distinguish liveness vs readiness, and drain in-flight requests within a configurable timeout.

Note

Containers and orchestrators will send SIGTERM. Ignoring it is a production bug, not a style choice.

Why this day exists

Typical stop sequence:

  1. SIGTERM
  2. grace period
  3. SIGKILL

If you ignore SIGTERM, users see dropped connections on every deploy. Health probes misconfigured cause restart storms. This day is mandatory production literacy—wired on Day 68 images that run your binary as PID 1.


Theory 1 — Process lifecycle

start → load config → open deps → serve
                              ↑
SIGTERM → stop readiness → Server.Shutdown → close deps → exit
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

signal.NotifyContext cancels when a signal arrives—clean integration with Shutdown. Prefer it over manual signal.Notify channels unless you need custom multiplexing.

Why both Interrupt and SIGTERM

Signal Source
SIGINT Ctrl-C in terminal
SIGTERM docker stop, Kubernetes, systemd

Handle both so laptop and cluster behave the same.


Theory 2 — http.Server.Shutdown

srv := &http.Server{
    Addr:              cfg.Addr,
    Handler:           mux,
    ReadHeaderTimeout: 5 * time.Second,
}

go func() {
    if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
        log.Error("serve", "err", err)
        // coordinate fatal via err channel in real main
    }
}()

<-ctx.Done()
log.Info("shutdown started")

shCtx, cancel := context.WithTimeout(context.Background(), time.Duration(cfg.ShutdownSec)*time.Second)
defer cancel()
if err := srv.Shutdown(shCtx); err != nil {
    log.Error("shutdown", "err", err)
    _ = srv.Close()
}
// close DB pool, flush metrics, otel Shutdown, etc.
Call Behavior
Shutdown Stop accept; wait in-flight up to ctx
Close Immediate; use if Shutdown times out

Ordering of teardown

  1. Mark not ready
  2. Optional short sleep for LB de-register
  3. Server.Shutdown
  4. Cancel worker contexts
  5. Close DB / message consumers
  6. OTel TracerProvider.Shutdown / flush metrics if needed
  7. Exit 0

Closing the DB before HTTP drain causes handler errors mid-flight.


Theory 3 — Liveness vs readiness

Probe Question Should fail when
Liveness /healthz Is process deadlocked / stuck? Only if process is beyond recovery
Readiness /readyz Can it serve traffic? DB down, warming, shutting down

Classic mistake: liveness fails when DB blips → Kubernetes restarts healthy pods that were temporarily unready → worse outage.

type Health struct {
    ready atomic.Bool
}

func (h *Health) Live(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("ok"))
}

func (h *Health) Ready(w http.ResponseWriter, r *http.Request) {
    if !h.ready.Load() {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    // optional: ping DB with short timeout
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("ready"))
}
h.ready.Store(true)  // after deps open
// on signal:
h.ready.Store(false) // fail readiness first
time.Sleep(1 * time.Second) // optional: let LB de-register
srv.Shutdown(...)

Optional DB ping on ready

func (h *Health) Ready(w http.ResponseWriter, r *http.Request) {
    if !h.ready.Load() {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    ctx, cancel := context.WithTimeout(r.Context(), 300*time.Millisecond)
    defer cancel()
    if err := h.db.PingContext(ctx); err != nil {
        http.Error(w, "db", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("ready"))
}

Keep the timeout tiny so probes do not pile up.


Theory 4 — In-flight work and contexts

Handlers should honor r.Context() cancellation when the client goes away. For shutdown, Shutdown waits for handlers to return—long Sleep without select on ctx blocks drain.

select {
case <-r.Context().Done():
    return
case <-time.After(work):
}

Background workers: listen on the same root context; exit on cancel.

go func() {
    for {
        select {
        case <-rootCtx.Done():
            return
        case job := <-jobs:
            process(rootCtx, job)
        }
    }
}()

Long requests vs grace period

If max handler time is 30s, ShutdownSec and Kubernetes terminationGracePeriodSeconds must be greater than that (plus de-register buffer). Otherwise SIGKILL cuts work mid-flight.


Theory 5 — Kubernetes sketch (awareness)

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
terminationGracePeriodSeconds: 30
lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]  # if you still have a shell; distroless may skip

With distroless, prefer app-side readiness flip + terminationGracePeriodSecondsShutdownSec.


Theory 6 — Testing shutdown

Automated tests can drive Shutdown without real signals:

func TestShutdownDrains(t *testing.T) {
    srv := &http.Server{Addr: "127.0.0.1:0", Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        time.Sleep(50 * time.Millisecond)
        w.WriteHeader(200)
    })}
    ln, err := net.Listen("tcp", srv.Addr)
    if err != nil {
        t.Fatal(err)
    }
    go srv.Serve(ln)

    // fire request...
    shCtx, cancel := context.WithTimeout(context.Background(), time.Second)
    defer cancel()
    if err := srv.Shutdown(shCtx); err != nil {
        t.Fatal(err)
    }
}

Manual demo remains required for SIGTERM path.


Worked example — compact main

func main() {
    cfg := mustConfig()
    rootCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    db := mustDB(cfg)
    defer db.Close()

    health := &Health{}
    mux := routes(db, health)
    srv := &http.Server{Addr: cfg.Addr, Handler: mux, ReadHeaderTimeout: 5 * time.Second}

    go func() {
        log.Printf("listen %s", cfg.Addr)
        if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
            log.Fatal(err)
        }
    }()
    health.ready.Store(true)

    <-rootCtx.Done()
    health.ready.Store(false)
    log.Printf("draining...")

    shCtx, cancel := context.WithTimeout(context.Background(), time.Duration(cfg.ShutdownSec)*time.Second)
    defer cancel()
    if err := srv.Shutdown(shCtx); err != nil {
        log.Printf("shutdown error: %v", err)
    }
}

Manual test:

go run .
# other terminal
curl -s localhost:8080/readyz
kill -TERM <pid>
# curl readyz should go 503; in-flight finishes; process exits 0

Config keys (Day 77)

Key Default Meaning
SHUTDOWN_SEC 15 Drain timeout
ADDR :8080 Listen address

Lab

Suggested workspace: your HTTP service.

  1. signal.NotifyContext + Server.Shutdown with timeout from config.
  2. /healthz and /readyz with readiness false during drain.
  3. Close DB/tracer after HTTP drain.
  4. Document probe semantics in README.
  5. Demo notes: command sequence + observed behavior.
  6. Prove with kill -TERM (or docker stop if containerized).

Stretch

  • Integrate OTel Shutdown and Prometheus gatherer flush if needed.
  • preStop sleep discussion for mesh/LB timing.
  • Integration test that hits a slow handler during Shutdown.

Common gotchas

Gotcha Fix
Liveness = DB ping Move DB to readiness
Shutdown timeout too short Align with longest request + grace
Ignoring ErrServerClosed Treat as success
Shell ENTRYPOINT eats signals Day 68 exec-form binary
Workers leak after HTTP stop Cancel shared context
Ready before migrations Run migrate then ready=true
Closing DB before Shutdown Reverse order
Logging FATAL on ErrServerClosed Filter it

Checkpoint

  • SIGTERM triggers drain
  • Shutdown timeout configurable
  • Distinct live vs ready endpoints
  • Readiness false on shutdown
  • Deps closed after HTTP drain
  • Demo steps written

Commit

git add .
git commit -m "day78: graceful shutdown health probes"

Write three personal gotchas before continuing.


Tomorrow

Day 79 — Feature flags: progressive delivery awareness with one real flag gate in code.


Day 79 — Feature flags

Stage VIII · ~3h
Goal: Implement one feature flag gate (boolean or percentage) with a clean evaluation API, safe defaults, and operational notes—without building a full flag SaaS.

Note

Deploy ≠ release. Today: one flag, well-structured, with a removal plan. Flag debt is real—treat flags as temporary code paths.

Why this day exists

Feature flags let you:

  • Ship dark code and enable later
  • Kill-switch a bad path quickly
  • Gradual rollout (10% → 50% → 100%)

Abuse creates untested combinations and permanent #ifdef culture. The skill is a small evaluation surface that application code can depend on without knowing LaunchDarkly vs env vars.


Theory 1 — Flag types

Kind Example
Release toggle New checkout flow on/off
Ops toggle Disable expensive export job
Experiment 5% see algorithm B
Permission Entitlement (often not a “flag service”)

Lifecycle: create → enable gradually → remove flag and dead code.

ship off → canary % → 100% → delete old path + flag

If a flag has been 100% for a sprint, schedule removal.


Theory 2 — Evaluation API

Keep evaluation boring and injectable:

package flags

import "context"

type Provider interface {
    Enabled(ctx context.Context, key string, subject Subject) bool
}

type Subject struct {
    UserID string
    // Attrs map[string]string // stretch
}

type Static map[string]bool

func (s Static) Enabled(_ context.Context, key string, _ Subject) bool {
    return s[key]
}

Env-based provider (good enough for many services):

type Env struct{}

func (Env) Enabled(_ context.Context, key string, _ Subject) bool {
    v := strings.ToLower(os.Getenv("FLAG_" + strings.ToUpper(key)))
    return v == "1" || v == "true" || v == "on"
}

Percentage rollout (stable bucketing):

func EnabledPercent(key, userID string, percent int) bool {
    if percent <= 0 {
        return false
    }
    if percent >= 100 {
        return true
    }
    sum := sha256.Sum256([]byte(key + ":" + userID))
    // use first 2 bytes → 0..65535
    n := int(sum[0])<<8 | int(sum[1])
    return n%100 < percent
}

Stability matters: same user should not flip every request. Include flag key in the hash so different flags do not correlate perfectly.

Composed provider

type PercentEnv struct {
    Percent map[string]int // key → 0..100
}

func (p PercentEnv) Enabled(_ context.Context, key string, sub Subject) bool {
    pct, ok := p.Percent[key]
    if !ok {
        // fall back to boolean env
        return Env{}.Enabled(context.Background(), key, sub)
    }
    return EnabledPercent(key, sub.UserID, pct)
}

Theory 3 — Where to branch

Prefer branching in application layer, not deep in SQL drivers:

if s.flags.Enabled(ctx, "new_search", flags.Subject{UserID: userID}) {
    return s.searchV2(ctx, q)
}
return s.searchV1(ctx, q)

HTTP adapter can pass subject from auth middleware.

Defaults

  • Fail closed for risky features (false if provider errors).
  • Fail open only for non-critical cosmetics—document choice.
func (s *Service) Enabled(ctx context.Context, key string, sub flags.Subject) bool {
    // if remote provider added later:
    // on timeout → return false for release toggles
    return s.flags.Enabled(ctx, key, sub)
}

Theory 4 — Config integration (Day 77)

type Config struct {
    // ...
    FlagNewSearch        bool
    FlagNewSearchPercent int
}

Or a JSON file:

{
  "flags": {
    "new_search": { "enabled": true, "percent": 10 }
  }
}

Do not require a SaaS vendor for the lab. Awareness of LaunchDarkly/Unleash/OpenFeature is enough:

Concept Note
OpenFeature Vendor-neutral API trend
Remote config Needs caching + timeouts
Audit log Who flipped what

Wire at composition root

fp := flags.Static{
    "new_search": cfg.FlagNewSearch,
}
// or flags.Env{} for ops kill switches without redeploy of binaries
// (still needs process restart unless you poll a file)
svc := app.New(deps, fp)

Theory 5 — Testing matrix

Flags double paths—test both:

func TestSearchFlag(t *testing.T) {
    svc := NewService(deps, flags.Static{"new_search": true})
    // assert v2 behavior
    got, err := svc.Search(context.Background(), "q")
    if err != nil || !got.FromV2 {
        t.Fatalf("v2: %#v %v", got, err)
    }

    svc2 := NewService(deps, flags.Static{"new_search": false})
    got2, err := svc2.Search(context.Background(), "q")
    if err != nil || got2.FromV2 {
        t.Fatalf("v1: %#v %v", got2, err)
    }
}

Percentage stability test

func TestPercentStable(t *testing.T) {
    a := EnabledPercent("exp", "user-1", 30)
    b := EnabledPercent("exp", "user-1", 30)
    if a != b {
        t.Fatal("unstable")
    }
}

Theory 6 — Ops and observability

Expose evaluation for operators carefully:

// internal admin only — auth required
mux.HandleFunc("GET /internal/flags", func(w http.ResponseWriter, r *http.Request) {
    // list configured keys + defaults — not every subject evaluation
})

Metrics (stretch):

flag_evaluation_total{key="new_search",result="true"} 120
flag_evaluation_total{key="new_search",result="false"} 880

Avoid per-user labels.

Kill switch runbook

# env-driven: restart with flag off
FLAG_REPORTS=false systemctl restart myservice
# or kubernetes set env + rollout restart

Document whether a restart is required. File-based flags can be re-read on interval—only if you implement it; do not pretend env vars hot-reload.


Worked example — kill switch for expensive endpoint

mux.HandleFunc("GET /v1/report", func(w http.ResponseWriter, r *http.Request) {
    if !fp.Enabled(r.Context(), "reports", subjectFrom(r)) {
        // 404 hides existence; 503 signals temporary — pick one and document
        http.Error(w, "reports disabled", http.StatusServiceUnavailable)
        return
    }
    writeReport(w, r)
})

FLAGS.md template

# Flags
| Key | Default | Owner | Type | Removal criteria |
|-----|---------|-------|------|------------------|
| reports | false | you | ops kill | N/A (permanent ops) or date |
| new_search | false | you | release | After 100% for 14 days + delete v1 |

Lab

Suggested workspace: your service.

  1. Define flags.Provider interface.
  2. Implement static + env (or file) provider.
  3. Gate one real path (feature or kill switch).
  4. Tests for on and off.
  5. FLAGS.md: key name, default, owner, removal criteria.
  6. Default is safe when unset (usually false for new features).

Stretch

  • Percentage rollout with stable hashing.
  • Expose active flags on an internal admin endpoint (auth required).
  • Metric flag_evaluation_total{key,result}.
  • File provider with mtime reload every 30s (document races).

Common gotchas

Gotcha Fix
Flag forever Removal date in FLAGS.md
UserID empty → random Explicit anonymous bucketing
Flag check in hot loop without cache Local snapshot; remote with TTL
Testing only “on” Matrix both sides
404 vs 403 vs 503 confusion Document semantics for clients
Branching inside SQL string builders Branch in app layer
Logging PII in flag evaluation Log key + bool only
Nested flags (A requires B) Avoid; simplify combinations

Checkpoint

  • Provider interface + implementation
  • One gated path in production code
  • On/off tests
  • Default safe when unset
  • Removal plan written
  • Ops note if restart required

Commit

git add .
git commit -m "day79: feature flag gate"

Write three personal gotchas before continuing.


Tomorrow

Day 80 — Load test: generate traffic, read metrics/profiles, and find a bottleneck with numbers.