Day 27 — Config, Graceful Shutdown & Feature Flags
Day 27 — Config, Graceful Shutdown & Feature Flags
Day 77 — Config layering
Stage VIII · ~3h
Goal: Implement a layered configuration loader—defaults → file → environment → flags—with clear precedence, typed config structs, validation, and no secret-in-repo footguns.
Only the composition root (and a dedicated config package) should read env/flags. Lower layers receive values.
Why this day exists
Production processes need configuration. Ad-hoc os.Getenv sprinkled across packages causes:
- Undocumented knobs
- Different defaults in tests vs prod
- Secrets committed to git
- Flags that fight env vars
12-factor-ish discipline: config from environment (and files for local), validated at startup, injected into the composition root. Capstone days 83–88 will copy this pattern.
Theory 1 — Precedence model
Recommended order (later wins):
1. Code defaults
2. Config file (optional YAML/JSON/TOML)
3. Environment variables
4. CLI flags
Document this in README. Surprises here are outages.
type Config struct {
Addr string
DatabaseURL string
LogLevel string
ShutdownSec int
}Why flags win last
Local overrides for debugging (--addr :9090) should beat shell env without editing files. Deployments usually set env and pass no flags.
Theory 2 — Stdlib-first loading
func Load(args []string) (Config, error) {
cfg := Config{
Addr: ":8080",
LogLevel: "info",
ShutdownSec: 15,
}
// optional file
if path := os.Getenv("CONFIG_FILE"); path != "" {
if err := loadFile(path, &cfg); err != nil {
return Config{}, err
}
}
if v := os.Getenv("ADDR"); v != "" {
cfg.Addr = v
}
if v := os.Getenv("DATABASE_URL"); v != "" {
cfg.DatabaseURL = v
}
if v := os.Getenv("LOG_LEVEL"); v != "" {
cfg.LogLevel = v
}
if v := os.Getenv("SHUTDOWN_SEC"); v != "" {
n, err := strconv.Atoi(v)
if err != nil {
return Config{}, fmt.Errorf("SHUTDOWN_SEC: %w", err)
}
cfg.ShutdownSec = n
}
fs := flag.NewFlagSet("service", flag.ContinueOnError)
fs.SetOutput(io.Discard) // or os.Stderr for user CLIs
addr := fs.String("addr", cfg.Addr, "listen address")
db := fs.String("database-url", cfg.DatabaseURL, "database URL")
logLevel := fs.String("log-level", cfg.LogLevel, "log level")
if err := fs.Parse(args); err != nil {
return Config{}, err
}
cfg.Addr = *addr
cfg.DatabaseURL = *db
cfg.LogLevel = *logLevel
if err := cfg.Validate(); err != nil {
return Config{}, err
}
return cfg, nil
}
func (c Config) Validate() error {
if c.DatabaseURL == "" {
return errors.New("DATABASE_URL required")
}
if c.ShutdownSec < 1 {
return errors.New("shutdown sec must be >= 1")
}
switch strings.ToLower(c.LogLevel) {
case "debug", "info", "warn", "error":
default:
return fmt.Errorf("invalid log level %q", c.LogLevel)
}
return nil
}File format
JSON is enough without deps:
func loadFile(path string, cfg *Config) error {
b, err := os.ReadFile(path)
if err != nil {
return err
}
return json.Unmarshal(b, cfg)
}{
"addr": ":8080",
"log_level": "debug",
"shutdown_sec": 20
}YAML needs a library—fine if you already use one. Prefer one format per project.
Theory 3 — Secrets
| Do | Don’t |
|---|---|
| Env / secret manager | Commit .env with real secrets |
| Document var names | Log full DATABASE_URL |
Separate Config vs Secrets if helpful |
Put tokens in config files in images |
// redact when logging
func (c Config) LogValue() slog.Value {
return slog.GroupValue(
slog.String("addr", c.Addr),
slog.String("log_level", c.LogLevel),
slog.Int("shutdown_sec", c.ShutdownSec),
slog.String("database_url", redactDSN(c.DatabaseURL)),
)
}
func redactDSN(dsn string) string {
// naive: hide password between : and @
// improve for your DSN shape
if i := strings.Index(dsn, "://"); i >= 0 {
rest := dsn[i+3:]
if at := strings.Index(rest, "@"); at >= 0 {
return dsn[:i+3] + "***@" + rest[at+1:]
}
}
return "***"
}.env for local dev is OK if gitignored; provide .env.example with empty values.
# .env.example
DATABASE_URL=
ADDR=:8080
LOG_LEVEL=info
SHUTDOWN_SEC=15
CONFIG_FILE=
Theory 4 — Injection at the root
func main() {
cfg, err := config.Load(os.Args[1:])
if err != nil {
fmt.Fprintf(os.Stderr, "config: %v\n", err)
os.Exit(2)
}
logger := newLogger(cfg.LogLevel)
logger.Info("starting", "cfg", cfg) // uses LogValue if slog.LogValuer
db := mustOpen(cfg.DatabaseURL)
// wire adapters with cfg fields — packages don't call os.Getenv
_ = db
}Rule: only config package (and main) reads env/flags. Lower layers receive values.
Why exit code 2
Convention: 2 for misuse/config; 1 for runtime failure. CLIs (Day 71) already care—services should too.
Theory 5 — Testing config
func TestLoadDefaults(t *testing.T) {
t.Setenv("DATABASE_URL", "sqlite://file.db")
t.Setenv("ADDR", "") // ensure not leaking from parent env if needed
cfg, err := Load(nil)
if err != nil {
t.Fatal(err)
}
if cfg.Addr != ":8080" {
t.Fatalf("addr %s", cfg.Addr)
}
}
func TestFlagBeatsEnv(t *testing.T) {
t.Setenv("DATABASE_URL", "sqlite://file.db")
t.Setenv("ADDR", ":9090")
cfg, err := Load([]string{"-addr", ":9091"})
if err != nil {
t.Fatal(err)
}
if cfg.Addr != ":9091" {
t.Fatalf("got %s", cfg.Addr)
}
}
func TestValidateMissingDB(t *testing.T) {
t.Setenv("DATABASE_URL", "")
_, err := Load(nil)
if err == nil {
t.Fatal("expected error")
}
}t.Setenv keeps tests hermetic (Go 1.17+).
Theory 6 — K8s / Docker alignment
| Surface | Maps to |
|---|---|
container env: |
process environment |
| ConfigMap file mount | CONFIG_FILE |
| Secret mount | env or file; never image layers |
| CLI debug | flags on command: |
Document one canonical name per knob (DATABASE_URL not sometimes DB_URL).
GOMEMLIMIT is process env that Go runtime reads—document alongside app config even if you do not parse it in Go.
Worked example — layered demo
export DATABASE_URL=postgres://localhost/app
export ADDR=:9090
go run ./cmd/service --addr :9091
# effective ADDR :9091 (flag wins)Document matrix in CONFIG.md:
| Key | Default | Env | Flag |
|---|---|---|---|
| addr | :8080 |
ADDR |
-addr |
| database | (none) | DATABASE_URL |
-database-url |
| log level | info |
LOG_LEVEL |
-log-level |
| shutdown | 15 |
SHUTDOWN_SEC |
-shutdown-sec |
Worked example — empty env must not clobber
// wrong: always assign
cfg.Addr = os.Getenv("ADDR") // empty string wipes file/default
// right: only override when set
if v := os.Getenv("ADDR"); v != "" {
cfg.Addr = v
}Lab
Suggested workspace: service composition root.
- Central
internal/config(orconfig) package.
- Precedence: defaults → file (optional) → env → flags.
Validate()fails fast on missing required values.
.env.example+ README table.
- Unit tests for precedence and validation errors.
- Redacted logging of secrets.
Stretch
GOMEMLIMITdocumented alongside app config.
- Feature flag file path field prepared for Day 79.
String()method that never prints secrets (for fmt debugging).
Common gotchas
| Gotcha | Fix |
|---|---|
| Env read deep in store package | Inject DSN from main |
| Empty string overwriting file values | Only override when env set |
flag.Parse global in library |
FlagSet in Load |
| Secrets in logs | Redact |
| Different names in K8s vs local | One canonical env name |
| Panic on bad config | Return error; main exits 2 |
| Bool flags defaulting wrong | Document true/false strings |
| Required vs optional confusion | Validate() is the source of truth |
Checkpoint
- Typed
Config+Load
- Documented precedence
- Validation at startup
- Tests for env/flag behavior
- Example env file without secrets
- Redaction on log path
Commit
git add .
git commit -m "day77: config layering flags env file"Write three personal gotchas before continuing.
Tomorrow
Day 78 — Graceful shutdown & health: production process lifecycle—SIGTERM, drain, liveness vs readiness.
Day 78 — Graceful shutdown & health
Stage VIII · ~3h
Goal: Implement signal-aware graceful shutdown for an HTTP service, distinguish liveness vs readiness, and drain in-flight requests within a configurable timeout.
Containers and orchestrators will send SIGTERM. Ignoring it is a production bug, not a style choice.
Why this day exists
Typical stop sequence:
- SIGTERM
- grace period
- SIGKILL
If you ignore SIGTERM, users see dropped connections on every deploy. Health probes misconfigured cause restart storms. This day is mandatory production literacy—wired on Day 68 images that run your binary as PID 1.
Theory 1 — Process lifecycle
start → load config → open deps → serve
↑
SIGTERM → stop readiness → Server.Shutdown → close deps → exit
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()signal.NotifyContext cancels when a signal arrives—clean integration with Shutdown. Prefer it over manual signal.Notify channels unless you need custom multiplexing.
Why both Interrupt and SIGTERM
| Signal | Source |
|---|---|
SIGINT |
Ctrl-C in terminal |
SIGTERM |
docker stop, Kubernetes, systemd |
Handle both so laptop and cluster behave the same.
Theory 2 — http.Server.Shutdown
srv := &http.Server{
Addr: cfg.Addr,
Handler: mux,
ReadHeaderTimeout: 5 * time.Second,
}
go func() {
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Error("serve", "err", err)
// coordinate fatal via err channel in real main
}
}()
<-ctx.Done()
log.Info("shutdown started")
shCtx, cancel := context.WithTimeout(context.Background(), time.Duration(cfg.ShutdownSec)*time.Second)
defer cancel()
if err := srv.Shutdown(shCtx); err != nil {
log.Error("shutdown", "err", err)
_ = srv.Close()
}
// close DB pool, flush metrics, otel Shutdown, etc.| Call | Behavior |
|---|---|
Shutdown |
Stop accept; wait in-flight up to ctx |
Close |
Immediate; use if Shutdown times out |
Ordering of teardown
- Mark not ready
- Optional short sleep for LB de-register
Server.Shutdown
- Cancel worker contexts
- Close DB / message consumers
- OTel
TracerProvider.Shutdown/ flush metrics if needed
- Exit 0
Closing the DB before HTTP drain causes handler errors mid-flight.
Theory 3 — Liveness vs readiness
| Probe | Question | Should fail when |
|---|---|---|
Liveness /healthz |
Is process deadlocked / stuck? | Only if process is beyond recovery |
Readiness /readyz |
Can it serve traffic? | DB down, warming, shutting down |
Classic mistake: liveness fails when DB blips → Kubernetes restarts healthy pods that were temporarily unready → worse outage.
type Health struct {
ready atomic.Bool
}
func (h *Health) Live(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ok"))
}
func (h *Health) Ready(w http.ResponseWriter, r *http.Request) {
if !h.ready.Load() {
http.Error(w, "not ready", http.StatusServiceUnavailable)
return
}
// optional: ping DB with short timeout
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ready"))
}h.ready.Store(true) // after deps open
// on signal:
h.ready.Store(false) // fail readiness first
time.Sleep(1 * time.Second) // optional: let LB de-register
srv.Shutdown(...)Optional DB ping on ready
func (h *Health) Ready(w http.ResponseWriter, r *http.Request) {
if !h.ready.Load() {
http.Error(w, "not ready", http.StatusServiceUnavailable)
return
}
ctx, cancel := context.WithTimeout(r.Context(), 300*time.Millisecond)
defer cancel()
if err := h.db.PingContext(ctx); err != nil {
http.Error(w, "db", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ready"))
}Keep the timeout tiny so probes do not pile up.
Theory 4 — In-flight work and contexts
Handlers should honor r.Context() cancellation when the client goes away. For shutdown, Shutdown waits for handlers to return—long Sleep without select on ctx blocks drain.
select {
case <-r.Context().Done():
return
case <-time.After(work):
}Background workers: listen on the same root context; exit on cancel.
go func() {
for {
select {
case <-rootCtx.Done():
return
case job := <-jobs:
process(rootCtx, job)
}
}
}()Long requests vs grace period
If max handler time is 30s, ShutdownSec and Kubernetes terminationGracePeriodSeconds must be greater than that (plus de-register buffer). Otherwise SIGKILL cuts work mid-flight.
Theory 5 — Kubernetes sketch (awareness)
livenessProbe:
httpGet: { path: /healthz, port: 8080 }
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet: { path: /readyz, port: 8080 }
periodSeconds: 5
terminationGracePeriodSeconds: 30
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 5"] # if you still have a shell; distroless may skipWith distroless, prefer app-side readiness flip + terminationGracePeriodSeconds ≥ ShutdownSec.
Theory 6 — Testing shutdown
Automated tests can drive Shutdown without real signals:
func TestShutdownDrains(t *testing.T) {
srv := &http.Server{Addr: "127.0.0.1:0", Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
time.Sleep(50 * time.Millisecond)
w.WriteHeader(200)
})}
ln, err := net.Listen("tcp", srv.Addr)
if err != nil {
t.Fatal(err)
}
go srv.Serve(ln)
// fire request...
shCtx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
if err := srv.Shutdown(shCtx); err != nil {
t.Fatal(err)
}
}Manual demo remains required for SIGTERM path.
Worked example — compact main
func main() {
cfg := mustConfig()
rootCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
db := mustDB(cfg)
defer db.Close()
health := &Health{}
mux := routes(db, health)
srv := &http.Server{Addr: cfg.Addr, Handler: mux, ReadHeaderTimeout: 5 * time.Second}
go func() {
log.Printf("listen %s", cfg.Addr)
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Fatal(err)
}
}()
health.ready.Store(true)
<-rootCtx.Done()
health.ready.Store(false)
log.Printf("draining...")
shCtx, cancel := context.WithTimeout(context.Background(), time.Duration(cfg.ShutdownSec)*time.Second)
defer cancel()
if err := srv.Shutdown(shCtx); err != nil {
log.Printf("shutdown error: %v", err)
}
}Manual test:
go run .
# other terminal
curl -s localhost:8080/readyz
kill -TERM <pid>
# curl readyz should go 503; in-flight finishes; process exits 0Config keys (Day 77)
| Key | Default | Meaning |
|---|---|---|
SHUTDOWN_SEC |
15 |
Drain timeout |
ADDR |
:8080 |
Listen address |
Lab
Suggested workspace: your HTTP service.
signal.NotifyContext+Server.Shutdownwith timeout from config.
/healthzand/readyzwith readiness false during drain.
- Close DB/tracer after HTTP drain.
- Document probe semantics in README.
- Demo notes: command sequence + observed behavior.
- Prove with
kill -TERM(ordocker stopif containerized).
Stretch
- Integrate OTel
Shutdownand Prometheus gatherer flush if needed.
preStopsleep discussion for mesh/LB timing.
- Integration test that hits a slow handler during Shutdown.
Common gotchas
| Gotcha | Fix |
|---|---|
| Liveness = DB ping | Move DB to readiness |
| Shutdown timeout too short | Align with longest request + grace |
Ignoring ErrServerClosed |
Treat as success |
| Shell ENTRYPOINT eats signals | Day 68 exec-form binary |
| Workers leak after HTTP stop | Cancel shared context |
| Ready before migrations | Run migrate then ready=true |
| Closing DB before Shutdown | Reverse order |
| Logging FATAL on ErrServerClosed | Filter it |
Checkpoint
- SIGTERM triggers drain
- Shutdown timeout configurable
- Distinct live vs ready endpoints
- Readiness false on shutdown
- Deps closed after HTTP drain
- Demo steps written
Commit
git add .
git commit -m "day78: graceful shutdown health probes"Write three personal gotchas before continuing.
Tomorrow
Day 79 — Feature flags: progressive delivery awareness with one real flag gate in code.
Day 79 — Feature flags
Stage VIII · ~3h
Goal: Implement one feature flag gate (boolean or percentage) with a clean evaluation API, safe defaults, and operational notes—without building a full flag SaaS.
Deploy ≠ release. Today: one flag, well-structured, with a removal plan. Flag debt is real—treat flags as temporary code paths.
Why this day exists
Feature flags let you:
- Ship dark code and enable later
- Kill-switch a bad path quickly
- Gradual rollout (10% → 50% → 100%)
Abuse creates untested combinations and permanent #ifdef culture. The skill is a small evaluation surface that application code can depend on without knowing LaunchDarkly vs env vars.
Theory 1 — Flag types
| Kind | Example |
|---|---|
| Release toggle | New checkout flow on/off |
| Ops toggle | Disable expensive export job |
| Experiment | 5% see algorithm B |
| Permission | Entitlement (often not a “flag service”) |
Lifecycle: create → enable gradually → remove flag and dead code.
ship off → canary % → 100% → delete old path + flag
If a flag has been 100% for a sprint, schedule removal.
Theory 2 — Evaluation API
Keep evaluation boring and injectable:
package flags
import "context"
type Provider interface {
Enabled(ctx context.Context, key string, subject Subject) bool
}
type Subject struct {
UserID string
// Attrs map[string]string // stretch
}
type Static map[string]bool
func (s Static) Enabled(_ context.Context, key string, _ Subject) bool {
return s[key]
}Env-based provider (good enough for many services):
type Env struct{}
func (Env) Enabled(_ context.Context, key string, _ Subject) bool {
v := strings.ToLower(os.Getenv("FLAG_" + strings.ToUpper(key)))
return v == "1" || v == "true" || v == "on"
}Percentage rollout (stable bucketing):
func EnabledPercent(key, userID string, percent int) bool {
if percent <= 0 {
return false
}
if percent >= 100 {
return true
}
sum := sha256.Sum256([]byte(key + ":" + userID))
// use first 2 bytes → 0..65535
n := int(sum[0])<<8 | int(sum[1])
return n%100 < percent
}Stability matters: same user should not flip every request. Include flag key in the hash so different flags do not correlate perfectly.
Composed provider
type PercentEnv struct {
Percent map[string]int // key → 0..100
}
func (p PercentEnv) Enabled(_ context.Context, key string, sub Subject) bool {
pct, ok := p.Percent[key]
if !ok {
// fall back to boolean env
return Env{}.Enabled(context.Background(), key, sub)
}
return EnabledPercent(key, sub.UserID, pct)
}Theory 3 — Where to branch
Prefer branching in application layer, not deep in SQL drivers:
if s.flags.Enabled(ctx, "new_search", flags.Subject{UserID: userID}) {
return s.searchV2(ctx, q)
}
return s.searchV1(ctx, q)HTTP adapter can pass subject from auth middleware.
Defaults
- Fail closed for risky features (
falseif provider errors).
- Fail open only for non-critical cosmetics—document choice.
func (s *Service) Enabled(ctx context.Context, key string, sub flags.Subject) bool {
// if remote provider added later:
// on timeout → return false for release toggles
return s.flags.Enabled(ctx, key, sub)
}Theory 4 — Config integration (Day 77)
type Config struct {
// ...
FlagNewSearch bool
FlagNewSearchPercent int
}Or a JSON file:
{
"flags": {
"new_search": { "enabled": true, "percent": 10 }
}
}Do not require a SaaS vendor for the lab. Awareness of LaunchDarkly/Unleash/OpenFeature is enough:
| Concept | Note |
|---|---|
| OpenFeature | Vendor-neutral API trend |
| Remote config | Needs caching + timeouts |
| Audit log | Who flipped what |
Wire at composition root
fp := flags.Static{
"new_search": cfg.FlagNewSearch,
}
// or flags.Env{} for ops kill switches without redeploy of binaries
// (still needs process restart unless you poll a file)
svc := app.New(deps, fp)Theory 5 — Testing matrix
Flags double paths—test both:
func TestSearchFlag(t *testing.T) {
svc := NewService(deps, flags.Static{"new_search": true})
// assert v2 behavior
got, err := svc.Search(context.Background(), "q")
if err != nil || !got.FromV2 {
t.Fatalf("v2: %#v %v", got, err)
}
svc2 := NewService(deps, flags.Static{"new_search": false})
got2, err := svc2.Search(context.Background(), "q")
if err != nil || got2.FromV2 {
t.Fatalf("v1: %#v %v", got2, err)
}
}Percentage stability test
func TestPercentStable(t *testing.T) {
a := EnabledPercent("exp", "user-1", 30)
b := EnabledPercent("exp", "user-1", 30)
if a != b {
t.Fatal("unstable")
}
}Theory 6 — Ops and observability
Expose evaluation for operators carefully:
// internal admin only — auth required
mux.HandleFunc("GET /internal/flags", func(w http.ResponseWriter, r *http.Request) {
// list configured keys + defaults — not every subject evaluation
})Metrics (stretch):
flag_evaluation_total{key="new_search",result="true"} 120
flag_evaluation_total{key="new_search",result="false"} 880
Avoid per-user labels.
Kill switch runbook
# env-driven: restart with flag off
FLAG_REPORTS=false systemctl restart myservice
# or kubernetes set env + rollout restartDocument whether a restart is required. File-based flags can be re-read on interval—only if you implement it; do not pretend env vars hot-reload.
Worked example — kill switch for expensive endpoint
mux.HandleFunc("GET /v1/report", func(w http.ResponseWriter, r *http.Request) {
if !fp.Enabled(r.Context(), "reports", subjectFrom(r)) {
// 404 hides existence; 503 signals temporary — pick one and document
http.Error(w, "reports disabled", http.StatusServiceUnavailable)
return
}
writeReport(w, r)
})FLAGS.md template
# Flags
| Key | Default | Owner | Type | Removal criteria |
|-----|---------|-------|------|------------------|
| reports | false | you | ops kill | N/A (permanent ops) or date |
| new_search | false | you | release | After 100% for 14 days + delete v1 |Lab
Suggested workspace: your service.
- Define
flags.Providerinterface.
- Implement static + env (or file) provider.
- Gate one real path (feature or kill switch).
- Tests for on and off.
FLAGS.md: key name, default, owner, removal criteria.
- Default is safe when unset (usually
falsefor new features).
Stretch
- Percentage rollout with stable hashing.
- Expose active flags on an internal admin endpoint (auth required).
- Metric
flag_evaluation_total{key,result}.
- File provider with mtime reload every 30s (document races).
Common gotchas
| Gotcha | Fix |
|---|---|
| Flag forever | Removal date in FLAGS.md |
| UserID empty → random | Explicit anonymous bucketing |
| Flag check in hot loop without cache | Local snapshot; remote with TTL |
| Testing only “on” | Matrix both sides |
| 404 vs 403 vs 503 confusion | Document semantics for clients |
| Branching inside SQL string builders | Branch in app layer |
| Logging PII in flag evaluation | Log key + bool only |
| Nested flags (A requires B) | Avoid; simplify combinations |
Checkpoint
- Provider interface + implementation
- One gated path in production code
- On/off tests
- Default safe when unset
- Removal plan written
- Ops note if restart required
Commit
git add .
git commit -m "day79: feature flag gate"Write three personal gotchas before continuing.
Tomorrow
Day 80 — Load test: generate traffic, read metrics/profiles, and find a bottleneck with numbers.