Day 25 — Observability, CLI & Security Scanning

Updated

July 30, 2025

Day 25 — Observability, CLI & Security Scanning

Day 69 — Prometheus metrics

Stage VII · ~3h
Goal: Instrument a Go service with the Prometheus client: process metrics, at least one business/request metric, a /metrics scrape endpoint, and a short note on cardinality discipline.

Why this day exists

Logs tell stories. Traces follow one request. Metrics answer:

How many requests? Error rate? Latency shape?
Is the process thrashing GC or goroutines?
Did the last deploy make p99 worse?

Prometheus is the de facto scrape model in cloud-native Go shops. You need enough instrumentation for Stage VIII load tests and the capstone—not a full SRE platform.

Theory 1 — Metric types

Type	Use for
Counter	Monotonic events (`requests_total`)
Gauge	Instant value (`queue_depth`)
Histogram	Distribution (`request_duration_seconds`)
Summary	Client-side quantiles (less common in new code; prefer histogram + PromQL)

Golden signals for HTTP services (RED):

Rate — http_requests_total
Errors — same counter with code label or separate error counter
Duration — histogram of latency

USE (utilization, saturation, errors) is the sibling model for resources—goroutines, heap, queue depth as gauges.

Theory 2 — Client library basics

Module: github.com/prometheus/client_golang

import (
    "net/http"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    httpRequests = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total HTTP requests",
        },
        []string{"method", "path", "code"},
    )
    httpDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "HTTP request latency",
            Buckets: prometheus.DefBuckets, // or custom
        },
        []string{"method", "path"},
    )
)

Middleware sketch (Go 1.22+ ServeMux patterns—normalize path labels carefully):

func instrument(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        ww := &statusWriter{ResponseWriter: w, code: 200}
        next.ServeHTTP(ww, r)
        path := r.Pattern // registered pattern when available
        if path == "" {
            path = "unknown"
        }
        httpRequests.WithLabelValues(r.Method, path, strconv.Itoa(ww.code)).Inc()
        httpDuration.WithLabelValues(r.Method, path).Observe(time.Since(start).Seconds())
    })
}

mux := http.NewServeMux()
mux.Handle("GET /metrics", promhttp.Handler())
mux.Handle("GET /v1/hello", instrument(http.HandlerFunc(hello)))

Warning

Cardinality bomb: never put unbounded values in labels (user_id, raw URL, email). Prefer route patterns (/v1/users/{id}) not concrete IDs.

Theory 3 — Default Go collector

promhttp.Handler() registers process/Go collectors by default when using the default registry:

go_goroutines
go_memstats_*
process_cpu_seconds_total
GC-related series

That alone is valuable for Day 67 insights in a long-running process. Hit /metrics after load and find go_memstats_heap_alloc_bytes.

Theory 4 — Scraping locally

curl -s localhost:8080/metrics | head
curl -s localhost:8080/metrics | grep http_requests_total

Optional dockerized Prometheus config (stretch):

scrape_configs:
  - job_name: day69
    scrape_interval: 5s
    static_configs:
      - targets: ["host.docker.internal:8080"]

You do not need Grafana today—raw /metrics text is enough.

Naming conventions

Suffix	Meaning
`_total`	Counter
`_seconds`	Time unit base
`_bytes`	Size unit base
`_ratio`	0–1 gauge (when used)

Theory 5 — Histograms and buckets

Default buckets are a starting point—not magic. For APIs aiming at sub-100ms:

Buckets: []float64{.001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5}

Too many buckets × too many label combos = expensive cardinality. Start narrow.

PromQL awareness (read-only for lab):

rate(http_requests_total[5m])
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

You need not run Prometheus server to understand the shape.

Theory 6 — Testing metrics

func TestMetricsInc(t *testing.T) {
    reg := prometheus.NewRegistry()
    c := prometheus.NewCounter(prometheus.CounterOpts{Name: "demo_total", Help: "x"})
    reg.MustRegister(c)
    c.Inc()
    mfs, err := reg.Gather()
    if err != nil {
        t.Fatal(err)
    }
    // assert metric family present
    _ = mfs
}

Avoid promauto global registration fighting tests—use a private registry in unit tests when you assert series.

Worked example — complete mini service

package main

import (
    "log"
    "net/http"
    "strconv"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    reqs = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "demo_requests_total",
        Help: "Demo requests",
    }, []string{"route", "code"})
    dur = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "demo_request_duration_seconds",
        Help:    "Demo latency",
        Buckets: []float64{.001, .005, .01, .025, .05, .1, .25, .5, 1},
    }, []string{"route"})
)

type statusWriter struct {
    http.ResponseWriter
    code int
}

func (w *statusWriter) WriteHeader(code int) {
    w.code = code
    w.ResponseWriter.WriteHeader(code)
}

func withMetrics(route string, next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        sw := &statusWriter{ResponseWriter: w, code: http.StatusOK}
        next(sw, r)
        reqs.WithLabelValues(route, strconv.Itoa(sw.code)).Inc()
        dur.WithLabelValues(route).Observe(time.Since(start).Seconds())
    }
}

func main() {
    mux := http.NewServeMux()
    mux.Handle("GET /metrics", promhttp.Handler())
    mux.HandleFunc("GET /healthz", withMetrics("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
        _, _ = w.Write([]byte("ok"))
    }))
    mux.HandleFunc("GET /v1/work", withMetrics("/v1/work", func(w http.ResponseWriter, r *http.Request) {
        time.Sleep(15 * time.Millisecond) // pretend work
        w.WriteHeader(http.StatusOK)
        _, _ = w.Write([]byte(`{"ok":true}`))
    }))
    log.Fatal(http.ListenAndServe(":8080", mux))
}

go get github.com/prometheus/client_golang@latest
go run .
# other terminal
curl -s localhost:8080/v1/work
curl -s localhost:8080/metrics | grep demo_

Worked example — label policy blurb

Paste into METRICS.md:

## Labels we allow
- method: GET|POST|...
- path: registered pattern only (e.g. /v1/items/{id})
- code: HTTP status as string

## Labels we refuse
- user_id, email, raw URL, request_id (use logs/traces)
- free-text error messages

## Series owned by this service
- demo_requests_total{route,code}
- demo_request_duration_seconds{route}
- plus default go_* / process_* from client_golang

That document is the review artifact—not a second monitoring platform.

Lab

Suggested workspace: ~/lab/90daysofx/01-go/day69

Add Prometheus client to your service (or the demo above).
Export /metrics.
At least one Counter and one Histogram (or latency summary) on a real route.
Hit the route several times; show series increasing in curl output (paste into METRICS.md).
Document label policy: which labels exist and which you refuse.
Confirm go_goroutines appears (default registry).

Stretch

Custom buckets matching your SLA (e.g. 50/100/200ms).
Counter for a domain event (orders_created_total) separate from HTTP.
Protect /metrics with network policy notes (not public internet).
Wire middleware into the Day 50/53 service instead of the demo main.

Common gotchas

Gotcha	Fix
High-cardinality labels	Route templates only
Histogram buckets useless	Tune to expected latency
Metrics on public internet unauthenticated	Network policy; sometimes separate port
Double-register metrics in tests	Use a fresh registry in unit tests
Ignoring units	`_seconds`, `_bytes`, `_total` suffixes
Measuring only `/metrics`	Instrument business paths
Labeling with raw `r.URL.Path`	Use `r.Pattern` or a fixed route name

Checkpoint

/metrics scrapes successfully
Counter + histogram (or gauge with clear story)
Label cardinality policy written
Sample scrape snippet in METRICS.md
Go runtime metrics visible
Know RED signals for HTTP

Commit

git add .
git commit -m "day69: prometheus metrics endpoint"

Write three personal gotchas before continuing.

Tomorrow

Day 70 — OpenTelemetry light: trace one request path end-to-end so latency has a story beyond histograms.

Day 70 — OpenTelemetry light

Stage VII · ~3h
Goal: Add a minimal OpenTelemetry trace path to one request: tracer provider, a root span + child span, context propagation, and export (stdout is enough). Understand how traces complement metrics and logs.

Note

Full mesh platforms are out of scope. Today is one path, correctly instrumented—not “adopt the whole observability platform.”

Why this day exists

Histograms say “p99 is 200ms.” Traces answer why this request was 200ms:

Time in handler vs DB vs outbound HTTP
Error attachment on a span
Correlation across services via W3C traceparent

Day 69 gave Prometheus metrics (aggregates). Day 47 gave structured logs. Traces complete the triangle for individual slow or failing requests.

Theory 1 — Three pillars (and boundaries)

Signal	Strength
Logs (`slog`)	Discrete events, cheap, high detail
Metrics	Aggregates, alerts, cheap cardinality if careful
Traces	Causal graph of a request

OpenTelemetry (OTel) standardizes APIs and exporters so you are not locked to one vendor SDK forever.

Core objects

TracerProvider — process-wide factory
Tracer — named instrumentation library
Span — unit of work with start/end, attributes, status
Context — carries span; must thread through context.Context

Request ctx
  └─ span: GET /v1/hello
       └─ span: doWork / db.query

Theory 2 — Minimal Go wiring (stdout exporter)

Modules evolve; pin versions that build on Go 1.26 when you run the lab. Pattern:

package main

import (
    "context"
    "log"
    "net/http"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func initTracer(ctx context.Context) (func(context.Context) error, error) {
    exp, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
    if err != nil {
        return nil, err
    }
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("day70-demo"),
        )),
    )
    otel.SetTracerProvider(tp)
    return tp.Shutdown, nil
}

func main() {
    ctx := context.Background()
    shutdown, err := initTracer(ctx)
    if err != nil {
        log.Fatal(err)
    }
    defer func() { _ = shutdown(context.Background()) }()

    mux := http.NewServeMux()
    mux.HandleFunc("GET /v1/hello", handleHello)
    log.Fatal(http.ListenAndServe(":8080", mux))
}

func handleHello(w http.ResponseWriter, r *http.Request) {
    ctx, span := otel.Tracer("day70/http").Start(r.Context(), "GET /v1/hello")
    defer span.End()

    if err := doWork(ctx); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        http.Error(w, "failed", 500)
        return
    }
    span.SetAttributes(attribute.String("handler.version", "v1"))
    w.WriteHeader(200)
    _, _ = w.Write([]byte(`{"ok":true}`))
}

func doWork(ctx context.Context) error {
    ctx, span := otel.Tracer("day70/http").Start(ctx, "doWork")
    defer span.End()
    // honor cancellation
    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-time.After(20 * time.Millisecond):
    }
    span.SetAttributes(attribute.Int("work.ms", 20))
    return nil
}

go get go.opentelemetry.io/otel@latest \
  go.opentelemetry.io/otel/sdk@latest \
  go.opentelemetry.io/otel/exporters/stdout/stdouttrace@latest
go run .
curl -s localhost:8080/v1/hello
# spans print to process stdout

Pinning note

In notes, record the module versions go.mod resolved. OTel packages move; your lab should still build on Go 1.26.

Theory 3 — Context is the bus

Always pass context.Context into downstream work:

// good
rows, err := db.QueryContext(ctx, q)

// bad — orphan child, broken cancellation
rows, err := db.Query(q)

HTTP clients should inject headers for outbound calls (contrib middleware / otelhttp). For one-process labs, parent→child spans via Start(ctx, name) is the essential skill.

Sampling (awareness)

Production systems sample (e.g. parent-based, ratio-based). Stdout labs can use default always-on for the demo. Never sample in a way that drops all errors if your platform supports tail sampling—out of scope, but know sampling exists.

// awareness only — not required today
// sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)))

Theory 4 — Correlation with logs and metrics

Put trace_id into slog attributes when a span is present (optional stretch).
Metrics remain for aggregates; traces for outliers.
Do not replace metrics with traces for alerting volume.

import "go.opentelemetry.io/otel/trace"

spanCtx := trace.SpanContextFromContext(ctx)
if spanCtx.IsValid() {
    logger.Info("handled",
        "trace_id", spanCtx.TraceID().String(),
        "span_id", spanCtx.SpanID().String(),
    )
}

Attribute discipline

Good attribute	Bad attribute
`http.route` = `/v1/users/{id}`	Full URL with user ids
`db.system` = `sqlite`	Entire SQL with PII
`handler.version` = `v2`	Unbounded free-text blobs

High-cardinality attributes explode backend cost the same way metric labels do.

Theory 5 — Auto vs manual instrumentation

Approach	Trade-off
Manual spans	Precise names; control attributes
`otelhttp` middleware	Fast coverage of HTTP edges
Vendor agents	Convenient; less portable

For this day: manual root + one child is mandatory. Middleware is optional polish.

// stretch sketch
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

mux := http.NewServeMux()
mux.Handle("GET /v1/hello", otelhttp.NewHandler(http.HandlerFunc(handleHello), "GET /v1/hello"))

Theory 6 — Errors, status, and shutdown

if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
    return err
}
// success: default status OK; optional:
// span.SetStatus(codes.Ok, "")

Process exit without Shutdown loses batched spans:

defer func() {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    _ = shutdown(ctx)
}()

Align with Day 78 graceful shutdown: flush tracer after HTTP drain.

Worked example — span tree narrative

Paste into TRACE.md something like:

Span: GET /v1/hello          25ms
  attributes: handler.version=v1
  Span: doWork               20ms
    attributes: work.ms=20

If stdout is JSON lines, keep a short excerpt—not megabytes of logs.

Worked example — outbound HTTP child (optional)

func fetchUpstream(ctx context.Context, client *http.Client, url string) error {
    ctx, span := otel.Tracer("day70/http").Start(ctx, "HTTP GET upstream")
    defer span.End()
    span.SetAttributes(attribute.String("peer.service", "upstream"))

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return err
    }
    resp, err := client.Do(req)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return err
    }
    defer resp.Body.Close()
    span.SetAttributes(attribute.Int("http.status_code", resp.StatusCode))
    return nil
}

Without otelhttp transport, you still get a local child span; cross-process traceparent needs proper propagation.

Lab

Suggested workspace: ~/lab/90daysofx/01-go/day70

Wire TracerProvider with stdout exporter (or OTLP if you already run a collector).
Instrument one route: root span + child span for a dependency call (DB, HTTP, or sleep-as-work).
Capture a sample span tree in TRACE.md (paste stdout JSON/pretty).
Call Shutdown on process exit (defer with timeout).
Note one attribute you added and why it is not high-cardinality.
Explain in two sentences (in TRACE.md) how this differs from Day 69 metrics.

Stretch

Use go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.
Propagate to a second tiny process with W3C headers.
Export OTLP to a local collector/Jaeger all-in-one.
Add trace_id to one slog line.

Common gotchas

Gotcha	Fix
Forgetting `span.End()`	`defer span.End()` immediately after Start
New context without parent	Always `Start(ctx, ...)` from request ctx
Export lost on exit	`Shutdown` flushes batcher
Span name = full URL with IDs	Use route pattern
“OTel is too heavy”	One tracer + stdout is enough for learning
Mixing metrics SDK complexity	Keep Day 69 Prometheus; add OTel traces only
Panic before End	Still prefer defer End; recover separately (Day 14)
Global provider race in tests	Set provider in TestMain or inject Tracer

Checkpoint

TracerProvider initialized and shut down
Root + child span on one path
Context threaded into child work
Example output saved in TRACE.md
Can explain metrics vs traces in two sentences
No high-cardinality attributes on the demo path

Commit

git add .
git commit -m "day70: otel light trace path"

Write three personal gotchas before continuing.

Tomorrow

Day 71 — CLI ergonomics: multi-command UX, flags, exit codes, and --help that respect users (and future you).

Day 71 — CLI ergonomics

Stage VII · ~3h
Goal: Ship a multi-command CLI (stdlib flag + subcommands or Cobra) with solid --help, exit codes, version, and stderr/stdout discipline—usable as capstone option C scaffolding.

Why this day exists

Many Go programs are CLIs first: devops tools, agents, migration runners, kubectl-style operators. Bad CLIs share traits:

Usage only discoverable by reading source
Errors printed to stdout
Exit code 0 on failure
Flags that collide with subcommands

Capstone option C needs this day; options A/B still need admin CLIs and cmd/ entrypoints.

Theory 1 — CLI UX contract

Rule	Practice
Help is free	`-h` / `--help` prints clear usage
Stdout vs stderr	Data → stdout; diagnostics → stderr
Exit codes	`0` ok; `2` usage; `1` runtime failure (common convention)
Version	`--version` or `version` subcommand
Non-interactive friendly	Flags over prompts for automation
Idempotent where claimed	Document side effects

if err != nil {
    fmt.Fprintf(os.Stderr, "error: %v\n", err)
    os.Exit(1)
}

Pipelines depend on this: day71 hash *.go | while read ... must not interleave errors into the hash stream.

Theory 2 — Stdlib approach: subcommands with `flag`

package main

import (
    "flag"
    "fmt"
    "os"
)

var version = "dev" // override with -ldflags

func main() {
    if len(os.Args) < 2 {
        usage()
        os.Exit(2)
    }
    switch os.Args[1] {
    case "hello":
        helloCmd(os.Args[2:])
    case "version":
        fmt.Println(version)
    case "-h", "--help", "help":
        usage()
    default:
        fmt.Fprintf(os.Stderr, "unknown command: %s\n", os.Args[1])
        usage()
        os.Exit(2)
    }
}

func usage() {
    fmt.Fprintf(os.Stderr, `usage: day71 <command> [flags]

commands:
  hello    greet someone
  hash     sha256 a file
  version  print version
`)
}

func helloCmd(args []string) {
    fs := flag.NewFlagSet("hello", flag.ExitOnError)
    name := fs.String("name", "world", "name to greet")
    fs.SetOutput(os.Stderr)
    _ = fs.Parse(args)
    fmt.Printf("hello, %s\n", *name)
}

Why flag.NewFlagSet: each subcommand owns flags; global flag.Parse fights subcommands.

Exit-on-error vs continue

flag.ExitOnError exits on bad flags (usage). For library-style parsing in tests, use flag.ContinueOnError and return errors from run(args []string) error.

Theory 3 — Cobra (optional, common in industry)

go get github.com/spf13/cobra@latest

var rootCmd = &cobra.Command{
    Use:   "day71",
    Short: "Day 71 demo CLI",
}

var helloCmd = &cobra.Command{
    Use:   "hello",
    Short: "Greet",
    RunE: func(cmd *cobra.Command, args []string) error {
        name, _ := cmd.Flags().GetString("name")
        fmt.Printf("hello, %s\n", name)
        return nil
    },
}

func init() {
    helloCmd.Flags().String("name", "world", "name to greet")
    rootCmd.AddCommand(helloCmd)
}

func main() {
    if err := rootCmd.Execute(); err != nil {
        os.Exit(1)
    }
}

Choose stdlib when	Choose Cobra when
1–3 commands, learning	Many commands, nested help, ecosystem (viper)
Zero deps	Team already uses Cobra

Both are valid for the lab. Prefer one well-finished tool over half of each.

Theory 4 — Structure for real tools

cmd/day71/main.go      # thin
internal/cli/root.go   # command wiring
internal/hello/        # business logic (testable without CLI)

// business logic pure
func Greet(name string) string {
    if name == "" {
        name = "world"
    }
    return "hello, " + name
}

// test without spawning process
func TestGreet(t *testing.T) {
    if Greet("a") != "hello, a" {
        t.Fatal()
    }
}

Config flags vs env (preview Day 77)

CLI flags win for one-shot tools; long-running services lean env/files. Same binary can support both later (--config + env override chain).

Completion and man pages (awareness)

Cobra generates shell completion; nice for option C stretch—not required today.

Theory 5 — Version injection

go build -ldflags "-X main.version=$(git describe --tags --always)" -o day71 .
./day71 version

// main.go
var version = "dev"

Ship version in --help footer or version subcommand so support tickets can cite a build.

Theory 6 — Testing CLIs

Prefer pure functions for logic. Optional process-level test:

func TestCLIHello(t *testing.T) {
    cmd := exec.Command("go", "run", ".", "hello", "--name", "Ada")
    cmd.Dir = // module root
    out, err := cmd.CombinedOutput()
    if err != nil {
        t.Fatalf("%v: %s", err, out)
    }
    if !strings.Contains(string(out), "hello, Ada") {
        t.Fatal(string(out))
    }
}

Slow; use sparingly. Better: run(args []string) error tested directly with buffer writers.

func run(args []string, stdout, stderr io.Writer) int {
    // return exit code; main calls os.Exit(run(...))
}

Worked example — useful mini tool

day71 hash — print sha256 of a file (stdlib only):

func hashCmd(args []string) {
    fs := flag.NewFlagSet("hash", flag.ExitOnError)
    fs.SetOutput(os.Stderr)
    _ = fs.Parse(args)
    if fs.NArg() != 1 {
        fmt.Fprintln(os.Stderr, "usage: day71 hash <file>")
        os.Exit(2)
    }
    path := fs.Arg(0)
    f, err := os.Open(path)
    if err != nil {
        fmt.Fprintf(os.Stderr, "error: %v\n", err)
        os.Exit(1)
    }
    defer f.Close()
    h := sha256.New()
    if _, err := io.Copy(h, f); err != nil {
        fmt.Fprintf(os.Stderr, "error: %v\n", err)
        os.Exit(1)
    }
    fmt.Printf("%x  %s\n", h.Sum(nil), path)
}

Cross-compile the CLI with Day 63 scripts for release artifacts (option C path).

Lab

Suggested workspace: ~/lab/90daysofx/01-go/day71

Build a CLI with ≥2 commands (e.g. hello, hash/version).
Correct exit codes for usage vs runtime errors.
--help / command help on stderr or Cobra default, but usable.
Unit-test pure logic without exec.Command (optional process test stretch).
Embed version via -ldflags from Day 63.

Acceptance demo

go build -ldflags "-X main.version=lab" -o day71 .
./day71 --help || ./day71 help
./day71 hello --name Ada
./day71 version
./day71 hash README.md
./day71 nope ; echo exit:$?   # expect non-zero
./day71 hash /no/such 2>/tmp/err ; echo exit:$?  # expect 1; err on stderr

Common gotchas

Gotcha	Fix
`flag.Parse` in subcommand world	Per-command `FlagSet`
Errors on stdout	`os.Stderr`
Exit 0 on failure	Explicit `os.Exit` / return err from Cobra
Logic only in `main`	`internal/` packages + tests
Interactive prompts only	Flags for CI
Ignoring Windows paths	`filepath` when touching files
Forgetting `SetOutput(os.Stderr)`	Help and flag errors on stderr
Version always `dev`	ldflags in release script

Checkpoint

Commit

git add .
git commit -m "day71: cli ergonomics multi-command"

Write three personal gotchas before continuing.

Tomorrow

Day 72 — govulncheck: scan for known vulnerable dependencies and remediate with evidence.

Day 72 — govulncheck & supply chain

Stage VII · ~3h
Goal: Run govulncheck on your module, understand call-graph reachability vs “CVE in go.mod”, remediate or document accepted risk, and capture a short supply-chain checklist for CI—on Go 1.26.

Why this day exists

Your code can be perfect while a dependency is not. Supply-chain hygiene is part of ship quality:

Known vulnerable modules
Pseudo-versions and abandoned libs
replace forks nobody audits
CI that never re-scans

govulncheck is the official Go vulnerability scanner, integrated with the Go vulnerability database. Day 74’s gate will expect a clean scan or an explicit accepted-risk note. Day 81’s security checklist will still cover your auth bugs—scanner ≠ secure app.

Theory 1 — What govulncheck does

go install golang.org/x/vuln/cmd/govulncheck@latest
govulncheck -version   # if supported; else: note binary path + date
govulncheck ./...

It analyzes your module’s code and reports vulnerabilities that are reachable in the call graph when possible—not merely present in the module graph. That reduces noise versus naïve “every CVE in the lockfile” tools.

Useful invocations

govulncheck ./...
govulncheck -show verbose ./...
# JSON for CI parsing when needed
govulncheck -json ./... > vuln.json

Binary mode

go build -o bin/app ./cmd/service
govulncheck -mode=binary ./bin/app

Binary mode helps when source and shipped artifact diverge (build tags, different main packages, stripped paths). Scan what you ship.

Mode	Strength
Source (`./...`)	Maps findings to your call sites
Binary	Matches the artifact operators run

Theory 2 — Reading a finding

A report typically includes:

Vulnerability ID (e.g. GO-20XX-XXXX)
Affected module and version range
Fixed version (if any)
Whether your code calls into the vulnerable symbol/path
Links to advisory detail

Situation	Action
Reachable + fixed version available	Upgrade; re-test
In module graph but not reachable	Still prefer upgrade; document if blocked
No fix yet	Mitigate (avoid code path), monitor, careful replace/fork
Only test-only dependency	Still upgrade if easy; else note scope
False sense of security	Scanner ≠ review of your auth bugs

Example narrative (write yours in SECURITY-SCAN.md)

ID: GO-2024-XXXX
Module: golang.org/x/net
Your version: v0.a.b
Fixed in: v0.x.y
Reachable: yes (HTTP/2 related import path)
Plan: go get golang.org/x/net@v0.x.y && go test ./...

Theory 3 — Remediation workflow

go get example.com/vulnerable@v1.2.4
go mod tidy
go test ./...
govulncheck ./...

If an upgrade breaks APIs:

Read upstream changelog / release notes.
Fix compile breaks in adapters first (prefer not to change domain).
Run unit + smoke of main paths.
Re-run govulncheck.
Update SECURITY-SCAN.md with before/after versions.

Partial upgrades

go list -m all | grep vulnerable
go get golang.org/x/net@v0.x.y
go mod tidy

Avoid go get -u ./... as a blind first move on a large graph—prefer targeted bumps for the advisory, then consider broader hygiene.

Toolchain and stdlib

go version
go env GOTOOLCHAIN

Keep the Go toolchain itself updated; stdlib CVEs happen too. Matching this volume’s Go 1.26 baseline means you already practice currency (Day 37 quality pass). If go.mod says go 1.26, CI should use 1.26.x as well.

Theory 4 — Reachability vs lockfile scanners

Scanner style	Reports	Noise
Lockfile/CVE match	Any vulnerable version in graph	Higher
govulncheck (source)	Prefer reachable symbols	Lower
Image OS scanner (Trivy etc.)	Distro packages in container	Different surface

Use both mindsets in production orgs. For this volume, govulncheck is the required Go-native bar; container image scan is stretch.

Your module code
   → imports package P@v1 (vulnerable func F)
   → if F is unreachable from your main graph, source mode may omit or deprioritize
Binary mode still encodes what was linked.

Theory 5 — CI placement

Recommended Stage VII pipeline shape:

test → vet → staticcheck → govulncheck → build → (image scan optional)

Example GitHub Actions step sketch:

- name: govulncheck
  run: |
    go install golang.org/x/vuln/cmd/govulncheck@latest
    govulncheck ./...

Policy options:

Policy	When
Fail on any finding	Main branch / release tags
Fail on reachable only	Early adoption
Annotated exceptions	Time-boxed, with owner + expiry

Script hook

Add to scripts/lint.sh or scripts/gate-vii.sh:

#!/usr/bin/env bash
set -euo pipefail
go test ./...
go vet ./...
staticcheck ./...
govulncheck ./...

Theory 6 — Other supply-chain controls (awareness)

Control	Role
`go.sum`	Integrity of downloaded modules
Checksum DB	Verified sums for public modules
Module proxy	Default `proxy.golang.org`; privacy via `GOPRIVATE`
`GOPRIVATE` / `GONOSUMDB`	Private modules not forced through public sum DB
`replace`	Local forks—audit like first-party code
SBOM / cosign	Org-level signing & inventory; optional stretch
Dependabot / Renovate	Automated upgrade PRs
Vendoring	`vendor/` can go stale—re-scan after updates

Private modules sketch

go env -w GOPRIVATE=github.com/yourorg/*
# ensure git credentials work in CI

Theory 7 — Accepted risk template

When you cannot upgrade today:

## Accepted risk

ID: GO-....
Module/version:
Reachable: yes/no
Why not fixed today:
Mitigation (feature flag, avoid path, WAF, etc.):
Owner:
Review by date:
Tracking issue:

Accepted risk without expiry is how permanent holes form. Day 74 gate should reject eternal waivers without dates.

Worked example — clean scan document

SECURITY-SCAN.md:

# govulncheck — day72

Date: YYYY-MM-DD
Go: go1.26.x
govulncheck: <version or install date>
Module: example.com/myservice

## Commands
govulncheck ./...
go build -o bin/app ./cmd/service
govulncheck -mode=binary ./bin/app

## Result
No vulnerabilities found.

## Notes
- Last dependency review: ...
- replace directives: none | listed below
- CI: scripts/lint.sh includes govulncheck

Finding + remediation example

## Finding
ID: GO-....
Module: golang.org/x/net @ v0.a.b
Fixed in: v0.x.y
Reachable: yes (import path ...)

## Remediation
go get golang.org/x/net@v0.x.y
go mod tidy
go test ./...
govulncheck ./... → clean

## Residual risk
None known.

Lab 1 — Install and baseline scan

Suggested workspace: your primary service or CLI module.

Confirm go version is 1.26.x.
Install govulncheck; record version/date.
Run govulncheck ./...; save full output to SECURITY-SCAN.md or attach summary.
Build main artifact; run binary mode.
If clean: still write the clean-scan document (evidence for Day 74).

go install golang.org/x/vuln/cmd/govulncheck@latest
govulncheck ./...
mkdir -p bin
go build -o bin/app ./cmd/...
govulncheck -mode=binary ./bin/app

Lab 2 — Remediate or accept

If findings exist: prefer upgrade path.
Re-test; re-scan.
If blocked: fill accepted-risk template with review date.
Search for replace in go.mod and justify each.

grep -n replace go.mod || true
go list -m all > /tmp/modules.txt
# skim for pseudo-versions on critical path libs

Lab 3 — CI / script integration

Add govulncheck to scripts/lint.sh (or create it).
Sketch Actions (or other CI) step in SECURITY-SCAN.md.
Run the script cold once.

chmod +x scripts/lint.sh
./scripts/lint.sh

Lab 4 — JSON output stretch (optional but useful)

govulncheck -json ./... > vuln.json
# count findings with jq if installed
jq '.. | .osv? // empty | .id' vuln.json 2>/dev/null || true

Document how CI could fail on non-empty findings.

Stretch — image scan awareness

# conceptual — tool must be installed
# trivy image myapp:dev

Note: OS packages in the container are outside govulncheck’s Go module graph. Distroless/static helps shrink that surface (Day 68).

Hour-by-hour suggestion

Time	Focus
0:00–0:20	Install tool; baseline source scan
0:20–1:00	Binary scan; interpret findings
1:00–2:00	Upgrade / remediate / accepted risk
2:00–2:30	lint.sh + CI sketch
2:30–3:00	SECURITY-SCAN.md polish

Common gotchas

Gotcha	Fix
Old govulncheck DB / binary	Reinstall `@latest`; ensure network to vuln DB
Clean scan ≠ secure app	Still do Day 81 checklist
Upgrading everything blindly	Read changelogs for major jumps
Ignoring binary mode	Scan what you ship
Vendoring stale code	Re-scan after `vendor` updates
Private modules fetch fail	`GOPRIVATE` + CI credentials
Accepting risk forever	Expiry date + owner
Only scanning `./cmd`	Prefer `./...` then binary
Committing `vuln.json` secrets	Usually none—but don’t commit env dumps

Checkpoint

govulncheck ./... run documented
Binary mode run or explicit N/A with reason
Findings remediated or explicitly accepted with review date
Tool version + Go 1.26 recorded
Script/CI hook sketched and executed once
Can explain reachability vs lockfile-only scanners
replace directives reviewed

Commit

git add .
git commit -m "day72: govulncheck supply chain pass"

Write three personal gotchas before continuing.

Tomorrow

Day 73 — Elective: Kubernetes client-go or deepen the service—choose deliberately and ship one coherent outcome. Bring SECURITY-SCAN.md into the Stage VII gate pack (Day 74).

Day 25 — Observability, CLI & Security Scanning

Day 69 — Prometheus metrics

Why this day exists

Theory 1 — Metric types

Theory 2 — Client library basics

Theory 3 — Default Go collector

Theory 4 — Scraping locally

Naming conventions

Theory 5 — Histograms and buckets

Theory 6 — Testing metrics

Worked example — complete mini service

Worked example — label policy blurb

Lab

Stretch

Common gotchas

Checkpoint

Commit

Tomorrow

Day 70 — OpenTelemetry light

Why this day exists

Theory 1 — Three pillars (and boundaries)

Core objects

Theory 2 — Minimal Go wiring (stdout exporter)

Pinning note

Theory 3 — Context is the bus

Sampling (awareness)

Theory 4 — Correlation with logs and metrics

Attribute discipline

Theory 5 — Auto vs manual instrumentation

Theory 6 — Errors, status, and shutdown

Worked example — span tree narrative

Worked example — outbound HTTP child (optional)

Lab

Stretch

Common gotchas

Checkpoint

Commit

Tomorrow

Day 71 — CLI ergonomics

Why this day exists

Theory 1 — CLI UX contract

Theory 2 — Stdlib approach: subcommands with flag

Exit-on-error vs continue

Theory 3 — Cobra (optional, common in industry)

Theory 4 — Structure for real tools

Config flags vs env (preview Day 77)

Completion and man pages (awareness)

Theory 5 — Version injection

Theory 6 — Testing CLIs

Worked example — useful mini tool

Lab

Acceptance demo

Common gotchas

Checkpoint

Commit

Tomorrow

Day 72 — govulncheck & supply chain

Why this day exists

Theory 1 — What govulncheck does

Useful invocations

Binary mode

Theory 2 — Reading a finding

Example narrative (write yours in SECURITY-SCAN.md)

Theory 3 — Remediation workflow

Partial upgrades

Toolchain and stdlib

Theory 4 — Reachability vs lockfile scanners

Theory 5 — CI placement

Script hook

Theory 6 — Other supply-chain controls (awareness)

Private modules sketch

Theory 7 — Accepted risk template

Worked example — clean scan document

Finding + remediation example

Lab 1 — Install and baseline scan

Lab 2 — Remediate or accept

Lab 3 — CI / script integration

Lab 4 — JSON output stretch (optional but useful)

Stretch — image scan awareness

Hour-by-hour suggestion

Theory 2 — Stdlib approach: subcommands with `flag`