Day 69 — Prometheus metrics

Updated

July 30, 2026

Day 69 — Prometheus metrics

Stage VII · ~3h
Goal: Instrument a Go service with the Prometheus client: process metrics, at least one business/request metric, a /metrics scrape endpoint, and a short note on cardinality discipline.

Why this day exists

Logs tell stories. Traces follow one request. Metrics answer:

  • How many requests? Error rate? Latency shape?
  • Is the process thrashing GC or goroutines?
  • Did the last deploy make p99 worse?

Prometheus is the de facto scrape model in cloud-native Go shops. You need enough instrumentation for Stage VIII load tests and the capstone—not a full SRE platform.

Theory 1 — Metric types

Type Use for
Counter Monotonic events (requests_total)
Gauge Instant value (queue_depth)
Histogram Distribution (request_duration_seconds)
Summary Client-side quantiles (less common in new code; prefer histogram + PromQL)

Golden signals for HTTP services (RED):

  • Rate — http_requests_total
  • Errors — same counter with code label or separate error counter
  • Duration — histogram of latency

USE (utilization, saturation, errors) is the sibling model for resources—goroutines, heap, queue depth as gauges.

Theory 2 — Client library basics

Module: github.com/prometheus/client_golang

import (
    "net/http"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    httpRequests = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total HTTP requests",
        },
        []string{"method", "path", "code"},
    )
    httpDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "HTTP request latency",
            Buckets: prometheus.DefBuckets, // or custom
        },
        []string{"method", "path"},
    )
)

Middleware sketch (Go 1.22+ ServeMux patterns—normalize path labels carefully):

func instrument(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        ww := &statusWriter{ResponseWriter: w, code: 200}
        next.ServeHTTP(ww, r)
        path := r.Pattern // registered pattern when available
        if path == "" {
            path = "unknown"
        }
        httpRequests.WithLabelValues(r.Method, path, strconv.Itoa(ww.code)).Inc()
        httpDuration.WithLabelValues(r.Method, path).Observe(time.Since(start).Seconds())
    })
}
mux := http.NewServeMux()
mux.Handle("GET /metrics", promhttp.Handler())
mux.Handle("GET /v1/hello", instrument(http.HandlerFunc(hello)))
Warning

Cardinality bomb: never put unbounded values in labels (user_id, raw URL, email). Prefer route patterns (/v1/users/{id}) not concrete IDs.

Theory 3 — Default Go collector

promhttp.Handler() registers process/Go collectors by default when using the default registry:

  • go_goroutines
  • go_memstats_*
  • process_cpu_seconds_total
  • GC-related series

That alone is valuable for Day 67 insights in a long-running process. Hit /metrics after load and find go_memstats_heap_alloc_bytes.

Theory 4 — Scraping locally

curl -s localhost:8080/metrics | head
curl -s localhost:8080/metrics | grep http_requests_total

Optional dockerized Prometheus config (stretch):

scrape_configs:
  - job_name: day69
    scrape_interval: 5s
    static_configs:
      - targets: ["host.docker.internal:8080"]

You do not need Grafana today—raw /metrics text is enough.

Naming conventions

Suffix Meaning
_total Counter
_seconds Time unit base
_bytes Size unit base
_ratio 0–1 gauge (when used)

Theory 5 — Histograms and buckets

Default buckets are a starting point—not magic. For APIs aiming at sub-100ms:

Buckets: []float64{.001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5}

Too many buckets × too many label combos = expensive cardinality. Start narrow.

PromQL awareness (read-only for lab):

rate(http_requests_total[5m])
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

You need not run Prometheus server to understand the shape.

Theory 6 — Testing metrics

func TestMetricsInc(t *testing.T) {
    reg := prometheus.NewRegistry()
    c := prometheus.NewCounter(prometheus.CounterOpts{Name: "demo_total", Help: "x"})
    reg.MustRegister(c)
    c.Inc()
    mfs, err := reg.Gather()
    if err != nil {
        t.Fatal(err)
    }
    // assert metric family present
    _ = mfs
}

Avoid promauto global registration fighting tests—use a private registry in unit tests when you assert series.

Worked example — complete mini service

package main

import (
    "log"
    "net/http"
    "strconv"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    reqs = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "demo_requests_total",
        Help: "Demo requests",
    }, []string{"route", "code"})
    dur = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "demo_request_duration_seconds",
        Help:    "Demo latency",
        Buckets: []float64{.001, .005, .01, .025, .05, .1, .25, .5, 1},
    }, []string{"route"})
)

type statusWriter struct {
    http.ResponseWriter
    code int
}

func (w *statusWriter) WriteHeader(code int) {
    w.code = code
    w.ResponseWriter.WriteHeader(code)
}

func withMetrics(route string, next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        sw := &statusWriter{ResponseWriter: w, code: http.StatusOK}
        next(sw, r)
        reqs.WithLabelValues(route, strconv.Itoa(sw.code)).Inc()
        dur.WithLabelValues(route).Observe(time.Since(start).Seconds())
    }
}

func main() {
    mux := http.NewServeMux()
    mux.Handle("GET /metrics", promhttp.Handler())
    mux.HandleFunc("GET /healthz", withMetrics("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
        _, _ = w.Write([]byte("ok"))
    }))
    mux.HandleFunc("GET /v1/work", withMetrics("/v1/work", func(w http.ResponseWriter, r *http.Request) {
        time.Sleep(15 * time.Millisecond) // pretend work
        w.WriteHeader(http.StatusOK)
        _, _ = w.Write([]byte(`{"ok":true}`))
    }))
    log.Fatal(http.ListenAndServe(":8080", mux))
}
go get github.com/prometheus/client_golang@latest
go run .
# other terminal
curl -s localhost:8080/v1/work
curl -s localhost:8080/metrics | grep demo_

Worked example — label policy blurb

Paste into METRICS.md:

## Labels we allow
- method: GET|POST|...
- path: registered pattern only (e.g. /v1/items/{id})
- code: HTTP status as string

## Labels we refuse
- user_id, email, raw URL, request_id (use logs/traces)
- free-text error messages

## Series owned by this service
- demo_requests_total{route,code}
- demo_request_duration_seconds{route}
- plus default go_* / process_* from client_golang

That document is the review artifact—not a second monitoring platform.

Lab

Suggested workspace: ~/lab/90daysofx/01-go/day69

  1. Add Prometheus client to your service (or the demo above).
  2. Export /metrics.
  3. At least one Counter and one Histogram (or latency summary) on a real route.
  4. Hit the route several times; show series increasing in curl output (paste into METRICS.md).
  5. Document label policy: which labels exist and which you refuse.
  6. Confirm go_goroutines appears (default registry).

Stretch

  • Custom buckets matching your SLA (e.g. 50/100/200ms).
  • Counter for a domain event (orders_created_total) separate from HTTP.
  • Protect /metrics with network policy notes (not public internet).
  • Wire middleware into the Day 50/53 service instead of the demo main.

Common gotchas

Gotcha Fix
High-cardinality labels Route templates only
Histogram buckets useless Tune to expected latency
Metrics on public internet unauthenticated Network policy; sometimes separate port
Double-register metrics in tests Use a fresh registry in unit tests
Ignoring units _seconds, _bytes, _total suffixes
Measuring only /metrics Instrument business paths
Labeling with raw r.URL.Path Use r.Pattern or a fixed route name

Checkpoint

  • /metrics scrapes successfully
  • Counter + histogram (or gauge with clear story)
  • Label cardinality policy written
  • Sample scrape snippet in METRICS.md
  • Go runtime metrics visible
  • Know RED signals for HTTP

Commit

git add .
git commit -m "day69: prometheus metrics endpoint"

Write three personal gotchas before continuing.

Tomorrow

Day 70 — OpenTelemetry light: trace one request path end-to-end so latency has a story beyond histograms.