Observability and SRE Overview

Updated

July 30, 2026

Observability and SRE Overview

This part explains how to make Go services operable in real incidents. Observability is a design property: if you cannot ask new questions of a running system, you do not have it — you have a fixed dashboard.

Why / Overview

Production failures are rarely “the server is off.” They are:

  • Elevated tail latency on one route
  • Error budgets burning from one dependency
  • A deploy that only breaks for a subset of tenants
  • Retries hiding the true failure rate

You need three complementary signals:

logs    — discrete events with context (what happened)
metrics — aggregates over time (is it bad, how bad)
traces  — causal latency across services (where is the time)

SRE practice ties those signals to SLIs/SLOs, alerts, and runbooks — not to collecting data for its own sake.

Learning Path

Chapter Focus Outcome
161 Structured logging slog, fields, correlation, redaction Debuggable event streams
162 Metrics / Prometheus RED/USE, labels, histograms Alertable, cheap aggregates
163 Tracing Spans, context propagation Cross-service latency attribution

The Observability Triangle

              metrics
             /       \
            /         \
         logs ------- traces
              correlation id / trace id

A single correlation key (request id / trace id) should appear in all three where possible.

SLI / SLO Primer

Term Meaning
SLI Quantitative measure (e.g. % of requests < 300ms)
SLO Target on that measure (e.g. 99.9% over 30d)
Error budget 1 − SLO; spend on velocity or freeze

Instrument what you SLO. Vanity metrics that nobody pages on are optional.

Go Building Blocks

Signal Libraries / stdlib
Logs log/slog (Go 1.21+)
Metrics Prometheus client, OpenTelemetry metrics
Traces OpenTelemetry Go SDK
Propagation context.Context + W3C headers 
slog.Info("checkout",
    "request_id", rid,
    "user_hash", hash(userID), // not raw PII if avoidable
    "total_ms", dur.Milliseconds(),
)

Production Principles

  1. Cardinality discipline — unbounded label values kill metric systems.
  2. Structured fields > string paste — enable search and aggregation.
  3. Propagate context — lost trace context is a permanent blind spot.
  4. Sample thoughtfully — full traces at 100% may be too expensive; errors often keep 100%.
  5. Alert on symptoms (user pain) more than causes (CPU high).
  6. Redact secrets — logs and spans are exfiltration channels.

Relationship to Other Parts

  • Network and distributed parts emit the signals designed here.
  • Performance engineering uses profiles (adjacent to metrics/traces).
  • Security hardening constrains what you may log.

Literacy Checklist

  • Emit JSON or structured logs with stable field names
  • Carry request/trace ids across goroutines and HTTP
  • Define RED metrics for a service
  • Choose histogram buckets intentionally
  • Avoid high-cardinality labels
  • Create and propagate spans on outbound calls
  • Tie an alert to an SLO and a runbook

Part Warm-Up Exercises

  1. List the top five questions you ask during incidents; map each to log/metric/trace.
  2. Find one high-cardinality metric risk in an existing codebase (user_id label?).
  3. Grep for log.Printf and plan migration of hot paths to slog.
  4. Check whether outbound HTTP clients forward trace headers.
  5. Write a draft SLO for your highest-traffic endpoint.

Instrumentation Order of Operations

When adding observability to a greenfield service, do this in order:

  1. Structured access logs with request id (chapter 161)
  2. RED metrics for public routes (chapter 162)
  3. Dependency client metrics (timeouts vs errors)
  4. Trace propagation on the hottest path (chapter 163)
  5. SLOs + alerts + runbooks for those SLIs

Skipping to distributed tracing without metrics leaves you with pretty waterfalls and no reliable pages. Skipping logs leaves metrics without narrative.

What “Good” Looks Like in an Incident

Within five minutes you should be able to:

1. See error rate and p99 on the affected route (metrics)
2. Grab a request_id / trace_id from a failing request (logs or edge)
3. Jump to a trace that shows the slow/failed span (traces)
4. Read surrounding log lines for that id (logs)
5. Open the runbook linked from the alert

If any hop is missing, fix instrumentation — not only the proximate bug.

Cardinality and Cost Budget

Observability has a cloud bill. Rough discipline:

Signal Cost driver Control
Logs volume × retention sample happy path; drop debug in prod
Metrics series count bounded labels; careful histograms
Traces spans × rate head/tail sampling

Review series count and log GB/day the same way you review CPU.

Anti-Patterns

  1. Unstructured logs that require regex archaeology
  2. Metrics labeled with user ids or raw URLs
  3. Traces without parent context on outbound HTTP
  4. Alerts with no runbook and no SLO
  5. Logging secrets “just for staging” (staging leaks too)

Minimum Dashboard Set

For each user-facing service, keep a default dashboard with:

  1. Request rate (global + top routes)
  2. Error rate / fraction (5xx and timeout class)
  3. Latency p50 / p95 / p99
  4. In-flight requests / saturation
  5. Dependency latency and error panels
  6. Go runtime: goroutines, heap, GC CPU (secondary)

If a panel cannot drive an action, demote it. Dashboards are for decisions, not decoration.

Alert Quality Bar

Before shipping an alert, answer:

  • What user pain does this detect?
  • What is the false-positive risk?
  • Who is paged and what do they run first?
  • Can we silence safely during known maintenance?

Symptom-based alerts (SLO burn) beat cause-based alerts (CPU > 80%) for most services. Cause alerts are still useful for capacity planning, usually at ticket severity not page severity.

Library Choices (2024–2026 pragmatic)

Need Common choice
Logs log/slog (+ JSON handler)
Metrics Prometheus client_golang
Traces OpenTelemetry Go
Bridges OTel → your backend vendor

Avoid inventing a fourth proprietary telemetry API inside your org without a migration plan.

Glossary

  • SLI / SLO / error budget — measure, target, remaining failure allowance
  • Cardinality — number of unique time series
  • Correlation id — joins signals for one request
  • Exemplar — metric sample pointing at a trace
  • Propagation — carrying context across process boundaries
  • RED / USE — request and resource metric recipes
  • Tail sampling — keep interesting traces after the fact

More examples

Tour: slog + counter + request id

mkdir -p /tmp/go-o11y-tour && cd /tmp/go-o11y-tour
go mod init example.com/o11y-tour

Save as main.go:

package main

import (
    "bytes"
    "fmt"
    "log/slog"
    "net/http"
    "net/http/httptest"
    "strings"
    "sync/atomic"
)

func main() {
    var hits atomic.Int64
    var buf bytes.Buffer
    log := slog.New(slog.NewJSONHandler(&buf, nil))

    h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        id := r.Header.Get("X-Request-ID")
        if id == "" {
            id = "gen-1"
        }
        hits.Add(1)
        log.Info("hit", "request_id", id, "path", r.URL.Path)
        w.Header().Set("X-Request-ID", id)
        fmt.Fprintln(w, "ok")
    })
    ts := httptest.NewServer(h)
    defer ts.Close()

    req, _ := http.NewRequest(http.MethodGet, ts.URL+"/x", nil)
    req.Header.Set("X-Request-ID", "abc")
    resp, _ := http.DefaultClient.Do(req)
    resp.Body.Close()
    fmt.Println("hits:", hits.Load(), "resp id:", resp.Header().Get("X-Request-ID"))
    fmt.Println("logged:", strings.Contains(buf.String(), "abc"))
}
go run .

Expected output:

hits: 1 resp id: abc
logged: true

Runnable example

Tour of the observability triangle: one JSON log line, one counter sample, one trace-id field on a context—stdlib only.

mkdir -p /tmp/go-o11y-tour && cd /tmp/go-o11y-tour
go mod init example.com/o11y-tour

Save as main.go:

package main

import (
    "bytes"
    "context"
    "fmt"
    "log/slog"
    "time"
)

type ctxKey string

const traceIDKey ctxKey = "trace_id"

func main() {
    var buf bytes.Buffer
    log := slog.New(slog.NewJSONHandler(&buf, nil))

    ctx := context.WithValue(context.Background(), traceIDKey, "tr-demo-1")
    start := time.Now()
    // fake work
    time.Sleep(5 * time.Millisecond)
    log.InfoContext(ctx, "request",
        "trace_id", ctx.Value(traceIDKey),
        "route", "GET /checkout",
        "status", 200,
        "dur_ms", time.Since(start).Milliseconds(),
    )

    // metric sample (manual exposition line)
    var requestsTotal = 1
    fmt.Printf("# TYPE demo_requests_total counter\ndemo_requests_total %d\n", requestsTotal)
    fmt.Println("--- log ---")
    fmt.Print(buf.String())
    fmt.Println("tour: logs + metrics + trace id field")
}
go run .

Expected output (illustrative):

# TYPE demo_requests_total counter
demo_requests_total 1
--- log ---
{"time":"...","level":"INFO","msg":"request","trace_id":"tr-demo-1",...}
tour: logs + metrics + trace id field

What to notice

  • The three signals answer different questions; this part teaches each without vendor lock-in first.
  • Stable field names make cross-service grepping possible during incidents.

Try next

  • Chapters 161–163: slog correlation, Prometheus-shaped metrics, W3C-ish traceparent propagation.

Next Chapter

Structured Logging and Correlation — the foundation every engineer greps first.