Day 70 — OpenTelemetry light

Updated

July 30, 2026

Day 70 — OpenTelemetry light

Stage VII · ~3h
Goal: Add a minimal OpenTelemetry trace path to one request: tracer provider, a root span + child span, context propagation, and export (stdout is enough). Understand how traces complement metrics and logs.

Note

Full mesh platforms are out of scope. Today is one path, correctly instrumented—not “adopt the whole observability platform.”

Why this day exists

Histograms say “p99 is 200ms.” Traces answer why this request was 200ms:

  • Time in handler vs DB vs outbound HTTP
  • Error attachment on a span
  • Correlation across services via W3C traceparent

Day 69 gave Prometheus metrics (aggregates). Day 47 gave structured logs. Traces complete the triangle for individual slow or failing requests.

Theory 1 — Three pillars (and boundaries)

Signal Strength
Logs (slog) Discrete events, cheap, high detail
Metrics Aggregates, alerts, cheap cardinality if careful
Traces Causal graph of a request

OpenTelemetry (OTel) standardizes APIs and exporters so you are not locked to one vendor SDK forever.

Core objects

  • TracerProvider — process-wide factory
  • Tracer — named instrumentation library
  • Span — unit of work with start/end, attributes, status
  • Context — carries span; must thread through context.Context
Request ctx
  └─ span: GET /v1/hello
       └─ span: doWork / db.query

Theory 2 — Minimal Go wiring (stdout exporter)

Modules evolve; pin versions that build on Go 1.26 when you run the lab. Pattern:

package main

import (
    "context"
    "log"
    "net/http"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func initTracer(ctx context.Context) (func(context.Context) error, error) {
    exp, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
    if err != nil {
        return nil, err
    }
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("day70-demo"),
        )),
    )
    otel.SetTracerProvider(tp)
    return tp.Shutdown, nil
}
func main() {
    ctx := context.Background()
    shutdown, err := initTracer(ctx)
    if err != nil {
        log.Fatal(err)
    }
    defer func() { _ = shutdown(context.Background()) }()

    mux := http.NewServeMux()
    mux.HandleFunc("GET /v1/hello", handleHello)
    log.Fatal(http.ListenAndServe(":8080", mux))
}

func handleHello(w http.ResponseWriter, r *http.Request) {
    ctx, span := otel.Tracer("day70/http").Start(r.Context(), "GET /v1/hello")
    defer span.End()

    if err := doWork(ctx); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        http.Error(w, "failed", 500)
        return
    }
    span.SetAttributes(attribute.String("handler.version", "v1"))
    w.WriteHeader(200)
    _, _ = w.Write([]byte(`{"ok":true}`))
}

func doWork(ctx context.Context) error {
    ctx, span := otel.Tracer("day70/http").Start(ctx, "doWork")
    defer span.End()
    // honor cancellation
    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-time.After(20 * time.Millisecond):
    }
    span.SetAttributes(attribute.Int("work.ms", 20))
    return nil
}
go get go.opentelemetry.io/otel@latest \
  go.opentelemetry.io/otel/sdk@latest \
  go.opentelemetry.io/otel/exporters/stdout/stdouttrace@latest
go run .
curl -s localhost:8080/v1/hello
# spans print to process stdout

Pinning note

In notes, record the module versions go.mod resolved. OTel packages move; your lab should still build on Go 1.26.

Theory 3 — Context is the bus

Always pass context.Context into downstream work:

// good
rows, err := db.QueryContext(ctx, q)

// bad — orphan child, broken cancellation
rows, err := db.Query(q)

HTTP clients should inject headers for outbound calls (contrib middleware / otelhttp). For one-process labs, parent→child spans via Start(ctx, name) is the essential skill.

Sampling (awareness)

Production systems sample (e.g. parent-based, ratio-based). Stdout labs can use default always-on for the demo. Never sample in a way that drops all errors if your platform supports tail sampling—out of scope, but know sampling exists.

// awareness only — not required today
// sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)))

Theory 4 — Correlation with logs and metrics

  • Put trace_id into slog attributes when a span is present (optional stretch).
  • Metrics remain for aggregates; traces for outliers.
  • Do not replace metrics with traces for alerting volume.
import "go.opentelemetry.io/otel/trace"

spanCtx := trace.SpanContextFromContext(ctx)
if spanCtx.IsValid() {
    logger.Info("handled",
        "trace_id", spanCtx.TraceID().String(),
        "span_id", spanCtx.SpanID().String(),
    )
}

Attribute discipline

Good attribute Bad attribute
http.route = /v1/users/{id} Full URL with user ids
db.system = sqlite Entire SQL with PII
handler.version = v2 Unbounded free-text blobs

High-cardinality attributes explode backend cost the same way metric labels do.

Theory 5 — Auto vs manual instrumentation

Approach Trade-off
Manual spans Precise names; control attributes
otelhttp middleware Fast coverage of HTTP edges
Vendor agents Convenient; less portable

For this day: manual root + one child is mandatory. Middleware is optional polish.

// stretch sketch
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

mux := http.NewServeMux()
mux.Handle("GET /v1/hello", otelhttp.NewHandler(http.HandlerFunc(handleHello), "GET /v1/hello"))

Theory 6 — Errors, status, and shutdown

if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
    return err
}
// success: default status OK; optional:
// span.SetStatus(codes.Ok, "")

Process exit without Shutdown loses batched spans:

defer func() {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    _ = shutdown(ctx)
}()

Align with Day 78 graceful shutdown: flush tracer after HTTP drain.

Worked example — span tree narrative

Paste into TRACE.md something like:

Span: GET /v1/hello          25ms
  attributes: handler.version=v1
  Span: doWork               20ms
    attributes: work.ms=20

If stdout is JSON lines, keep a short excerpt—not megabytes of logs.

Worked example — outbound HTTP child (optional)

func fetchUpstream(ctx context.Context, client *http.Client, url string) error {
    ctx, span := otel.Tracer("day70/http").Start(ctx, "HTTP GET upstream")
    defer span.End()
    span.SetAttributes(attribute.String("peer.service", "upstream"))

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return err
    }
    resp, err := client.Do(req)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return err
    }
    defer resp.Body.Close()
    span.SetAttributes(attribute.Int("http.status_code", resp.StatusCode))
    return nil
}

Without otelhttp transport, you still get a local child span; cross-process traceparent needs proper propagation.

Lab

Suggested workspace: ~/lab/90daysofx/01-go/day70

  1. Wire TracerProvider with stdout exporter (or OTLP if you already run a collector).
  2. Instrument one route: root span + child span for a dependency call (DB, HTTP, or sleep-as-work).
  3. Capture a sample span tree in TRACE.md (paste stdout JSON/pretty).
  4. Call Shutdown on process exit (defer with timeout).
  5. Note one attribute you added and why it is not high-cardinality.
  6. Explain in two sentences (in TRACE.md) how this differs from Day 69 metrics.

Stretch

  • Use go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.
  • Propagate to a second tiny process with W3C headers.
  • Export OTLP to a local collector/Jaeger all-in-one.
  • Add trace_id to one slog line.

Common gotchas

Gotcha Fix
Forgetting span.End() defer span.End() immediately after Start
New context without parent Always Start(ctx, ...) from request ctx
Export lost on exit Shutdown flushes batcher
Span name = full URL with IDs Use route pattern
“OTel is too heavy” One tracer + stdout is enough for learning
Mixing metrics SDK complexity Keep Day 69 Prometheus; add OTel traces only
Panic before End Still prefer defer End; recover separately (Day 14)
Global provider race in tests Set provider in TestMain or inject Tracer

Checkpoint

  • TracerProvider initialized and shut down
  • Root + child span on one path
  • Context threaded into child work
  • Example output saved in TRACE.md
  • Can explain metrics vs traces in two sentences
  • No high-cardinality attributes on the demo path

Commit

git add .
git commit -m "day70: otel light trace path"

Write three personal gotchas before continuing.

Tomorrow

Day 71 — CLI ergonomics: multi-command UX, flags, exit codes, and --help that respect users (and future you).