Day 42 — Observability pattern

Updated

July 30, 2026

Day 42 — Observability pattern

Stage IV · ~4h
Goal: Add an observability pattern for the lab slice—either Prometheus + Grafana (or Prometheus alone) or a deliberate logs-first minimal stack—so one service is inspectable without SSH archaeology alone.

Why this day exists

Services without observability fail quietly:

  • Disk fills
  • Proxy 502s for hours
  • Postgres connections exhaust
  • ACME renewals fail

You do not need a full SRE platform. You need a repeatable pattern: what you measure, where it lives, who can access it, and how it is firewalled.

Theory 1 — Three pillars (light)

Pillar Lab approach today
Logs journald (journalctl) first-class
Metrics Prometheus node exporter + app exporters optional
Traces Optional later; not required

Pick a primary for the day: metrics stack or logs discipline. Doing both shallowly is fine; doing neither is not.

Theory 2 — Logs-first pattern (valid Stage IV)

If resources are tight:

journalctl -u caddy -b --no-pager | tail
journalctl -u postgresql -b --no-pager | tail
journalctl -p err -b --no-pager | tail -50

Declarative helpers:

{
  # ensure persistent journal if you want history across reboots
  services.journald.extraConfig = ''
    SystemMaxUse=500M
  '';
}

Document a debug runbook for proxy, DB, and sops activation units. That can satisfy the pattern if you explicitly choose logs-first in docs/observability.md.

Theory 3 — Prometheus pattern

# modules/services/prometheus.nix
{
  services.prometheus = {
    enable = true;
    port = 9090;
    listenAddress = "127.0.0.1";
    scrapeConfigs = [
      {
        job_name = "node";
        static_configs = [{
          targets = [ "127.0.0.1:9100" ];
        }];
      }
    ];
  };

  services.prometheus.exporters.node = {
    enable = true;
    port = 9100;
    listenAddress = "127.0.0.1";
  };

  # Do not open 9090/9100 to the world by default
}

Verify option names for 26.05. Access via SSH tunnel:

ssh -L 9090:127.0.0.1:9090 lab
# browse localhost:9090

Or reverse-proxy with authentication (basic auth / SSO)—never raw public Prometheus without auth.

Theory 4 — Grafana (optional layer)

{
  services.grafana = {
    enable = true;
    settings = {
      server = {
        http_addr = "127.0.0.1";
        http_port = 3000;
      };
    };
  };
}

Initial admin password: use sops / first-boot procedures—not plaintext in git. Some modules support password file options; prefer those.

Dashboard: import Node Exporter essentials or build one panel showing CPU/memory.

Theory 5 — What to scrape for your slice

Target Why
node exporter Host health
Caddy/Nginx metrics If exporter/module available
Postgres exporter Optional; more moving parts
Blackbox / probes Optional

Minimum bar: node exporter + Prometheus or excellent journald runbooks + one automated log check.

Theory 6 — Firewall and multi-tenant risk

Metrics endpoints often expose sensitive operational detail. Defaults:

  • Listen on 127.0.0.1
  • SSH tunnel or authenticated proxy
  • Update docs/ports.md

Theory 7 — Alerting (awareness only)

Alertmanager is real production work. Today: optional single recording of “what would page me.” Do not block the gate on paging infrastructure.

Worked example — Minimal metrics module set

modules/services/observability.nix
  - prometheus listen localhost
  - node exporter localhost
docs/observability.md
  - how to tunnel
  - example PromQL: up, node_load1
  - log commands for proxy/db

Sample queries:

up
node_memory_MemAvailable_bytes

Lab 1 — Choose path

Write top of docs/observability.md:

Primary path: metrics | logs-first | both
Rationale: …

Lab 2A — Metrics path

Enable Prometheus + node exporter (Grafana optional). Rebuild.

curl -sS http://127.0.0.1:9090/-/ready
curl -sS http://127.0.0.1:9100/metrics | head

Lab 2B — Logs-first path

Create a runbook with copy-pasteable journalctl lines for:

  1. Proxy
  2. PostgreSQL
  3. sops-related activation
  4. OCI container unit if any

Add journald retention config if useful.

Lab 3 — One service deep dive

Pick proxy or postgres or node. Answer:

  • How do I know it is up?
  • How do I see last error?
  • What metric or log line changes when I break it deliberately?

Perform a controlled break/fix (stop unit; observe; start).

Lab 4 — Access path

Document SSH tunnel or authenticated proxy. Prove you can view metrics/logs without opening world-readable admin UIs.

Lab 5 — Update ports and isolation docs

Mark 9090/9100/3000 as localhost-only (if used).

Lab 6 — Capstone foreshadow

List three signals you would want before Stage VIII wipe/rebuild proof (e.g. free disk, proxy up, DB accepting connections).

Theory 8 — Retention and disk

Metrics and journal logs fill disks.

# journald examples — tune to lab disk
services.journald.extraConfig = ''
  SystemMaxUse=500M
'';

Prometheus local TSDB also needs a size budget. A lab that OOMs the disk mid-capstone is a self-own.

Theory 9 — Authn in front of Grafana

Never expose Grafana 3000 to the world bare. Patterns:

  1. Localhost + SSH tunnel
  2. Reverse proxy + SSO/basic auth
  3. VPN-only interface

Update docs/ports.md if any metrics UI is reachable remotely.

Worked example — node exporter only

services.prometheus.exporters.node = {
  enable = true;
  openFirewall = false; # scrape from localhost prometheus only
  listenAddress = "127.0.0.1";
};
services.prometheus = {
  enable = true;
  scrapeConfigs = [{
    job_name = "node";
    static_configs = [{ targets = [ "127.0.0.1:9100" ]; }];
  }];
};

Verify option names on 26.05; bind scrape targets to localhost by default.

Lab 7 — One dashboard or one journal recipe

Either:

  • Export/save a minimal Grafana dashboard JSON path, or
  • Write a docs/runbooks/service-down.md with exact journalctl filters

Ship one—not zero.

Lab 8 — Disk use snapshot

journalctl --disk-usage
du -sh /var/lib/prometheus2 2>/dev/null || du -sh /var/lib/prometheus 2>/dev/null || true

Record numbers in observability notes.

Common gotchas

Symptom / mistake What to do
Grafana/Prometheus on 0.0.0.0 public Bind localhost; add auth
Scrape target wrong Check listen addresses
Huge cardinality later Keep lab exporters few
Logs-only with no runbook Write Lab 2B
Password in flake for Grafana sops / setup procedure
Thinking Nix replaces monitoring It does not

Checkpoint

  • Observability path chosen and documented
  • Metrics stack running or logs runbook complete
  • One controlled break/fix observed
  • Admin interfaces not world-open
  • ports.md updated
  • Signals list for future ops

Commit

git add .
git commit -m "day42: observability pattern for lab slice"

Write three personal gotchas before continuing.

Tomorrow

Day 43 — Optional k3s: single-node Kubernetes only if you need it—otherwise skip cleanly with a written decision.