Observability patterns

Updated

July 30, 2026

Observability patterns

Goal: Add an observability pattern for the lab slice—either Prometheus + Grafana (or Prometheus alone) or a deliberate logs-first minimal stack—so one service is inspectable without SSH archaeology alone.

Why this chapter exists

Services without observability fail quietly:

  • Disk fills
  • Proxy 502s for hours
  • Postgres connections exhaust
  • ACME renewals fail

You do not need a full SRE platform. You need a repeatable pattern: what you measure, where it lives, who can access it, and how it is firewalled.

Theory 1 — Three pillars (light)

Pillar Lab approach
Logs journald (journalctl) first-class
Metrics Prometheus node exporter + app exporters optional
Traces Optional later; not required

Pick a primary: metrics stack or logs discipline. Doing both shallowly is fine; doing neither is not.

Theory 2 — Logs-first pattern (valid)

If resources are tight:

journalctl -u caddy -b --no-pager | tail
journalctl -u postgresql -b --no-pager | tail
journalctl -p err -b --no-pager | tail -50

Declarative helpers:

{
  services.journald.extraConfig = ''
    SystemMaxUse=500M
  '';
}

Document a debug runbook for proxy, DB, and sops activation units. That can satisfy the pattern if you explicitly choose logs-first in docs/observability.md.

Theory 3 — Prometheus pattern

# modules/services/prometheus.nix
{
  services.prometheus = {
    enable = true;
    port = 9090;
    listenAddress = "127.0.0.1";
    scrapeConfigs = [
      {
        job_name = "node";
        static_configs = [{
          targets = [ "127.0.0.1:9100" ];
        }];
      }
    ];
  };

  services.prometheus.exporters.node = {
    enable = true;
    port = 9100;
    listenAddress = "127.0.0.1";
    openFirewall = false;
  };
}

Verify option names for 26.05. Access via SSH tunnel:

ssh -L 9090:127.0.0.1:9090 lab
# browse localhost:9090

Or reverse-proxy with authentication—never raw public Prometheus without auth.

Theory 4 — Grafana (optional layer)

{
  services.grafana = {
    enable = true;
    settings = {
      server = {
        http_addr = "127.0.0.1";
        http_port = 3000;
      };
    };
  };
}

Initial admin password: use sops / first-boot procedures—not plaintext in git. Prefer password-file options when available.

Theory 5 — What to scrape for your slice

Target Why
node exporter Host health
Caddy/Nginx metrics If exporter/module available
Postgres exporter Optional; more moving parts
Blackbox / probes Optional

Minimum bar: node exporter + Prometheus or excellent journald runbooks + one automated log check.

Sample queries:

up
node_memory_MemAvailable_bytes
node_load1

Theory 6 — Firewall and multi-tenant risk

Metrics endpoints often expose sensitive operational detail. Defaults:

  • Listen on 127.0.0.1
  • SSH tunnel or authenticated proxy
  • Update docs/ports.md

Theory 7 — Alerting (awareness only)

Alertmanager is real production work. Optional: record “what would page me.” Do not block on paging infrastructure.

Theory 8 — Retention and disk

Metrics and journal logs fill disks.

services.journald.extraConfig = ''
  SystemMaxUse=500M
'';

Prometheus local TSDB also needs a size budget. A lab that fills the disk mid-capstone is a self-own.

journalctl --disk-usage
du -sh /var/lib/prometheus2 2>/dev/null || du -sh /var/lib/prometheus 2>/dev/null || true

Theory 9 — Authn in front of Grafana

Never expose Grafana 3000 to the world bare. Patterns:

  1. Localhost + SSH tunnel
  2. Reverse proxy + SSO/basic auth
  3. VPN-only interface

Worked example — Minimal metrics module set

modules/services/observability.nix
  - prometheus listen localhost
  - node exporter localhost
docs/observability.md
  - how to tunnel
  - example PromQL: up, node_load1
  - log commands for proxy/db

Worked example — Runbook skeleton

# docs/runbooks/service-down.md

## Proxy 502
journalctl -u caddy -b --no-pager | tail -100
systemctl status lab-backend

## Postgres
journalctl -u postgresql -b --no-pager | tail -100
sudo -u postgres psql -c 'SELECT 1;'

## Secrets activation
journalctl -b --no-pager | rg -i sops | tail

Exercises

Exercise 1 — Choose path

Write top of docs/observability.md:

Primary path: metrics | logs-first | both
Rationale: …

Exercise 2A — Metrics path

Enable Prometheus + node exporter (Grafana optional). Rebuild.

curl -sS http://127.0.0.1:9090/-/ready
curl -sS http://127.0.0.1:9100/metrics | head

Exercise 2B — Logs-first path

Create a runbook with copy-pasteable journalctl lines for:

  1. Proxy
  2. PostgreSQL
  3. sops-related activation
  4. OCI container unit if any

Add journald retention config if useful.

Exercise 3 — One service deep dive

Pick proxy or postgres or node. Answer:

  • How do I know it is up?
  • How do I see last error?
  • What metric or log line changes when I break it deliberately?

Perform a controlled break/fix (stop unit; observe; start).

Exercise 4 — Access path

Document SSH tunnel or authenticated proxy. Prove you can view metrics/logs without opening world-readable admin UIs.

Exercise 5 — Update ports and isolation docs

Mark 9090/9100/3000 as localhost-only (if used).

Exercise 6 — Capstone foreshadow

List three signals you would want before a wipe/rebuild proof (e.g. free disk, proxy up, DB accepting connections).

Exercise 7 — One dashboard or one journal recipe

Either export a minimal Grafana dashboard path or ship docs/runbooks/service-down.md.

Exercise 8 — Disk use snapshot

journalctl --disk-usage
du -sh /var/lib/prometheus2 2>/dev/null || true

Record numbers in observability notes.

Exercise 9 — PromQL basics (metrics path)

In Prometheus UI: run up and one node metric. Screenshot optional; notes required.

Exercise 10 — Password hygiene for Grafana

If Grafana enabled, confirm no admin password in git.

Exercise 11 — Unit list for the slice

systemctl list-units --type=service --state=running | rg -i 'caddy|nginx|postgres|prometheus|grafana|podman' || true

Exercise 12 — Error priority journal

journalctl -p err -b --no-pager | tail -30

Exercise 13 — Tunnel muscle memory

ssh -L 9090:127.0.0.1:9090 lab

Common gotchas

Symptom / mistake What to do
Grafana/Prometheus on 0.0.0.0 public Bind localhost; add auth
Scrape target wrong Check listen addresses
Huge cardinality later Keep lab exporters few
Logs-only with no runbook Write runbook
Password in flake for Grafana sops / setup procedure
Thinking Nix replaces monitoring It does not
Disk full of journals/TSDB Cap retention
No break/fix Do Exercise 3
Metrics without ports.md Update docs

Checkpoint

  • Observability path chosen and documented
  • Metrics stack running or logs runbook complete
  • One controlled break/fix observed
  • Admin interfaces not world-open
  • ports.md updated
  • Signals list for future ops
  • Disk usage noted
  • Access path (tunnel/auth) documented

Journal (optional)

Write three personal gotchas before continuing.