Day 15 — Observability, K3s & Gate III

Updated

July 30, 2025

Day 15 — Observability, K3s & Gate III

Day 42 — Observability pattern

Stage IV · ~4h
Goal: Add an observability pattern for the lab slice—either Prometheus + Grafana (or Prometheus alone) or a deliberate logs-first minimal stack—so one service is inspectable without SSH archaeology alone.

Why this day exists

Services without observability fail quietly:

  • Disk fills
  • Proxy 502s for hours
  • Postgres connections exhaust
  • ACME renewals fail

You do not need a full SRE platform. You need a repeatable pattern: what you measure, where it lives, who can access it, and how it is firewalled.

Theory 1 — Three pillars (light)

Pillar Lab approach today
Logs journald (journalctl) first-class
Metrics Prometheus node exporter + app exporters optional
Traces Optional later; not required

Pick a primary for the day: metrics stack or logs discipline. Doing both shallowly is fine; doing neither is not.

Theory 2 — Logs-first pattern (valid Stage IV)

If resources are tight:

journalctl -u caddy -b --no-pager | tail
journalctl -u postgresql -b --no-pager | tail
journalctl -p err -b --no-pager | tail -50

Declarative helpers:

{
  # ensure persistent journal if you want history across reboots
  services.journald.extraConfig = ''
    SystemMaxUse=500M
  '';
}

Document a debug runbook for proxy, DB, and sops activation units. That can satisfy the pattern if you explicitly choose logs-first in docs/observability.md.

Theory 3 — Prometheus pattern

# modules/services/prometheus.nix
{
  services.prometheus = {
    enable = true;
    port = 9090;
    listenAddress = "127.0.0.1";
    scrapeConfigs = [
      {
        job_name = "node";
        static_configs = [{
          targets = [ "127.0.0.1:9100" ];
        }];
      }
    ];
  };

  services.prometheus.exporters.node = {
    enable = true;
    port = 9100;
    listenAddress = "127.0.0.1";
  };

  # Do not open 9090/9100 to the world by default
}

Verify option names for 26.05. Access via SSH tunnel:

ssh -L 9090:127.0.0.1:9090 lab
# browse localhost:9090

Or reverse-proxy with authentication (basic auth / SSO)—never raw public Prometheus without auth.

Theory 4 — Grafana (optional layer)

{
  services.grafana = {
    enable = true;
    settings = {
      server = {
        http_addr = "127.0.0.1";
        http_port = 3000;
      };
    };
  };
}

Initial admin password: use sops / first-boot procedures—not plaintext in git. Some modules support password file options; prefer those.

Dashboard: import Node Exporter essentials or build one panel showing CPU/memory.

Theory 5 — What to scrape for your slice

Target Why
node exporter Host health
Caddy/Nginx metrics If exporter/module available
Postgres exporter Optional; more moving parts
Blackbox / probes Optional

Minimum bar: node exporter + Prometheus or excellent journald runbooks + one automated log check.

Theory 6 — Firewall and multi-tenant risk

Metrics endpoints often expose sensitive operational detail. Defaults:

  • Listen on 127.0.0.1
  • SSH tunnel or authenticated proxy
  • Update docs/ports.md

Theory 7 — Alerting (awareness only)

Alertmanager is real production work. Today: optional single recording of “what would page me.” Do not block the gate on paging infrastructure.

Worked example — Minimal metrics module set

modules/services/observability.nix
  - prometheus listen localhost
  - node exporter localhost
docs/observability.md
  - how to tunnel
  - example PromQL: up, node_load1
  - log commands for proxy/db

Sample queries:

up
node_memory_MemAvailable_bytes

Lab 1 — Choose path

Write top of docs/observability.md:

Primary path: metrics | logs-first | both
Rationale: …

Lab 2A — Metrics path

Enable Prometheus + node exporter (Grafana optional). Rebuild.

curl -sS http://127.0.0.1:9090/-/ready
curl -sS http://127.0.0.1:9100/metrics | head

Lab 2B — Logs-first path

Create a runbook with copy-pasteable journalctl lines for:

  1. Proxy
  2. PostgreSQL
  3. sops-related activation
  4. OCI container unit if any

Add journald retention config if useful.

Lab 3 — One service deep dive

Pick proxy or postgres or node. Answer:

  • How do I know it is up?
  • How do I see last error?
  • What metric or log line changes when I break it deliberately?

Perform a controlled break/fix (stop unit; observe; start).

Lab 4 — Access path

Document SSH tunnel or authenticated proxy. Prove you can view metrics/logs without opening world-readable admin UIs.

Lab 5 — Update ports and isolation docs

Mark 9090/9100/3000 as localhost-only (if used).

Lab 6 — Capstone foreshadow

List three signals you would want before Stage VIII wipe/rebuild proof (e.g. free disk, proxy up, DB accepting connections).

Theory 8 — Retention and disk

Metrics and journal logs fill disks.

# journald examples — tune to lab disk
services.journald.extraConfig = ''
  SystemMaxUse=500M
'';

Prometheus local TSDB also needs a size budget. A lab that OOMs the disk mid-capstone is a self-own.

Theory 9 — Authn in front of Grafana

Never expose Grafana 3000 to the world bare. Patterns:

  1. Localhost + SSH tunnel
  2. Reverse proxy + SSO/basic auth
  3. VPN-only interface

Update docs/ports.md if any metrics UI is reachable remotely.

Worked example — node exporter only

services.prometheus.exporters.node = {
  enable = true;
  openFirewall = false; # scrape from localhost prometheus only
  listenAddress = "127.0.0.1";
};
services.prometheus = {
  enable = true;
  scrapeConfigs = [{
    job_name = "node";
    static_configs = [{ targets = [ "127.0.0.1:9100" ]; }];
  }];
};

Verify option names on 26.05; bind scrape targets to localhost by default.

Lab 7 — One dashboard or one journal recipe

Either:

  • Export/save a minimal Grafana dashboard JSON path, or
  • Write a docs/runbooks/service-down.md with exact journalctl filters

Ship one—not zero.

Lab 8 — Disk use snapshot

journalctl --disk-usage
du -sh /var/lib/prometheus2 2>/dev/null || du -sh /var/lib/prometheus 2>/dev/null || true

Record numbers in observability notes.

Common gotchas

Symptom / mistake What to do
Grafana/Prometheus on 0.0.0.0 public Bind localhost; add auth
Scrape target wrong Check listen addresses
Huge cardinality later Keep lab exporters few
Logs-only with no runbook Write Lab 2B
Password in flake for Grafana sops / setup procedure
Thinking Nix replaces monitoring It does not

Checkpoint

  • Observability path chosen and documented
  • Metrics stack running or logs runbook complete
  • One controlled break/fix observed
  • Admin interfaces not world-open
  • ports.md updated
  • Signals list for future ops

Commit

git add .
git commit -m "day42: observability pattern for lab slice"

Write three personal gotchas before continuing.

Tomorrow

Day 43 — Optional k3s: single-node Kubernetes only if you need it—otherwise skip cleanly with a written decision.

Day 43 — Optional k3s

Stage IV · ~4h (or ~30m skip path)
Goal: Either stand up a single-node k3s for literacy or skip cleanly with a written decision—Kubernetes is not required for Stage IV gate.

Important

If you do not need k8s for work or curiosity this month, skip. A forced k3s install can destabilize a small lab (CPU, disk, conceptual overload) right before the gate.

Why this day exists

Some learners hit Stage IV and ask: “Should everything move to Kubernetes?” On NixOS the honest answer is usually no—not for a one-node lab slice with Caddy + Postgres.

This day exists to:

  1. Prevent FOMO-driven architecture
  2. Give a safe optional on-ramp if you truly need k8s APIs
  3. Practice skipping as a professional skill

Theory 1 — Decision rubric

Choose skip if… Choose k3s lite if…
Stage IV slice already works natively You must learn kubectl for a job soon
Host is small on RAM/disk You have ≥8–16 GB RAM lab resources
You have not finished secrets/proxy/DB You already finished Days 33–42 solidly
Curiosity only, no time You will actually deploy one manifest

Default recommendation: skip, write the decision, move to Day 44.

Theory 2 — What k3s is

k3s is a lightweight Kubernetes distribution. On NixOS:

# modules/services/k3s.nix — only if you opt in
{
  services.k3s = {
    enable = true;
    role = "server";
    # extraFlags = [ "--write-kubeconfig-mode=0644" ]; # understand security implications
  };
}

Options and defaults change—read current module docs for 26.05 before enabling.

Implications:

Area Effect
Ports API server, kubelet, etc.—firewall carefully
CPU/RAM Non-trivial idle cost
Storage Container images + etcd/sqlite data
Mental model YAML/Helm vs NixOS modules

Theory 3 — NixOS vs k8s configuration

Concern NixOS native k3s
System packages flake modules images inside cluster
Host firewall networking.firewall + k8s Services/NodePorts
Secrets sops-nix on host also k8s Secrets (different!)
Rollback generations cluster state is separate

Running k3s does not replace host flake discipline. You now have two control planes: NixOS + cluster.

Theory 4 — Single-node honesty

Single-node k3s is fine for learning APIs. It is not:

  • Multi-AZ HA
  • A reason to abandon PostgreSQL module for “cloud native” on one VPS without backups
  • Automatically more secure

Theory 5 — Minimal literacy tasks (if enabled)

  1. kubectl get nodes
  2. Deploy a tiny Deployment + Service
  3. Expose via host proxy or port-forward—not random NodePort to the world
  4. Delete the workload
  5. Document how to disable services.k3s and reclaim resources

Theory 6 — Skip path is first-class

Skipping is documented output:

# docs/k3s-decision.md
Decision: SKIP
Date: …
Reasons:
- Stage IV gate does not require k8s
- Host resources: …
- Will revisit if: job requirement / multi-service mesh need

That file passes the day.

Worked example — Disable / enable toggles

Prefer a role flag:

# modules/roles/server.nix
{ config, lib, ... }:
{
  # imports = lib.optionals config.lab.enableK3s [ ../services/k3s.nix ];
}

Or simply do not import k3s module. Avoid half-enabled clusters.

Lab S — Skip path (most readers)

  1. Write docs/k3s-decision.md with SKIP + reasons
  2. Skim what k3s would open (ports, disk) from docs—no install
  3. List two future triggers to revisit
  4. Commit and proceed to Day 44

Time box: 30 minutes. Do not guilt-install.

Lab K1 — Opt-in: enable k3s

Only if rubric says go:

# add module; ensure plenty of disk
sudo nixos-rebuild switch --flake .#lab
sudo systemctl status k3s
sudo k3s kubectl get nodes

Kubeconfig location per module—follow module notes. Restrict access modes; do not world-chmod casually without understanding.

Lab K2 — Tiny workload

# manifests/hello.yaml — apply via kubectl
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 1
  selector:
    matchLabels: { app: hello }
  template:
    metadata:
      labels: { app: hello }
    spec:
      containers:
        - name: hello
          image: docker.io/library/nginx:alpine
          ports:
            - containerPort: 80
kubectl apply -f manifests/hello.yaml
kubectl get pods
kubectl port-forward deploy/hello 8088:80
curl -sS http://127.0.0.1:8088/ | head

Lab K3 — Tear-down plan

Document:

kubectl delete -f manifests/hello.yaml
# disable services.k3s; rebuild; confirm resources freed

Update firewall if you opened API ports. Prefer API on localhost / VPN only.

Lab K4 — Compare control planes

Write a short note: what still belongs in the NixOS flake vs what you put in the cluster. If the answer is “everything in cluster,” push back—host still owns kernel, disks, proxy in many designs.

Theory 7 — When NixOS modules beat k8s on a single node

Workload Prefer
One reverse proxy + one DB NixOS modules (Days 38–39)
Many identical microservices with k8s API dependency k3s/k8s
GPU training cluster Often specialized schedulers
Homelab “because CNCF” Usually modules + compose/podman

NixOS is already a declarative control plane for one machine. k3s adds a second control plane—justify it.

Theory 8 — Resource floor honesty

Single-node k3s wants real RAM/CPU. On a 2 GB VM:

k3s + postgres + caddy + prometheus = thrash

Skip is the professional choice when resources are tight before Day 44 gate.

Worked example — Feature flag module

# modules/services/k3s.nix
{ config, lib, ... }:
let cfg = config.myLab.k3s; in {
  options.myLab.k3s.enable = lib.mkEnableOption "optional single-node k3s";
  config = lib.mkIf cfg.enable {
    services.k3s = {
      enable = true;
      role = "server";
      # extraFlags = [ "--disable traefik" ]; # if you keep Day 38 proxy as edge
    };
    # networking.firewall.allowedTCPPorts = [ 6443 ]; # admin API — lock down
  };
}

Default enable = false on the gate host.

Lab K5 — API surface inventory (if enabled)

sudo k3s kubectl get nodes
sudo k3s kubectl get pods -A | head
ss -tlnp | grep -E '6443|10250' || true

Add every k3s port to docs/ports.md with “cluster-internal / admin-only” labels.

Lab S2 — Skip artifact quality bar

Your skip note must include:

  1. Decision (skip)
  2. Resource or complexity reason
  3. What you use instead (NixOS modules / podman)
  4. Revisit criteria (“if job requires CKA lab…”)

A one-word “skipped” is not enough for the gate dossier.

Common gotchas

Symptom / mistake What to do
Guilt install before gate Use skip path
Opened k8s API to world Close; bind/firewall
Disk full of images Prune; disable k3s
Secrets only in k8s, host ignored Host sops still matters
Two proxies fighting One ingress story
Undocumented cluster Decision doc mandatory either way

Checkpoint

If skipped

  • docs/k3s-decision.md with SKIP + reasons
  • Future revisit triggers listed
  • No half-installed k3s units left

If enabled

  • Single-node ready; kubectl get nodes works
  • One workload applied and removed or still intentional
  • API not casually public
  • Disable/tear-down plan written
  • ports.md updated

Commit

git add .
git commit -m "day43: k3s optional — decision recorded"

Write three personal gotchas before continuing.

Tomorrow

Day 44 — Stage IV gate: prove a secret-safe lab slice—proxy + secret + data/app path + firewall story—rebuildable from flake + keys.

Day 44 — Stage IV gate

Stage IV · ~4h
Goal: Prove the Stage IV exit criteria: a lab slice that is declarative, secret-safe, firewalled, and rebuildable—proxy + secrets tooling + one app/data path—with evidence that grep finds no live secrets and a fresh VM story is clear.

Important

Gates are proof days. Do not start greenfield k3s or a second database here. Close gaps from Days 33–43.

Why this day exists

Stage IV added production-shaped risk:

Risk Your control
Secret leakage via store/git sops-nix (house default)
Network exposure firewall + localhost binds
Service sprawl one proxy pattern + one DB
“Works on my SSH session” flake rebuild + docs

Stage V packaging will add more moving parts. Exit Stage IV clean.

Theory 1 — Syllabus exit criteria

grep-able repo has no live secrets; slice rebuilds on a fresh VM from flake + keys.

Break down:

  1. No live secrets in git (encrypted OK; plaintext not)
  2. Slice = proxy + secret consumption + data or app path + firewall story
  3. Fresh VM = clone flake, place age key, generate hardware, rebuild
  4. Keys = age/sops host key + admin SSH—not passwords in the repo

Theory 2 — Slice definition (minimum)

Your lab must demonstrate:

Component Evidence
Secrets sops-nix (or declared house default) decrypts at least one secret to /run/...
Proxy / TLS Front door pattern (public ACME or documented lab TLS)
Data or app path Postgres or OCI/native app behind proxy
Firewall docs/ports.md matches reality
Hardening SSH keys-only baseline still true
Layout Still hosts/modules/homes—not a relapsed monolith

Optional: observability, containers, k3s—only if already done and stable.

Theory 3 — Evidence pack

Create docs/stage-iv-evidence.md:

# Stage IV evidence

## Architecture diagram (ASCII)
[clients] -> :443 proxy -> localhost app
                |
             postgres (localhost)
secrets: sops -> /run/secrets/...

## Commands that passed
- rg secret audit
- nix flake check
- nixos-rebuild switch
- curl proxy
- psql / app check
- sudo ls /run/secrets (no values pasted)

## Pins
- nixpkgs: …
- sops-nix: …

## Fresh VM steps
1. …
2. …

## Debt carried to later stages
- …

Theory 4 — Secret audit protocol

cd /path/to/your-flake
rg -n -i 'password\s*=\s*\"[^\"]+\"' .
rg -n -i 'BEGIN (OPENSSH|RSA|EC) PRIVATE KEY' .
rg -n -i 'api[_-]?key\s*=\s*\"' .
rg -n 'AKIA[0-9A-Z]{16}' .  # cloud key shape heuristic
# encrypted sops files should look like ciphertext, not plain tokens

Classify hits. Fix before claiming the gate.

Theory 5 — Fresh VM narrative (real or dry)

Ideal: second VM completes rebuild.

Dry-run acceptable if resource-limited—must be step-complete:

  1. Clone repo
  2. Install NixOS minimal / use existing installer path
  3. Copy/generate hardware-configuration.nix
  4. Install host age private key to configured path
  5. nixos-rebuild switch --flake .#lab
  6. Verify proxy + secret path + DB

List which secrets/keys are out of band (not in git).

Theory 6 — What Stage V will assume

You can:

  • Navigate module layout
  • Update locks selectively
  • Keep secrets out of derivations
  • Run a networked service responsibly

Packaging craft should not fight a burning host config.

Lab 1 — Architecture freeze

Draw the ASCII diagram of the actual slice (not aspirational). If something is missing, implement the smallest fix or demote it from the gate claim.

Lab 2 — Secret audit

Run Theory 4 searches. Record clean result or fixes. Do not paste secret values into the evidence file.

Lab 3 — Full rebuild

sudo nixos-rebuild switch --flake .#lab
nix flake check -L

Fix failures.

Lab 4 — Functional probes

# SSH
ssh lab true

# Proxy
curl -k -sS -o /dev/null -w '%{http_code}\n' https://localhost/ 2>/dev/null || curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1/

# Secret path exists (no cat of values)
sudo test -e /run/secrets && sudo find /run/secrets -type f | head

# DB
sudo -u postgres psql -d labapp -c 'SELECT 1;' 2>/dev/null || true

Adjust to your hostnames and paths.

Lab 5 — Firewall reconcile

ss -tulpn
# compare to docs/ports.md — fix either reality or docs

Lab 6 — Fresh VM write-up

Complete the step list in evidence. If you perform a real second VM, attach notes on time taken and issues.

Lab 7 — Gate commit / tag

git add .
git commit -m "day44: stage IV gate — secret-safe lab slice"
git tag -a stage-iv-gate -m "Stage IV exit"

Lab 8 — Personal retrospective

Three questions:

  1. What almost leaked a secret?
  2. What service would you not put on this host yet?
  3. What will you simplify before Stage V?

Theory 7 — Slice dependency graph

age key (out of band)
    → sops decrypt at activation
        → app/DB credentials
            → data service
                → app/backend
                    → reverse proxy :443
                        → clients
firewall policy wraps the graph

If any node is imperative-only, the fresh VM story fails.

Theory 8 — Evidence that greps clean

# run from flake root; tune patterns to your threat model
rg -n --hidden -g '!.git' \
  -e 'BEGIN OPENSSH PRIVATE KEY' \
  -e 'BEGIN RSA PRIVATE KEY' \
  -e 'AGE-SECRET-KEY' \
  -e 'AKIA[0-9A-Z]{16}' \
  || true

Encrypted sops blobs may still match naive patterns—know what is ciphertext vs plaintext. Live passwords and private keys are the fail condition.

Worked example — Gate eval probes

nix eval .#nixosConfigurations.lab.config.services.openssh.enable
nix eval .#nixosConfigurations.lab.config.networking.firewall.enable
# proxy enable path — adjust to caddy/nginx
nix flake check -L || true

Lab 9 — Rollback confidence after secrets

Confirm you still know:

sudo nixos-rebuild switch --rollback

Rolling back config does not roll back DB rows or rotated secrets—write that sentence in the evidence pack.

Lab 10 — ports.md final reconcile

Every listening non-localhost port must appear in docs/ports.md. Delete docs rows that no longer exist. Gate fails on fiction.

Common gotchas

Symptom / mistake What to do
Gate with plaintext “lab” passwords in git Remove + rotate; fail the gate until fixed
Proxy works; secrets demo missing Wire one sops secret consumer
ports.md fantasy Reconcile Lab 5
Monolith returned Re-apply Day 23 structure
k3s half-broken eating the day Disable; skip path; gate native slice
Evidence only oral Write docs/stage-iv-evidence.md

Checkpoint — Stage IV exit

  • sops-nix (or documented house default) E2E
  • No live plaintext secrets in repo (audit clean)
  • Hardening baseline retained
  • Firewall documented and true
  • TLS/proxy front door pattern live
  • Data service or app path declarative
  • Containers/k3s either solid or out of the claim
  • Observability at least logs runbook or metrics
  • Fresh VM steps written
  • Evidence doc committed
  • nix flake check + rebuild pass

Commit

git add .
git commit -m "day44: stage IV gate"

Write three personal gotchas before continuing.

Tomorrow

Day 45 — stdenv phases: Stage V packaging craft begins—build software with the stdenv phase model, not only consume it.