Day 86 — Capstone 4/5

Updated

July 30, 2026

Day 86 — Capstone 4/5 — tests & operability docs

Stage VIII · ~4h (implementation)
Goal: Leave Capstone with a sufficient test suite, a RUNBOOK an exhausted future-you can follow, and architecture diagrams that match reality—operability, not feature sprawl.

Note

Baseline: NixOS 26.05, flakes, sops-nix (or documented agenix), deploy-rs or Colmena. Today hardens proof and human interface—not new roles.

Why this day exists

Code without operable docs fails Day 88. Tests without assertions that match the proxy/secrets design fail Day 89. Today closes Capstone A quality on tests + runbook + diagrams: the difference between a demo host and a rebuildable system.

Theory 1 — “Sufficient” tests for Capstone A

Minimum bar:

Test Intent
Eval/build in CI Configuration still composes
≥1 nixosTest or strong VM-less check Behavioral regression
Secret-safe No prod keys required in CI
Proxy/app path Front door works as designed
Optional negative Firewall deny or failed assertion fixture

Stretch (only if already green):

  • Multi-node client→server
  • Deploy dry-run / eval in CI
  • Dead-man checks for deprecated options

Flake outputs shape

# flake fragment — adapt to your layout
checks.x86_64-linux = {
  lab-proxy = pkgs.nixosTest (import ./tests/lab-proxy.nix {
    inherit pkgs;
  });
  # Optional pure eval / package checks:
  # cooltool = self.packages.x86_64-linux.cooltool;
};
nix flake show
nix flake check -L
nix build .#checks.x86_64-linux.lab-proxy -L

Wait discipline (Day 76 reprise)

# good
machine.wait_for_unit("myapp.service")
machine.wait_for_unit("nginx.service")
machine.wait_for_open_port(80)
machine.succeed("curl -f http://lab/")

# bad sole strategy
machine.sleep(30)

No blind sleeps as the only wait. Prefer unit + port readiness.

Theory 2 — Sharing modules between test and host

Tests that invent a second, unrelated nginx config prove little. Prefer:

# tests/lab-proxy.nix (sketch)
import ./make-test-python.nix ({ pkgs, ... }: {
  name = "lab-proxy";
  nodes.lab = {
    imports = [
      ../modules/common.nix
      ../modules/proxy.nix
      ../modules/myapp.nix
      ./fixtures/test-secrets.nix  # dummy secrets, not prod sops keys
    ];
    # test-only overrides:
    networking.hostName = "lab";
  };
  testScript = ''
    lab.wait_for_unit("multi-user.target")
    lab.wait_for_unit("myapp.service")
    lab.wait_for_open_port(80)
    lab.succeed("curl -f http://127.0.0.1/")
  '';
})
Pattern Use when
Import real modules + test overrides Default Capstone
Fully synthetic minimal node Teaching isolation; weaker proof
Full host configuration.nix Heavy; only if modules are thin

Secrets in tests: use a fixture module that drops a known token into a path, or a test age key committed for CI only—never require the production private key in GitHub Actions.

Theory 3 — RUNBOOK contents

docs/RUNBOOK.md should answer without chat history:

  1. Purpose of the lab host (one paragraph)
  2. Access — SSH users, keys, console/IPMI if any
  3. Deploy — exact commands (link docs/DEPLOY.md)
  4. Rollback — generation switch + tool-specific notes
  5. Secrets — where keys live offline; re-encrypt steps pointer
  6. Health probes — curl, systemctl --failed, critical units
  7. Logs — journalctl -u … units of interest
  8. Backup/restore — pointers (Day 69 / Day 88)
  9. Common incidents — link Day 81 playbook entries
  10. Upgrade — link Day 80 runbook

RUNBOOK anti-patterns

Anti-pattern Better
“Deploy as usual” Exact deploy .#lab + prerequisites
Diagrams only Copy-paste commands
Secrets inline Paths to key custody docs
Three conflicting READMEs One RUNBOOK + links

Keep it boring. Future-you at 02:00 is the audience.

Theory 4 — Diagrams that stay true

Update docs/ARCHITECTURE.md so pictures match freeze candidates:

[Admin laptop]
    | deploy-rs / colmena (SSH)
    v
[lab NixOS 26.05]
    | sops-nix activation → /run/secrets/...
    | myapp.service (localhost:8080)
    v
[edge] nginx/caddy :80/:443 → proxy_pass app

[CI] → nix build / flake check → (optional) binary cache
    → substituters on admin + lab

If a diagram disagrees with code, code wins—fix the diagram today.

Also record:

Trust edge Who/what
Cache signatures Which public keys
SSH deploy Which keys/users
Secret decrypt Host age/ssh key + admin key

Theory 5 — Operability as a product feature

Artifact Capstone A role
Tests Prove regressions before wipe
RUNBOOK Prove humans can operate
ARCHITECTURE Prove design matches deploy
Playbook Prove incidents are repeatable fixes
CAPSTONE-A.md Prove honesty on scope

Feature count does not substitute for these. A thin host with green tests and a real runbook beats a kitchen-sink host with tribal knowledge.

Theory 6 — Assertion fixtures (optional teaching aid)

Keep a small “must fail eval” example for module assertions (Day 72), either under tests/bad/ (not wired into default flake check) or as documented snippets:

# docs/examples/bad-proxy.nix — must NOT be imported by lab
{
  # e.g. enable proxy without upstream port — triggers assertions
}

Optional advanced CI job: expect failure. Not required for Capstone A if time is short—document instead.

Worked example — Minimal proxy test script

lab.start()
lab.wait_for_unit("network-online.target")
lab.wait_for_unit("myapp.service")
lab.wait_for_unit("nginx.service")
lab.wait_for_open_port(80)
# front door
lab.succeed("curl -f http://127.0.0.1/ | grep -q ok")
# optional: upstream not exposed if firewall tested via second node
nix build .#checks.x86_64-linux.lab-proxy -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day86/lab-proxy.log
# run twice for flake hunting
nix build .#checks.x86_64-linux.lab-proxy -L

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day86
cd /path/to/capstone-flake
mkdir -p docs playbook tests
git status

Lab 1 — Inventory and gap list

nix flake show 2>&1 | tee ~/lab/90daysofx/02-nixos/day86/flake-show.txt

Write docs/TEST-SUITE.md (or section in CAPSTONE-A):

Check attr Layer Maps to Capstone item Notes
lab-proxy VM proxy + app
… build/eval host composes

List gaps vs Capstone A items 6–7 (tests + docs).

Lab 2 — Test suite completion

  1. Fill gaps: at least one behavioral check for proxy/app; keep package/eval checks if useful.
  2. Ensure test fixtures do not need prod sops keys.
  3. Run critical checks twice:
nix build .#checks.x86_64-linux.lab-proxy -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day86/test1.log
nix build .#checks.x86_64-linux.lab-proxy -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day86/test2.log
  1. Wire into CI if not already (nix flake check or explicit nix build of check attrs).
  2. If a check is flaky, fix waits—do not “hope Day 89 is lucky.”

Lab 3 — RUNBOOK.md

Write or finish docs/RUNBOOK.md per Theory 3. Include copy-paste blocks for:

# health
ssh lab 'systemctl --failed; systemctl is-active myapp nginx; curl -fsS http://127.0.0.1/ | head'

# deploy (example)
deploy .#lab
# rollback pointer: see docs/DEPLOY.md and boot generations

Rubber-duck rule: follow only the runbook to go from “SSH works” to “health green.” Note friction; fix docs, not your memory.

Lab 4 — Playbook merge + diagrams

  1. Copy Day 81 playbook entries into repo playbook/ if they lived only under ~/lab/....
  2. Index in playbook/README.md (symptom → file).
  3. Refresh docs/ARCHITECTURE.md diagrams and trust edges.
  4. Tick tests/docs rows on docs/CAPSTONE-A.md with proof paths.
  5. List residual bugs for Day 87—no new features unless P0.

Lab 5 — Operability drill (30 min)

Simulate “forgot how this works”:

# From cold notes / repo docs only:
# 1) find deploy command
# 2) find health curl
# 3) find secret key backup location (document path; do not expose key)
# 4) find how to run the main nixosTest

Record time-to-confidence in ~/lab/90daysofx/02-nixos/day86/NOTES.md. If >15 minutes to find deploy+health, the runbook is not done.

Lab 6 — Failure injection (docs + tests)

Pick one:

  1. Break upstream port in a branch; confirm test fails; revert.
  2. Follow runbook rollback section on lab (generation switch); confirm app returns; note any missing steps.

Keep host green at end of day.

Common gotchas

Symptom What to do
Tests green, prod config different Share modules between test nodes and host
Runbook lists old hostnames Grep docs when renaming
flake check too heavy for laptop Still must pass in CI; local attr build OK
Docs in three places disagree Single canonical RUNBOOK + links
Tests need production age key Fixture/dummy secrets module
Sleep-only waits wait_for_unit / ports
Architecture art, wrong ports Update diagram to match nginx/app

Checkpoint

  • Behavioral check(s) green twice
  • CI runs tests or explicit check builds
  • docs/RUNBOOK.md complete and rubber-duck tested
  • Architecture diagram matches code
  • Playbook indexed in repo
  • CAPSTONE-A tests/docs rows honest with proof
  • Three personal gotchas

Commit

git add .
git commit -m "day86: capstone tests runbook and operability docs"

Write three personal gotchas before continuing.

Tomorrow

Day 87 — Capstone 5/5 freeze: feature freeze, bugfix only, checklist nearly green before wipe/rebuild.