Capstone: prove rebuildability

Updated

July 30, 2026

Capstone: prove rebuildability

Goal: Leave Capstone with a sufficient test suite, a RUNBOOK an exhausted future-you can follow, and architecture diagrams that match reality—operability, not feature sprawl.

Note

Baseline: NixOS 26.05, flakes, sops-nix (or documented agenix), deploy-rs or Colmena. This chapter hardens proof and human interface—not new roles.

Why this chapter exists

Code without operable docs fails disaster recovery. Tests without assertions that match the proxy/secrets design fail rollback drills. This chapter closes Capstone A quality on tests + runbook + diagrams: the difference between a demo host and a rebuildable system.

Theory 1 — “Sufficient” tests for Capstone A

Minimum bar:

Test Intent
Eval/build in CI Configuration still composes
≥1 nixosTest or strong VM-less check Behavioral regression
Secret-safe No prod keys required in CI
Proxy/app path Front door works as designed
Optional negative Firewall deny or failed assertion fixture

Stretch (only if already green):

  • Multi-node client→server
  • Deploy dry-run / eval in CI
  • Dead-man checks for deprecated options

Flake outputs shape

# flake fragment — adapt to your layout
{
  checks.x86_64-linux.proxy = pkgs.nixosTest { /* … */ };
  # packages / apps optional
}
nix build .#checks.x86_64-linux.proxy -L
nix flake check -L

Theory 2 — RUNBOOK audience

Write for tired you at 2am after a bad switch:

Section Content
Identity Host name, role, IP/DNS
Health probes curl, systemctl, logs
Deploy One-liner + prerequisites
Rollback Generation + git
Secrets Where keys live; what not to do
Escalation Console path; snapshot restore

Avoid tutorial prose. Use numbered steps and copy-paste commands.

Theory 3 — Diagrams that match reality

Minimum diagrams (ASCII OK):

  1. Runtime: client → proxy → app → secret path
  2. Deploy: admin → tool → host activation
  3. Trust: who decrypts; which caches

If the diagram shows ACME but lab is HTTP-only, fix diagram or TLS mode.

Theory 4 — Rebuild proof steps (pre-wipe)

You are not wiping yet. Prove paper rebuildability:

Step Evidence
Fresh clone builds CI or clean machine nix build
Docs list install path DR-PLAN draft exists
Keys offline SECRETS.md offline section
stateVersion recorded ARCHITECTURE / host module
Disk layout documented disko or hardware notes

Theory 5 — Assertion fixtures

Keep at least one bad config snippet that must fail eval (from module deep dive). CI can later evaluate it expecting failure, or you re-run manually before freeze.

# expect non-zero
nix eval -f tests/bad-host.nix config 2>&1 | tee bad-host.err

Concrete checklists

Tests checklist

  • Suite inventory (SUITE.md or RUNBOOK section)
  • Critical path without blind sleep
  • Twice-green local run
  • CI runs checks or documents KVM waiver
  • Regression assertion present

Docs checklist

  • RUNBOOK complete
  • ARCHITECTURE matches runtime
  • Diagrams updated
  • CAPSTONE-A ticks with log paths

Lab 0 — Workspace

mkdir -p ~/lab/nixos/capstone/prove
cd /path/to/capstone-flake

Lab 1 — Harden tests

  1. Inventory checks.
  2. Remove sleep-only waits.
  3. Add/verify proxy + app assertions.
  4. Run twice:
nix build .#checks.<system>.<name> -L 2>&1 | tee ~/lab/nixos/capstone/prove/check1.log
nix build .#checks.<system>.<name> -L 2>&1 | tee ~/lab/nixos/capstone/prove/check2.log

Acceptance: two consecutive greens; no sole blind sleep on critical path.

Lab 2 — RUNBOOK

Write or finish docs/RUNBOOK.md with deploy, health, rollback, secrets pointers.

Read it once without the repo open on a second machine if possible—note gaps.

Lab 3 — Diagrams

Update ARCHITECTURE with three diagrams. Paste into CAPSTONE notes.

Lab 4 — Paper DR draft

Create docs/DR-PLAN.md draft (steps only; wipe later):

  1. Confirm target disk
  2. Boot installer
  3. Partition (disko command or manual)
  4. Install flake
  5. Place keys
  6. Deploy/switch
  7. Health probes
  8. Timing log template

Lab 5 — Freeze readiness review

List P0s remaining. If any block health, fix now. Else schedule freeze chapter.

Lab 6 — Negative eval fixture

Commit tests/fixtures/bad-*.nix that fails type or assertion; document expected error fingerprint.

Common gotchas

Mistake Fix
Docs describe yesterday’s layout Diff ARCHITECTURE vs flake show
Tests only curl localhost app Include front door
RUNBOOK assumes muscle memory Numbered commands
CI never runs tests Wire or waive with reason
Secret keys in CI Dummy secrets / testMode

Checkpoint

  • Sufficient tests green twice
  • RUNBOOK exhausted-you ready
  • Diagrams match reality
  • DR-PLAN draft exists
  • CAPSTONE-A updated
  • Three personal gotchas

Commit

git add .
git commit -m "capstone: tests runbook and rebuild proof docs"

Write three personal gotchas before continuing.

Next

→ Optional multi-node bar — feature freeze (and optional second node).