Capstone and recovery

Updated

July 30, 2026

Capstone and recovery

This part of the standalone NixOS book is organized by topic (normal book chapters), not by a calendar. Complete Capstone A before optional multi-node work. Study pacing is yours.

Prerequisites. Architecture review complete (ARCHITECTURE.md, GAPS.md, seeded CAPSTONE-A.md). You can rebuild a flake host, manage secrets without store leakage, and run at least a stub flake check. Internals literacy (eval, modules, store, tests) is assumed when debugging Capstone failures.

What this part is for. Capstone A is not “another service.” It is proof that your repository rebuilds a real host role end-to-end: secrets, edge, deploy, tests, CI, wipe/rebuild, and rollback—with documentation an exhausted future-you can follow.

Capstone A vs optional bar

Track Required? Bar
Capstone A Yes Single host role (or primary lab guest) with full checklist
Multi-node / freeze discipline Strongly recommended before DR Feature freeze + tag; optional second node only if A green
Capstone B ideas No Fleet patterns—park in retrospective roadmap
Important

Capstone A (required): flake host · HM or service users · sops/agenix · reverse proxy · deploy-rs/colmena · nixosTest · CI + cache preferred · wipe/rebuild · rollback + green suite · architecture + runbook docs.

Learning outcomes

When you leave this part you should be able to:

  1. Close P0 gaps into a building host skeleton with honest checklist ticks.
  2. Wire secrets E2E and a reverse-proxy front door without plaintext secrets in git.
  3. Deploy remotely and run CI that builds (and preferably caches) Capstone artifacts.
  4. Freeze a tagged baseline with known waivers.
  5. Wipe and rebuild from flake + offline keys alone; time the recovery.
  6. Roll back a bad generation and keep the full test suite green.
  7. Write a retrospective and teach plan without starting a rewrite.

Chapter map

Chapter Theme Deliverable
Scope and plan Close structural P0s Green host + users + module hygiene
Implement Secrets + proxy E2E Encrypted secrets + front door proof
Harden and document Deploy + CI + cache One-command deploy + CI green
Prove rebuildability Tests + runbook + diagrams Sufficient suite + RUNBOOK
Optional multi-node bar Feature freeze (+ optional node) Annotated freeze tag
Disaster recovery Wipe + rebuild timed DR log + fixed docs
Rollback and tests Generation rollback + suite Rollback log + green CI
Retrospective Teach + roadmap Retro notes; no rewrite

Suggested sequence

Architecture review (internals)
        │
        ▼
  Scope & plan  →  Implement secrets/proxy
        │
        ▼
  Harden deploy/CI  →  Prove tests/runbook
        │
        ▼
  Freeze tag  →  Wipe/rebuild DR
        │
        ▼
  Rollback + full suite  →  Retrospective

Do not wipe before freeze. Do not start multi-region mesh mid-Capstone A.

Dossier structure (canonical)

Keep these paths (or equivalent) current throughout Capstone:

docs/
  ARCHITECTURE.md
  CAPSTONE-A.md          # live checklist
  GAPS.md                # remaining / waived
  RUNBOOK.md             # deploy, rollback, health
  SECRETS.md             # key custody
  TLS.md                 # ACME vs lab mode
  DEPLOY.md              # remote path
  DR-PLAN.md             # wipe/rebuild steps
  TRUST.md               # caches, trusted-users
  POST-FREEZE.md         # ideas deferred after freeze
  playbooks/             # PL-*.md from drills
  RETRO.md               # retrospective
Doc Owner chapter
CAPSTONE-A.md All; start in scope
SECRETS / TLS Implement
DEPLOY / CI notes Harden
RUNBOOK / diagrams Prove rebuild
Freeze tag notes Multi-node / freeze
DR-PLAN + timing Disaster recovery
Rollback log Rollback and tests
RETRO Retrospective

Live Capstone A checklist (copy into repo)

# CAPSTONE-A.md
- [ ] Host role rebuilds from flake
- [ ] HM user OR service user story
- [ ] sops-nix OR agenix wired; service reads secret
- [ ] Reverse proxy edge with TLS or lab HTTP mode documented
- [ ] deploy-rs OR colmena path works from admin machine
- [ ] nixosTest/check catches a real regression
- [ ] CI builds toplevel (and checks)
- [ ] CI pushes to binary cache (preferred)
- [ ] ARCHITECTURE.md current
- [ ] RUNBOOK.md for deploy/rollback
- [ ] Feature freeze tag (recommended before wipe)
- [ ] Wipe + rebuild timed from flake + offline keys
- [ ] Rollback drill + suite green

Tick only with proof commands recorded (build log path, curl, deploy log).

Lab workspace convention

mkdir -p ~/lab/nixos/capstone/{scope,implement,harden,prove,freeze,dr,rollback,retro}

Prefer a single Capstone git branch (capstone-a) over calendar-named branches.

Acceptance bar for Capstone A

  • Every required checklist row has a demo command
  • Secrets never appear as Nix strings in the store graph
  • Front door health probe documented
  • Remote deploy works from admin machine (or documented exception with local-only waiver)
  • At least one behavioral check fails when you break a module
  • DR wipe/rebuild timed with doc fixes committed
  • Rollback proven after intentional bad change
  • Freeze tag (or explicit waiver why not) before DR

Elitebook / KVM lab notes

Concern Capstone guidance
16 GB RAM Capstone A on one primary guest; optional second node only if headroom
Snapshots Snapshot before implement breaks, freeze, and DR
Console Required before firewall / wipe experiments
Disk Free space for two system closures + installer ISO
KVM Needed for nixosTest in prove/rollback chapters

Policy reminders

Policy Rule
Pin nixpkgs 26.05 for Capstone A
Store Default input-addressed; no experimental CA on Capstone
Secrets Offline private key backup before wipe
Scope Non-goals stay non-goals until retrospective roadmap
Day framing None—these are topic chapters

Next

→ Capstone: scope and plan — close structural P0s and make the checklist real.