Day 29 — Capstone: Deploy, CI & Disaster Recovery

Updated

July 30, 2025

Day 29 — Capstone: Deploy, CI & Disaster Recovery

Day 86 — Capstone 4/5 — tests & operability docs

Stage VIII · ~4h (implementation)
Goal: Leave Capstone with a sufficient test suite, a RUNBOOK an exhausted future-you can follow, and architecture diagrams that match reality—operability, not feature sprawl.

Note

Baseline: NixOS 26.05, flakes, sops-nix (or documented agenix), deploy-rs or Colmena. Today hardens proof and human interface—not new roles.

Why this day exists

Code without operable docs fails Day 88. Tests without assertions that match the proxy/secrets design fail Day 89. Today closes Capstone A quality on tests + runbook + diagrams: the difference between a demo host and a rebuildable system.

Theory 1 — “Sufficient” tests for Capstone A

Minimum bar:

Test	Intent
Eval/build in CI	Configuration still composes
≥1 nixosTest or strong VM-less check	Behavioral regression
Secret-safe	No prod keys required in CI
Proxy/app path	Front door works as designed
Optional negative	Firewall deny or failed assertion fixture

Stretch (only if already green):

Multi-node client→server
Deploy dry-run / eval in CI
Dead-man checks for deprecated options

Flake outputs shape

# flake fragment — adapt to your layout
checks.x86_64-linux = {
  lab-proxy = pkgs.nixosTest (import ./tests/lab-proxy.nix {
    inherit pkgs;
  });
  # Optional pure eval / package checks:
  # cooltool = self.packages.x86_64-linux.cooltool;
};

nix flake show
nix flake check -L
nix build .#checks.x86_64-linux.lab-proxy -L

Wait discipline (Day 76 reprise)

# good
machine.wait_for_unit("myapp.service")
machine.wait_for_unit("nginx.service")
machine.wait_for_open_port(80)
machine.succeed("curl -f http://lab/")

# bad sole strategy
machine.sleep(30)

No blind sleeps as the only wait. Prefer unit + port readiness.

Theory 2 — Sharing modules between test and host

Tests that invent a second, unrelated nginx config prove little. Prefer:

# tests/lab-proxy.nix (sketch)
import ./make-test-python.nix ({ pkgs, ... }: {
  name = "lab-proxy";
  nodes.lab = {
    imports = [
      ../modules/common.nix
      ../modules/proxy.nix
      ../modules/myapp.nix
      ./fixtures/test-secrets.nix  # dummy secrets, not prod sops keys
    ];
    # test-only overrides:
    networking.hostName = "lab";
  };
  testScript = ''
    lab.wait_for_unit("multi-user.target")
    lab.wait_for_unit("myapp.service")
    lab.wait_for_open_port(80)
    lab.succeed("curl -f http://127.0.0.1/")
  '';
})

Pattern	Use when
Import real modules + test overrides	Default Capstone
Fully synthetic minimal node	Teaching isolation; weaker proof
Full host configuration.nix	Heavy; only if modules are thin

Secrets in tests: use a fixture module that drops a known token into a path, or a test age key committed for CI only—never require the production private key in GitHub Actions.

Theory 3 — RUNBOOK contents

docs/RUNBOOK.md should answer without chat history:

Purpose of the lab host (one paragraph)
Access — SSH users, keys, console/IPMI if any
Deploy — exact commands (link docs/DEPLOY.md)
Rollback — generation switch + tool-specific notes
Secrets — where keys live offline; re-encrypt steps pointer
Health probes — curl, systemctl --failed, critical units
Logs — journalctl -u … units of interest
Backup/restore — pointers (Day 69 / Day 88)
Common incidents — link Day 81 playbook entries
Upgrade — link Day 80 runbook

RUNBOOK anti-patterns

Anti-pattern	Better
“Deploy as usual”	Exact `deploy .#lab` + prerequisites
Diagrams only	Copy-paste commands
Secrets inline	Paths to key custody docs
Three conflicting READMEs	One RUNBOOK + links

Keep it boring. Future-you at 02:00 is the audience.

Theory 4 — Diagrams that stay true

Update docs/ARCHITECTURE.md so pictures match freeze candidates:

[Admin laptop]
    | deploy-rs / colmena (SSH)
    v
[lab NixOS 26.05]
    | sops-nix activation → /run/secrets/...
    | myapp.service (localhost:8080)
    v
[edge] nginx/caddy :80/:443 → proxy_pass app

[CI] → nix build / flake check → (optional) binary cache
    → substituters on admin + lab

If a diagram disagrees with code, code wins—fix the diagram today.

Also record:

Trust edge	Who/what
Cache signatures	Which public keys
SSH deploy	Which keys/users
Secret decrypt	Host age/ssh key + admin key

Theory 5 — Operability as a product feature

Artifact	Capstone A role
Tests	Prove regressions before wipe
RUNBOOK	Prove humans can operate
ARCHITECTURE	Prove design matches deploy
Playbook	Prove incidents are repeatable fixes
CAPSTONE-A.md	Prove honesty on scope

Feature count does not substitute for these. A thin host with green tests and a real runbook beats a kitchen-sink host with tribal knowledge.

Theory 6 — Assertion fixtures (optional teaching aid)

Keep a small “must fail eval” example for module assertions (Day 72), either under tests/bad/ (not wired into default flake check) or as documented snippets:

# docs/examples/bad-proxy.nix — must NOT be imported by lab
{
  # e.g. enable proxy without upstream port — triggers assertions
}

Optional advanced CI job: expect failure. Not required for Capstone A if time is short—document instead.

Worked example — Minimal proxy test script

lab.start()
lab.wait_for_unit("network-online.target")
lab.wait_for_unit("myapp.service")
lab.wait_for_unit("nginx.service")
lab.wait_for_open_port(80)
# front door
lab.succeed("curl -f http://127.0.0.1/ | grep -q ok")
# optional: upstream not exposed if firewall tested via second node

nix build .#checks.x86_64-linux.lab-proxy -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day86/lab-proxy.log
# run twice for flake hunting
nix build .#checks.x86_64-linux.lab-proxy -L

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day86
cd /path/to/capstone-flake
mkdir -p docs playbook tests
git status

Lab 1 — Inventory and gap list

nix flake show 2>&1 | tee ~/lab/90daysofx/02-nixos/day86/flake-show.txt

Write docs/TEST-SUITE.md (or section in CAPSTONE-A):

Check attr	Layer	Maps to Capstone item	Notes
lab-proxy	VM	proxy + app
…	build/eval	host composes

List gaps vs Capstone A items 6–7 (tests + docs).

Lab 2 — Test suite completion

Fill gaps: at least one behavioral check for proxy/app; keep package/eval checks if useful.
Ensure test fixtures do not need prod sops keys.
Run critical checks twice:

nix build .#checks.x86_64-linux.lab-proxy -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day86/test1.log
nix build .#checks.x86_64-linux.lab-proxy -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day86/test2.log

Wire into CI if not already (nix flake check or explicit nix build of check attrs).
If a check is flaky, fix waits—do not “hope Day 89 is lucky.”

Lab 3 — RUNBOOK.md

Write or finish docs/RUNBOOK.md per Theory 3. Include copy-paste blocks for:

# health
ssh lab 'systemctl --failed; systemctl is-active myapp nginx; curl -fsS http://127.0.0.1/ | head'

# deploy (example)
deploy .#lab
# rollback pointer: see docs/DEPLOY.md and boot generations

Rubber-duck rule: follow only the runbook to go from “SSH works” to “health green.” Note friction; fix docs, not your memory.

Lab 4 — Playbook merge + diagrams

Copy Day 81 playbook entries into repo playbook/ if they lived only under ~/lab/....
Index in playbook/README.md (symptom → file).
Refresh docs/ARCHITECTURE.md diagrams and trust edges.
Tick tests/docs rows on docs/CAPSTONE-A.md with proof paths.
List residual bugs for Day 87—no new features unless P0.

Lab 5 — Operability drill (30 min)

Simulate “forgot how this works”:

# From cold notes / repo docs only:
# 1) find deploy command
# 2) find health curl
# 3) find secret key backup location (document path; do not expose key)
# 4) find how to run the main nixosTest

Record time-to-confidence in ~/lab/90daysofx/02-nixos/day86/NOTES.md. If >15 minutes to find deploy+health, the runbook is not done.

Lab 6 — Failure injection (docs + tests)

Pick one:

Break upstream port in a branch; confirm test fails; revert.
Follow runbook rollback section on lab (generation switch); confirm app returns; note any missing steps.

Keep host green at end of day.

Common gotchas

Symptom	What to do
Tests green, prod config different	Share modules between test nodes and host
Runbook lists old hostnames	Grep docs when renaming
`flake check` too heavy for laptop	Still must pass in CI; local attr build OK
Docs in three places disagree	Single canonical RUNBOOK + links
Tests need production age key	Fixture/dummy secrets module
Sleep-only waits	`wait_for_unit` / ports
Architecture art, wrong ports	Update diagram to match nginx/app

Checkpoint

Behavioral check(s) green twice
CI runs tests or explicit check builds
docs/RUNBOOK.md complete and rubber-duck tested
Architecture diagram matches code
Playbook indexed in repo
CAPSTONE-A tests/docs rows honest with proof
Three personal gotchas

Commit

git add .
git commit -m "day86: capstone tests runbook and operability docs"

Write three personal gotchas before continuing.

Tomorrow

Day 87 — Capstone 5/5 freeze: feature freeze, bugfix only, checklist nearly green before wipe/rebuild.

Day 87 — Capstone 5/5 freeze

Stage VIII · ~4h
Goal: Feature freeze the Capstone A MVP: no new services or toys—only bugfixes, doc fixes, and checklist honesty—so Days 88–89 test a stable, tagged artifact.

Important

After today’s tag, the only legitimate code changes before Day 88 wipe are P0 blockers that make rebuild or health checks impossible. Everything else goes in docs/POST-FREEZE.md.

Why this day exists

Wipe/rebuild (Day 88) against a moving target is chaos. Freeze creates a known baseline: annotated git tag, green CI, known gaps explicitly waived with risk. Without freeze, disaster-recovery drills measure luck, not process.

Experts freeze early relative to demos. Amateurs keep “just one more module” until the disk wipe and then invent history.

Theory 1 — What “freeze” means

Allowed	Forbidden
Bugfixes for P0/P1 with issue id	New apps / node roles / hostnames
Doc corrections that match code	“Quick” plugin or package installs
Test stability fixes (flakes, waits)	Refactors without bug id
Checklist updates (honest ticks)	Starting Capstone B
Dependency bumps only if build broken	Casual `nix flake update`
Typo/typo-level RUNBOOK fixes	New sops keys without re-documenting DR

Tag shape

git tag -a capstone-a-freeze -m "Capstone A feature freeze"
git push origin capstone-a-freeze  # if remote exists
git rev-parse capstone-a-freeze
git rev-parse HEAD

Record the full commit hash in docs/CAPSTONE-A.md and docs/FREEZE.md. Annotated tags (-a) carry a message and date; lightweight tags are easier to lose meaning.

Theory 2 — Near-green checklist policy

Every Capstone A box must be one of:

[x] done with a proof command (or CI URL) written next to it, or
[ ] waived with written reason + risk + follow-up date

No silent partials. “Mostly works” is not a checkbox state.

Example waiver (copy pattern)

- [ ] CI cache push — WAIVED: no Cachix token in personal forge;
      CI still builds toplevel (`https://…/actions/runs/…`).
      Revisit: 26.11 window. Risk: slow cold CI; deploy still local-builds.

Example done-with-proof

- [x] Reverse proxy edge — proof: `curl -f http://127.0.0.1/` → 200
      from day84/proxy.log; TLS mode: lab HTTP documented in docs/TLS.md

Waivers that are usually unacceptable at freeze

Waiver	Why it fails freeze
“Secrets later”	Day 88 needs keys; Day 84 was the path
“Deploy only on the box”	Capstone A requires remote deploy story
“No tests, trust me”	Day 89 cannot verify
“CI red but laptop green”	Impurity; not frozen

Preferred-not-met items (e.g. cache push) may be waived once with risk—prefer that over lying.

Theory 3 — Bug triage before freeze

Keep docs/BUGS.md or forge issues with a short table:

ID	Sev	Symptom	Repro	Fix / defer
B01	P0	myapp fails without secret path	remove sops key	must fix pre-88
B02	P1	deploy fails second time	redeploy without GC	fix or document
B03	P2	README host typo	visual	defer post-90

Severity guide

Sev	Meaning for freeze
P0	Blocks deploy, boot, secret decrypt, or health probe
P1	Degrades ops; workaround exists; fix if time
P2	Polish / docs nits

P0 open at end of day ⇒ you are not frozen. Fix, reduce scope (disable the feature and update checklist), or postpone Day 88.

Theory 4 — Pre-DR readiness (prepare Day 88)

Before freeze ends, verify you have offline (not only on the VM you will wipe):

Asset	Why
Git remote or `git bundle`	Rebuild source of truth
`flake.lock` (in git)	Exact pins
Age/sops private keys	Decrypt secrets on rebuild
Disko / partition notes	Disk layout without guessing
Network/IP/DNS notes	Reachability after boot
Hardware/VM config (CPU, disk, firmware)	Recreate lab shape
deploy SSH keys / known_hosts policy	Optional if rebuild is local-only first
Backup repo password (Day 69) if state restore is in scope	State half of DR

# Example: offline git bundle on admin machine or encrypted USB
git bundle create /media/usb/capstone-a-freeze.bundle --all
# verify:
git clone /media/usb/capstone-a-freeze.bundle /tmp/capstone-verify

Put a sealed checklist in docs/DR-ASSETS.md: paths to key storage, not the keys themselves in git.

Warning

If your only copy of the age private key is on the lab disk you will wipe tomorrow, stop. Copy it offline today. Day 88 cannot invent lost keys.

Theory 5 — What “green” means at freeze

Minimum freeze gate (all should pass on the freeze commit):

nix flake metadata          # lock present, inputs known
nix flake check -L          # or explicit critical checks
nix build .#nixosConfigurations.<lab>.config.system.build.toplevel -L
# deploy once more if remote path is Capstone-claimed
# CI green on the same commit (or note forge lag)

Optional but strong:

nix build .#checks.x86_64-linux.<proxy-or-app-test> -L

If flake check is too heavy, document the exact attrs CI and you both run—still no red builds.

Theory 6 — Freeze documentation set

File	Role
`docs/FREEZE.md`	Date, hash, tag name, waivers summary, “no new features” rule
`docs/DR-ASSETS.md`	Offline asset locations for Day 88
`docs/DR-PLAN.md`	Ordered wipe/rebuild steps (mental rehearsal)
`docs/POST-FREEZE.md`	Parking lot for ideas after Day 90
`docs/BUGS.md`	Open/closed with severity
`docs/CAPSTONE-A.md`	Checklist done/waived with proof

FREEZE.md sketch:

# Capstone A freeze

- Date: YYYY-MM-DD
- Tag: capstone-a-freeze
- Commit: <full sha>
- Nix: <nix --version>
- nixpkgs: <from flake metadata>
- Host: lab (x86_64-linux)

## Green proof
- CI: <url>
- Local: flake check + toplevel (logs under ~/lab/.../day87/)

## Waivers
- …

## DR assets confirmed
- [x] keys offline
- [x] bundle or remote
- [x] Disko notes

Worked example — Reducing scope instead of shipping broken

You planned ACME + public DNS; lab has neither; proxy works on HTTP localhost.

Wrong freeze move: leave ACME enabled, CI red, “fix during Day 88.”
Right freeze move: set lab HTTP mode, document prod ACME path in docs/TLS.md, tick proxy with proof, waiver “public ACME” with risk “no real cert automation proven.”

Scope reduction is a feature freeze skill, not failure.

Lab 0 — Status snapshot

mkdir -p ~/lab/90daysofx/02-nixos/day87
cd /path/to/capstone-flake
git status | tee ~/lab/90daysofx/02-nixos/day87/status.txt
git log --oneline -15 | tee ~/lab/90daysofx/02-nixos/day87/log.txt
nix flake metadata | tee ~/lab/90daysofx/02-nixos/day87/metadata.txt
nix --version | tee ~/lab/90daysofx/02-nixos/day87/nix-version.txt

Confirm branch is the Capstone branch you intend to freeze (not a random experiment).

Lab 1 — Full green gate

cd /path/to/capstone-flake
nix flake check -L 2>&1 | tee ~/lab/90daysofx/02-nixos/day87/check.log
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day87/build.log

Deploy once more if remote deploy is claimed on Capstone A:

# deploy-rs
deploy .#lab 2>&1 | tee ~/lab/90daysofx/02-nixos/day87/deploy.log
# OR Colmena
# colmena apply --on lab 2>&1 | tee ~/lab/90daysofx/02-nixos/day87/deploy.log

Post-deploy health:

ssh lab 'systemctl --failed; curl -fsS http://127.0.0.1/ | head'

Fix failures only—no drive-by dependency updates.

Lab 2 — Checklist pass (no ambiguity)

Open docs/CAPSTONE-A.md. For each line:

Mark done with proof command/log path, or
Formal waiver with reason + risk + follow-up

Cross-check against Day 83–86 claims (secrets, proxy, deploy, CI, tests, runbook). If architecture diagram disagrees with code, update the diagram today—code is truth at freeze.

Lab 3 — Bug bash (timebox 90–120 min)

Sort docs/BUGS.md by severity.
Reproduce top P0/P1.
Fix or reclassify (with evidence).
Re-run checks after each fix.

When tempted to add a feature: append one line to docs/POST-FREEZE.md and stop.

# POST-FREEZE.md
- [ ] Add second vhost for metrics (idea only)
- [ ] Try Attic instead of Cachix
- [ ] Capstone B: third node

Lab 4 — DR assets offline

Confirm age/sops private keys exist off the lab disk.
Confirm git remote push works or write a bundle to USB/admin.
Fill docs/DR-ASSETS.md with locations (password manager item names, USB label)—not secret material.
Confirm Disko/partition and network notes match last known-good boot.

Optional integrity check:

# On admin machine, after cloning/bundling:
nix flake lock --no-update-lock-file
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L

Lab 5 — Tag, FREEZE note, mental rehearsal

git add docs
git status
git commit -m "day87: capstone A feature freeze documentation"
git tag -a capstone-a-freeze -m "Capstone A feature freeze"
git rev-parse capstone-a-freeze | tee ~/lab/90daysofx/02-nixos/day87/freeze-sha.txt

Write docs/FREEZE.md (Theory 6). Write docs/DR-PLAN.md: ordered steps you will run tomorrow (wipe → install/disko → clone → keys → rebuild → health), with rough time estimates. Do not wipe yet.

Lab 6 — Freeze audit (pair or rubber-duck)

Without looking at git history, answer from docs only:

What commit is frozen?
How do you decrypt secrets after wipe?
What is the one-command deploy?
What is waived and why?

If any answer needs “I just know,” fix the docs before you trust Day 88.

Common gotchas

Mistake	Fix
“One more service” at 23:00	`POST-FREEZE.md` only
Untagged mush of commits	Annotated tag + hash in FREEZE.md
Waivers without risk	Add risk + date
Keys only on the VM you’ll wipe	Offline backup now
CI red at freeze	Not frozen—fix or waive honestly
Freeze then `flake update`	Only if build is broken; retag if you must
DR plan lives only in your head	`DR-PLAN.md`
Checkbox theatre	Proof commands or explicit waiver

Checkpoint

CI/check/build green on freeze commit (or honest limited attr set)
Annotated tag capstone-a-freeze
CAPSTONE-A fully done or waived with proof/risk
docs/FREEZE.md + docs/DR-ASSETS.md + docs/DR-PLAN.md
P0 bugs closed or scope reduced
Deploy health verified once more if claimed
Three personal gotchas

Commit

git add .
git commit -m "day87: capstone A feature freeze"
# tag if not already tagged on this commit

Write three personal gotchas before continuing.

Tomorrow

Day 88 — Disaster recovery: wipe the lab disk/VM; rebuild solely from flake + keys; time the recovery against today’s freeze tag.

Day 88 — Disaster recovery wipe/rebuild

Stage VIII · ~4h (runbook execution)
Goal: Wipe the lab VM/disk and rebuild to a working Capstone A host using only git/flake + offline keys + your docs—time the recovery and fix documentation gaps that the wipe reveals.

Warning

This day destroys the lab system disk by design. Confirm target identity three times. Do not wipe a daily driver, NAS, or the disk holding your only key copies. Capstone freeze tag + NixOS 26.05-aligned installer when possible.

Why this day exists

Syllabus completeness: prove rebuildability. “It worked once” is not NixOS expertise. Day 88 is the exam for Stages II–VII—Disko, secrets, deploy, images, docs—under stress.

Teams that skip wipe drills ship folklore. You will find the missing step that only lived in your head.

Theory 1 — What “from flake + keys alone” means

Allowed inputs	Disallowed crutches
Git remote / USB bundle of repo	Old disk “just mount and copy /etc”
`flake.lock` pins	Random unpinned `nix-env` debris
Offline age/sops private keys	Re-typing secrets from memory into git
Installer ISO / nixos-anywhere / disko	Undocumented manual partitioning folklore
Your `docs/DR-PLAN.md` + RUNBOOK	Discord scrollback
Freeze tag (`capstone-a-freeze`)	“Whatever was on main yesterday”

Ideal proof: new empty disk → installer → disko or manual partitions per docs → install flake → decrypt secrets → deploy/services healthy.

Theory 2 — Recovery paths (pick what you documented)

Path A — Interactive installer + flake

Boot NixOS installer ISO (26.05-aligned if possible).
Network + git clone.
Partition (disko or scripted).
nixos-install --flake .#lab (or current equivalent).
Boot, enter keys if needed, switch/activate secrets.

Path B — nixos-anywhere / remote install

From admin machine to empty VM—if you already use it, this is excellent proof and closer to fleet reality.

Path C — Impermanence-aware

If root is ephemeral, prove “empty root + persistent datasets + flake” comes back. Still a wipe of what you claim is disposable; do not cheat by only rebooting without reset.

Path D — Image restore

If Day 62 produces golden images: rehydrate from image, then deploy day-2 config. Valid if documented as primary DR for that host class.

Theory 3 — Metrics

Record in docs/DR-REPORT.md:

Metric	Value
Start timestamp
First boot timestamp
Services healthy timestamp
Total RTO	minutes
Blockers hit	list
Doc fixes made	list
Freeze tag hash used
Installer / path used	A/B/C/D
Cache help?	yes/no + notes

Also note download vs build time. Multi-hour cold builds are still a successful DR if the runbook works—but record the pain so Capstone B can add cache.

Theory 4 — Secret restoration

Common failure: host SSH host key changes → sops age recipient mismatch.

Plan options:

Strategy	Notes
Persist host key on non-wiped medium	Reinject after install
Re-encrypt secrets to new host key	Needs admin age key
Use admin key as additional recipient always	Recommended
Separate bootstrap secret set	Lab-only; still document

If you did not set additional recipients, Day 88 will hurt—that is educational. Fix docs/SECRETS.md after.

# offline key preflight (example locations)
ls ~/offline-keys/age/  # NOT on the VM disk
# confirm sops can decrypt with admin key on admin machine

Theory 5 — Abort criteria

Abort and restore VM snapshot only if:

Wrong disk selected
Keys missing and irrecoverable mid-flight (you failed preflight)

If keys exist but docs suck: continue and improve docs—that is the point.

Situation	Action
Typo in hostname, right disk	Continue; fix docs
Realized key only lived on wiped disk	Abort if no offline copy; document failure honestly
Network down in installer	Fix network; do not invent partitions from memory

Theory 6 — What “healthy” means

Do not stop at “I can SSH.” Capstone health is RUNBOOK probes:

Probe class	Examples
System	`systemctl --failed` empty (or known)
Edge	`curl -f` proxy / ACME path
Data	App feature needing secret works
Secrets	sops decrypt path present, modes correct
Deploy identity	hostname, generation, freeze tag visible

Define healthy before wipe in DR-PLAN.md so you do not move goalposts mid-panic.

Preflight (mandatory, before wipe)

# 1) Identity
hostname; lsblk; readlink -f /dev/disk/by-id/* | head

# 2) Repo available offline/online
git -C /path/to/capstone-flake remote -v
git -C /path/to/capstone-flake rev-parse capstone-a-freeze

# 3) Keys exist OFF the VM disk
ls <offline-key-location>   # age master, etc.

# 4) Snapshot optional safety net on hypervisor (allowed)
# 5) Installer ISO downloaded
# 6) RUNBOOK + DR-PLAN printed or on second device

Sign off in docs/DR-REPORT.md: “Preflight OK at <time>.” No sign-off → no wipe.

Execution runbook

Phase 0 — Start clock

date -Is | tee ~/lab/90daysofx/02-nixos/day88/start.txt

Phase 1 — Wipe

Hypervisor: delete disk / dd / re-create empty volume—per your lab norms.

Confirm empty boot fails or installer-only boots. Photograph/screenshot identity if that helps you sleep.

Phase 2 — Install media

Boot installer. Configure network.

# example
sudo -i
# load keys onto installer env if needed (USB)
git clone <url> /tmp/capstone && cd /tmp/capstone
git checkout capstone-a-freeze

Phase 3 — Disk layout

# if disko:
# sudo nix run github:nix-community/disko -- --mode disko ./hosts/lab/disk.nix
# Prefer by-id device paths from your docs—not /dev/sda folklore
# else follow docs/RUNBOOK partitioning section exactly

Phase 4 — Install system

# illustrative — match current nixos-install flake UX on 26.05
nixos-install --flake .#lab
# set root password if required by your design

Phase 5 — First boot

Apply secrets keys (if not embedded).
nixos-rebuild switch --flake .#lab if needed.
Or deploy .#lab / Colmena from admin once SSH up.

Phase 6 — Health

systemctl --failed
curl -f …   # proxy probe from RUNBOOK
# app secret-dependent feature works
nixos-version

Phase 7 — Stop clock & report

date -Is | tee ~/lab/90daysofx/02-nixos/day88/end.txt

Fill docs/DR-REPORT.md completely. Fix every doc gap in the same study block with commits.

Lab — Post-recovery hardening of docs

For each blocker:

Reproduce the missing step in RUNBOOK/DR-PLAN.
Commit doc fix.
Optionally re-run a partial drill (re-install secrets only) if time.

Update CAPSTONE-A Day 88 row to done with RTO number.

Suggested `DR-REPORT.md` skeleton

# DR Report — Day 88

## Identity
- host:
- disk:
- freeze tag:

## Timeline
- start:
- first boot:
- healthy:
- RTO minutes:

## Path used
- A/B/C/D:

## Blockers
1. …
2. …

## Doc fixes (commits)
- …

## Secrets outcome
- …

## Honest residual risks
- …

Common gotchas

Symptom	What to do
Wiped key storage	Restore from offline backup; if none, Capstone secrets must be re-created—document failure
sops can’t decrypt	New host key; re-encrypt with admin key
disko device name differs	Use by-id paths in config
Network not up in installer	Firmware/bridge; fix before blaming Nix
Flake inputs uncached, multi-hour build	Note RTO; improve cache for next time
“I’ll wipe later”	Does not pass this day
Wrong freeze / dirty main	Checkout tag; refuse improvisation
Healthy = SSH only	Finish RUNBOOK probes

Checkpoint

Commit

git add .
git commit -m "day88: disaster recovery wipe-rebuild report and doc fixes"

Write three personal gotchas before continuing.

Tomorrow

Day 89 — Rollback & full tests: generation rollback drill + full suite green + troubleshooting checklist finalized.

Day 89 — Rollback & full tests

Stage VIII · ~4h
Goal: Prove generation rollback works after a bad change, run the full test/CI suite green, and finalize the troubleshooting checklist that ties Day 81 playbooks to Capstone reality (post-DR host on NixOS 26.05).

Note

Day 88 proved rebuild from zero. Day 89 proves you need not always go to zero: generations + tests + a crisp checklist cover the common middle ground. Do the real break—theater fails the day.

Why this day exists

Most production incidents are not “disk gone.” They are “bad deploy an hour ago.” Without rollback muscle and a short checklist, you re-image for a typo. Capstone A is incomplete until both DR and rollback are proven with logs.

Theory 1 — Rollback layers

Layer	When to use	How
Bootloader generation	Unbootable / broken after reboot	Select previous generation at boot
Runtime rollback	System boots but config bad	`nixos-rebuild switch --rollback` or activate previous profile
Git revert	Want source truth back	`git revert` / deploy freeze tag
Deploy tool	Remote	Redeploy known-good attr/tag
Full DR	Disk/keys/corruption	Day 88 path

Capstone must document which you try first for remote lab (console vs SSH).

Commands (lab)

# list generations
sudo nix-env -p /nix/var/nix/profiles/system --list-generations
ls -la /nix/var/nix/profiles/system*

# rollback switch
sudo nixos-rebuild switch --rollback
# or
sudo /nix/var/nix/profiles/system-GEN-link/bin/switch-to-configuration switch

With flakes, rebuild a known commit:

sudo nixos-rebuild switch --flake github:you/capstone?rev=<good>#lab
# or local checkout of freeze tag
sudo nixos-rebuild switch --flake .#lab

Theory 2 — Safe bad-change protocol

Note current generation number.
Introduce deliberate recoverable fault (typo unit, bad proxy upstream).
switch and confirm break.
Rollback.
Confirm health.
Re-deploy good config from git so source matches running system.

Never leave “rolled back runtime + dirty git” undocumented.

Good faults for drill	Bad faults for drill
Wrong upstream port on proxy	`rm -rf /nix`
Broken oneshot that fails activation	Disk wipe (already Day 88)
Deliberately failing health unit	Firewall lockout without console

Theory 3 — Full suite definition

“Full” means everything Capstone claims:

nix flake check -L
# explicit if check is subset:
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L
nix build .#checks.<system>.<each> -L
# CI: confirm latest main/freeze pipeline green

Post-DR host should also pass runtime probes from RUNBOOK.

Suite piece	Pass criteria
`nix flake check`	Exit 0; log saved
System toplevel build	Exit 0
CI pipeline	Green on freeze or main+fixes
Runtime probes	Same as Day 88 healthy bar

Theory 4 — Final troubleshooting checklist

Produce docs/TROUBLESHOOTING.md (one pager + links):

# Troubleshooting checklist (Capstone)

## 1. Is it eval/build?
- [ ] `nix build .#nixosConfigurations.lab.config.system.build.toplevel`
- [ ] Read error; module bisect

## 2. Is it activation/service?
- [ ] `systemctl --failed`
- [ ] `journalctl -u … -b`

## 3. Is it network?
- [ ] console access?
- [ ] `ss -lntp`; firewall; proxy

## 4. Is it secrets?
- [ ] secret path exists; mode; sops key

## 5. Rollback?
- [ ] previous generation
- [ ] redeploy freeze tag

## 6. DR?
- [ ] docs/DR-PLAN.md

See playbook/PL-*.md for worked incidents.

Tie to Day 81

If you wrote playbooks on Day 81, link each PL-* from this checklist. Capstone should not have two competing “how to debug” docs.

Theory 5 — Source of truth after rollback

profile rollback  ≠  git fixed
        │
        ▼
you must redeploy good flake so next boot/rebuild matches intent

State	Risk
Good profile, dirty bad git	Next deploy re-breaks
Good git, bad profile	Fixed by switch/deploy
Both good, untested	Still run probes

Document the post-rollback “realign” step in RUNBOOK § Rollback.

Theory 6 — Remote rollback realities

Access	Prefer
SSH still works	`switch --rollback` or redeploy
SSH dead, console lives	Bootloader generation
No console, SSH dead	Avoidable with magic rollback (deploy-rs) / out-of-band
Disk corrupted	Day 88

Practice bootloader selection once on the VM before Capstone freeze if you never have.

Lab 0 — Baseline after Day 88

mkdir -p ~/lab/90daysofx/02-nixos/day89
cd /path/to/capstone-flake
git checkout capstone-a-freeze  # or main if fast-forwarded with fixes
sudo nix-env -p /nix/var/nix/profiles/system --list-generations \
  | tee ~/lab/90daysofx/02-nixos/day89/gens-before.txt

Health probe baseline from RUNBOOK → baseline-health.txt.

Lab 1 — Inject recoverable failure

Example:

# temporarily wrong proxy upstream port in your proxy module
# or a deliberately failing systemd oneshot for the drill

sudo nixos-rebuild switch --flake .#lab 2>&1 | tee bad-switch.log
curl -vf … || true
systemctl --failed | tee failed-bad.txt

Record generation number of bad state in notes.

Lab 2 — Rollback drill

sudo nixos-rebuild switch --rollback 2>&1 | tee rollback.log
# health
curl -f …
systemctl --failed | tee failed-after-rollback.txt

If using deploy-rs/colmena, also practice redeploying freeze tag from admin machine.

Document exact commands in docs/RUNBOOK.md § Rollback (verify accuracy against what you just typed).

Lab 3 — Realign source

git status
# ensure working tree is good config (revert drill commit or reset file)
sudo nixos-rebuild switch --flake .#lab
# or deploy .#lab / colmena apply --on lab

Running system and git should agree. Record nixos-version + generation after realign.

Lab 4 — Full tests

nix flake check -L 2>&1 | tee ~/lab/90daysofx/02-nixos/day89/full-check.log

Fix any flakes introduced by Day 88 doc-only commits or drift. CI must be green.

Lab 5 — Finalize artifacts

docs/TROUBLESHOOTING.md
CAPSTONE-A Day 89 rows
Short docs/PROOF.md:

Proof	Evidence path
Deploy	day85 log / command
Secrets+proxy	day84 notes
CI/cache	CI URL
Wipe/rebuild	DR-REPORT RTO
Rollback	day89 rollback.log
Tests	full-check.log

Lab 6 — Teach-back (5 minutes)

Without notes, say out loud:

First three steps when SSH works but service is wrong
When you choose bootloader vs --rollback
When you escalate to Day 88

If you stumble, fix TROUBLESHOOTING.md until you do not.

Common gotchas

Symptom	What to do
Rollback works but flake still bad	Redeploy good git; don’t stop at profile rollback
No previous generation	You only have one—make two before drill
Boot menu rollback needed	Practice on VM with console
Tests need old host keys	Keep test dummy secrets
Skipping real break	Drill fails the day
CI green, runtime red	Runtime probes are part of “full”
Checklist too long	One page + links; details in playbooks

Checkpoint

Bad generation induced and observed
Rollback restores health
Git and running system realigned
Full nix flake check (or documented full suite) green
TROUBLESHOOTING.md + PROOF.md
CAPSTONE-A nearly complete

Commit

git add .
git commit -m "day89: rollback drill full tests troubleshooting checklist"

Write three personal gotchas before continuing.

Tomorrow

Day 90 — Retrospective & roadmap: teach-back plan, personal roadmap (26.11, PRs, Capstone B), volume close.

Day 29 — Capstone: Deploy, CI & Disaster Recovery

Day 86 — Capstone 4/5 — tests & operability docs

Why this day exists

Theory 1 — “Sufficient” tests for Capstone A

Flake outputs shape

Wait discipline (Day 76 reprise)

Theory 2 — Sharing modules between test and host

Theory 3 — RUNBOOK contents

RUNBOOK anti-patterns

Theory 4 — Diagrams that stay true

Theory 5 — Operability as a product feature

Theory 6 — Assertion fixtures (optional teaching aid)

Worked example — Minimal proxy test script

Lab 0 — Workspace

Lab 1 — Inventory and gap list

Lab 2 — Test suite completion

Lab 3 — RUNBOOK.md

Lab 4 — Playbook merge + diagrams

Lab 5 — Operability drill (30 min)

Lab 6 — Failure injection (docs + tests)

Common gotchas

Checkpoint

Commit

Tomorrow

Day 87 — Capstone 5/5 freeze

Why this day exists

Theory 1 — What “freeze” means

Freeze is a social contract with yourself

Tag shape

Theory 2 — Near-green checklist policy

Example waiver (copy pattern)

Example done-with-proof

Waivers that are usually unacceptable at freeze

Theory 3 — Bug triage before freeze

Severity guide

Theory 4 — Pre-DR readiness (prepare Day 88)

Theory 5 — What “green” means at freeze

Theory 6 — Freeze documentation set

Worked example — Reducing scope instead of shipping broken

Lab 0 — Status snapshot

Lab 1 — Full green gate

Lab 2 — Checklist pass (no ambiguity)

Lab 3 — Bug bash (timebox 90–120 min)

Lab 4 — DR assets offline

Lab 5 — Tag, FREEZE note, mental rehearsal

Lab 6 — Freeze audit (pair or rubber-duck)

Common gotchas

Checkpoint

Commit

Tomorrow

Day 88 — Disaster recovery wipe/rebuild

Why this day exists

Theory 1 — What “from flake + keys alone” means

Theory 2 — Recovery paths (pick what you documented)

Path A — Interactive installer + flake

Path B — nixos-anywhere / remote install

Path C — Impermanence-aware

Path D — Image restore

Theory 3 — Metrics

Theory 4 — Secret restoration

Theory 5 — Abort criteria

Theory 6 — What “healthy” means

Preflight (mandatory, before wipe)

Execution runbook

Phase 0 — Start clock

Phase 1 — Wipe

Phase 2 — Install media

Phase 3 — Disk layout

Phase 4 — Install system

Phase 5 — First boot

Phase 6 — Health

Phase 7 — Stop clock & report

Lab — Post-recovery hardening of docs

Suggested DR-REPORT.md skeleton

Common gotchas

Checkpoint

Commit

Tomorrow

Day 89 — Rollback & full tests

Why this day exists

Suggested `DR-REPORT.md` skeleton