Day 88 — Disaster recovery wipe/rebuild

Updated

July 30, 2026

Day 88 — Disaster recovery wipe/rebuild

Stage VIII · ~4h (runbook execution)
Goal: Wipe the lab VM/disk and rebuild to a working Capstone A host using only git/flake + offline keys + your docs—time the recovery and fix documentation gaps that the wipe reveals.

Warning

This day destroys the lab system disk by design. Confirm target identity three times. Do not wipe a daily driver, NAS, or the disk holding your only key copies. Capstone freeze tag + NixOS 26.05-aligned installer when possible.

Why this day exists

Syllabus completeness: prove rebuildability. “It worked once” is not NixOS expertise. Day 88 is the exam for Stages II–VII—Disko, secrets, deploy, images, docs—under stress.

Teams that skip wipe drills ship folklore. You will find the missing step that only lived in your head.

Theory 1 — What “from flake + keys alone” means

Allowed inputs Disallowed crutches
Git remote / USB bundle of repo Old disk “just mount and copy /etc”
flake.lock pins Random unpinned nix-env debris
Offline age/sops private keys Re-typing secrets from memory into git
Installer ISO / nixos-anywhere / disko Undocumented manual partitioning folklore
Your docs/DR-PLAN.md + RUNBOOK Discord scrollback
Freeze tag (capstone-a-freeze) “Whatever was on main yesterday”

Ideal proof: new empty disk → installer → disko or manual partitions per docs → install flake → decrypt secrets → deploy/services healthy.

Theory 2 — Recovery paths (pick what you documented)

Path A — Interactive installer + flake

  1. Boot NixOS installer ISO (26.05-aligned if possible).
  2. Network + git clone.
  3. Partition (disko or scripted).
  4. nixos-install --flake .#lab (or current equivalent).
  5. Boot, enter keys if needed, switch/activate secrets.

Path B — nixos-anywhere / remote install

From admin machine to empty VM—if you already use it, this is excellent proof and closer to fleet reality.

Path C — Impermanence-aware

If root is ephemeral, prove “empty root + persistent datasets + flake” comes back. Still a wipe of what you claim is disposable; do not cheat by only rebooting without reset.

Path D — Image restore

If Day 62 produces golden images: rehydrate from image, then deploy day-2 config. Valid if documented as primary DR for that host class.

Theory 3 — Metrics

Record in docs/DR-REPORT.md:

Metric Value
Start timestamp
First boot timestamp
Services healthy timestamp
Total RTO minutes
Blockers hit list
Doc fixes made list
Freeze tag hash used
Installer / path used A/B/C/D
Cache help? yes/no + notes

Also note download vs build time. Multi-hour cold builds are still a successful DR if the runbook works—but record the pain so Capstone B can add cache.

Theory 4 — Secret restoration

Common failure: host SSH host key changes → sops age recipient mismatch.

Plan options:

Strategy Notes
Persist host key on non-wiped medium Reinject after install
Re-encrypt secrets to new host key Needs admin age key
Use admin key as additional recipient always Recommended
Separate bootstrap secret set Lab-only; still document

If you did not set additional recipients, Day 88 will hurt—that is educational. Fix docs/SECRETS.md after.

# offline key preflight (example locations)
ls ~/offline-keys/age/  # NOT on the VM disk
# confirm sops can decrypt with admin key on admin machine

Theory 5 — Abort criteria

Abort and restore VM snapshot only if:

  • Wrong disk selected
  • Keys missing and irrecoverable mid-flight (you failed preflight)

If keys exist but docs suck: continue and improve docs—that is the point.

Situation Action
Typo in hostname, right disk Continue; fix docs
Realized key only lived on wiped disk Abort if no offline copy; document failure honestly
Network down in installer Fix network; do not invent partitions from memory

Theory 6 — What “healthy” means

Do not stop at “I can SSH.” Capstone health is RUNBOOK probes:

Probe class Examples
System systemctl --failed empty (or known)
Edge curl -f proxy / ACME path
Data App feature needing secret works
Secrets sops decrypt path present, modes correct
Deploy identity hostname, generation, freeze tag visible

Define healthy before wipe in DR-PLAN.md so you do not move goalposts mid-panic.

Preflight (mandatory, before wipe)

# 1) Identity
hostname; lsblk; readlink -f /dev/disk/by-id/* | head

# 2) Repo available offline/online
git -C /path/to/capstone-flake remote -v
git -C /path/to/capstone-flake rev-parse capstone-a-freeze

# 3) Keys exist OFF the VM disk
ls <offline-key-location>   # age master, etc.

# 4) Snapshot optional safety net on hypervisor (allowed)
# 5) Installer ISO downloaded
# 6) RUNBOOK + DR-PLAN printed or on second device

Sign off in docs/DR-REPORT.md: “Preflight OK at <time>.” No sign-off → no wipe.

Execution runbook

Phase 0 — Start clock

date -Is | tee ~/lab/90daysofx/02-nixos/day88/start.txt

Phase 1 — Wipe

Hypervisor: delete disk / dd / re-create empty volume—per your lab norms.

Confirm empty boot fails or installer-only boots. Photograph/screenshot identity if that helps you sleep.

Phase 2 — Install media

Boot installer. Configure network.

# example
sudo -i
# load keys onto installer env if needed (USB)
git clone <url> /tmp/capstone && cd /tmp/capstone
git checkout capstone-a-freeze

Phase 3 — Disk layout

# if disko:
# sudo nix run github:nix-community/disko -- --mode disko ./hosts/lab/disk.nix
# Prefer by-id device paths from your docs—not /dev/sda folklore
# else follow docs/RUNBOOK partitioning section exactly

Phase 4 — Install system

# illustrative — match current nixos-install flake UX on 26.05
nixos-install --flake .#lab
# set root password if required by your design

Phase 5 — First boot

  1. Apply secrets keys (if not embedded).
  2. nixos-rebuild switch --flake .#lab if needed.
  3. Or deploy .#lab / Colmena from admin once SSH up.

Phase 6 — Health

systemctl --failed
curl -f …   # proxy probe from RUNBOOK
# app secret-dependent feature works
nixos-version

Phase 7 — Stop clock & report

date -Is | tee ~/lab/90daysofx/02-nixos/day88/end.txt

Fill docs/DR-REPORT.md completely. Fix every doc gap in the same study block with commits.

Lab — Post-recovery hardening of docs

For each blocker:

  1. Reproduce the missing step in RUNBOOK/DR-PLAN.
  2. Commit doc fix.
  3. Optionally re-run a partial drill (re-install secrets only) if time.

Update CAPSTONE-A Day 88 row to done with RTO number.

Suggested DR-REPORT.md skeleton

# DR Report — Day 88

## Identity
- host:
- disk:
- freeze tag:

## Timeline
- start:
- first boot:
- healthy:
- RTO minutes:

## Path used
- A/B/C/D:

## Blockers
1. …
2. …

## Doc fixes (commits)
- …

## Secrets outcome
- …

## Honest residual risks
- …

Common gotchas

Symptom What to do
Wiped key storage Restore from offline backup; if none, Capstone secrets must be re-created—document failure
sops can’t decrypt New host key; re-encrypt with admin key
disko device name differs Use by-id paths in config
Network not up in installer Firmware/bridge; fix before blaming Nix
Flake inputs uncached, multi-hour build Note RTO; improve cache for next time
“I’ll wipe later” Does not pass this day
Wrong freeze / dirty main Checkout tag; refuse improvisation
Healthy = SSH only Finish RUNBOOK probes

Checkpoint

  • Preflight signed
  • Disk actually wiped
  • System rebuilt from freeze tag + keys
  • Health probes pass
  • RTO recorded in DR-REPORT.md
  • Doc gaps fixed and committed
  • CAPSTONE-A Day 88 ticked

Commit

git add .
git commit -m "day88: disaster recovery wipe-rebuild report and doc fixes"

Write three personal gotchas before continuing.

Tomorrow

Day 89 — Rollback & full tests: generation rollback drill + full suite green + troubleshooting checklist finalized.