Disaster recovery notes

Updated

July 30, 2026

Disaster recovery notes

Goal: Wipe the lab VM/disk and rebuild to a working Capstone A host using only git/flake + offline keys + your docs—time the recovery and fix documentation gaps that the wipe reveals.

Warning

This chapter destroys the lab system disk by design. Confirm target identity three times. Do not wipe a daily driver, NAS, or the disk holding your only key copies. Capstone freeze tag + NixOS 26.05-aligned installer when possible.

Why this chapter exists

Syllabus completeness: prove rebuildability. “It worked once” is not NixOS expertise. DR is the exam for host, secrets, deploy, images, and docs—under stress.

Teams that skip wipe drills ship folklore. You will find the missing step that only lived in your head.

Theory 1 — What “from flake + keys alone” means

Allowed inputs Disallowed crutches
Git remote / USB bundle of repo Old disk “just mount and copy /etc”
flake.lock pins Random unpinned nix-env debris
Offline age/sops private keys Re-typing secrets from memory into git
Installer ISO / nixos-anywhere / disko Undocumented manual partitioning folklore
Your docs/DR-PLAN.md + RUNBOOK Chat scrollback
Freeze tag (capstone-a-freeze) “Whatever was on main yesterday”

Ideal proof: new empty disk → installer → disko or manual partitions per docs → install flake → decrypt secrets → deploy/services healthy.

Theory 2 — Recovery paths (pick what you documented)

Path A — Interactive installer + flake

  1. Boot NixOS installer ISO (26.05-aligned if possible).
  2. Network + partition (disko or manual per DR-PLAN).
  3. nixos-install --flake /path/to/flake#lab (or documented equivalent).
  4. Reboot; place keys if not on disk yet.
  5. Switch/deploy; health probes.

Path B — nixos-anywhere / remote install

Follow your ops docs; still need keys + freeze tag.

Path C — Image rehydrate

If Capstone ships generators/ISO product path, use only if documented as DR path.

Theory 3 — Timing and RTO

Record wall clock:

Phase Start End Minutes
Confirm target + backups of keys
Wipe / reinstall base
First flake build/substitute
Secrets decrypt + services up
Health green
Total

Lab RTO target: pick a number (e.g. < 2–4 hours for solo lab) and note whether you met it.

Theory 4 — What wipe proves / does not prove

Proves Does not prove
Docs completeness Multi-region HA
Key custody Perfect security
Pin/lock sufficiency Infinite disk
Deploy path after install Capstone B

Theory 5 — Data vs system

Layer DR handling
System closure Rebuild from flake
Secrets private keys Offline restore
App state (DB) Restore from backups chapter artifacts if any
Machine-id / SSH host keys May change; update sops recipients if needed

If host SSH key is a sops recipient, re-encrypt or use dedicated age key that you restore from offline.

Concrete DR checklist

Pre-wipe (mandatory)

  • Freeze tag exists and is the intended recover point
  • Offline keys verified readable
  • Repo pushed / USB clone available
  • DR-PLAN printed or on second device
  • Target disk identity confirmed three times
  • Snapshot optional safety net (know if you will refuse to use it)
  • Second machine for docs/timer

Post-rebuild health

  • hostname correct
  • SSH works
  • systemctl --failed clean
  • Secret path exists; app active
  • Proxy front door probe
  • Deploy path still works from admin

Lab 0 — Confirm freeze and keys

git rev-parse capstone-a-freeze
# open offline key medium
# read docs/DR-PLAN.md on a second device
mkdir -p ~/lab/nixos/capstone/dr

Lab 1 — Execute wipe per DR-PLAN

Follow your plan. Example shape only:

# ON INSTALLER — example placeholders
# lsblk   # confirm disk
# disko or manual partition
# nixos-install --flake github:you/capstone?ref=capstone-a-freeze#lab

Log every deviation from DR-PLAN in ~/lab/nixos/capstone/dr/DEVIATIONS.md.

Lab 2 — Time the recovery

Fill the timing table. Be honest about waiting on downloads.

Lab 3 — Secrets rehydrate

# restore age key to documented path if needed
# rebuild/switch
systemctl is-active myapp.service

If SSH host key recipients broke sops, document fix in SECRETS.md and commit on a post-DR branch carefully.

Lab 4 — Health + deploy proof

curl -fsS http://127.0.0.1/ || true
# from admin:
deploy .#lab 2>&1 | tee ~/lab/nixos/capstone/dr/redeploy.log

Lab 5 — Fix docs (the real deliverable)

Every DEVIATIONS item becomes a DR-PLAN/RUNBOOK patch. Commit doc fixes.

git checkout -b capstone/dr-doc-fixes
# edit docs
git commit -m "docs: fix DR gaps found during wipe rebuild"

Lab 6 — Optional: refuse snapshot

If you took pre-dr snapshot, either:

  • Hard mode: do not restore it; live with rebuilt system
  • Teaching mode: note that snapshot was available but unused

Document which mode you chose.

Common gotchas

Mistake Fix
Wiped wrong disk Triple-check; hypervisor snapshot on host only
Keys only on wiped disk Offline first—always
Installer channel ≠ freeze pin Prefer flake install with lock
sops fails after new host keys Dedicated age key or re-encrypt
“Fixed” by mounting old root Invalidates proof; redo
No timing log Fill table now

Checkpoint

  • Wipe executed on correct target
  • Rebuild to healthy Capstone A
  • Timing table filled
  • DEVIATIONS → doc commits
  • CAPSTONE-A DR row ticked with time
  • Three personal gotchas

Commit

git add docs/
git commit -m "docs: disaster recovery proof and gap fixes"

Write three personal gotchas before continuing.

Next

→ Rollback and tests — generation rollback + full suite green.