Rollback and tests

Updated

July 30, 2026

Rollback and tests

Goal: Prove generation rollback works after a bad change, run the full test/CI suite green, and finalize the troubleshooting checklist that ties playbooks to Capstone reality (post-DR host on NixOS 26.05).

Note

Disaster recovery proved rebuild from zero. This chapter proves you need not always go to zero: generations + tests + a crisp checklist cover the common middle ground. Do the real break—theater fails the exercise.

Why this chapter exists

Most production incidents are not “disk gone.” They are “bad deploy an hour ago.” Without rollback muscle and a short checklist, you re-image for a typo. Capstone A is incomplete until both DR and rollback are proven with logs.

Theory 1 — Rollback layers

Layer When to use How
Bootloader generation Unbootable / broken after reboot Select previous generation at boot
Runtime rollback System boots but config bad nixos-rebuild switch --rollback or activate previous profile
Git revert Want source truth back git revert / deploy freeze tag
Deploy tool Remote Redeploy known-good attr/tag
Full DR Disk/keys/corruption Disaster recovery path

Capstone must document which you try first for remote lab (console vs SSH).

Commands (lab)

# list generations
sudo nix-env -p /nix/var/nix/profiles/system --list-generations
ls -la /nix/var/nix/profiles/system*

# rollback switch
sudo nixos-rebuild switch --rollback

# boot previous (if needed)
# use bootloader menu; or:
# sudo /nix/var/nix/profiles/system-<n>-link/bin/switch-to-configuration switch

Theory 2 — Bad change protocol

  1. Record current generation number.
  2. Introduce one deliberate bad change (firewall locks SSH only with console, or break proxy upstream).
  3. Deploy/switch.
  4. Observe failure with checklist.
  5. Rollback.
  6. Confirm health.
  7. Revert git if needed so main is clean.
Good bad-change Bad bad-change
Wrong proxy upstream rm -rf /
Typo in unit ExecStart Disk wipe
Assertion-triggering option Destroying offline keys

Theory 3 — Full suite definition

Layer Command
Local checks nix flake check -L or explicit check attrs
Toplevel nix build .#nixosConfigurations.lab.config.system.build.toplevel -L
CI Green workflow on post-DR commit
Manual probes RUNBOOK health section

All must be green after rollback drill.

Theory 4 — Troubleshooting checklist (finalize)

One page docs/TROUBLESHOOT.md (or RUNBOOK section):

1. Console access?
2. systemctl --failed
3. journalctl -p err -b
4. Generation / rollback?
5. Eval flake?
6. Secrets path permissions?
7. Firewall / ss -lntp?
8. Playbook index?

Link PL entries from internals troubleshooting.

Theory 5 — State vs generation (again)

Rollback restores system configuration, not necessarily:

  • Database contents
  • Accidental deletes under /var
  • Lost age keys

If the bad change was data, use backups—not only generations.

Concrete checklists

Rollback drill

  • Generation number before break recorded
  • Bad change applied and failed health
  • Rollback restored health
  • Log saved
  • Git left clean (or intentional fix commit)

Suite

  • flake check / tests green
  • CI green
  • Manual probes green
  • CAPSTONE-A rollback + tests rows ticked

Lab 0 — Workspace

mkdir -p ~/lab/nixos/capstone/rollback
cd /path/to/capstone-flake
# confirm post-DR host healthy

Lab 1 — Record baseline generation

sudo nix-env -p /nix/var/nix/profiles/system --list-generations \
  | tee ~/lab/nixos/capstone/rollback/generations-before.txt
readlink -f /run/current-system | tee ~/lab/nixos/capstone/rollback/current-system.txt

Lab 2 — Intentional break

Example (adjust to your stack):

# temporarily wrong upstream
# roles.proxy.upstream = "http://127.0.0.1:1";
sudo nixos-rebuild switch --flake .#lab 2>&1 | tee break.log
curl -fsS http://127.0.0.1/ || echo "expected fail"

Lab 3 — Rollback

sudo nixos-rebuild switch --rollback 2>&1 | tee ~/lab/nixos/capstone/rollback/rollback.log
# health
curl -fsS http://127.0.0.1/ | head
systemctl --failed

Acceptance: health restored without full reinstall.

Lab 4 — Git hygiene

git checkout -- path/to/broken.nix   # or revert commit
# ensure freeze tag still meaningful

Lab 5 — Full suite green

nix flake check -L 2>&1 | tee ~/lab/nixos/capstone/rollback/check.log
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L \
  2>&1 | tee ~/lab/nixos/capstone/rollback/toplevel.log

Confirm CI green if remote.

Lab 6 — Finalize troubleshooting checklist

Write docs/TROUBLESHOOT.md; link playbooks; tick CAPSTONE-A.

Lab 7 — Bootloader path (optional)

If safe, reboot and select previous generation once. Document for remote hosts with console.

Common gotchas

Mistake Fix
Theater rollback without real break Do real break
Rollback then leave bad git committed Clean source
Expecting DB undo from generation Backups
Skipping CI after DR Run suite
No generation listed Profile permissions; use sudo

Checkpoint

  • Rollback log with before/after health
  • Full suite green
  • TROUBLESHOOT checklist finalized
  • CAPSTONE-A complete for rollback+tests
  • Three personal gotchas

Commit

git add .
git commit -m "capstone: rollback drill and suite green"

Write three personal gotchas before continuing.

Next

→ Retrospective — teach plan and roadmap without rewrite thrash.