Day 23 — GC, Secure Boot, Backups & Gate V

Updated

July 30, 2025

Day 23 — GC, Secure Boot, Backups & Gate V

Day 67 — GC & store hygiene

Stage VI · ~4h
Goal: Measure store size, explain GC roots, set a retention policy for generations and CI, and run optimise/GC safely without deleting live systems.

Why this day exists

Deploy + CI + packaging fill disks silently. Out-of-disk mid-activate is a self-inflicted outage. Hygiene is an operational skill equal to deploy tooling—Capstone hosts die from full /nix as often as from bad modules.

Theory 1 — What GC deletes

Garbage collection deletes store paths that are not reachable from any GC root.

Root examples Keeps alive
/run/current-system Running NixOS closure
/nix/var/nix/profiles/* System generations, user profiles
~/.local/state/nix/profiles Modern user profiles
/nix/var/nix/gcroots/* Explicit pins
Result symlinks in cwd ./result from nix build
Indirect roots / registries Various tooling pins 
nix-store --gc --print-roots | head -n 50
nix-collect-garbage --dry-run
nix store gc --dry-run  # modern CLI shape; flags evolve—check --help

Mental model

GC root → profile/generation/result link → closure of store paths
Anything not reachable is eligible for deletion

Deleting a project directory does not free the store if roots still point at builds.

Theory 2 — Generations vs store GC

# List system generations
sudo nix-env -p /nix/var/nix/profiles/system --list-generations
# or inspect profile links
ls -l /nix/var/nix/profiles/system*

# Delete old generations then GC
sudo nix-collect-garbage --delete-older-than 14d
# aggressive: remove all old gens of profiles
# sudo nix-collect-garbage -d
Command Effect
nix-collect-garbage GC unreachable paths
--delete-older-than Drop old gens then GC
-d Aggressive old gen removal
nix store gc Modern CLI GC
nix store optimise Hardlink identical files
Important

Deleting generations removes rollback options. Keep enough to survive a bad deploy week—especially before Capstone freeze and DR drills.

User vs system profiles

Home-Manager and user nix profile installs have their own generation chains. System GC policy does not automatically equal user hygiene.

nix profile list 2>/dev/null || true
ls ~/.local/state/nix/profiles 2>/dev/null || true

Theory 3 — Policy template

Host class Generation retention Notes
Laptop 7–14 days Disk pressure
Prod VM 14–30 days + keep last N Rollback window
CI runner Daily GC; few profiles Ephemeral preferred
Builder Weekly GC; keep hot cache set Balance disk vs rebuild
Capstone lab ≥14 days through Day 89 Rollback drills need gens 
# NixOS automatic GC example
{ ... }:
{
  nix.gc = {
    automatic = true;
    dates = "weekly";
    options = "--delete-older-than 14d";
  };
  nix.optimise.automatic = true;
  # proactive free-space GC thresholds (bytes)
  nix.settings.min-free = 10 * 1024 * 1024 * 1024; # 10 GiB
  nix.settings.max-free = 50 * 1024 * 1024 * 1024;
}

min-free triggers proactive GC when disk is low—excellent on small roots and cloud boxes.

Theory 4 — Finding what holds space

df -h /nix
nix path-info -Sh /run/current-system
nix path-info -rS /run/current-system | sort -k2 -n | tail -n 20

# Why is path alive?
nix-store --query --roots /nix/store/…-something
# Old result links (common disk vampires)
find "$HOME" -maxdepth 3 -type l \( -name result -o -name 'result-*' \) 2>/dev/null
# Largest store paths (approx; tools vary)
nix path-info -rS /run/current-system | sort -k2 -h | tail

CI-specific roots

Runners accumulate result links and old profiles if jobs are not ephemeral. Prefer clean workspaces per job; document GC for self-hosted runners in docs/CI.md.

Theory 6 — Incident patterns

Symptom First checks
No space left on device during build df -h /nix /; roots; old gens; result links
Activate failed mid-switch Free space; incomplete profile; repair with boot generation
GC “deleted my package” Missing root; rebuild; avoid bare -d in prod
Rollback missing Generations deleted by policy
Disk full only on CI Self-hosted runner hygiene

Never run destructive GC on prod without:

  1. Confirming current-system root
  2. Confirming retention window
  3. Preferring --delete-older-than over blind -d

Worked example — hygiene report script

#!/usr/bin/env bash
# scripts/store-report.sh
set -euo pipefail
echo "== disk =="
df -h /nix 2>/dev/null || df -h /
echo "== current system closure =="
nix path-info -Sh /run/current-system 2>/dev/null || true
echo "== gc roots (sample) =="
nix-store --gc --print-roots 2>/dev/null | head -n 40 || true
echo "== result symlinks =="
find "$HOME" -maxdepth 2 -type l -name 'result*' 2>/dev/null || true
echo "== system generations =="
sudo nix-env -p /nix/var/nix/profiles/system --list-generations 2>/dev/null | tail -n 15 || true
chmod +x scripts/store-report.sh
./scripts/store-report.sh | tee ~/lab/90daysofx/02-nixos/day67/before.txt

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day67

Step 1 — Baseline metrics

Record df -h /nix (or /) and current-system size via nix path-info -Sh.

Step 2 — Map roots

Print roots; identify unexpected result links and old profiles. Screenshot or tee a sample.

Step 3 — Safe cleanup

Remove stale ./result from lab dirs; GC without aggressive -d first:

nix store gc
df -h /nix

Step 4 — Generation policy

List generations; delete only those outside your written policy window.

sudo nix-collect-garbage --delete-older-than 14d

Step 5 — Enable automatic GC on lab host

Commit nix.gc (+ optional min-free / optimise) settings; deploy or switch.

Step 6 — optimise

Run optimise; record space reclaimed if measurable.

df -h /nix | tee ~/lab/90daysofx/02-nixos/day67/before-optimise.txt
nix store optimise
df -h /nix | tee ~/lab/90daysofx/02-nixos/day67/after-optimise.txt

Step 7 — CI runner note

Add GC step for self-hosted runners or rely on ephemeral runners; document in docs/CI.md.

Step 8 — Policy doc

Write docs/STORE-HYGIENE.md (or section in RUNBOOK):

Host Retention Auto GC Notes
lab 14d weekly
laptop 7d weekly

Step 9 — Rollback still possible?

Confirm at least one previous generation remains on lab after cleanup. If not, you over-pruned—adjust policy.

Step 10 — Profile audit (user + HM)

# User profiles / modern nix profile
nix profile list 2>/dev/null | tee ~/lab/90daysofx/02-nixos/day67/user-profile.txt || true
# Home-Manager generations if present
ls -la /nix/var/nix/profiles/per-user/"$USER"/ 2>/dev/null || true
ls -la ~/.local/state/home-manager 2>/dev/null || true

Write one sentence: how user-profile retention differs from system nix.gc on this host.

Step 11 — Dry-run before aggressive delete

# Prefer dry-run / list before destroy
nix-collect-garbage --delete-older-than 30d --dry-run 2>&1 \
  | tee ~/lab/90daysofx/02-nixos/day67/gc-dry-run.txt || true

If dry-run is unavailable on your Nix version, list generations explicitly and delete only named old gens.

Worked example — min-free incident story

Disk at 98% on small cloud root:
  1) df -h /nix
  2) remove stale result links in ~ and CI workspaces
  3) nix-collect-garbage --delete-older-than 7d   # laptop policy, not prod
  4) enable nix.settings.min-free so next time GC starts earlier
  5) consider separate /nix partition next reprovision (Disko)

Record whether you hit this during Stage V/VI. Capstone hosts on 20 GiB disks fail predictably without policy.

Common gotchas

Symptom / mistake What to do
GC “does nothing” Roots still hold paths
Deleted all rollbacks Stop bare -d in prod without policy
/nix on small partition Separate partition; min-free
optimise confuses backups Hardlinks; tool configuration
Multi-user permissions GC as root for system profiles
HM gens forgotten User profile hygiene
Full disk mid-CI Ephemeral runners; job GC; slimmer checks

Checkpoint

  • Measured store before/after
  • Explained roots with real examples from your machine
  • Retention policy written and applied on lab
  • optimise run once
  • Auto GC or min-free configured
  • At least one rollback generation still present
  • User/HM profile note written
  • docs/STORE-HYGIENE.md (or RUNBOOK section) committed
  • Three personal gotchas

Commit

git add .
git commit -m "day67: GC policy and store hygiene"

Write three personal gotchas before continuing.

Tomorrow

Day 68 — Secure Boot / TPM path: Lanzaboote and LUKS unlock—hardware permitting.

Day 68 — Secure Boot & TPM path

Stage VI · ~4h (hardware-dependent)
Goal: Document and, if hardware allows, implement a Lanzaboote Secure Boot path and/or TPM-backed LUKS unlock—or produce a lab-limits dossier that is honest about what you cannot do on VMs.

Important

Copy-paste without the upstream guide bricks boots. Treat this day as process + documentation first; enable on a disposable machine second. Recovery USB and recovery keys before enrollment.

Why this day exists

Fleet images and metal boxes increasingly expect Secure Boot and disk encryption that does not beg for a passphrase at every reboot in a datacenter. NixOS’s community answer for SB is primarily Lanzaboote (uki/systemd-boot + signed generations). TPM2 unlock pairs with LUKS (Day 57 Disko often declares the encrypted volumes).

Capstone B (stretch) lists SB/TPM; Capstone A does not require full enablement—but Stage VI experts must know the path and the failure modes.

Theory 1 — Problem statement

Threat Control
Evil maid bootloader swap Secure Boot + enrolled keys
Disk theft LUKS (or vendor disk encryption—know the difference)
Passphrase at scale TPM2 PCR-bound unlock (with enrollment process)
Unsigned kernel modules policies SB + your key hierarchy
Silent rollback to unsigned attacker kernel Sign each generation you boot

NixOS’s generation model must sign new boots—not only ship a distro shim forever. That is why Lanzaboote exists relative to “turn on SB and hope.”

Theory 2 — Lanzaboote overview (2026)

Lanzaboote integrates Secure Boot with NixOS (systemd-boot oriented flows).

Typical story (always verify against current README for your pin):

  1. Enable Secure Boot in firmware (setup mode / enroll keys carefully)
  2. Generate or enroll machine keys (PK/KEK/db as relevant to the guide)
  3. Import Lanzaboote NixOS modules so boots are signed
  4. Keep recovery path (USB installer + known recovery keys)
  5. Re-test boot after each major change before trusting prod
# Conceptual only — follow current Lanzaboote README for 26.05-compatible paths
{
  imports = [ lanzaboote.nixosModules.lanzaboote ];
  boot.loader.systemd-boot.enable = lib.mkForce false; # lanzaboote replaces aspects
  boot.lanzaboote = {
    enable = true;
    pkiBundle = "/etc/secureboot"; # path patterns per docs
  };
}

Flake input sketch:

# inputs.lanzaboote.url = "github:nix-community/lanzaboote/…";
# inputs.lanzaboote.inputs.nixpkgs.follows = "nixpkgs";
Warning

Firmware Setup Mode, key enrollment, and Microsoft/3rd-party key politics vary by vendor. Laptop OEM SB is not the same as a server you control end-to-end.

Theory 3 — TPM2 + LUKS

Goals:

  • Unlock root (or data) volume without typing a passphrase every boot
  • Still retain a recovery passphrase or recovery key file offline
  • Survive policy: which PCRs bind the sealed key
# Sketch — systemd-cryptenroll / auto-unlock patterns evolve
{
  boot.initrd.systemd.enable = true; # often required for modern enroll flows
  # boot.initrd.luks.devices."crypted".crypttabExtraOpts = [ "tpm2-device=auto" ];
  # Exact options: verify on your nixpkgs 26.05 pin and systemd version
}

Manual enrollment often looks like:

# list TPM
systemd-cryptenroll --tpm2-device=list
ls /sys/class/tpm

# enroll (EXAMPLE — adjust device + PCR set per threat model)
# systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7 /dev/nvme0n1p2

PCR trade-offs

Choice Security Fragility
Fewer PCRs Weaker binding Survives more firmware changes
PCR 7 (Secure Boot state) common Ties to SB Firmware/SB policy changes may require re-enroll
Many PCRs Stronger Breaks after BIOS updates more often

Document which PCRs and the re-enroll procedure after BIOS updates. Never TPM-only without recovery credentials.

Theory 4 — VM lab limits

Environment Secure Boot TPM
Bare metal modern laptop Often yes Often yes
Cloud VM Sometimes virtual SB/TPM Vendor-specific
libvirt QEMU Emulated possible swtpm possible
Nested cheap VPS Often no Often no

QEMU/swtpm note (advanced lab)

You can practice pieces with OVMF Secure Boot and swtpm—but it is a project of its own. Stage VI accepts a document-only track if hardware refuses you. Do not fake “SB enabled” in notes without evidence.

Theory 5 — Recovery & break-glass

Asset Storage
Disk recovery key / passphrase Password manager / sealed paper / offline encrypted store
SB setup mode knowledge Runbook steps
Unsigned rescue ISO / installer USB Physical media
Age/sops keys Separate from disk encryption keys
Old generation unsigned Know when SB will reject boot 
Failure Friday scenario:
  BIOS update → PCR mismatch → TPM unlock fails
  → boot installer USB
  → unlock with recovery key
  → re-enroll TPM
  → document time-to-recover

Never enroll TPM-only without a recovery passphrase backup you have tested.

Theory 6 — Relation to Disko, deploy, and images

Concern Link
LUKS layout Day 57 Disko declares partitions; TPM enroll often post-install
Impermanence Day 58: encrypted root + persist datasets
Images Day 62 generators may not equal metal SB story
Deploy Remote deploy does not replace firmware SB enrollment
Capstone B Fleet policy: which classes require SB

Write one paragraph: which host classes require SB vs best-effort.

Worked example — dossier template

# SECUREBOOT-TPM.md

## Hardware
- Machine:
- Firmware vendor/version:
- Firmware SB state: SetupMode / User / Disabled / Unknown
- TPM: 2.0 visible? (commands + output summary)
- Lab type: metal / libvirt / cloud

## Goals
- [ ] Secure Boot with Lanzaboote
- [ ] LUKS passphrase only
- [ ] LUKS + TPM2
- [ ] Document-only (limits)

## Upstream sources
- Lanzaboote README URL + commit/date consulted:
- NixOS wiki / manual notes:

## Steps performed
1. …
2. …

## Recovery
- Recovery key location (not the key itself):
- USB installer location:
- Re-enroll after BIOS update:
- Estimated RTO if TPM fails:

## Residual risks
- …

## Fleet policy
- Host classes requiring SB:
- Best-effort only:

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day68

Step 1 — Probe hardware

mkdir -p ~/lab/90daysofx/02-nixos/day68
bootctl status 2>/dev/null | tee ~/lab/90daysofx/02-nixos/day68/bootctl.txt || true
efi-readvar 2>/dev/null | tee ~/lab/90daysofx/02-nixos/day68/efi-vars.txt || true
ls -la /sys/class/tpm 2>&1 | tee ~/lab/90daysofx/02-nixos/day68/tpm-sysfs.txt
systemd-cryptenroll --tpm2-device=list 2>&1 | tee ~/lab/90daysofx/02-nixos/day68/tpm-list.txt || true

Step 2 — Choose track

Track Criteria
A Metal + SB available → Lanzaboote guided enable on disposable
B TPM + LUKS available → enroll with recovery key tested
C VM only → emulate with swtpm or document-only dossier

Write the track choice first; do not improvise on your daily driver.

Step 3 — Read upstream (required even for Track C)

Open current Lanzaboote README + Secure Boot notes relevant to your generation. Save URLs in the dossier. Take notes—do not invent key enrollment sequences from memory.

Step 4 — Recovery assets first

Before any enrollment:

  1. Confirm installer USB or rescue media boots
  2. Confirm LUKS recovery key unlocks (lab disk)
  3. Write locations in dossier

Step 5 — Implement or simulate

If A/B: snapshot/backup first; proceed carefully; capture commands and outcomes.
If C: write why cloud images still matter (signed provenance vs local SB) and what you would do on metal.

Step 6 — Failure plan drill (written)

Write the exact steps to boot installer and decrypt disk if TPM fails after firmware update. Include who holds recovery keys.

Step 8 — Fleet policy paragraph

Which host classes require SB? Which are best-effort? Capstone A vs B expectations.

Step 9 — Tabletop: evil maid + firmware update

Write answers in the dossier (no need to re-brick hardware):

  1. Attacker replaces bootloader on stolen laptop with SB off historically—what changes with Lanzaboote enrolled?
  2. After OEM BIOS update, TPM unlock fails at initrd—ordered recovery steps?
  3. How do age/sops keys relate to disk recovery keys (separate custody)?
  4. Can deploy-rs fix a machine that will not boot? (Usually no—firmware/recovery path first.)

Step 10 — Capstone A vs B honesty line

One sentence each:

  • Capstone A: SB/TPM not required—document limits OK.
  • Capstone B: which pieces become mandatory for your stretch goals?

Worked example — recovery USB checklist

## Recovery USB
- [ ] Boots on target hardware (tested date: …)
- [ ] Contains or can fetch flake tools / nix installer as needed
- [ ] Operator knows LUKS recovery unlock command
- [ ] Network plan if flake must be cloned (ethernet? Wi-Fi firmware?)
- [ ] Time estimate from power-on to decrypted root shell: …

If you never tested the USB on this machine, Track A/B is incomplete even if enrollment “worked once.”

Common gotchas

Symptom / mistake What to do
Bricked boot Installer USB; setup mode; recover keys
PCR mismatch after update Re-enroll; reconsider PCR set
Lost recovery key Undecryptable disk; prevention only
Enabled SB without signing new gens Use Lanzaboote properly
Assumed cloud disk encryption = LUKS Understand vendor encryption separately
Experiment on only laptop Use disposable or accept data-loss risk consciously
Docs lag code Date-stamp upstream guide you followed

Checkpoint

  • Hardware probed with saved outputs
  • Track A/B/C completed with artifacts
  • SECUREBOOT-TPM.md filled
  • Recovery path written and assets exist
  • No production host left without recovery keys
  • Upstream sources recorded
  • Tabletop answers written
  • Recovery USB checklist honest
  • Three personal gotchas

Commit

git add .
git commit -m "day68: secure boot / TPM path dossier"

Write three personal gotchas before continuing.

Tomorrow

Day 69 — Backup story: Borg/Restic/ZFS—pick one; restore drill of state.

Day 69 — Backup story

Stage VI · ~4h (lab-heavy)
Goal: Pick one backup system (Restic, Borg, or ZFS send), declare it in NixOS, and complete a restore drill of state (not only a successful backup job log).

Note

Flakes rebuild systems; they do not recreate Postgres, SQLite, photos, or /persist. Pair with Day 88 later: rebuild OS from git+keys, restore state from backups.

Why this day exists

Capstones fail when “declarative” is confused with “immortal data.” Today is the state half of disaster recovery. A green restic snapshots line without a restore is theatre.

Theory 1 — What to back up

Include Exclude
/persist / service state dirs /nix/store (rebuild from flake)
Database dumps or consistent snapshots Ephemeral caches, /tmp
Secrets materials you cannot reissue easily Cloud provider API caches
User home select paths result symlinks, build trees
Backup tool config references The entire OS image as only strategy

Rule: If wipe+reinstall cannot recreate it from git+keys+cache, it needs a backup story.

Inventory questions

  1. What does Day 58 impermanence already send to /persist?
  2. What does your Day 39 data service write?
  3. What keys exist only once (age, SSH CA, ACME account)?
  4. What would embarrass you if lost (homelab photos vs throwaway metrics)?

Theory 2 — Tool choice (pick one)

Tool Strengths Notes
Restic Simple; many backends (S3, B2, SFTP, local) Encrypted snapshots; prune policies
Borg Dedup; mature; SSH repos Borgmatic helper common on NixOS
ZFS send/recv Excellent for ZFS hosts Not an application-level portable tool

This book accepts any one done well. Switching tools mid-day is thrash—pick and finish.

3-2-1 reminder

Idea Lab minimum
3 copies Live + backup + offsite note
2 media Disk + remote/object
1 offsite Even “friend’s SFTP” or second VPS

Local-only lab is fine for learning if you write the offsite sentence for real systems.

Theory 3 — Restic on NixOS (shape)

{ config, pkgs, ... }:
{
  # Prefer sops-nix for repository password / access keys (Day 34)
  # sops.secrets.restic-password = { … };

  services.restic.backups.lab = {
    initialize = true;
    passwordFile = config.sops.secrets.restic-password.path;
    repository = "sftp:backup@backup-host:/backups/lab";
    # or "s3:s3.amazonaws.com/bucket/lab"
    paths = [
      "/persist"
      "/var/lib/myapp"
    ];
    timerConfig = {
      OnCalendar = "daily";
      Persistent = true;
    };
    pruneOpts = [
      "--keep-daily 7"
      "--keep-weekly 4"
      "--keep-monthly 6"
    ];
  };
}

Exact module options: check man configuration.nix / search.nixos.org for your 26.05 pin if attribute names shift.

Borgmatic sketch

{
  services.borgmatic.enable = true;
  # services.borgmatic.settings = { … };
  # or environment.etc."borgmatic.d/lab.yaml".text = …;
}

ZFS

zfs snapshot tank/persist@day69
zfs send -v tank/persist@day69 | ssh backup zfs recv -F backup/lab/persist

Declare snapshot timers via Nix modules or sanoid/syncoid if you go that ecosystem. Still do a restore drill (recv to a scratch dataset, mount, verify files).

Theory 4 — Consistency

Data Technique
Postgres backupPrepareCommand dump or brief stop / native snapshot
SQLite Stop service or .backup API; avoid mid-write copies
VM disks Hypervisor snapshot or fs freeze
Mutable logs Optional; size-cap
Mail / queues App-native export 
services.restic.backups.lab.backupPrepareCommand = ''
  mkdir -p /var/backup
  ${pkgs.sudo}/bin/sudo -u postgres \
    ${pkgs.postgresql}/bin/pg_dumpall > /var/backup/pg.sql
'';
# include /var/backup/pg.sql in paths

Inconsistent DB files that “restored” but won’t start are not a successful drill.

Theory 5 — Restore is the product

# Restic example
export RESTIC_REPOSITORY=…
export RESTIC_PASSWORD_FILE=…
restic snapshots
restic restore latest --target /tmp/restore-drill
# verify files; for DB: restore dump into disposable instance

Success criteria

Level Meaning
Files on disk Necessary but insufficient
Service starts Better
Application-usable state Done for Day 69
Timed RTO/RPO notes Ops maturity

Write RESTORE.md with commands a tired operator can paste.

Theory 6 — Secrets and backup encryption

Concern Practice
Repo password sops-nix / agenix path; not world-readable
Encryption keys only on source host Fails disaster recovery—offsite custody
Backing up decrypted /run/secrets Usually wrong; back up encrypted materials + key custody story
Same disk as source Not a disaster backup

Restic/Borg encrypt client-side; you still need the password after wipe (Day 88).

Worked example — local restic lab (no cloud)

{
  services.restic.backups.local = {
    initialize = true;
    passwordFile = "/persist/secrets/restic-password"; # lab only; prefer sops
    repository = "/mnt/backup-disk/restic-lab";
    paths = [ "/persist/PROOF" "/var/lib/myapp" ];
    timerConfig.OnCalendar = "hourly";
  };
}
echo important-state | sudo tee /persist/PROOF
sudo systemctl start restic-backups-local.service
sudo systemctl status restic-backups-local.service --no-pager
sudo restic -r /mnt/backup-disk/restic-lab snapshots
sudo rm /persist/PROOF
sudo restic -r /mnt/backup-disk/restic-lab restore latest --target /tmp/r
sudo install -D /tmp/r/persist/PROOF /persist/PROOF
cat /persist/PROOF

Worked example — RTO/RPO honesty

## Lab targets (not production promises)
- RPO: up to 24h (daily timer) — last successful snapshot time: …
- RTO: ~30–60 min to restore PROOF + app dir on empty disk (measured: …)
- Not included: full bare-metal with SB/TPM re-enroll (Day 68)

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day69

Step 1 — Choose tool

Write choice + backend (local disk, SFTP, S3-compatible) in NOTES.

Step 2 — State inventory

List paths tied to services you run (reuse Day 58 inventory). Mark each: rebuildable from git vs must backup.

Step 3 — Declare backup job in Nix

Wire secrets properly if possible (sops). Commit module + docs.

Step 4 — First backup

Run service once; list snapshots; capture logs.

mkdir -p ~/lab/90daysofx/02-nixos/day69
sudo systemctl start restic-backups-lab.service  # name may vary
journalctl -u restic-backups-lab.service -n 100 --no-pager \
  | tee ~/lab/90daysofx/02-nixos/day69/backup.log

Step 5 — Restore drill (required)

Delete or move a lab file; restore; verify content byte-for-byte or known string.

Step 6 — DB drill (if you have Day 39 DB)

Dump → backup → drop table/schema in lab → restore dump → query. If no DB, write N/A with reason.

Step 7 — Offsite note

Even local-only lab needs a sentence on offsite/3-2-1 for real systems.

Step 8 — Runbook

docs/RESTORE.md:

  • Prerequisites (password location, repo URL)
  • List snapshots
  • Restore commands
  • Service restart order
  • Lab RTO/RPO numbers

Step 9 — Negative cases table

Failure Detection Response
Timer silent fail monitoring / systemctl --failed
Repo full prune / expand
Wrong password after wipe key custody
Restored inconsistent DB prepare hooks

Common gotchas

Symptom / mistake What to do
Backed up /nix only Useless for state; fix paths
Password in world-readable file sops/agenix; modes
Never pruned Fill disk; set keep policy
Restored files, app still broken Consistent DB story
Same disk as source Not a disaster backup
Encryption keys only on source host Offsite key custody
“Backup works” without restore Fail the day until restore proves
Hardlink confusion with optimise Know tool behavior (Day 67)

Checkpoint

  • One tool chosen and declared in NixOS
  • Backup job succeeds with log proof
  • Restore drill completed with proof
  • RESTORE.md written with RTO/RPO honesty
  • Store vs state distinction clear in notes
  • Secrets for repo password handled sanely
  • Three personal gotchas

Commit

git add .
git commit -m "day69: backup + restore drill"

Write three personal gotchas before continuing.

Tomorrow

Day 70 — Stage VI gate: remote lab with deploy + CI + cache + tests green.

Day 70 — Stage VI gate

Stage VI · ~4h (integration)
Goal: Prove fleet habits: a flake-defined lab that deploys remotely, is tested, built in CI, and supported by a cache story—with Disko/impermanence/backup notes complete enough that a new host of an existing role is boring.

Why this day exists

Stage VI without a gate is a pile of demos. Exit only when the boring path works: clone → build/check → deploy → rollback plan → restore state plan.

Gate bar (explicit)

# Requirement Evidence
H1 Remote deploy works (deploy-rs or Colmena) Transcript + remote nixos-version
H2 ≥2 hosts or 1 remote + documented second role path Hive/nodes
H3 Shared baseline + ≥1 role module Tree layout Day 61
H4 Flake checks ≥3 meaningful, green locally nix flake check
H5 CI runs Nix build/check on PR/push Actions/GitLab URL
H6 Binary cache push/pull or file-store proof Day 53/66
H7 Disko layout for at least one host type Committed module
H8 Impermanence proof or written skip Day 58
H9 Backup restore drill done Day 69 RESTORE.md
H10 SB/TPM dossier exists Day 68
H11 TF boundary sketch exists Day 63
H12 Dossier GATE-VI.md self-scored This day

Stretch: image boot (Day 62) linked from gate dossier.

Theory — Reference architecture

                    ┌────────── CI (GHA) ──────────┐
                    │ nix flake check / build      │
                    │ cachix push                  │
                    └─────────────┬────────────────┘
                                  │ substitutes
┌────────────┐   deploy-rs/colmena v
│ laptop/dev │ ───────────────────► edge host (remote VM)
│ flake.dev  │ ───────────────────► core host (optional)
└────────────┘
      │
      ├─ modules/profiles/baseline.nix
      ├─ modules/roles/web.nix
      ├─ hosts/*/disko.nix
      ├─ tests/*.nix
      └─ RESTORE.md / BOUNDARY.md / SECUREBOOT-TPM.md

Worked example — gate dossier skeleton

# GATE-VI.md

## Inventory
| Host | Role | Deploy tool | IP | Disko | Persist |
|------|------|-------------|----|-------|---------|

## Commands that must work
```bash
nix flake check -L
deploy .#edge   # or colmena apply --on edge
ssh edge cat /etc/role

CI

  • Workflow:
  • Latest green URL:

Cache

  • Substituter:
  • Push method:

State

  • Backup tool:
  • Last restore drill date:
  • Link RESTORE.md

Scores

ID Pass? Notes
H1

…


---

## Lab — gate protocol (multi-step)

Suggested workspace: promote your fleet flake; notes `~/lab/90daysofx/02-nixos/day70`

### Step 1 — Freeze scope

No new tools today. Only glue and proof.

### Step 2 — Remote deploy proof

```bash
# change a harmless etc file text with timestamp
deploy .#edge   # or colmena
ssh edge cat /etc/day70

Step 3 — Checks green

nix flake check -L

Fix or quarantine with written reason (not silent delete).

Step 4 — CI green

Ensure main branch workflow green; link it.

Step 5 — Cache proof

Cold-ish machine or nix store delete careful path on lab only—prefer second VM pull through cache.

Step 6 — Role composition review

New host checklist written:

  1. Copy hosts/_template
  2. Set disko device
  3. Enable roles
  4. Add to colmena/deploy nodes
  5. Apply
  6. Attach backup paths

Step 7 — Fill GATE-VI.md

Score H1–H12 ruthlessly.

Step 8 — Tag

git tag -a stage-vi-gate -m "Stage VI fleet habits gate"

Step 9 — Rollback drill (required)

On the remote host, after a successful deploy:

ssh edge 'ls /nix/var/nix/profiles/system*'
# switch to previous generation once, then back
ssh edge 'sudo /nix/var/nix/profiles/system-N-link/bin/switch-to-configuration switch'
# or boot-menu / nixos-rebuild --rollback depending on setup

Record: how long until SSH returned, whether services flapped, whether magic-rollback would have helped.

Step 10 — Retrospective (short)

Three things that will break in Stage VII/VIII; put them in BACKLOG-stage-vii.md.

House defaults freeze (record yours)

By Day 70 you should stop re-picking tools every week. Fill and commit:

Concern Your default Runner-up Why
Pinning flakes + lock
Secrets sops-nix / agenix
Deploy deploy-rs / Colmena
Cache Cachix / Attic / file
Images nixos-generators format
Backup restic / borg / zfs
Tests nixosTest + pkg checks
Disks Disko layout family
Persist full / partial / skip

This table is part of the gate dossier—not optional flavor text.

Acceptance ceremony (30 minutes)

Run in order; paste outputs into GATE-VI.md:

# 1. Metadata
nix flake metadata
git rev-parse HEAD
git describe --tags --always

# 2. Checks
nix flake check -L

# 3. Package (Stage V still alive)
nix build .#cooltool -L || nix build .#default -L

# 4. Host eval
nix build .#nixosConfigurations.edge.config.system.build.toplevel -L

# 5. Deploy
deploy .#edge   # or: colmena apply --on edge

# 6. Live proof
ssh edge 'hostname; nixos-version; cat /etc/role 2>/dev/null || true'

# 7. Backup pointer
test -f RESTORE.md && head -n 20 RESTORE.md

If any step fails, the gate fails—fix or document a time-boxed exception with a ticket ID.

What “new host is boring” means

You pass the spirit of Stage VI when this checklist is mechanical:

  1. Allocate VM/metal (TF or console)
  2. Set disko.devices.disk.main.device
  3. Import profiles.baseline + role(s)
  4. Add SSH bootstrap key
  5. Add node to deploy/colmena inventory
  6. First deploy
  7. Attach backup paths for that role’s state
  8. CI already covers shared modules—no new workflow required

If step 3 still means “copy 400 lines from edge,” return to Day 61.

Theory — Deploy path proof matrix

Path Command / tool Proven?
Local rebuild nixos-rebuild switch --flake
Remote deploy deploy-rs / colmena
Image path generators / disko install story
Test path nix flake check subset

Gate requires honesty: blank cells need a written “not used” with reason.

Theory — Impermanence / Disko status

Even if optional in your lab:

Topic Status
Disko config exists / N/A
Persist list written
Backup of state dirs
New host install notes

“We use default ext4 installer” is a valid status if documented.

Worked example — gate freeze commit message

day70: stage VI gate

- deploy tool: …
- shared modules: roles/…
- checks: list
- cache: …
- known debt: …

Lab — cross-host dry run

If you only have one machine, still write:

  1. How a second host reuses modules/roles
  2. What differs (hardware, secrets, networking.hostName)
  3. How inventory assigns roles

Lab — CI badge honesty

If CI exists, paste latest green URL or log hash into the dossier. If not, write the blocker (secrets, runners, time) and the minimal check you run locally instead.

Common gotchas

Symptom / mistake What to do
“Works on my laptop only” Fail H1/H5
Checks skipped in CI Fail H5
Deploy without backup story Fail H9
Secret docs missing Fix before claiming gate
Tool zoo (deploy-rs+colmena+tf+k8s) unfinished Narrow; finish boring path
Impermanence half-broken SSH Fix keys before scoring pass
Gate scored green without remote Invalid—H1 is remote
Stage V package abandoned Rebuild once; prove cache still helps
Rollback never practiced Do Step 9 before tag

Checkpoint

  • H1–H12 scored in GATE-VI.md
  • Remote deploy demonstrated
  • Rollback drill recorded
  • CI URL recorded
  • House defaults table frozen
  • New-host checklist exists
  • Acceptance ceremony outputs attached
  • Tag stage-vi-gate (or equivalent milestone)
  • Ready for Stage VII internals without fleet shame

Commit

git add .
git commit -m "day70: stage VI gate dossier"

Write three personal gotchas before continuing.

Tomorrow

Day 71 — Evaluator: thunks, infinite recursion, and tracing tricky evals (Stage VII begins).