Day 69 — Backup story

Updated

July 30, 2026

Day 69 — Backup story

Stage VI · ~4h (lab-heavy)
Goal: Pick one backup system (Restic, Borg, or ZFS send), declare it in NixOS, and complete a restore drill of state (not only a successful backup job log).

Note

Flakes rebuild systems; they do not recreate Postgres, SQLite, photos, or /persist. Pair with Day 88 later: rebuild OS from git+keys, restore state from backups.

Why this day exists

Capstones fail when “declarative” is confused with “immortal data.” Today is the state half of disaster recovery. A green restic snapshots line without a restore is theatre.

Theory 1 — What to back up

Include Exclude
/persist / service state dirs /nix/store (rebuild from flake)
Database dumps or consistent snapshots Ephemeral caches, /tmp
Secrets materials you cannot reissue easily Cloud provider API caches
User home select paths result symlinks, build trees
Backup tool config references The entire OS image as only strategy

Rule: If wipe+reinstall cannot recreate it from git+keys+cache, it needs a backup story.

Inventory questions

  1. What does Day 58 impermanence already send to /persist?
  2. What does your Day 39 data service write?
  3. What keys exist only once (age, SSH CA, ACME account)?
  4. What would embarrass you if lost (homelab photos vs throwaway metrics)?

Theory 2 — Tool choice (pick one)

Tool Strengths Notes
Restic Simple; many backends (S3, B2, SFTP, local) Encrypted snapshots; prune policies
Borg Dedup; mature; SSH repos Borgmatic helper common on NixOS
ZFS send/recv Excellent for ZFS hosts Not an application-level portable tool

This book accepts any one done well. Switching tools mid-day is thrash—pick and finish.

3-2-1 reminder

Idea Lab minimum
3 copies Live + backup + offsite note
2 media Disk + remote/object
1 offsite Even “friend’s SFTP” or second VPS

Local-only lab is fine for learning if you write the offsite sentence for real systems.

Theory 3 — Restic on NixOS (shape)

{ config, pkgs, ... }:
{
  # Prefer sops-nix for repository password / access keys (Day 34)
  # sops.secrets.restic-password = { … };

  services.restic.backups.lab = {
    initialize = true;
    passwordFile = config.sops.secrets.restic-password.path;
    repository = "sftp:backup@backup-host:/backups/lab";
    # or "s3:s3.amazonaws.com/bucket/lab"
    paths = [
      "/persist"
      "/var/lib/myapp"
    ];
    timerConfig = {
      OnCalendar = "daily";
      Persistent = true;
    };
    pruneOpts = [
      "--keep-daily 7"
      "--keep-weekly 4"
      "--keep-monthly 6"
    ];
  };
}

Exact module options: check man configuration.nix / search.nixos.org for your 26.05 pin if attribute names shift.

Borgmatic sketch

{
  services.borgmatic.enable = true;
  # services.borgmatic.settings = { … };
  # or environment.etc."borgmatic.d/lab.yaml".text = …;
}

ZFS

zfs snapshot tank/persist@day69
zfs send -v tank/persist@day69 | ssh backup zfs recv -F backup/lab/persist

Declare snapshot timers via Nix modules or sanoid/syncoid if you go that ecosystem. Still do a restore drill (recv to a scratch dataset, mount, verify files).

Theory 4 — Consistency

Data Technique
Postgres backupPrepareCommand dump or brief stop / native snapshot
SQLite Stop service or .backup API; avoid mid-write copies
VM disks Hypervisor snapshot or fs freeze
Mutable logs Optional; size-cap
Mail / queues App-native export 
services.restic.backups.lab.backupPrepareCommand = ''
  mkdir -p /var/backup
  ${pkgs.sudo}/bin/sudo -u postgres \
    ${pkgs.postgresql}/bin/pg_dumpall > /var/backup/pg.sql
'';
# include /var/backup/pg.sql in paths

Inconsistent DB files that “restored” but won’t start are not a successful drill.

Theory 5 — Restore is the product

# Restic example
export RESTIC_REPOSITORY=…
export RESTIC_PASSWORD_FILE=…
restic snapshots
restic restore latest --target /tmp/restore-drill
# verify files; for DB: restore dump into disposable instance

Success criteria

Level Meaning
Files on disk Necessary but insufficient
Service starts Better
Application-usable state Done for Day 69
Timed RTO/RPO notes Ops maturity

Write RESTORE.md with commands a tired operator can paste.

Theory 6 — Secrets and backup encryption

Concern Practice
Repo password sops-nix / agenix path; not world-readable
Encryption keys only on source host Fails disaster recovery—offsite custody
Backing up decrypted /run/secrets Usually wrong; back up encrypted materials + key custody story
Same disk as source Not a disaster backup

Restic/Borg encrypt client-side; you still need the password after wipe (Day 88).

Worked example — local restic lab (no cloud)

{
  services.restic.backups.local = {
    initialize = true;
    passwordFile = "/persist/secrets/restic-password"; # lab only; prefer sops
    repository = "/mnt/backup-disk/restic-lab";
    paths = [ "/persist/PROOF" "/var/lib/myapp" ];
    timerConfig.OnCalendar = "hourly";
  };
}
echo important-state | sudo tee /persist/PROOF
sudo systemctl start restic-backups-local.service
sudo systemctl status restic-backups-local.service --no-pager
sudo restic -r /mnt/backup-disk/restic-lab snapshots
sudo rm /persist/PROOF
sudo restic -r /mnt/backup-disk/restic-lab restore latest --target /tmp/r
sudo install -D /tmp/r/persist/PROOF /persist/PROOF
cat /persist/PROOF

Worked example — RTO/RPO honesty

## Lab targets (not production promises)
- RPO: up to 24h (daily timer) — last successful snapshot time: …
- RTO: ~30–60 min to restore PROOF + app dir on empty disk (measured: …)
- Not included: full bare-metal with SB/TPM re-enroll (Day 68)

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day69

Step 1 — Choose tool

Write choice + backend (local disk, SFTP, S3-compatible) in NOTES.

Step 2 — State inventory

List paths tied to services you run (reuse Day 58 inventory). Mark each: rebuildable from git vs must backup.

Step 3 — Declare backup job in Nix

Wire secrets properly if possible (sops). Commit module + docs.

Step 4 — First backup

Run service once; list snapshots; capture logs.

mkdir -p ~/lab/90daysofx/02-nixos/day69
sudo systemctl start restic-backups-lab.service  # name may vary
journalctl -u restic-backups-lab.service -n 100 --no-pager \
  | tee ~/lab/90daysofx/02-nixos/day69/backup.log

Step 5 — Restore drill (required)

Delete or move a lab file; restore; verify content byte-for-byte or known string.

Step 6 — DB drill (if you have Day 39 DB)

Dump → backup → drop table/schema in lab → restore dump → query. If no DB, write N/A with reason.

Step 7 — Offsite note

Even local-only lab needs a sentence on offsite/3-2-1 for real systems.

Step 8 — Runbook

docs/RESTORE.md:

  • Prerequisites (password location, repo URL)
  • List snapshots
  • Restore commands
  • Service restart order
  • Lab RTO/RPO numbers

Step 9 — Negative cases table

Failure Detection Response
Timer silent fail monitoring / systemctl --failed
Repo full prune / expand
Wrong password after wipe key custody
Restored inconsistent DB prepare hooks

Common gotchas

Symptom / mistake What to do
Backed up /nix only Useless for state; fix paths
Password in world-readable file sops/agenix; modes
Never pruned Fill disk; set keep policy
Restored files, app still broken Consistent DB story
Same disk as source Not a disaster backup
Encryption keys only on source host Offsite key custody
“Backup works” without restore Fail the day until restore proves
Hardlink confusion with optimise Know tool behavior (Day 67)

Checkpoint

  • One tool chosen and declared in NixOS
  • Backup job succeeds with log proof
  • Restore drill completed with proof
  • RESTORE.md written with RTO/RPO honesty
  • Store vs state distinction clear in notes
  • Secrets for repo password handled sanely
  • Three personal gotchas

Commit

git add .
git commit -m "day69: backup + restore drill"

Write three personal gotchas before continuing.

Tomorrow

Day 70 — Stage VI gate: remote lab with deploy + CI + cache + tests green.