Backups and Generation Hygiene

Updated

September 12, 2026

Backups and Generation Hygiene

A flake rebuilds the operating system. It does not recreate Postgres, /persist, or the Age key you generated once. The boring default is: restic (or borg) of state, a restore drill, and GC that keeps two weeks of generations — not rm -rf /nix.

Mental model

Rebuild from git + cache Must be backed up
/nix/store /persist (impermanence)
Kernel, units, packages Database files / dumps
Most of /etc Secrets material you cannot reissue
Home Manager managed files Unmanaged $HOME (photos, SSH private keys if not in sops)

3-2-1 still applies: live + backup + offsite. A green restic snapshots line without a restore is theatre.

Generation hygiene is the other disk: old NixOS generations are rollback insurance and gigabytes. Keep a window. Automatic GC with --delete-older-than 14d is the boring timer. -d is for incidents.

wipe disk
   ├── nixos-anywhere / installer  →  OS from flake
   └── restic restore              →  /persist + DB

Worked examples

Case 1: What to include

Save as backup-paths.md in the desk repo (documentation, not Nix):

# backup-paths.md
Include:
  /persist
  /var/lib/postgresql
  /var/lib/desk-api
Exclude:
  /nix/store
  /tmp
  /var/cache
  result
  .direnv

If you use impermanence, /persist is the list. Do not also back up the ephemeral root.

Case 2: Restic on NixOS

Save as restic.nix:

# restic.nix
{ config, pkgs, ... }:

{
  services.restic.backups.desk = {
    initialize = true;
    passwordFile = config.sops.secrets.restic-password.path;
    repository = "sftp:backup@backup.desk.internal:/backups/desk-vm";
    paths = [
      "/persist"
      "/var/lib/desk-api"
    ];
    pruneOpts = [
      "--keep-daily 7"
      "--keep-weekly 4"
      "--keep-monthly 6"
    ];
    timerConfig = {
      OnCalendar = "daily";
      Persistent = true;
    };
  };
}

The repository password is a sops secret, not a string in this file. sftp: is one backend; s3:s3.amazonaws.com/desk-backups is the same module.

# restic.nix extras
{ config, pkgs, ... }:

{
  services.restic.backups.desk = {
    user = "root"; # default. pg_dumpall needs to become postgres.
    environmentFile = config.sops.secrets.restic-env.path; # AWS keys if s3:
    extraBackupArgs = [ "--exclude-caches" "--tag" "desk" ];
    exclude = [ "/persist/backups/*.tmp" ];
    backupPrepareCommand = ''
      set -euo pipefail
      mkdir -p /persist/backups
      ${pkgs.sudo}/bin/sudo -u postgres \
        ${config.services.postgresql.package}/bin/pg_dumpall -c \
        -f /persist/backups/pg.dump
    '';
    backupCleanupCommand = ''
      rm -f /persist/backups/pg.dump
    '';
    runCheck = true;
  };
}

Dump before restic walks /persist. A running cluster’s data files without a dump are a crash-consistent maybe. backupPrepareCommand is a script NixOS wraps with pkgs.writeScript — interpolate store paths (config.services.postgresql.package, not pg_dump from PATH). set -euo pipefail so a failed dump fails the unit instead of uploading yesterday’s file.

user defaults to "root". If you set user = "restic", sudo -u postgres needs a sudoers rule, or skip sudo and user = "postgres" only when the paths are that uid’s. Do not invent a wrapper until the default root unit is proven.

backupCleanupCommand removes the dump from disk after the snapshot (the dump still lives in the snapshot). runCheck is restic check after prune — catch a silent repo corruption on the timer, not at restore time.

createWrapper = true (module default on many pins) installs a restic-desk wrapper with the same env as the unit. Humans restore with that, not a bare restic that is missing RESTIC_PASSWORD_FILE.

sudo nixos-rebuild switch
sudo systemctl start restic-backups-desk.service
sudo journalctl -u restic-backups-desk.service -n 40

Case 3: Restore drill (the part people skip)

sudo restic -r sftp:backup@backup.desk.internal:/backups/desk-vm \
  --password-file /run/secrets/restic-password \
  snapshots

sudo restic -r sftp:backup@backup.desk.internal:/backups/desk-vm \
  --password-file /run/secrets/restic-password \
  restore latest --target /tmp/desk-restore-drill
ls /tmp/desk-restore-drill/persist | head

On a lab VM, go further: restore /var/lib/desk-api onto a second VM and start the service. A file listing is not a drill. A responding API is.

Schedule this quarterly. Write the date in the desk repo.

Case 4: Generation window on NixOS

Save as gc.nix:

# gc.nix
{
  nix.gc = {
    automatic = true;
    dates = "weekly";
    options = "--delete-older-than 14d";
    persistent = true; # catch up after the box was off
  };
  nix.optimise.automatic = true;
  boot.loader.systemd-boot.configurationLimit = 8;
}

configurationLimit caps bootloader entries so /boot (often a small ESP) does not fill. GC and the bootloader limit work together: deleting generations without limiting entries leaves stale boot menu rows; limiting entries without GC leaves the store fat.

sudo nixos-rebuild switch
sudo nix-env --list-generations --profile /nix/var/nix/profiles/system

Case 5: /boot full is a different incident

df -h /boot
sudo nixos-rebuild switch

If /boot is 100%, switch fails while /nix still has space. configurationLimit plus removing old entries:

sudo nix-collect-garbage --delete-older-than 7d
sudo nixos-rebuild switch

Do not rm /boot/EFI/nixos/* by hand unless you know which generation you are running.

The trap

The trap is backing up /nix/store. It is huge, it is reconstructable, and restic will spend all night hashing NARs. Back up state. Substitute the store from the flake + cache.

The other trap is nix-collect-garbage -d on Friday before a Monday upgrade. Keep 14 days. Use -d when df is the outage.

A third trap: restic password in repository URL query string inside configuration.nix. That string is in the world-readable store.

A fourth: backupPrepareCommand = "pg_dumpall …" with no store path — the unit’s PATH is not your login shell. A fifth: restoring the dump over a live cluster that already initdb’d empty (DR chapter: --no-reboot, restore /persist first).

The boring rule

  • Flake + cache = OS. Restic (or borg) = state. You need both.
  • pg_dumpall via store-path backupPrepareCommand before the walk. runCheck after.
  • Restore drill with a second VM (empty Disko), not only restic snapshots.
  • Weekly GC with --delete-older-than 14d. configurationLimit on the ESP.
  • Never back up /nix/store as the backup strategy.
  • Backup passwords via sops, not Nix strings. Use the generated wrapper.

Try this

  1. Inventory /persist (or /var/lib) on a lab VM and write the include/exclude list.
  2. Enable Case 4, systemctl status nix-gc.timer, and list generations.
  3. Fill a dummy /persist/desk-note.txt, take a restic snapshot to a local repository = "/var/backup/restic"; (lab), delete the file, restore, confirm the text.
  4. Set configurationLimit = 2 on a VM, switch twice, reboot, and count systemd-boot entries. Put it back to 8.
  5. systemctl cat restic-backups-desk.service and confirm pg_dumpall is a /nix/store/…-postgresql-…/bin/pg_dumpall path.