Day 20 — Disko, Impermanence & deploy-rs

Updated

July 30, 2025

Day 20 — Disko, Impermanence & deploy-rs

Day 57 — Disko for real

Stage VI · ~4h (lab-heavy, destructive-capable)
Goal: Describe disks declaratively with Disko, install or reinstall a lab host from that description, and wire Disko into your flake so storage stops being tribal knowledge.

Important

Disko can wipe disks. Use a disposable VM, spare drive, or cloud instance. Never experiment on a disk with irreplaceable data.

Disko disk layout ESP root optional

Disko layout

Why this day exists

Stage II may have used the installer GUI or a one-shot hardware-configuration.nix. Fleet habits require repeatable storage: partitions, LVM, btrfs/zfs layouts, LUKS—as code. Disko is the 2026 default answer in many NixOS fleets.

Theory 1 — What Disko is

Disko turns a Nix attrset into:

  1. Partitioning / formatting scripts
  2. Mount configuration compatible with NixOS
  3. Optionally integrated into nixos-rebuild / installer flows
disko.devices = { … }
        │
        ├─► disko script (destroy/create/mount)
        └─► fileSystems / boot / swap NixOS options

Theory 2 — Minimal GPT + ext4 root (UEFI)

# modules/disko-lab.nix
{
  disko.devices = {
    disk = {
      main = {
        type = "disk";
        device = "/dev/vda"; # CHANGE per machine: nvme0n1, sda, …
        content = {
          type = "gpt";
          partitions = {
            ESP = {
              size = "512M";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
                mountOptions = [ "umask=0077" ];
              };
            };
            root = {
              size = "100%";
              content = {
                type = "filesystem";
                format = "ext4";
                mountpoint = "/";
              };
            };
          };
        };
      };
    };
  };
}
Field Notes
device Host-specific; inject via host module
type = "gpt" UEFI-friendly
ESP EF00 EFI system partition type
mountpoint Becomes NixOS fileSystems

Theory 3 — Flake integration

# flake.nix fragment
{
  inputs.disko.url = "github:nix-community/disko";
  inputs.disko.inputs.nixpkgs.follows = "nixpkgs";

  outputs = { self, nixpkgs, disko, ... }: {
    nixosConfigurations.lab = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        disko.nixosModules.disko
        ./hosts/lab/disko.nix
        ./hosts/lab/configuration.nix
      ];
    };
  };
}
# From installer ISO or rescue, after networking + clone:
sudo nix --extra-experimental-features "nix-command flakes" run github:nix-community/disko -- \
  --mode disown \
  --flake .#lab
# modes/flags evolve—check current Disko CLI help; common modes: destroy, format, mount, disown

Then install:

sudo nixos-install --flake .#lab

Or use documented disko-install flows when available in your Disko version.

Theory 4 — LUKS sketch (common homelab)

root = {
  size = "100%";
  content = {
    type = "luks";
    name = "crypted";
    settings.allowDiscards = true;
    content = {
      type = "filesystem";
      format = "ext4";
      mountpoint = "/";
    };
  };
};

Password vs TPM unlock: passphrase for today’s lab; TPM path appears Day 68.

Theory 5 — btrfs subvolumes (impermanence-ready)

content = {
  type = "btrfs";
  subvolumes = {
    "@" = { mountpoint = "/"; };
    "@home" = { mountpoint = "/home"; };
    "@nix" = { mountpoint = "/nix"; };
    "@persist" = { mountpoint = "/persist"; };
  };
};

Day 58 (impermanence) loves @persist + wipe @ on boot. You can create the layout today even if you skip impermanence.

Theory 6 — Host-specific devices without copy-paste hell

# hosts/lab/disko.nix
{ lib, ... }:
{
  disko.devices.disk.main.device = lib.mkDefault "/dev/vda";
}
# hosts/metal/disko.nix
{
  imports = [ ../../modules/disko-btrfs-template.nix ];
  disko.devices.disk.main.device = "/dev/nvme0n1";
}

Never commit a fleet-wide wrong device path.

Worked example — VM lab checklist

1. Create empty qcow2 / virt disk
2. Boot NixOS 26.05 installer ISO
3. Git clone flake (or nix copy)
4. Identify disk: lsblk
5. Set device in disko config
6. Run disko (destructive)
7. nixos-install --flake .#lab
8. Reboot; verify findmnt / lsblk
lsblk -f
findmnt /
cat /etc/fstab   # generated view; source of truth is Nix

Lab — multi-step

Suggested workspace: ~/lab/90daysofx/02-nixos/day57 (flake fragments live in your main lab flake)

Step 1 — Snapshot safety

Confirm target disk is disposable. lsblk screenshot/paste into notes.

Step 2 — Add Disko input

Pin via flake; follows nixpkgs; update lock.

Step 3 — Write simplest UEFI layout

ext4 root + ESP. Parameterize device.

Step 4 — Dry evaluation

nix eval .#nixosConfigurations.lab.config.fileSystems --json | jq 'keys'
# ensure / and /boot appear as expected after disko module merge

Step 5 — Apply on VM

Run Disko + install or nixos-rebuild path if reinstalling an already-Disko host per docs.

Step 6 — Reinstall proof (core of “for real”)

Wipe once more (VM) and reinstall from same flake. Time it. Note pain points.

Step 7 — Optional LUKS or btrfs

Add one complexity only if Step 6 succeeded cleanly.

Step 8 — Document recovery

DISK.md: device names per host, LUKS key location, disko command that saved you.

Common gotchas

Symptom / mistake What to do
Wiped wrong disk Device IDs; prefer /dev/disk/by-id/… when possible
No EFI boot Missing ESP / wrong bootloader module
Device path differs after hardware change by-id; update host module
Forgot disko module import fileSystems empty/wrong
Swap forgotten Add swap partition or zram intentionally
Cloud images Device often /dev/vda or /dev/nvme0n1; check vendor docs

Checkpoint

  • Disko input wired in flake
  • Declarative layout for lab host committed
  • At least one real format/install or reinstall performed on disposable media
  • fileSystems matches intent after boot
  • DISK.md recovery notes written

Commit

git add .
git commit -m "day57: disko layout and reinstall proof"

Write three personal gotchas before continuing.

Tomorrow

Day 58 — Impermanence: persist deliberately or document why you skip.

Day 58 — Impermanence

Stage VI · ~4h
Goal: Implement or consciously reject root filesystem impermanence using impermanence patterns—proving that unpersisted state disappears on reboot while /persist (or equivalent) keeps what you declare.

Tip

Impermanence is optional as a lifestyle. It is not optional to understand. Skipping still requires a written “why” for the Stage VI gate. Disk layouts pair with Day 57 Disko; secrets with Days 34–35. Target: NixOS 26.05.

Why this day exists

Snowflakes are not only packages—they are mutated roots: /etc edits, /var surprises, leftover packages. Impermanence makes the root a camping tent and /persist the shipping container. Combined with Disko btrfs/ZFS layouts, reinstalls become boring—and Day 88 disaster recovery gets a cleaner story.

Even if you never wipe root on a daily driver, the inventory habit (what must survive reboot?) is permanent skill.

Theory 1 — Model

boot
  │
  ├─ tmpfs or fresh subvolume for /
  ├─ bind/mount persisted paths from /persist
  └─ everything else is recreated by activation
Path class Examples Strategy
Store /nix Own subvolume/partition; never wipe carelessly
Persist /persist/var/lib/postgres, ssh host keys Explicit list
Ephemeral /var/cache, junk in /root Accept loss
Secrets sops age keys Persist carefully; permissions matter
Boot EFI, kernels, generations Separate story; do not “wipe /boot” casually

What “disappears” means

On reboot, anything not in the persist list is gone if you actually wipe/reset root. Soft modes that only document policy without wipe are still useful—but do not claim impermanence until the reboot proof works.

Theory 2 — impermanence module (nix-community)

# flake input
inputs.impermanence.url = "github:nix-community/impermanence";
# follows nixpkgs when the project supports it
{
  imports = [ impermanence.nixosModules.impermanence ];

  environment.persistence."/persist" = {
    hideMounts = true;
    directories = [
      "/var/log"
      "/var/lib/nixos"
      "/var/lib/systemd/coredump"
      { directory = "/var/lib/tailscale"; mode = "0700"; }
    ];
    files = [
      "/etc/machine-id"
      # ssh host keys often:
      "/etc/ssh/ssh_host_ed25519_key"
      "/etc/ssh/ssh_host_ed25519_key.pub"
    ];
    users.alice = {
      directories = [ "Downloads" ".ssh" ];
      files = [ ".config/sops/age/keys.txt" ]; # only if you accept this layout
    };
  };
}

Exact option paths can track module versions—confirm against current impermanence README when options shift.

neededForBoot

fileSystems."/persist".neededForBoot = true;

If /persist is not available early enough, activation ordering fails in mysterious ways. Treat this as non-negotiable for real setups.

Theory 3 — btrfs rollback-on-boot pattern

Common blog/production pattern:

  1. Root subvolume @
  2. On boot, optionally reset @ to empty snapshot
  3. Mount @persist at /persist
  4. Impermanence binds declared paths
# Conceptual boot.initrd or systemd unit logic—use a maintained pattern
# (e.g. known btrfs wipe scripts in community flakes). Prefer copying a
# pattern you understand over opaque one-liners.
Layout piece Role
@ Ephemeral root
@persist Declared state
@nix Store (never auto-wipe)
EFI Bootloader

ZFS native datasets with mountpoint and rollback are an alternative school—pick one for the lab. Mixing half-understood btrfs blog posts with half-understood ZFS is how you reinstall for the wrong reason.

Theory 4 — What breaks first (expect it)

Symptom Cause Fix
Cannot SSH after reboot Host keys not persisted Persist /etc/ssh/ssh_host_*
Machine ID churn /etc/machine-id ephemeral Persist or accept
sops fails Age key path wiped Persist key path; mode 600
Docker/podman volumes gone State under /var/lib Persist selected dirs
User linger/services odd systemd user state Persist carefully or redesign
Tailscale identity reset State dir wiped Persist with tight mode
NetworkManager connections gone Connection profiles ephemeral Persist or declarative NM

Iterate: wipe/reboot is a test suite, not a one-shot.

Theory 5 — Soft impermanence

Not ready to wipe root? Still gain discipline:

# Document-only policy
# - no manual /etc edits
# - all state dirs listed in inventory
# - weekly diff against expected paths

Or tmpfs for /tmp only—training wheels.

Level What you prove
A Full Wipe + persist list + reboot proof
B Partial Persist mounts without wipe; inventory honest
C Skip Written architecture why mutable root is OK

All three are valid if documented. Silent skip is not.

Theory 6 — Interaction with generations and secrets

Impermanence does not replace Nix generations. Store rollbacks and root wipes are different axes:

Generation rollback  →  previous system closure
Root wipe            →  empty / except binds from /persist
Secrets              →  must live on persist or offline media
Scenario Tool
Bad module activation Generation rollback (Day 13 / Day 89)
Unknown files on root Impermanence wipe
Lost age key Offline backup; Day 88 pain if missing

Theory 7 — Home Manager alignment

If HM manages ~/.config/... on an impermanent home, either:

  • Persist the HM-managed paths you care about, or
  • Accept regenerate-from-flake behavior, or
  • Keep home on a persistent dataset

Mismatch symptoms: “my HM files vanished” after reboot. Align Day 25–27 layouts with today’s persist lists.

Worked example — minimal VM proof

# hosts/imperm-lab/configuration.nix fragments
{
  imports = [
    ./disko-btrfs.nix
    ./persist.nix
  ];

  fileSystems."/persist".neededForBoot = true;

  users.users.root.openssh.authorizedKeys.keys = [
    "ssh-ed25519 AAAA… lab"
  ];
  services.openssh.enable = true;
  system.stateVersion = "26.05";
}

Proof protocol:

echo "ephemeral-should-die" | sudo tee /root/SCRATCH
echo "persist-me" | sudo tee /persist/PROOF
sudo reboot
# after boot:
cat /root/SCRATCH     # should fail
cat /persist/PROOF    # should work

Capture both outcomes in notes. A proof that only tests the success path is incomplete.

Lab — multi-step

Suggested workspace notes: ~/lab/90daysofx/02-nixos/day58

Step 1 — Decide commit level

Level Action today
A Full btrfs/ZFS + impermanence module + wipe proof
B Partial Persist module without wipe; list future wipe
C Skip Written architecture review why not (still do path inventory)

Record choice in IMPERMANENCE.md.

Step 2 — State inventory

On current lab host:

sudo du -xh /var/lib | sort -h | tail -n 30
ls /etc/ssh
systemctl --user list-units --failed 2>/dev/null || true

Classify each important path: persist / regenerate / ignore.

Step 3 — Implement level A or B

Wire flake input; persist SSH keys + one service state dir you actually use. Set modes deliberately.

Step 4 — Proof reboot

Create SCRATCH vs PROOF files; reboot; record outcomes with timestamps.

Step 5 — Secrets path

Confirm sops-nix/agenix key path survives (or document regenerate procedure and offline backup location).

Step 6 — Rollback story interaction

Write 5 lines: store generations vs root wipes. Link Day 89 thinking early.

Step 7 — If Level C

Write one page: threat model, backup story (Day 69), why mutable root is acceptable for this host class, and what would trigger revisiting (e.g. multi-user, kiosk, fleet sameness).

Step 8 — Persist list review

Print effective list; remove anything that is “comfort mutable root.” Goal is intentional survival, not nostalgia.

Common gotchas

Symptom / mistake What to do
/persist not neededForBoot Boot fails ordering; set flag
Persisted too little Iterate after each surprise reboot
Persisted too much You rebuilt a mutable root in disguise
Home manager vs system paths Align HM files with persist lists
LUKS + wipe scripts Order of unlock vs mounts
Forgetting /nix separation Do not wipe the store subvolume
SSH host key regenerate Clients scream TOFU; persist keys or automate known_hosts
Age key mode 644 Tighten; treat as root-equivalent

Checkpoint

  • Level A/B/C chosen and documented
  • State inventory exists
  • Reboot proof (or skip rationale) recorded
  • SSH still works after reboot if remote
  • Secrets path understood
  • Generations vs wipe note written

Commit

git add .
git commit -m "day58: impermanence proof or documented skip"

Write three personal gotchas before continuing.

Tomorrow

Day 59 — deploy-rs: push a lab config to a remote VM the boring way.

Day 59 — deploy-rs

Stage VI · ~4h (lab-heavy)
Goal: Deploy a flake-defined NixOS host to a remote machine with deploy-rs, understand activation profiles, and succeed at least once end-to-end over SSH. Target config: NixOS 26.05.

Note

nixos-rebuild --target-host works; fleets still standardize on deploy tools for profiles, magic rollback, and multi-node later. deploy-rs is a common choice for small-to-medium flakes before (or beside) Colmena (Day 60).

Flake build deploy switch flow

Deploy flow

Why this day exists

Pushing a closure and activating it without sitting on the machine is the core of fleet ops. Today installs the boring muscle memory: SSH auth, profile activation, proof file, second delta, and a safe failure story.

Theory 1 — Deploy pipeline

local eval (flake) → build (local/remote) → copy closure → activate → health check → keep/rollback
Step Failure mode
Eval Module errors
Build Compile / missing cache
Copy SSH/disk
Activate Boot/switch scripts fail
Confirm Custom checks fail → auto rollback (when configured)

Understanding where it failed is half the job—read the stage in the log before rewriting modules blindly.

Theory 2 — Flake skeleton with deploy-rs

{
  description = "Day 59 deploy-rs lab";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
    deploy-rs.url = "github:serokell/deploy-rs";
    deploy-rs.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, deploy-rs, ... }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };
    in {
      nixosConfigurations.lab-vm = nixpkgs.lib.nixosSystem {
        inherit system;
        modules = [ ./hosts/lab-vm/configuration.nix ];
      };

      deploy.nodes.lab-vm = {
        hostname = "lab-vm.example.internal"; # or IP
        profiles.system = {
          user = "root";
          path = deploy-rs.lib.${system}.activate.nixos
            self.nixosConfigurations.lab-vm;
        };
      };

      checks = builtins.mapAttrs
        (system: deployLib: deployLib.deployChecks self.deploy)
        deploy-rs.lib;
    };
}
nix run github:serokell/deploy-rs -- .#lab-vm
# or if deploy-rs is in the flake / shell:
deploy .#lab-vm

Exact CLI entrypoints: prefer deploy .#node after nix shell github:serokell/deploy-rs. Pin via flake input so CI and laptop match.

Theory 3 — SSH prerequisites

ssh root@lab-vm.example.internal nix --version
# or deploy user with passwordless sudo / root activation per your policy
Requirement Notes
Nix on target NixOS has it; confirm version band
Flakes enabled On target as needed for activation
Disk space Closures are large first time
Network to substituters Or pre-pushed cache
Non-interactive auth Keys, not passwords 
# target snippets
services.openssh.enable = true;
users.users.root.openssh.authorizedKeys.keys = [
  "ssh-ed25519 AAAA… deploy"
];
# Prefer dedicated deploy key; Day 36 hardening still applies
system.stateVersion = "26.05";

Theory 4 — Magic rollback / confirmation

deploy-rs can require confirmation that the new system is healthy; if SSH dies, it rolls back. Read current serokell docs for confirmTimeout / magic-rollback options for your version—enable something before testing firewall changes remotely.

deploy.nodes.lab-vm = {
  hostname = "192.0.2.50";
  # sshOpts = [ "-p" "22" ];
  profiles.system = {
    user = "root";
    path = deploy-rs.lib.x86_64-linux.activate.nixos
      self.nixosConfigurations.lab-vm;
    # magicRollback / confirmTimeout per current docs
  };
};

Always keep a console/VNC path on the first remote firewall experiment. Magic rollback is not a substitute for out-of-band access.

Theory 5 — Build locality

Mode Idea
Build local, copy Laptop/CI builds; target only activates
Build on target Sometimes via remote builder / deploy options
Build on remote builder + cache Stage V machinery

For lab: local build + copy is fine if architectures match. Cross-arch without a builder is pain—fix Day 54 first if needed.

Theory 6 — Profiles beyond system

deploy-rs supports multiple profiles per node (system is the common first). Later you might activate user/home profiles; today master one system profile end-to-end.

Profile idea Role
system NixOS activation
Future custom App-only activations (advanced)

Do not invent multi-profile complexity until single profile is boring.

Worked example — trivial visible change

# hosts/lab-vm/configuration.nix
{ pkgs, ... }:
{
  networking.hostName = "lab-vm";
  environment.systemPackages = [ pkgs.hello pkgs.git ];
  environment.etc."day59".text = "deployed-by-deploy-rs\n";
  services.openssh.enable = true;
  system.stateVersion = "26.05";
}
deploy .#lab-vm
ssh lab-vm cat /etc/day59

Second deploy: change the text; redeploy; confirm update. Two successful deploys beat one lucky shot.

Lab — multi-step

Suggested workspace: your fleet flake; notes in ~/lab/90daysofx/02-nixos/day59

Step 1 — Target ready

Disposable NixOS VM with SSH key auth. Snapshot the VM before first deploy.

Step 2 — Add deploy-rs input

follows nixpkgs; lock; commit.

Step 3 — Wire deploy.nodes

Match nixosConfigurations name; set hostname/IP. Prefer IP in lab DNS chaos.

Step 4 — First deploy

nix flake check   # includes deployChecks if wired
deploy .#lab-vm
# -s / skip-checks flags: know them; don’t live on skip forever

Capture full terminal output in notes.

Step 5 — Prove activation

ssh lab-vm cat /etc/day59
ssh lab-vm nixos-version
ssh lab-vm readlink /run/current-system

Step 6 — Second deploy delta

Change environment.etc text; redeploy; confirm update. Note wall time vs first deploy (cache effects).

Step 7 — Failure drill (safe)

Deploy a config that adds a harmless broken oneshot or bad package, observe failure handling; restore from snapshot if needed. Do not lock yourself out on purpose without console.

Step 8 — Compare one-liner

Once, run classic:

nixos-rebuild switch --flake .#lab-vm --target-host root@lab-vm

Write 5 lines: when you prefer deploy-rs vs rebuild.

Step 9 — Document RUNBOOK seed

# Deploy lab-vm
1. VPN/SSH path
2. deploy .#lab-vm
3. health: cat /etc/day59

Step 10 — Generation awareness after deploy

ssh lab-vm -- sudo nix-env -p /nix/var/nix/profiles/system --list-generations | tail

Note generation number before/after deploy. This is Day 89 prep: you cannot rollback what you never listed.

Step 11 — Cache interaction

Deploy once with cold cache (or after GC on target), once with warm substitutes. Record wall times. Link Day 53: deploy without cache is a social incident for fleets.

Profile activation mental model

deploy-rs
  builds profile path P (NixOS system)
  copies closure of P to target
  activates P (switch-to-configuration)
  optional confirm / magic rollback window
If this fails… Look at
Eval on laptop modules, flake inputs
Build caches, builders, arch
Copy SSH, disk, bandwidth
Activate systemd, scripts, secrets paths
Confirm timeout network cut by your own firewall

Common gotchas

Symptom / mistake What to do
Arch mismatch Build on same system or use remote builder
Permission denied root keys / sudo deploy user
Out of disk on target GC; grow disk
Hang on activation Console; magic rollback timeouts
deploy not found nix shell the tool
Hostname DNS Use IP for lab
Checks fail locally Fix deployChecks or understand impure needs
follows forgotten Mixed nixpkgs between deploy-rs and host

Checkpoint

  • deploy-rs in flake
  • Successful remote activation
  • Visible file/package change verified over SSH
  • Second delta deploy succeeded
  • Comparison note vs nixos-rebuild --target-host

Commit

git add .
git commit -m "day59: deploy-rs remote lab activation"

Write three personal gotchas before continuing.

Tomorrow

Day 60 — Colmena: multi-host tags and parallel deploy.