Day 21 — Filesystems intro

Updated

July 30, 2026

Day 21 — Filesystems intro

Stage II · ~4h (theory-heavy)
Goal: Inventory mounts and partitions; separate store, state, and ephemeral paths; understand the impermanence idea (persist deliberately) without implementing full impermanence yet.

Why this day exists

Nix makes the system closure rebuildable; it does not automatically make your data policy sane. Ops failures often come from “I rolled back the system but the database is gone” or the reverse. Label what must persist.

Theory 1 — Three layers of “the disk”

Layer Examples Rebuild replaces?
Store /nix/store Via new paths; GC collects dead
Config activation much of /etc Yes, from generation
State /var, /home, databases No (unless you design it) 
┌────────────────────────────┐
│ /home  /var  (state)       │  ← backup & persist policy
├────────────────────────────┤
│ /etc (mixed; mostly managed)│
├────────────────────────────┤
│ /nix/store (immutable pkgs) │
└────────────────────────────┘

Theory 2 — fileSystems options

From hardware-configuration.nix typically:

fileSystems."/" = {
  device = "/dev/disk/by-uuid/…";
  fsType = "ext4";
};

fileSystems."/boot" = {
  device = "/dev/disk/by-uuid/…";
  fsType = "vfat";
};
Field Role
device UUID/LABEL preferred over raw /dev/sdX
fsType ext4, btrfs, xfs, vfat, zfs, …
options mount flags (noatime, …)
neededForBoot initrd critical mounts

Swap:

swapDevices = [{ device = "/dev/disk/by-uuid/…"; }];

Theory 3 — Why UUID/LABEL

Device names (vda2, sda1) can reorder. By-uuid is stable:

ls -l /dev/disk/by-uuid
lsblk -f

Theory 4 — State directories that matter

Path Typical content
/var/lib Service state (postgres, docker, …)
/var/log Logs (sometimes ephemeral OK)
/var/cache Regenerable caches
/home User data, secrets, keys
/root Root’s home
/etc/ssh host keys Machine identity (special)
/nix Store + nix state database

When you enable services (Stage IV+), each service’s state dir is part of your persist list.

Theory 5 — Impermanence idea (concept only)

Impermanence (pattern/tooling, Day 58 deep dive): treat root as mostly ephemeral; bind-mount or symlink only chosen paths from a persistent volume.

boot → empty-ish root
     → restore/persist: /home, /var/lib/postgres, /etc/ssh, …
     → everything else regenerated by NixOS activation

Why people want it

Benefit Meaning
Force declarative truth Untracked snowflakes disappear
Cleaner systems Less drift
Clear backup targets Persist volume = backup unit

Costs

Cost Meaning
Design work Must list persist paths correctly
Host keys / secrets Easy to forget; painful
Debugging “Where did my file go?”

Today: you do not need to enable impermanence. You do need a written list of what would be persisted.

Theory 6 — Store is not a backup of state

nixos-rebuild switch
  → new system closure
  ≠ snapshot of /var/lib/postgresql

Backups are a separate story (Day 69). Generations help software; not data.

Theory 7 — tmpfs and ephemeral mounts (awareness)

# conceptual examples — do not apply casually
# boot.tmp.useTmpfs = true;
# fileSystems."/tmp" = { device = "tmpfs"; fsType = "tmpfs"; };

Useful later with impermanence. Understanding: not every path needs disk durability.

Worked examples bank

Example A — Mount inventory

findmnt -R /
lsblk -f
df -hT

Example B — What’s big?

sudo du -xhd1 / 2>/dev/null | sort -h
sudo du -sh /nix/store /var /home 2>/dev/null

Example C — SSH host key paths

ls -l /etc/ssh/ssh_host_*

These are state even though under /etc.

Lab 1 — Full inventory table

In docs/filesystems.md:

Mountpoint device/UUID fsType options role (root/boot/state/…)

Fill from findmnt + lsblk -f.

Lab 2 — Persist classification

Label each:

Path Persist? Why Backup?
/nix/store rebuildable no (or cache strategy)
/home/alice yes yes
/var/log maybe optional
/var/lib mostly yes yes
/etc/ssh host keys yes identity yes
/etc/nixos if still used prefer git flake git

Be specific to your lab services.

Lab 3 — State created by your Stage II work

List state produced so far:

  • user homes
  • SSH host keys
  • timer logs you wrote
  • any DB? (probably not yet)

Lab 4 — Impermanence reading (light)

Skim what Day 58 will cover (impermanence module concepts) from memory of Theory 5. Write five sentences: Would I want impermanence on this lab VM? Why/why not?

No implementation required.

Lab 5 — Danger checklist

Confirm you will never:

  • rm -rf /nix/store as “cleanup”
  • Format ESP casually
  • Delete /var/lib because “Nix will regenerate” (it will not for app data)

Theory 8 — Btrfs / ZFS awareness without deploying them

You may see labs using btrfs snapshots or ZFS datasets. For Stage II:

FS Persist story
ext4 default VM Simple; backups = copy/dump
btrfs Subvolumes + snapshots possible
ZFS Datasets; often external to vanilla hardware-config

Do not switch root FS casually mid-lab. Understand that snapshot tools are about state, not store purity.

Theory 9 — /nix on its own mount (optional design)

Some installs put /nix on a dedicated volume:

Benefit Cost
Reinstall root without losing store cache More mount points to get right
Clear size accounting Hardware-config complexity

Record whether your lab is single-root or split.

Theory 10 — Bind mounts as proto-impermanence

# conceptual — only if you knowingly add a data disk
# fileSystems."/var/lib/postgres" = {
#   device = "/data/postgres";
#   options = [ "bind" ];
# };

Bind mounts are how many impermanence setups pin state onto a durable dataset. Note the idea; implement later (Day 58).

Worked example D — neededForBoot meaning

fileSystems."/" = {
  device = "/dev/disk/by-uuid/…";
  fsType = "ext4";
  neededForBoot = true; # default true for /
};

Extra mounts required to reach multi-user (e.g. separate /var) may need neededForBoot = true or the system reaches emergency targets.

Lab 6 — What survives nixos-rebuild switch?

Create a marker:

echo "state-$(date -Is)" | sudo tee /var/tmp/day21-marker
echo "etc-try" | sudo tee /etc/day21-hand-edit 2>/dev/null || true

Rebuild switch. Check which markers remain. Note: hand-edited /etc files may be replaced by activation—another reason flake is SoT.

Lab 7 — Size budget sentence

From Lab 2 du output, write one budget sentence:

“If I wipe the VM disk, I lose X (state) but can rebuild Y (system) from git+flake.lock.”

Common gotchas

Symptom Theory
Rolled back OS, data weird Data ≠ generation
Copied hardware-config UUIDs wrong Unbootable
Thought /etc fully ephemeral Host keys, some static files
No backup of home Lab loss on disk wipe
Impermanence without persist list Instant footgun

Checkpoint

  • Mount inventory table complete
  • Persist classification written
  • Store vs state explained in own words
  • Impermanence idea summarized
  • Host keys noted as state
  • Docs committed

Commit

cd ~/nixos-config
git add .
git commit -m "day21: filesystems inventory persist vs store"

Tomorrow

Day 22 — Stage II gate. Minimal production-shaped host: users, SSH, firewall, one service, one custom module—all in the flake; cold boot + rollback proof.