Day 8 — Boot, Filesystems & Gate I

Updated

July 30, 2025

Day 8 — Boot, Filesystems & Gate I

Day 20 — Boot & hardware

Stage II · ~4h
Goal: Document and intentionally configure the lab’s boot loader, kernel packages, and firmware—so recovery and hardware quirks are not mysteries.

Important

Boot experiments can brick a bad disk layout. Prefer the disposable VM, snapshot first, and change one variable at a time.

Why this day exists

When generations fail to boot, the difference between “NixOS is broken” and “I know how to pick last generation / fix EFI” is this literacy. Hardware config is part of ownership.

Theory 1 — Boot chain (UEFI lab)

UEFI firmware
  → EFI system partition (ESP) files
  → boot loader (systemd-boot or GRUB)
  → kernel + initrd (from a generation)
  → stage-1 (find root, unlock, …)
  → stage-2 / systemd

NixOS generations typically install loader entries pointing at store paths for kernel/initrd.

Theory 2 — systemd-boot vs GRUB

Loader Common on Notes
systemd-boot UEFI VMs/laptops Simple, generation entries
GRUB BIOS + UEFI More knobs; legacy friendly 
# systemd-boot (UEFI)
boot.loader.systemd-boot.enable = true;
boot.loader.efi.canTouchEfiVariables = true;

# GRUB example (do not enable both casually)
# boot.loader.grub.enable = true;
# boot.loader.grub.device = "/dev/vda";

Pick one model. Mixing without care is a classic footgun.

Generation UI

  • systemd-boot: machine-id entries, often need key to show menu
  • GRUB: menu with generation list

Document how your VM displays the menu (ESC, hold space, etc.).

Theory 3 — Kernel packages

boot.kernelPackages = pkgs.linuxPackages;          # default channel kernel
# boot.kernelPackages = pkgs.linuxPackages_latest;
# boot.kernelPackages = pkgs.linuxPackages_6_12; # example versioned attr — check 26.05
Choice Trade-off
Default Matches nixpkgs expectations
_latest Newer hardware; more churn
Pinned series Stability / specific needs

Kernel modules:

boot.kernelModules = [ ];
boot.initrd.availableKernelModules = [ /* often from hardware-config */ ];
boot.extraModulePackages = [ ];

hardware-configuration.nix usually fills initrd modules for disks/filesystems—do not delete blindly.

Theory 4 — Firmware

hardware.enableRedistributableFirmware = true;
# hardware.enableAllFirmware = true; # broader; licensing awareness

VMs often need little firmware; bare metal Wi-Fi/GPU might need redistributable firmware enabled.

Theory 5 — CPU / virt hardware bits

Common NixOS knobs:

# virtualisation-friendly examples (enable only if true for you)
# services.qemuGuest.enable = true;
# virtualisation.hypervGuest.enable = true;

Also: microcode updates on real Intel/AMD machines (hardware.cpu.*.updateMicrocode)—relevant on bare metal, less so on many VMs.

Theory 6 — hardware-configuration.nix discipline

Do Don’t
Import it Hand-edit UUIDs casually
Regenerate after disk changes Copy another machine’s hardware file
Review initrd modules Assume it never needs refresh

Regenerate carefully (live media or docs)—wrong root UUID → unbootable.

Theory 7 — Temporary boot parameters

Kernel params:

boot.kernelParams = [ "console=ttyS0" ]; # example serial console for some VMs

Useful for debugging; remove noise when done. One change per rebuild when experimenting.

Worked examples bank

Example A — Document current loader

bootctl status 2>/dev/null | head -40
ls /boot 2>/dev/null | head
ls /boot/loader/entries 2>/dev/null | head

Example B — Kernel identity

uname -r
nixos-version
ls /run/current-system/kernel

Example C — Safe param experiment

Add a harmless boot.kernelParams comment-level change only after snapshot; or set a documented console param if your hypervisor uses serial.

Lab 1 — Boot path document

Create docs/boot.md in the flake repo:

# Boot path — lab
- Firmware: UEFI / BIOS
- Loader: systemd-boot / GRUB
- ESP mount: /boot (options)
- Kernel package option: …
- uname -r: …
- How to open generation menu: …
- Snapshot name: …

Fill with real commands’ output.

Lab 2 — Verify loader config matches reality

Compare flake boot.loader.* with bootctl status / GRUB presence. Fix mismatches.

Lab 3 — Kernel package awareness

  1. Record current boot.kernelPackages (implicit default if unset).
  2. Optionally try linuxPackages_latest in a VM with snapshot.
  3. Reboot; uname -r; note difference.
  4. Revert if not needed.

If you skip the change, still write how you would do it.

Lab 4 — Firmware decision

State in docs:

  • VM only → firmware settings chosen
  • Bare metal → enable redistributable firmware if devices need it

Rebuild if you change it.

Lab 5 — Recovery dry-run (no panic)

Without breaking the system:

  1. List generations in bootloader docs
  2. Rehearse verbally: “If black screen after upgrade, I will…”
  3. Update Day 13 rollback card with boot-menu specifics

Theory 8 — boot.loader.timeout and generation menu

boot.loader.timeout = 3; # seconds; 0 may hide menu unless key held

For labs that practice rollback, a short non-zero timeout is friendlier than zero. Document the key to force the menu on your firmware (often Esc or hold Space for systemd-boot).

Theory 9 — Initrd and LUKS awareness (preview)

Encrypted roots need initrd unlock configuration. Even if your VM is plaintext:

Concept Why note it
boot.initrd.luks.devices Maps UUID → mapped name
Prompt vs keyfile Physical vs automated unlock
Wrong UUID Emergency shell at stage-1

Disko/impermanence (Stage VI) will revisit layout; today only name the risk.

Theory 10 — hardware-configuration.nix vs flake purity

hardware-configuration.nix is machine-specific generated state. It belongs in git for the lab host or is generated on first install and kept private per machine.

Approach Trade-off
Commit lab VM hardware file Easy clone of this VM
Generate per machine Correct for fleet; clone drill needs care

Do not copy UUIDs between bare metal and VM casually.

Worked example D — QEMU guest agent (virt)

services.qemuGuest.enable = true; # if hypervisor is QEMU/KVM

Helps shutdown/reboots from host tooling. Enable only when accurate.

Lab 6 — Loader entries inventory

bootctl list 2>/dev/null | head -60
ls -la /boot/loader/entries 2>/dev/null | head

Map one entry to a NixOS generation number in docs/boot.md.

Lab 7 — Kernel param one-shot (snapshot)

With a VM snapshot:

boot.kernelParams = [ "loglevel=7" ]; # noisier logs; lab only

Rebuild, reboot, confirm cat /proc/cmdline, revert.

Common gotchas

Symptom Theory
Two boot loaders enabled Conflicts
canTouchEfiVariables false in some virt May need alternate install path
Removed initrd modules Disk not found
Latest kernel + guest tools mismatch Virt drivers
Editing /boot by hand Lost on rebuild

Checkpoint

  • docs/boot.md complete
  • Loader in flake matches reality
  • Kernel identity recorded
  • Firmware policy chosen
  • Rollback card updated with boot menu
  • Snapshot available before any kernel play

Commit

cd ~/nixos-config
git add .
git commit -m "day20: boot hardware documentation"

Tomorrow

Day 21 — Filesystems intro. Mounts, state vs store, and the impermanence idea (persist deliberately)—without a full impermanence deployment yet.

Day 21 — Filesystems intro

Stage II · ~4h (theory-heavy)
Goal: Inventory mounts and partitions; separate store, state, and ephemeral paths; understand the impermanence idea (persist deliberately) without implementing full impermanence yet.

Why this day exists

Nix makes the system closure rebuildable; it does not automatically make your data policy sane. Ops failures often come from “I rolled back the system but the database is gone” or the reverse. Label what must persist.

Theory 1 — Three layers of “the disk”

Layer Examples Rebuild replaces?
Store /nix/store Via new paths; GC collects dead
Config activation much of /etc Yes, from generation
State /var, /home, databases No (unless you design it) 
┌────────────────────────────┐
│ /home  /var  (state)       │  ← backup & persist policy
├────────────────────────────┤
│ /etc (mixed; mostly managed)│
├────────────────────────────┤
│ /nix/store (immutable pkgs) │
└────────────────────────────┘

Theory 2 — fileSystems options

From hardware-configuration.nix typically:

fileSystems."/" = {
  device = "/dev/disk/by-uuid/…";
  fsType = "ext4";
};

fileSystems."/boot" = {
  device = "/dev/disk/by-uuid/…";
  fsType = "vfat";
};
Field Role
device UUID/LABEL preferred over raw /dev/sdX
fsType ext4, btrfs, xfs, vfat, zfs, …
options mount flags (noatime, …)
neededForBoot initrd critical mounts

Swap:

swapDevices = [{ device = "/dev/disk/by-uuid/…"; }];

Theory 3 — Why UUID/LABEL

Device names (vda2, sda1) can reorder. By-uuid is stable:

ls -l /dev/disk/by-uuid
lsblk -f

Theory 4 — State directories that matter

Path Typical content
/var/lib Service state (postgres, docker, …)
/var/log Logs (sometimes ephemeral OK)
/var/cache Regenerable caches
/home User data, secrets, keys
/root Root’s home
/etc/ssh host keys Machine identity (special)
/nix Store + nix state database

When you enable services (Stage IV+), each service’s state dir is part of your persist list.

Theory 5 — Impermanence idea (concept only)

Impermanence (pattern/tooling, Day 58 deep dive): treat root as mostly ephemeral; bind-mount or symlink only chosen paths from a persistent volume.

boot → empty-ish root
     → restore/persist: /home, /var/lib/postgres, /etc/ssh, …
     → everything else regenerated by NixOS activation

Why people want it

Benefit Meaning
Force declarative truth Untracked snowflakes disappear
Cleaner systems Less drift
Clear backup targets Persist volume = backup unit

Costs

Cost Meaning
Design work Must list persist paths correctly
Host keys / secrets Easy to forget; painful
Debugging “Where did my file go?”

Today: you do not need to enable impermanence. You do need a written list of what would be persisted.

Theory 6 — Store is not a backup of state

nixos-rebuild switch
  → new system closure
  ≠ snapshot of /var/lib/postgresql

Backups are a separate story (Day 69). Generations help software; not data.

Theory 7 — tmpfs and ephemeral mounts (awareness)

# conceptual examples — do not apply casually
# boot.tmp.useTmpfs = true;
# fileSystems."/tmp" = { device = "tmpfs"; fsType = "tmpfs"; };

Useful later with impermanence. Understanding: not every path needs disk durability.

Worked examples bank

Example A — Mount inventory

findmnt -R /
lsblk -f
df -hT

Example B — What’s big?

sudo du -xhd1 / 2>/dev/null | sort -h
sudo du -sh /nix/store /var /home 2>/dev/null

Example C — SSH host key paths

ls -l /etc/ssh/ssh_host_*

These are state even though under /etc.

Lab 1 — Full inventory table

In docs/filesystems.md:

Mountpoint device/UUID fsType options role (root/boot/state/…)

Fill from findmnt + lsblk -f.

Lab 2 — Persist classification

Label each:

Path Persist? Why Backup?
/nix/store rebuildable no (or cache strategy)
/home/alice yes yes
/var/log maybe optional
/var/lib mostly yes yes
/etc/ssh host keys yes identity yes
/etc/nixos if still used prefer git flake git

Be specific to your lab services.

Lab 3 — State created by your Stage II work

List state produced so far:

  • user homes
  • SSH host keys
  • timer logs you wrote
  • any DB? (probably not yet)

Lab 4 — Impermanence reading (light)

Skim what Day 58 will cover (impermanence module concepts) from memory of Theory 5. Write five sentences: Would I want impermanence on this lab VM? Why/why not?

No implementation required.

Lab 5 — Danger checklist

Confirm you will never:

  • rm -rf /nix/store as “cleanup”
  • Format ESP casually
  • Delete /var/lib because “Nix will regenerate” (it will not for app data)

Theory 8 — Btrfs / ZFS awareness without deploying them

You may see labs using btrfs snapshots or ZFS datasets. For Stage II:

FS Persist story
ext4 default VM Simple; backups = copy/dump
btrfs Subvolumes + snapshots possible
ZFS Datasets; often external to vanilla hardware-config

Do not switch root FS casually mid-lab. Understand that snapshot tools are about state, not store purity.

Theory 9 — /nix on its own mount (optional design)

Some installs put /nix on a dedicated volume:

Benefit Cost
Reinstall root without losing store cache More mount points to get right
Clear size accounting Hardware-config complexity

Record whether your lab is single-root or split.

Theory 10 — Bind mounts as proto-impermanence

# conceptual — only if you knowingly add a data disk
# fileSystems."/var/lib/postgres" = {
#   device = "/data/postgres";
#   options = [ "bind" ];
# };

Bind mounts are how many impermanence setups pin state onto a durable dataset. Note the idea; implement later (Day 58).

Worked example D — neededForBoot meaning

fileSystems."/" = {
  device = "/dev/disk/by-uuid/…";
  fsType = "ext4";
  neededForBoot = true; # default true for /
};

Extra mounts required to reach multi-user (e.g. separate /var) may need neededForBoot = true or the system reaches emergency targets.

Lab 6 — What survives nixos-rebuild switch?

Create a marker:

echo "state-$(date -Is)" | sudo tee /var/tmp/day21-marker
echo "etc-try" | sudo tee /etc/day21-hand-edit 2>/dev/null || true

Rebuild switch. Check which markers remain. Note: hand-edited /etc files may be replaced by activation—another reason flake is SoT.

Lab 7 — Size budget sentence

From Lab 2 du output, write one budget sentence:

“If I wipe the VM disk, I lose X (state) but can rebuild Y (system) from git+flake.lock.”

Common gotchas

Symptom Theory
Rolled back OS, data weird Data ≠ generation
Copied hardware-config UUIDs wrong Unbootable
Thought /etc fully ephemeral Host keys, some static files
No backup of home Lab loss on disk wipe
Impermanence without persist list Instant footgun

Checkpoint

  • Mount inventory table complete
  • Persist classification written
  • Store vs state explained in own words
  • Impermanence idea summarized
  • Host keys noted as state
  • Docs committed

Commit

cd ~/nixos-config
git add .
git commit -m "day21: filesystems inventory persist vs store"

Tomorrow

Day 22 — Stage II gate. Minimal production-shaped host: users, SSH, firewall, one service, one custom module—all in the flake; cold boot + rollback proof.

Day 22 — Stage II gate

Stage II · ~4h (review + proof)
Goal: Prove a minimal production-shaped flake-managed NixOS lab: users, SSH keys, firewall, one service, one custom module—cold boot and rollback work; the flake is the only source of truth.

Why this day exists

Stage III multiplies structure (layouts, Home Manager). Entering it with a snowflake host guarantees pain. Today is an honesty gate: the machine is rebuildable, operable, and documented.

Theory 1 — Syllabus exit criteria

You may leave Stage II when:

  1. From cold boot you can roll back a bad generation
  2. The flake is the only source of truth for the lab host
  3. Host includes: declarative users, SSH key access, firewall policy, at least one service/timer, one custom module
  4. Notes cover boot path + network + recovery

Theory 2 — Minimal production-shaped host (definition for this gate)

Capability Evidence
Flake host nixosConfigurations.<name> rebuilds
Pin flake.lock committed; nixpkgs 26.05
User Non-root admin in wheel
Remote access SSH key login; password SSH off (or justified)
Firewall Enabled; SSH allowed; extras documented
Packages Daily tools declarative
Service Custom or high-level service active
Module Local options+config module imported
Docs rebuild, rollback, boot, network
Recovery Rollback card tested

Not required yet: secrets machinery, Home Manager, Disko redesign, multi-host, reverse proxy zoo.

Theory 3 — Cold boot proof

running system (gen N)
  → reboot
  → firmware → loader → kernel → multi-user
  → SSH/console login works
  → systemctl is-system-running (or degraded explained)

Cold boot catches “worked in this session only” activation myths.

Theory 4 — Source of truth test

Signal flake is SoT Smell of drift
Rebuild from git repo only Hand edits in /etc that vanish/surprise
README rebuild command works “Also run these shell fixes”
hardware-config in git Hardware only on machine, never backed up

Theory 5 — Gate knowledge verbalization

Prepare short answers:

  1. switch vs test vs boot
  2. What rollback does not restore
  3. Why SSH public keys in git are OK but passwords are not
  4. Store vs state paths
  5. Why custom modules use mkIf cfg.enable
  6. How you open the boot generation menu

Worked examples bank

Example A — Gate rebuild

cd ~/nixos-config
sudo nixos-rebuild switch --flake .#lab

Example B — Eval sanity

nix flake show
nix eval .#nixosConfigurations.lab.config.networking.hostName
nix eval .#nixosConfigurations.lab.config.networking.firewall.allowedTCPPorts

Example C — Service proof

systemctl is-enabled sshd || systemctl status sshd
systemctl list-timers --all | head

Lab 1 — Gate repository hygiene

cd ~/nixos-config
git status
git log --oneline | head
test -f flake.lock && echo lock-ok

Ensure structure roughly:

flake.nix
flake.lock
hosts/lab/configuration.nix
hosts/lab/hardware-configuration.nix
modules/*.nix
docs/{boot,network,rollback,filesystems,modules-study}.md
README.md

Missing docs → write them now (brief is fine).

Lab 2 — Checklist (tick only if true)

Stage II gate checklist

  • NixOS 26.05 lab boots (VM preferred)
  • nix --version / nixos-version recorded
  • Flake nixosConfigurations rebuild switch works
  • flake.lock committed
  • Hostname + flake attr documented
  • Admin user declarative + wheel
  • SSH key login works
  • PasswordAuthentication disabled or exception documented
  • Firewall enabled; SSH port allowed
  • Port audit exists
  • environment.systemPackages / programs.* used intentionally
  • ≥1 custom systemd service or timer or meaningful services.*
  • ≥1 custom module with enable option
  • Assertion or deliberate option validation somewhere
  • Generations listed; rollback performed at least once in Stage II
  • Boot loader + generation menu documented
  • Mount inventory + persist list written
  • README rebuild instructions accurate
  • No private keys or live passwords in git
  • Cold boot → login succeeds

Any empty box is a stop for Stage III.

Lab 3 — Cold boot ceremony

  1. VM snapshot labeled day22-pre-coldboot (optional safety)
  2. Reboot lab
  3. Login console and/or SSH
  4. systemctl is-system-running || true
  5. hostname; curl localhost service if any
  6. Record timestamp + generation

Lab 4 — Rollback proof (again)

  1. Make a tiny visible change (motd package, environment.etc."day22".text, …)
  2. switch
  3. Confirm change
  4. sudo nixos-rebuild switch --rollback
  5. Confirm reversion
  6. Re-apply good config if needed

Lab 5 — Architecture one-pager

docs/host-diagram.md:

# lab host
- flake: ~/nixos-config #lab
- users: …
- ssh: keys only …
- firewall: …
- services/timers: …
- modules: …
- boot: …
- disks: …
- what is state: …

Lab 6 — Teach-back (10–15 min)

Explain aloud or written:

“Here is how I rebuild this host from git; here is how I recover from a bad generation; here is what I must back up because Nix will not.”

If rough, revisit Days 13–15 and 21.

Theory 6 — Generation evidence commands

sudo nix-env -p /nix/var/nix/profiles/system --list-generations | tail -5
readlink -f /run/current-system
nixos-version

Gate notes should include current generation number and flake shortrev if you embed git info (optional).

Theory 7 — What “flake is SoT” excludes

Still allowed outside the flake (state):

  • Home directory contents (until HM owns them)
  • SSH host keys
  • Service databases (when you add them)
  • Hypervisor snapshots

Not allowed as required ops procedure:

  • “Also apt install…”
  • “Then manually systemctl edit…”
  • “Copy this binary into /usr/local”

Worked example D — Eval probes for gate

cd ~/nixos-config
nix eval .#nixosConfigurations.lab.config.users.users.alice.isNormalUser
nix eval .#nixosConfigurations.lab.config.services.openssh.enable
nix eval .#nixosConfigurations.lab.config.networking.firewall.enable

Boolean/string outputs are quick CI-less checks that config intent is present.

Lab 7 — Secret smell re-scan

cd ~/nixos-config
rg -n "initialPassword|password\s*=\s*\"" || true
rg -n "PRIVATE KEY|AKIA" || true

Stage IV will add secret machinery; Stage II must not already leak.

Lab 8 — Offline notes pack

Bundle in docs/ or a gate/ folder:

  1. Rebuild one-liner
  2. Rollback one-liner
  3. Boot menu steps
  4. Console lockout recovery
  5. Network identity (IP/interface)

Print or keep on the host that is not the VM (so a broken VM does not eat the runbook).

Common gotchas

Symptom Theory
Gate “passes” but docs missing Future you fails
Secrets in history Stage IV will not save past leaks
Service only started manually Not declarative
Hardware file not in git Disk loss → pain
Still using channels for system Diverges from flake story

Checkpoint

  • All Stage II checklist boxes ticked
  • Cold boot verified
  • Rollback re-verified
  • Host diagram written
  • Teach-back completed
  • Ready for Stage III layout + Home Manager

Commit

cd ~/nixos-config
git add .
git commit -m "day22: stage II gate production-shaped host"

Tag optional:

git tag stage-ii-gate

Tomorrow

Day 23 — Flake layout. Split hosts/ / modules/ / homes/ cleanly; stop growing a single tangled config as Stage III begins (Home Manager arrives soon after).