Day 26 — Ecosystem, Comparisons & Upgrade Planning
Day 26 — Ecosystem, Comparisons & Upgrade Planning
Day 77 — Ecosystem map
Stage VII · ~4h
Goal: Map the adjacent tooling landscape, try exactly one of devenv / Snowfall / nix-darwin in a disposable trial, and write a keep/drop decision against your flake layout—without rewriting Capstone mid-flight.
Syllabus rule: try one, don’t collect all. Tool tourism is how 90-day programs die on day 77.
Why this day exists
By now your layout is intentional: flake hosts, modules, HM, deploy, tests. The ecosystem constantly offers frameworks that promise less boilerplate. Some are worth it; many are expensive rewrites. Experts sample with a comparison table, not a weekend migration of Capstone main.
Theory 1 — Layers of “ecosystem”
| Layer | Examples | Problem they target |
|---|---|---|
| Devshell UX | devenv, devbox, nix develop, direnv |
Fast project environments |
| Flake structure frameworks | flake-parts, Snowfall, digga (legacy) | Scale modules/systems/homes |
| Darwin / non-NixOS | nix-darwin, HM standalone | Same declarative story on macOS |
| Deploy | deploy-rs, Colmena, nixinate | Already in Stage VI |
| Secrets | sops-nix, agenix | Already in Stage IV |
| Caches | Cachix, Attic, Harmonia | Stage V–VI |
| Install / metal | nixos-anywhere, disko, images | Stage VI |
| UX wrappers | nh, nixos-rebuild-ng experiments | Faster day-2 commands |
| Editors / templates | templates repos, blueprint flakes | Speed of scaffolding |
Today’s forced choice is among devenv, Snowfall, nix-darwin—three different layers. Pick the one that matches your pain:
| If your pain is… | Prefer trying |
|---|---|
| Polyglot project shells, services in dev | devenv |
| Many hosts/homes, naming convention fatigue | Snowfall (or double-down on flake-parts you may already use) |
| You live on macOS half the time | nix-darwin |
| None of the above | Still pick one; learn by contrast |
Theory 2 — devenv (project environments)
What it is: A Nix-powered dev environment tool (CLI + modules) focused on languages, processes, pre-commit, and local services—closer to “compose for dev” than to NixOS hosts.
Typical win: New contributors run one tool and get DB + language toolchain.
Typical cost: Another module system dialect; CI must speak devenv; overlap with devShells you already have (Day 28).
# illustrative — follow current devenv docs for install
# nix profile install nixpkgs#devenv
# devenv init
# devenv shellCompare to house default
| Concern | nix develop + direnv |
devenv |
|---|---|---|
| Mental model | Flake devShells |
devenv modules + CLI |
| Services (postgres…) | Manual / process-compose DIY | First-class modules often |
| Capstone impact | Low if already working | Medium if dual standards |
Theory 3 — Snowfall (flake library / structure)
What it is: A convention-heavy library for laying out flakes: systems, modules, packages, homes under standard paths, auto-discovery.
Typical win: Multi-host monorepos with consistent naming and less flake.nix glue.
Typical cost: Framework lock-in; debugging indirection; migration of a working hand layout (Days 23–29).
Compare to flake-parts
| Concern | Hand layout / flake-parts | Snowfall |
|---|---|---|
| Structure | Explicit imports | Convention + library |
| Learning | Your code is visible | Magic paths |
| Capstone rewrite cost | Low if already structured | Often high mid-flight |
Snowfall and flake-parts both attack “structure,” differently. Trying Snowfall does not require abandoning flake-parts forever—decision is local.
Theory 4 — nix-darwin
What it is: Nix modules for macOS system configuration—spiritual cousin of NixOS modules, not a Linux VM.
Typical win: One repo language for laptop (Darwin) + servers (NixOS) + Home Manager.
Typical cost: Darwin modules ≠ NixOS modules; GUI/casks world; CI harder; not a substitute for your lab VM.
# only on a Mac you own — follow current nix-darwin + flakes docs
# darwin-rebuild switch --flake .#hostnameIf you have no Mac, you may still read a nix-darwin sample config and write a comparison—or pick devenv/Snowfall for a hands-on trial instead.
| Shared idea | Divergence |
|---|---|
| Module options, activation | Different option trees |
| HM integration common | GUI apps / Homebrew friction |
| Flakes | Same pin discipline |
Theory 5 — Decision framework (keep/drop)
Score 1–5 for your context:
| Criterion | Weight idea |
|---|---|
| Reduces your real weekly pain | High |
| Clear primary docs | High |
| Plays with flakes + lock discipline | High |
| Compatible with sops/deploy/tests | High |
| Rewrite cost of Capstone | Critical penalty |
| Bus factor / maintenance | Medium |
| Fashion / social media mindshare | Zero |
Default Capstone policy: adopt only if score says “yes” and migration is a branch, not a rewrite of main before Day 88.
decision ∈ { keep-current, adopt-later, trial-only-drop }
Theory 6 — Ecosystem postcard (vocabulary)
You need names for Day 78–82 conversations even without installs:
| Name | One-line purpose |
|---|---|
| flake-parts | Modular flake output composition |
| nixos-anywhere | Remote install from existing SSH |
| nh | Friendlier rebuild/search UX |
| Attic | Self-hosted binary cache |
| digga | Legacy flake framework (historical literacy) |
| blueprint / templates | Scaffold speed |
≤15 lines total in notes is enough—depth belongs to the one trial.
Worked example — COMPARE.md skeleton
# COMPARE.md — devenv vs house layout (example)
## Context
- Capstone flake with nixosConfigurations.lab, sops-nix, deploy-rs
- Pain: polyglot shells inconsistent across repos
## Scores (1–5)
| Criterion | House | devenv | Weight |
|-----------|-------|--------|--------|
| Weekly pain relief | 2 | 4 | 5 |
| Docs clarity | 4 | 4 | 4 |
| Flake/lock fit | 5 | 4 | 5 |
| sops/deploy/tests | 5 | 3 | 5 |
| Capstone rewrite cost | 5 | 2 | 5 |
## Decision
We will **not** adopt before Day 90 because rewrite cost outweighs shell pain;
revisit after Capstone freeze if side projects still hurt.
## Evidence from trial/
- commands run:
- what broke:
- time spent:Lab 0 — Workspace
mkdir -p ~/lab/90daysofx/02-nixos/day77/trial
cd ~/lab/90daysofx/02-nixos/day77Do not trial on the only copy of Capstone; branch or separate directory.
Lab 1 — Map your current layout (30 min)
In LAYOUT.md:
flake.nix outputs:
nixosConfigurations: …
homeConfigurations / HM-as-module: …
devShells: …
checks: …
packages: …
deploy / colmena: …
modules/: …
hosts/: …
secrets/: …
List top 3 pains (e.g. “shells inconsistent”, “host boilerplate”, “no Mac story”).
Lab 2 — Pick one tool and trial
If devenv
devenv initintrial/appor import docs’ minimal flake.
- Add one language toolchain you actually use.
- Optional: one service (postgres/redis) if docs make it easy.
- Enter shell; run a real command (tests/build).
- Note direnv integration if any.
- Time spent + friction log.
If Snowfall
- Scaffold a tiny Snowfall flake (one system stub, one module).
- Do not migrate all hosts.
- Trace: where does a host definition live? How is it discovered?
- Compare to your
hosts/lab/default.nixpattern.
- Eval something small; capture errors as learning.
If nix-darwin
- On Mac: flake with one
darwinConfigurations; apply to a throwaway aspect if possible.
- Or read-only deep dive: clone an example, map options vs NixOS cousins.
- Document what cannot be shared as-is with NixOS modules.
Lab 3 — Comparison table (required)
Complete COMPARE.md per Theory 5–worked example. Include explicit paragraph:
“We will / will not adopt before Day 90 because…”
Lab 4 — Ecosystem postcard
In ≤15 lines, define in your words: flake-parts, nixos-anywhere, nh, Attic. No install required.
Lab 5 — Capstone protection check
# Capstone main should be untouched by trial frameworks
cd /path/to/capstone-flake
git status
git log --oneline -5If you “accidentally” started migrating main, revert or move work to a branch named experiment/… and restore main to last known good.
Common gotchas
| Mistake | What to do |
|---|---|
| Trying all three “a little” | Violates the day; pick one |
| Rewriting Capstone main branch | Use trial/ only |
| Adopting for aesthetics | Score against pain list |
| devenv and heavy custom shells forever duplicated | Choose a house standard |
| Assuming Snowfall ≈ NixOS | It’s flake structure, not an OS |
| nix-darwin as lab substitute | Still need Linux NixOS for Capstone |
| Infinite template shopping | Timebox trial to hours, not days |
Checkpoint
LAYOUT.md+ top pains
- One tool trial notes or read-only darwin depth
COMPARE.mdwith keep/drop decision
- Ecosystem postcard vocabulary
- Capstone main layout untouched (or only trivial fixes)
- Three personal gotchas
Commit
git add .
git commit -m "day77: ecosystem trial and keep/drop decision"Write three personal gotchas before continuing.
Tomorrow
Day 78 — Comparative landscape: Guix, image-based distros, and Ansible—honest trade-offs vs NixOS for you.
Day 78 — Comparative landscape
Stage VII · ~4h
Goal: Write an honest, concrete trade-off note comparing NixOS to Guix System, image-based/immutable distros, and Ansible-style config management—grounded in your workloads, not tribal slogans.
Why this day exists
If you cannot say when NixOS is the wrong tool, you do not understand it. Capstone commitment is real effort; Day 78 is the “steelman the alternatives” day so your retrospective (Day 90) is adult, not fanfic.
Theory 1 — Comparison dimensions
Score systems on dimensions that matter to operators:
| Dimension | Question |
|---|---|
| Reproducibility | Can two machines converge from the same artifact/definition? |
| Atomic rollback | Bad update → previous generation quickly? |
| Abstraction | Packages + services + users in one language? |
| Learning curve | Time to first safe production change |
| Ecosystem breadth | Packaging coverage, cloud images, vendor support |
| Secrecy & compliance | Secrets, audit, SBOM stories |
| Day-2 ops | Observability, patches, multi-node |
| Escape hatches | Can you drop to imperative when burning? |
| Team skill fit | What does your org already know? |
| Mac/desktop story | Not everything is a Linux server |
| Migration cost | From your current estate |
No system wins all columns. Weighted scores beat vibes.
Theory 2 — NixOS (your baseline)
Strengths you have felt (cite your volume days):
- Declarative modules + generations (II, 13, 89)
- Flake pins for inputs (III, 24)
- Same language from package to OS (V–VII)
- Binary caches + remote builders (53–54)
- Strong “rebuild from repo” story (Day 88 will prove it)
- sops-nix / deploy-rs / Colmena / nixosTest ecosystem (IV, VI)
Weaknesses you should admit:
- Steep language + module learning curve
- Eval/build complexity; footguns (IFD, priorities, infinite recursion)
- Docs fragmented across eras (channels vs flakes)
- Some proprietary/vendor agents painful to package
- GUI desktop polish varies by investment
- Experimental features (CA derivations, etc.) not free wins
- Team hiring: fewer “NixOS natives” than Ansible generalists
Write two scars from your lab, not from memes.
Theory 3 — Guix System
What it is: GNU Guix—functional package manager + Guix System; Scheme (Guile) instead of Nix language; strong free-software ethos; similar store/generation ideas.
| Topic | Guix-ish shape | NixOS-ish shape |
|---|---|---|
| Language | Scheme | Nix |
| Init | shepherd | systemd |
| Channels / pins | channels | flakes/channels |
| FHS / binary blob culture | Often stricter free software | Pragmatic unfree options common |
| Community size | Smaller | Larger |
| Commercial surface | Smaller | Larger blog/job footprint |
When Guix may win: Scheme fluency, GNU purity goals, shepherd preference, research alignment.
When NixOS may win for you: systemd ecosystem, larger package set, more operational write-ups, your already-built Capstone investment.
You do not need to install Guix today—read primary docs enough to avoid strawmen. If you already know Scheme, say so in the essay.
Theory 4 — Image-based / immutable distros
Examples of the family (not a full product review): Fedora CoreOS / Silverblue-style, Bottlerocket, Talos, some appliance OS designs—image A/B, ostree or similar, workloads often in containers.
| Topic | Image-based | NixOS |
|---|---|---|
| Unit of deploy | OS image | Config evaluation → system closure |
| App delivery | Containers/k8s common | Packages + modules + optional containers |
| Rollback | Previous image slot | Generations |
| Host customization | Minimal by design | Extreme (sometimes too much) |
| “Pets” | Discouraged | Possible (homelab pets common) |
| Provisioning | Ignition/cloud-init class tools | Disko, nixos-anywhere, images (Day 62) |
When images win: Large fleets of identical minimal nodes, Kubernetes workers, locked-down appliances, org already standardized on container platforms.
When NixOS wins: Highly specialized hosts, cross-cutting host services, desire for one repo to express weird topology without baking images every change (though NixOS can also build images).
Hybrid reality: NixOS build of images + k8s workers is a valid architecture—not purity cosplay.
Theory 5 — Ansible (and friends)
What it is: Imperative/convergent config management over SSH; YAML playbooks; enormous module library; agentless common. Cousins: Salt, Puppet, Chef—today’s comparison centers Ansible as the industry default many teams already run.
| Topic | Ansible | NixOS |
|---|---|---|
| Model | Converge state with tasks | Specify desired system; build closure |
| Partial apply | Easy, sometimes too easy | Whole-system generations |
| Drift | Easy without discipline | Harder if you only use nix |
| Secrets | Vault, etc. | sops/agenix + store discipline |
| Non-NixOS estate | Excellent | Weak (HM/darwin partial) |
| Rollback | You build it | Built-in generations |
| Network gear / mixed OS | Strong ecosystem | Not the point |
Hybrid reality: Many orgs use Terraform/cloud for create, Ansible for classic distros, NixOS for greenfield highly automated slices.
When Ansible wins: Heterogeneous fleets (Ubuntu + network gear + weird appliances), existing playbook investment, teams without Nix skills, “change this one file on 50 boxes” urgency.
When NixOS wins: Greenfield Linux hosts you fully own, strong need for rebuildable closures, rollback as a product feature, package+service unity.
Theory 6 — Honesty checklist (avoid tribal takes)
| Bad argument | Better |
|---|---|
| “Ansible is YAML spaghetti, Nix is pure.” | “Our fleet is 200 Ubuntu boxes with existing roles; migration cost exceeds benefits this year.” |
| “Guix is just Nix with Scheme.” | “Shared functional ideas; different language, init, and community trade-offs.” |
| “Immutable OS means no SSH ever.” | “SSH policy is orthogonal; image distros still need day-2 access models.” |
| “NixOS can’t do containers.” | “NixOS can; sometimes image-first platforms optimize for that path harder.” |
| “Nix solves secrets.” | “sops-nix helps; key custody and policy still human.” |
Theory 7 — Decision time-bounds
Prefer time-bounded decisions:
Through 26.11: NixOS Capstone lab remains source of truth for my learning hosts.
Ubuntu estate at $work: Ansible remains unless a greenfield slice appears.
Re-evaluate Guix: only if Scheme project requires it.
Eternal loyalty is not an engineering output.
Worked example — weighted scorecard fragment
Context: solo homelab + one day-job Ubuntu fleet I don’t control.
| Dimension | Weight | NixOS | Guix | Image-family | Ansible |
|-----------|--------|-------|------|--------------|---------|
| Reproducibility | 5 | 5 | 5 | 4 | 2 |
| Rollback | 5 | 5 | 5 | 4 | 2 |
| Learning curve | 3 | 2 | 2 | 3 | 4 |
| Team skill (job) | 4 | 1 | 1 | 3 | 5 |
| Specialized hosts | 5 | 5 | 4 | 2 | 3 |Compute crude weighted sums; note uncertainty bands (“±1 on Guix—I read, didn’t run”).
Lab 0 — Workspace
mkdir -p ~/lab/90daysofx/02-nixos/day78
cd ~/lab/90daysofx/02-nixos/day78Lab 1 — Primary sources (skimming with intent)
Collect links in SOURCES.md (official or project docs, not only Reddit):
- Guix System manual intro / features
- One image-based OS docs page (e.g. CoreOS update model or Talos architecture)
- Ansible “how it works” / idempotence intro
Spend ~45–60 min reading, not installing everything. Optional: one throwaway VM if time remains—not required.
Lab 2 — Scorecard for your context
SCORECARD.md — rate NixOS / Guix / Image-family / Ansible 1–5 on Theory 1 dimensions. Add “Weight for me.” Compute weighted sum. Note uncertainty.
Define your context in 5 bullets (homelab, job, course-only, multi-OS, etc.).
Lab 3 — Essay (required)
ESSAY.md — 400–800 words:
- Where NixOS clearly wins for you (with concrete examples from this volume).
- Where it loses or is overkill.
- One scenario you would pick Ansible or image-based instead.
- Whether Guix is interesting to you personally (even if “not now”).
- What you will not rewrite after this essay (protect Capstone).
Tone: steelman alternatives; no meme warfare.
Lab 4 — Elevator version
Compress the essay into a 90-second talk outline (bullet list) for a skeptical SRE. Save as ELEVATOR.md.
Include one sentence: “Here’s when I would not use NixOS…”
Lab 5 — Capstone implication
One short paragraph in IMPLICATIONS.md: does this change Capstone A scope? (Default: no—essay is clarity, not a rewrite trigger.)
Lab 6 — Scenario cards (forced choice)
Write three short cards (5–8 sentences each) in SCENARIOS.md. For each, pick a primary tool and justify with dimensions from Theory 1:
Card A — Homelab “weird topology”
One edge proxy, one data host, sops secrets, occasional GPU box. You already have Capstone muscle.
Card B — Employer fleet
80 Ubuntu VMs, existing Ansible roles, two network engineers who refuse new DSLs, compliance questionnaire next quarter.
Card C — Kubernetes worker pool
Stateless nodes, image A/B updates preferred by platform team, apps only in containers, host should be minimal.
Do not use these cards to reopen Capstone A scope. They exist to prove you can switch criteria with context.
Lab 7 — Quote hygiene
From your primary sources, copy one short quotation or paraphrased claim (with URL) that surprised you for each of: Guix, image-based, Ansible. In SOURCES.md, note whether you agree after Capstone experience.
This blocks empty essays that restate Twitter without contact with docs.
Worked example — elevator closer
NixOS wins for me when I fully own the Linux host and need rebuildable
closures plus rollback. I would not force it onto our Ubuntu estate this
year—the migration cost dwarfs the benefit. Image-based OS wins for
identical k8s workers. Guix is intellectually close but not my ops path.
Questions?
Adapt to your scorecard; keep under 90 seconds spoken.
Common gotchas
| Mistake | Fix |
|---|---|
| Comparing marketing to your scars | Use Capstone experience as data |
| Installing four OSes today | Read + score; optional one VM if time |
| Declaring eternal loyalty | Prefer time-bounded decisions |
| Ignoring team skills | Skills are a real dimension |
| Strawmanning Guix without reading | Primary sources in SOURCES.md |
| Using Day 78 to restart layout thrash | Protect Capstone; Day 77 already covered tools |
Checkpoint
SOURCES.mdwith primary links
- Weighted
SCORECARD.md
ESSAY.md400–800 words, honest trade-offs
ELEVATOR.mdincluding when not NixOS
- Capstone protection note
- Three scenario cards
- One primary-source surprise note per alternative
- Three personal gotchas
Commit
git add .
git commit -m "day78: comparative landscape essay and scorecard"Write three personal gotchas before continuing.
Tomorrow
Day 79 — 26.05 release notes deep: primary-source read; annotate impact on your flake.
Day 79 — 26.05 release notes
Stage VII · ~4h
Goal: Read the official NixOS 26.05 “Yarara” release notes (and related nixpkgs/Nix notes as needed), then produce an annotated impact checklist for your flake—not a blog summary.
Baseline for this volume: NixOS / nixpkgs 26.05. Primary source beats social media threads. If a note conflicts with a blog, trust the release notes + your eval.
Why this day exists
Upgrades fail when people skip release notes and only chase build errors one by one. Day 80 plans 26.05→26.11; today you learn the method on the release you should already be targeting, and capture breakages that still lurk in your modules.
Reading notes is an operator skill equal to writing modules.
Theory 1 — What “a NixOS release” actually is
| Artifact | Role |
|---|---|
nixpkgs branch nixos-26.05 |
Package + module tree for the release |
| Release notes | Human-facing breaking changes, renames, deprecations |
State version system.stateVersion |
Migration behavior for stateful options |
| ISO / channel artifacts | Install media (flakes pin git/github instead) |
| Nix version shipped/tested | Sometimes separate from nixpkgs cadence |
| flake.lock | Your actual pinned revisions—not the marketing branch name alone |
Flakes pin revisions. Your flake.lock may already be on 26.05 while your brain still follows 24.11/25.05 blog habits.
nix flake metadata
# note nixpkgs locked rev / ref / lastModifiednix eval --raw .#nixosConfigurations.lab.pkgs.lib.version 2>/dev/null || true
# or inspect lock:
# jq '.nodes.nixpkgs' flake.lockTheory 2 — How to read release notes like an operator
Do not read linearly once and forget. Use passes:
Pass A — Inventory (30–45 min)
Skim sections:
- Breaking changes
- Notable renames / option moves
- Service-specific notes for software you run
- Deprecated options
- Kernel / boot / systemd highlights
- Security-relevant defaults
- NixOS module system / installer notes
Pass B — Personal filter (45–60 min)
For each note, tag:
| Tag | Meaning |
|---|---|
N/A |
You don’t use it |
WATCH |
Indirect risk |
ACTION |
Must change config |
DONE |
Already handled |
VERIFY |
Needs rebuild/test to confirm |
Pass C — Prove with eval/build
nixos-rebuild build --flake .#<host>
# or
nix build .#nixosConfigurations.<host>.config.system.build.toplevel -LSearch your repo for renamed options:
rg -n "oldOption|services\.foo" -g '*.nix'Pass D — Warnings are future breakages
evaluation warning: …
trace: obsolete option …
Paste warnings into IMPACT.md as VERIFY/ACTION. Next release often hard-errors them.
Theory 3 — stateVersion is not a fashion number
system.stateVersion = "26.05"; # example — only change with understandingstateVersion tells modules how to migrate on-disk state defaults. Blindly bumping it to “match release” can break stateful services (Postgres data dirs, user settings, etc.). Release notes and module comments matter more than aesthetics.
Rule for this volume: record your current stateVersion; do not bump it casually on Day 79. If notes require a migration path, put it on the Day 80 runbook.
| Action | When |
|---|---|
| Leave unchanged | Default |
| Bump with migration plan | Notes + module docs say so; backups ready |
| Mismatch with nixpkgs branch | Allowed and common; understand why |
Theory 4 — Cross-check Nix itself
Your syllabus pins modern Nix (book baseline mentions 2.34.x class—verify yours). Release notes for Nix (the evaluator/daemon) are separate from NixOS notes.
nix --version
nix config show | rg -i 'experimental|system|substituter' || trueIf CI uses a different Nix than your laptop, note the skew in the checklist—flake eval can differ on experimental defaults and sandbox settings.
| Surface | Record |
|---|---|
| Laptop Nix | version |
| CI Nix | installer action / version |
| Lab host Nix | from nix --version on host |
Theory 5 — Module renames and aliases (kinds of change)
Common historical patterns (examples of kinds, not a complete 26.05 dump—you must read the real notes):
| Kind | Example pattern | Response |
|---|---|---|
| Option rename | services.foo → services.foo-ng |
Update; don’t live on aliases forever |
Nested settings |
freeform attr moved under settings |
Follow module type |
| Package rename | overlay breaks | Fix overlay attr |
| Default flip | enable default changed |
Explicit mkForce/set |
| Kernel package set | boot fails on old nvidia/out-of-tree | Pin or update module |
| Removed service | module gone | Replace or package yourself |
When notes say an option is renamed, prefer the new path. Aliases expire.
Theory 6 — Inputs beyond nixpkgs
Your flake may pin:
| Input | Why check notes/changelogs |
|---|---|
| home-manager | release-26.05 pairing |
| sops-nix | module option moves |
| deploy-rs / colmena | activation quirks |
| disko | device option changes |
| lanzaboote | SB flow changes |
Release notes for NixOS are necessary but not sufficient. Skim changelogs for your inputs when IMPACT is large.
nix flake metadata
# for each input: note ref + whether it follows nixpkgsWorked example — IMPACT.md row
| Note / change | Tag | Repo paths | Action | Owner/date |
|---------------|-----|------------|--------|------------|
| services.foo renamed to services.foo-ng | ACTION | modules/foo.nix:12 | rename option; rebuild | you / today |
| postgres default major | WATCH | hosts/lab/db.nix | verify stateVersion; backup | Day 80 |
| plasma something | N/A | — | no desktop | — |
| evaluation warning: obsolete bar | VERIFY | modules/common.nix | rebuild log line 480 | today |Aim for every ACTION and VERIFY item to reference a file path or explicit N/A reason.
Lab 0 — Workspace
mkdir -p ~/lab/90daysofx/02-nixos/day79/notes-raw
cd ~/lab/90daysofx/02-nixos/day79Lab 1 — Capture current pin
cd /path/to/your/flake
nix flake metadata | tee ~/lab/90daysofx/02-nixos/day79/metadata.txt
nix --version | tee ~/lab/90daysofx/02-nixos/day79/nix-version.txt
rg -n "stateVersion" -g '*.nix' | tee ~/lab/90daysofx/02-nixos/day79/stateVersion-hits.txtOptional lock peek:
jq -r '.nodes.nixpkgs.original,.nodes.nixpkgs.locked' flake.lock \
| tee ~/lab/90daysofx/02-nixos/day79/nixpkgs-lock.txtLab 2 — Official notes intake
- Open the official NixOS 26.05 release notes (nixos.org manual / nixpkgs
nixos/doc/manualrelease notes for 26.05).
- Save the URL in
SOURCES.md.
- Export or copy key sections into
notes-raw/if you want offline annotation (optional).
- Create
IMPACT.mdwith the table from the worked example.
Also add:
- URL for any Nix release notes you consulted
- HM release notes if you use HM
Lab 3 — Repo audit scripts
cd /path/to/your/flake
rg -n "mkEnableOption|services\.|programs\." -g '*.nix' | head -n 100 \
| tee ~/lab/90daysofx/02-nixos/day79/options-sample.txt
# search for deprecated names you found in notes:
# rg -n "deprecatedOptionName" -g '*.nix'For each ACTION, either:
- fix now on a branch, or
- file as Day 80/83 backlog with severity (blocker / nice)
Avoid giant drive-by refactors while annotating—annotate first; fix blockers only.
Lab 4 — Build verification
nix build .#nixosConfigurations.<host>.config.system.build.toplevel -L \
2>&1 | tee ~/lab/90daysofx/02-nixos/day79/build.logIf warnings appear, paste them into IMPACT.md as VERIFY/ACTION.
Run relevant checks:
nix build .#checks.<system>.<name> -L \
2>&1 | tee ~/lab/90daysofx/02-nixos/day79/check.logLab 5 — Personal “gotchas from 26.05”
Write 5 bullets in IMPACT.md under “What I almost missed.” These feed Day 81 troubleshooting and Day 90 teach-back.
Also write stateVersion policy in one sentence:
## stateVersion policy
We run stateVersion = "…" on host lab. We will not bump it without: backup, notes citation, and Day 80 runbook step.Lab 6 — Feed Day 80
Create UPGRADE-SEEDS.md with:
| Item | Why it matters for 26.05→26.11 |
|---|---|
| ACTION leftovers | Must clear or plan |
| WATCH services | Higher test priority |
| CI Nix skew | Align before upgrade PR |
Do not perform the upgrade today unless you deliberately expand scope.
Common gotchas
| Mistake | Fix |
|---|---|
| Only reading unofficial summaries | Start from official notes |
Bumping stateVersion for fun |
Read migration semantics first |
| Assuming lock is 26.05 without checking | nix flake metadata |
| Ignoring warnings that aren’t errors | Warnings become breakages next release |
| Giant drive-by refactors while annotating | Annotate first; fix blockers only |
| Forgetting HM/sops inputs | Check paired release branches |
| Treating blog “Yarara tips” as canonical | Primary notes + your build log |
Checkpoint
metadata.txt+nix-version.txt
- Official 26.05 URL in
SOURCES.md
IMPACT.mdtagged N/A/WATCH/ACTION/DONE/VERIFY
- Build (and checks) log captured
stateVersionpolicy recorded
- Five “almost missed” bullets
- Day 80 seeds noted
- Three personal gotchas
Commit
git add .
git commit -m "day79: annotate 26.05 release notes against flake"Write three personal gotchas before continuing.
Tomorrow
Day 80 — Upgrade planning 26.05→26.11: written dry-run runbook, risks, tests-first, rollback—no requirement to upgrade prod yet.
Day 80 — Upgrade planning 26.05→26.11
Stage VII · ~4h
Goal: Produce a complete dry-run upgrade runbook from 26.05 → 26.11 (next cadence release): input bumps, test gates, staged rollout, rollback, and risk register—without requiring a production upgrade today.
NixOS releases target roughly May/November cadence (.05 / .11). 26.11 may be upcoming, in beta, or stable depending on when you read this. The deliverable is the plan; execute only on lab when ready.
Why this day exists
Day 79 taught release-note literacy on the current baseline. Day 80 turns that into operational muscle: upgrades are projects, not nix flake update on Friday night.
Theory 1 — What actually changes in a flake upgrade
flake.nix inputs (nixpkgs url/ref)
↓
flake.lock (revs, narHashes)
↓
eval modules against new nixpkgs
↓
realize new toplevel
↓
switch / reboot if kernel
↓
stateful service migrations (stateVersion-aware)
| Step | Failure mode |
|---|---|
| Lock bump | Unrelated input moves (HM, sops, deploy-rs) break together |
| Eval | Option removed/renamed; type errors |
| Build | Package broken; FODs; your overlay |
| Switch | Service fails activation |
| Runtime | Data migration, socket changes, API versions |
| Reboot | Bootloader/kernel/initrd issues |
Theory 2 — Staging strategy
Environments
| Stage | Purpose |
|---|---|
| Dev laptop / lab VM | First eval/build/test |
| CI on branch | Checks + cache populate |
| Non-prod server | Real networking/secrets |
| Prod | Last; maybe canary host |
Git strategy
main (26.05 pin)
└─ branch upgrade/26.11
├─ bump nixpkgs only
├─ fix eval
├─ bump companion inputs (HM, etc.)
└─ PR with checklist
Avoid one commit that bumps six inputs and refactors modules.
Companion inputs
Home Manager, sops-nix, deploy-rs, disko, agenix—each has branches/tags tracking nixpkgs. Read their upgrade notes. Mismatched HM vs nixpkgs is a classic footgun.
Theory 3 — Tests-first gate
Minimum gates before switch on anything important:
nix flake check
nix build .#nixosConfigurations.<host>.config.system.build.toplevel
nix build .#checks.<system>.<integration>
# deploy to lab only:
# deploy .#lab -- or colmena apply --on labOptional: nixos-rebuild build-vm / smoke VM.
Policy: red checks ⇒ no promote.
Theory 4 — Rollback is part of the plan
| Layer | Mechanism |
|---|---|
| Boot | Previous generation in bootloader |
| Runtime | nixos-rebuild switch --rollback / rollback generation |
| Git | Revert lock on branch |
| Secrets | Keys unchanged; ciphertext may need re-encrypt only if tooling changes |
| Data | DB dumps / zfs snapshots before upgrade (Day 69) |
Document RTO goals for lab vs prod. Day 88–89 practice wipe/rebuild and rollback drills.
Theory 5 — stateVersion and stateful services
Upgrading nixpkgs ≠ automatically bumping stateVersion.
Plan section required:
- Current
stateVersion
- Whether 26.11 notes require migration
- Snapshot/backup steps for PostgreSQL, Vaultwarden, etc. you actually run
- Who verifies app-level health post-switch
Theory 6 — Communication & windows
Even for homelab:
- Maintenance window
- What breaks for family/users
- Where logs live
- Abort criteria (e.g. “if proxy down >15m, rollback”)
Runbook template (you will fill this)
Create UPGRADE-26.11.md with these sections:
0. Metadata
- Author, date, target hosts
- Current pins (
flake.locknixpkgs rev)
- Target ref (
github:NixOS/nixpkgs/nixos-26.11or unstable interim)
1. Preconditions
- Backups verified restorable
- CI green on main
- Disk space for two closures
- Access to console/IPMI if remote
- sops/age keys available offline
2. Release notes
- Links to 26.11 notes (when published) + diff against your 26.05
IMPACT.mdmethod
- ACTION items list
3. Technical steps (lab first)
git checkout -b upgrade/26.11
# edit flake.nix input for nixpkgs
nix flake lock --update-input nixpkgs
# then carefully update HM / sops / etc.
nix flake check
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L
# deploy lab
# health checks4. Health verification
- HTTP endpoints
systemctl --failed
- Specific app probes
- Logs
5. Rollback
Exact commands for your deploy path (deploy-rs/colmena/rebuild).
6. Risks
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| … |
7. Decision
Go / no-go criteria for prod.
Lab 0 — Workspace
mkdir -p ~/lab/90daysofx/02-nixos/day80
cd ~/lab/90daysofx/02-nixos/day80Lab 1 — Inventory pins
cd /path/to/flake
nix flake metadata --json > ~/lab/90daysofx/02-nixos/day80/metadata.json
cp flake.lock ~/lab/90daysofx/02-nixos/day80/flake.lock.baselineList every input and its tracking branch/tag in INPUTS.md.
Lab 2 — Write the full runbook
Complete UPGRADE-26.11.md from the template. Mark unknown 26.11 specifics as TBD when notes ship but keep the structure solid. Reuse Day 79 method for notes intake.
Lab 3 — Dry-run mechanics on lab (optional execution)
If 26.11 exists and you have time:
- Branch
- Update nixpkgs only
- Eval and fix until
toplevelbuilds
- Do not require prod switch
If 26.11 is not out: dry-run against nixos-unstable on a disposable branch/VM only, or stop at lock-file rehearsal:
# rehearsal of commands without pushing
nix flake lock --update-input nixpkgs --output-lock-file /tmp/test.lockDocument what you did / skipped.
Lab 4 — Test plan matrix
TEST-PLAN.md:
| Check | Command | Blocks promote? |
|---|---|---|
| flake check | nix flake check |
Yes |
| host toplevel | nix build … |
Yes |
| nixosTest X | … | Yes |
| manual proxy curl | … | Yes for edge host |
| backup restore drill | … | Prod only |
Lab 5 — Abort & rollback tabletop
Without breaking prod, write the exact rollback commands for:
- Failed switch, system still booting old generation
- Booted new generation, services bad
- Git revert of lock after partial deploy
Common gotchas
| Mistake | Fix |
|---|---|
| Big-bang all inputs | Stage nixpkgs first |
| No backup verification | Test restore once |
| Forgetting companion inputs | HM/sops/disko matrix |
| Upgrading prod first | Lab → CI → prod |
| No abort criteria | Write them cold |
| Bumping stateVersion with release | Separate decision |
Checkpoint
INPUTS.md+ lock baseline
- Complete
UPGRADE-26.11.mdrunbook
TEST-PLAN.mdwith blockers
- Rollback tabletop written
- Explicit statement: prod upgraded or not (expected: not required)
- Three personal gotchas
Commit
git add .
git commit -m "day80: 26.05 to 26.11 upgrade runbook dry-run"Write three personal gotchas before continuing.
Tomorrow
Day 81 — Troubleshooting drill: inject failures across boot/build/service/network; fix; write playbook entries.