Day 24 — Evaluator Internals, Module System & Store Daemon

Updated

July 30, 2025

Day 24 — Evaluator Internals, Module System & Store Daemon

Day 71 — Evaluator internals

Stage VII · ~4h (theory-heavy)
Goal: Explain thunks, force evaluation intentionally, reproduce and fix infinite recursion, and use trace / tryEval / seq / deepSeq without cargo-culting them.

Note

This day is language/evaluator depth. Prefer a disposable nix repl and tiny pure expressions before touching your host flake. A bad recursive module option can hang eval of the whole system.

Why this day exists

By Stage VI you already use laziness every rebuild: modules only force what activation needs. When eval explodes—infinite recursion encountered, mysterious null, or a multi-minute wait before any build starts—the bug is often a thunk cycle or accidental full-tree force, not a broken package.

Internals last (syllabus rule) means you can now map symptoms → mechanisms.

Theory 1 — Evaluation is a graph of thunks

Values vs thunks

Nix does not compute every subexpression when you write it. An unevaluated expression is a thunk: a promise to compute later, holding the expression body plus an environment of free variables.

Concept	Meaning
Thunk	Deferred computation; may be forced 0 or 1 times (then memoized)
WHNF	Weak head normal form: outer constructor known (`{…}`, `[…]`, lambda, int, …) without forcing nested fields
NF	Full normal form: nested values forced recursively
Memoization	After force, the thunk is replaced by the result (shared)

let
  x = builtins.trace "forcing x" (1 + 1);
  y = { a = x; b = x; };
in
  y.a

You should see "forcing x" once. Both a and b share the same thunk identity until forced.

Why modules love laziness

# Sketch of the idea — not the real module system
config = {
  services.foo.enable = true;
  # huge attrset of options you never touch...
};
# Only options that are *demanded* by evaluation of the final system get forced.

If every option were strict, evaluating a NixOS config would always pay the full cost of every default expression, including unused services.

Eager edges you already know

Construct	Eager?	Notes
`import ./file.nix`	Path resolve + parse/eval of the file	File body still may return lazy values
Function application `f x`	Forces `f` to a function; forces args per function body	Body of `f` decides strictness
`&&` / `\\|\\|`	Short-circuit	Second arm may never force
`if c then a else b`	Forces `c`	Only one branch forces
`builtins.seq a b`	Forces `a` to WHNF, returns `b`	Classic “strictness injection”
`builtins.deepSeq a b`	Forces `a` fully, returns `b`	Expensive; tests and asserts
`builtins.trace msg v`	Forces nothing about `v` except returns it	Side effect for debugging

Theory 2 — Infinite recursion patterns

Pattern A — Direct self-reference

let x = x + 1; in x
# error: infinite recursion encountered

Pattern B — Mutual recursion without a base case

let
  a = b + 1;
  b = a + 1;
in a

Pattern C — Attrset self-reference (legal shape, dangerous force)

rec {
  a = 1;
  b = a + 1;      # OK
  # c = c;        # infinite if forced
}

rec is not evil; forcing a cycle is.

Pattern D — Module / option cycles (production shape)

Classic failure:

# module A
config.services.foo.settings.bar = config.services.baz.settings.qux;

# module B
config.services.baz.settings.qux = config.services.foo.settings.bar;

Or subtler:

options.my.paths.dataDir = lib.mkOption {
  type = lib.types.path;
  default = config.my.paths.root + "/data";  # forces config while defining defaults
};

Rule of thumb: defaults and apply functions should depend on options (or pure inputs), not on deep config that re-enters the same option.

Pattern E — `with` + shadowing confusion

let pkgs = { hello = "ok"; }; in
with pkgs;
let hello = hello; in  # which hello?
hello

Prefer explicit attr access in teaching examples; with is a readability tax on recursion debugging.

Theory 3 — Debugging toolkit

`builtins.trace`

builtins.trace "here" value

Prints to stderr when the trace expression itself is forced (i.e. when something needs value through the trace).

`builtins.tryEval`

builtins.tryEval (throw "boom")
# → { success = false; value = false; }  # value is dummy on failure

builtins.tryEval 42
# → { success = true; value = 42; }

Use for soft probing—not for swallowing production errors silently.

`seq` vs `deepSeq`

builtins.seq { a = throw "late"; } "outer-ok"
# → "outer-ok"   # only WHNF of the attrset

builtins.deepSeq { a = throw "late"; } "never"
# → error: late

`lib.debug.traceVal` / `traceSeq` / `traceValSeq` (nixpkgs)

When you have lib from nixpkgs:

lib.debug.traceVal someAttr
lib.debug.traceValSeqN 2 nested

Prefer library helpers over ad-hoc nested trace once you are in a flake with pkgs/lib.

`NIX_SHOW_STATS` and evaluator metrics

NIX_SHOW_STATS=1 nix eval --impure --expr '1+1'

Look for counts of thunks, allocations, and GC—useful when comparing “eval is slow” hypotheses, not as a daily dashboard.

`nix-instantiate --eval` / `nix eval`

nix eval --expr 'let x = x; in x'          # expect infinite recursion
nix eval --json .#nixosConfigurations.lab.config.networking.hostName

For module cycles, force one leaf option rather than the entire config.

Theory 4 — When to force (and when not to)

Goal	Tool	Risk
Ensure side-effect config is checked early	`assertions` / `warnings` in modules	Good default
Fail fast if a secret path missing	assert in module, or activation script	Prefer module-time when pure
Make a function strict in an arg	`seq` at the start of the body	Can defeat sharing
Serialize entire config for tests	`deepSeq` or `lib.generators.toJSON`	Huge cost if done casually
“Fix” mysterious laziness bugs with deepSeq everywhere	Don’t	Hides design issues; tanks eval

Warning

deepSeq is a diagnostic scalpel and a test hammer—not a seasoning for every module.

Worked example — Construct, diagnose, fix

Broken

# day71-broken.nix
rec {
  root = "/var/lib/app";
  data = "${data}/files";   # oops: meant root
}

nix eval -f day71-broken.nix data
# infinite recursion encountered

Fixed

rec {
  root = "/var/lib/app";
  data = "${root}/files";
}

Module-shaped fix

Prefer:

{ lib, config, ... }:
{
  options.my.app.root = lib.mkOption {
    type = lib.types.path;
    default = "/var/lib/app";
  };
  options.my.app.dataDir = lib.mkOption {
    type = lib.types.path;
    default = "${config.my.app.root}/files";  # depends on sibling option via config — OK if no cycle
  };
}

If root’s default ever depended on dataDir, you create a cycle. Draw a DAG of option dependencies when stuck.

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day71
cd ~/lab/90daysofx/02-nixos/day71

Lab 1 — Watch memoization

Create thunks.nix:

let
  expensive = builtins.trace "EXPENSIVE" (builtins.foldl' (a: b: a + b) 0 (builtins.genList (i: i) 100));
  bag = { x = expensive; y = expensive; };
in
{
  first = bag.x;
  second = bag.y;
}

nix eval -f thunks.nix first
nix eval -f thunks.nix second
nix eval -f thunks.nix

Record how many times EXPENSIVE prints in each case. Explain sharing in one paragraph in NOTES.md.

Lab 2 — Infinite recursion zoo

Create cycles.nix with three named attributes that each fail differently when forced:

Direct let cycle
Mutual a/b cycle
rec set cycle

Force each with nix eval -f cycles.nix <name> and paste the error fingerprints into NOTES.md.

Then fix all three; leave the broken versions commented for teaching.

Lab 3 — `seq` vs `deepSeq`

# force.nix
let
  t = {
    a = builtins.trace "a" 1;
    b = builtins.trace "b" (throw "b-fail");
  };
in {
  onlyWhnf = builtins.seq t "ok-whnf";
  full = builtins.deepSeq (builtins.removeAttrs t ["b"]) "ok-deep";
  # fullBoom = builtins.deepSeq t "never";
}

Evaluate onlyWhnf and full. Uncomment fullBoom once and capture the error. Write when WHNF is enough for module work.

Lab 4 — Trace a tricky real eval (your flake)

Pick one leaf from your lab host:

cd /path/to/your/flake
nix eval .#nixosConfigurations.<host>.config.networking.hostName
# or a service option you care about
time nix eval .#nixosConfigurations.<host>.config.system.build.toplevel --raw >/dev/null

If eval is slow, binary-search:

Comment out recently added modules (or use a slimmed modules = [ ... ] copy).
Force smaller slices with nix eval …config.services.<name>.enable.
Add a temporary lib.trace only around the suspect option default.

Document the path of the investigation—even if the conclusion is “eval cost is nixpkgs baseline.”

Lab 5 — Intentional soft failure with `tryEval`

# try.nix
{
  ok = builtins.tryEval (1 + 1);
  bad = builtins.tryEval (1 + "x");
}

nix eval -f try.nix --json

Note: type errors vs throw vs infinite recursion behave differently under tryEval—catalog what you observe.

Common gotchas

Symptom	Theory link
`infinite recursion encountered` with no file	Cycle in options/`rec`; force a smaller attr
Trace never prints	Expression never forced; wrap the used value
Trace prints thousands of times	You forced a list map; narrow the traced value
Eval hangs, no error	Likely cycle; interrupt and bisect modules
`deepSeq` “fixed” a bug	You hid a lazy design smell; redesign dependencies
`with pkgs;` + name collision	Shadowing; drop `with` while debugging
Module default uses `config` of same option	Define default from other options or pure constants

Checkpoint

Explain thunk / WHNF / memoization in your own words
Three infinite-recursion examples constructed and fixed
seq vs deepSeq demo committed
Trace notes on one real flake option
NOTES.md with three personal gotchas
No permanent deepSeq left in production modules

Commit

git add .
git commit -m "day71: evaluator thunks, cycles, and trace discipline"

Write three personal gotchas before continuing.

Tomorrow

Day 72 — Module system deep: full type catalog, mkMerge / priorities, and redesigning one module properly.

Day 72 — Module system deep

Stage VII · ~4h (theory-heavy)
Goal: Use a real type catalog, compose with mkIf / mkMerge / priorities correctly, add assertions, and redesign one production-shaped module so bad configs fail at eval time.

Note

You already consumed modules (Stage II) and wrote a custom one (Day 19). Today is the Zai-style types + merge pass—not another “enable nginx” tutorial.

Why this day exists

Untyped attrs options accept anything until activation or first request fails. Priority mistakes silently lose settings (mkDefault overridden by accident, or mkForce used as a hammer). Capstone hosts need modules that reject nonsense early and merge predictably across roles.

Theory 1 — Anatomy of a module (recall with precision)

A module is a function (or attrset) producing some of:

{ config, lib, pkgs, ... }:
{
  imports = [ ./peer.nix ];
  options = { /* option declarations */ };
  config = { /* option definitions */ };
}

Field	Role
`imports`	DAG of other modules
`options`	Schema: types, defaults, descriptions
`config`	Values / definitions (often under `mkIf`)
freeform / `_module`	Advanced; respect unless you know why

Evaluation merges all definitions according to type + priority, then checks types and assertions.

Theory 2 — Type catalog (practical reference)

Import lib.types as types in real modules. Below is the catalog you should be able to choose from, not memorize constructor source.

Scalars & paths

Type	Accepts	Notes
`types.bool`	`true`/`false`	Prefer over “stringly” enables
`types.int` / `types.ints.positive` / `types.ints.u16` …	Integers with bounds	Ports, counts
`types.float`	Floats	Rare in OS config
`types.str`	String	Most text
`types.nonEmptyStr`	Non-empty string	Hostnames, user names
`types.singleLineStr`	No newlines	Avoid injection in unit files
`types.lines` / `types.commas`	Multi-line / comma merge	Distinct merge semantics
`types.path`	Path-like	Coerces some strings; store paths common
`types.package`	Derivation/package	For `environment.systemPackages` style
`types.shellPackage`	Package with shell	Specialized
`types.passwdEntry types.str`	Shadow-safe strings	Passwords (still don’t put secrets in store)
`types.strMatching "regex"`	Pattern	Domains, tokens shape
`types.enum [ "a" "b" ]`	Closed set	Modes
`types.either t1 t2`	Union	e.g. `bool` or `str`
`types.oneOf [ t1 t2 … ]`	Multi union	Prefer small sets
`types.nullOr t`	`null` or `t`	Optional values
`types.uniq t`	Unique merge	Only one definition wins (strict)
`types.anything`	Escape hatch	Avoid in public modules
`types.raw`	Untyped raw	Advanced / internal

Collections

Type	Merge behavior (intuition)
`types.listOf t`	Concatenate definitions
`types.attrsOf t`	Deep-merge by attr name
`types.lazyAttrsOf t`	Lazier attrs (perf / recursion)
`types.attrs`	Freeform attrs (weak)
`types.attrsWith { … }`	Configurable attrs (modern)
`types.loaOf t`	Deprecated list-or-attrs legacy

Structure

Type	Use
`types.submodule { options = …; }`	Nested option trees
`types.submoduleWith { modules = …; }`	Modular submodules
`types.deferredModule`	Module value delayed
`types.optionType`	Type of types (rare)

Function-ish / paths to callables

Type	Use
`types.functionTo t`	Options that are functions
`types.raw` + convention	When types can’t express it

Useful helpers

types.addCheck t predicate
types.coercedTo from coerceFn to
lib.mkOption {
  type = types.submodule {
    options.enable = lib.mkEnableOption "this feature";
    options.port = lib.mkOption {
      type = types.port;  # alias around ints
      default = 8080;
    };
  };
  default = { };
}

lib.mkEnableOption "desc" → bool option, default false, standard description.

Theory 3 — Definitions: `mkIf`, `mkMerge`, priorities

`mkIf`

config = lib.mkIf config.services.myapp.enable {
  users.users.myapp = { isSystemUser = true; group = "myapp"; };
  # ...
};

False conditions drop definitions (they do not define null everywhere).

`mkMerge`

config = lib.mkMerge [
  (lib.mkIf cfg.enable { … })
  (lib.mkIf cfg.tls { … })
  { assertions = [ … ]; }
];

Prefer mkMerge over huge nested // trees when multiple independent conditionals apply.

Priority ladder (high level)

Helper	Intent
`mkOptionDefault`	Lowest-ish option default layer
`mkDefault`	Soft default; user config should win
`mkOverride n`	Numeric priority (`mkDefault` ≈ 1000, normal 100, …)
`mkForce`	High priority “I really mean this”
`mkFixStrictly` / order-related	Advanced ordering

Discipline:

Modules ship mkDefault for opinionated defaults.
Hosts use plain assignments or selective mkForce.
If you need mkForce everywhere, your module boundaries are wrong.

Example conflict

# module
services.openssh.enable = lib.mkDefault true;

# host
services.openssh.enable = false;  # wins over mkDefault

# role module
networking.firewall.enable = lib.mkForce true;

# host tries
networking.firewall.enable = false;  # loses unless higher override

Theory 4 — Assertions & warnings

config = {
  assertions = [
    {
      assertion = !(cfg.enable && cfg.package == null);
      message = "myapp: enable requires package";
    }
    {
      assertion = cfg.port != 22 || !config.services.openssh.enable;
      message = "myapp: port 22 conflicts with OpenSSH on this host pattern";
    }
  ];
  warnings = lib.optional cfg.legacyMode ''
    myapp.legacyMode is deprecated; migrate to settings.v2
  '';
};

Assertions fail evaluation—perfect for capstone CI (nix flake check).

Theory 5 — Submodules & `attrsOf` services

Pattern for multi-instance:

options.services.myapp.instances = lib.mkOption {
  type = types.attrsOf (types.submodule ({ name, ... }: {
    options = {
      enable = lib.mkEnableOption "instance ${name}";
      port = lib.mkOption { type = types.port; };
    };
  }));
  default = { };
};

Then map instances to systemd units with lib.mapAttrs'.

Worked example — Redesign sketch

Before (weak)

{ config, lib, ... }:
{
  options.my.proxy = lib.mkOption {
    type = lib.types.attrs;
    default = { };
  };
  config = {
    # hope the attrs are right...
  };
}

After (typed)

{ config, lib, pkgs, ... }:
let
  inherit (lib) types mkIf mkOption mkEnableOption;
  cfg = config.my.proxy;
in {
  options.my.proxy = {
    enable = mkEnableOption "edge reverse proxy front door";
    domain = mkOption {
      type = types.strMatching "[a-z0-9.-]+";
      example = "lab.example.com";
      description = "Primary public hostname";
    };
    upstream = mkOption {
      type = types.str;
      example = "http://127.0.0.1:8080";
    };
    listenPort = mkOption {
      type = types.port;
      default = 443;
    };
    acme = mkOption {
      type = types.bool;
      default = true;
    };
  };

  config = mkIf cfg.enable {
    assertions = [
      {
        assertion = cfg.listenPort != 0;
        message = "my.proxy.listenPort must be non-zero";
      }
    ];
    # wire services.nginx / caddy / traefik here with mkDefault
  };
}

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day72
cd ~/lab/90daysofx/02-nixos/day72

Copy or reference one real module from your lab flake (proxy, app, or secrets-adjacent).

Lab 1 — Type catalog cheat sheet (you write it)

In TYPES.md, list at least 15 types from the catalog with:

one-line purpose
one good example value
one bad value that should fail

Do not paste the table from this chapter blindly—rewrite in your words.

Lab 2 — Redesign one module

Identify untyped attrs, raw strings for ports, or missing mkEnableOption.
Rewrite options with proper types + descriptions.
Move conditionals under mkIf / mkMerge.
Add ≥2 assertions (conflict + required field).
Ensure host still evaluates:

nix eval .#nixosConfigurations.<host>.config.system.build.toplevel --raw >/dev/null
# or
nixos-rebuild build --flake .#<host>

Lab 3 — Priority drill

Create a tiny three-file demo (can be pure lib.evalModules):

# eval-demo.nix — sketch
{ lib ? (import <nixpkgs> { }).lib }:
lib.evalModules {
  modules = [
    ({ lib, ... }: {
      options.value = lib.mkOption { type = lib.types.int; };
      config.value = lib.mkDefault 1;
    })
    ({ lib, ... }: {
      config.value = 2; # plain
    })
    ({ lib, ... }: {
      # toggle these in experiments:
      # config.value = lib.mkForce 3;
    })
  ];
}

nix eval -f eval-demo.nix config.value

Record the winner for: default vs plain vs force. Paste results into PRIORITIES.md.

Lab 4 — Intentional bad config

Against your redesigned module, write bad-host.nix snippets that must fail eval (wrong type, failed assertion). Run them and save stderr excerpts—these become CI fixtures later (Day 86).

Common gotchas

Symptom	What to do
`The option … is used but not defined`	Missing option decl or wrong path
Infinite recursion in modules	Option default depends on own config cycle (Day 71)
Settings silently ignored	Lost priority war; check `mkDefault`/`mkForce`
`listOf` duplicates explode	Dedup with `lib.unique` or design set-like options
Submodule names vs `name` arg	Use `{ name, ... }:` pattern carefully
Secrets in typed options	Types don’t encrypt; still use sops/agenix
Over-using `types.attrs`	Prefer `attrsOf submodule` or explicit options

Checkpoint

TYPES.md with ≥15 types in your words
One production module redesigned with real types
≥2 assertions
Priority demo understood (PRIORITIES.md)
Bad configs fail at eval with clear messages
Host/toplevel still builds

Commit

git add .
git commit -m "day72: module types catalog and redesigned module"

Write three personal gotchas before continuing.

Tomorrow

Day 73 — Store & daemon: multi-user trust, sandbox namespaces, substituter signatures, and reading a sandbox build log.

Day 73 — Store & daemon

Stage VII · ~4h
Goal: Explain multi-user Nix trust boundaries, GC roots, sandbox isolation on Linux, and substituter signature policy—then prove it by reading a real sandbox build log and mapping live roots on your machine.

Why this day exists

“It came from the cache” and “sandbox off fixed my build” are not strategies. Capstone CI, remote builders, and disaster recovery all assume you understand who can write the store, what a root pins, and what isolation actually guarantees.

Theory 1 — Single-user vs multi-user

Mode	Builder identity	Typical install	Risk profile
Single-user	Your UID owns `/nix/store`	Rare on servers; some dev installs	User compromise ≈ store compromise
Multi-user	`nix-daemon` (root) builds; clients speak to daemon	Default multi-user installer / NixOS	Trusted users can influence builds; untrusted cannot freely poison store

On NixOS you almost always run multi-user with nix-daemon.service.

systemctl status nix-daemon
ps aux | grep '[n]ix-daemon'

Clients (nix build, nixos-rebuild) send build goals to the daemon over a Unix socket (/nix/var/nix/daemon-socket/socket).

Trusted users

nix.settings.trusted-users = [ "root" "@wheel" ];
# or classic nix.conf: trusted-users = root @wheel

Trusted clients may set extra substituters, import unsigned paths (depending on config), and change certain build parameters. Do not put random CI runners in trusted-users without understanding that.

Allowed users

allowed-users gates who may talk to the daemon at all. Default is often *.

Theory 2 — Store paths, validity, and roots

Path anatomy (recall)

/nix/store/<hash>-<name>

The hash fingerprints the derivation inputs (input-addressed world—the default you live in). Day 75 contrasts content-addressed experimental paths.

Validity

A path is valid if the daemon registered it (built or substituted). Query:

nix path-info /nix/store/…-hello-…
nix path-info -r /nix/store/…-hello-…   # closure
nix path-info -S /nix/store/…           # size

GC roots

Garbage collection deletes unreferenced valid paths. Roots include:

Root class	Examples
Profiles	`/nix/var/nix/profiles/system`, per-user profiles
Generations	System generation symlinks
Result symlinks	`./result`, `./result-1` from `nix build`
Runtime roots	Running processes’ mapped files (when gcroots scanning enabled)
Explicit roots	`nix-store --add-root`, GC root dirs under `/nix/var/nix/gcroots`

nix-store --gc --print-roots | head -50
# modern:
nix store gc --dry-run   # careful; prefer print-roots first
ls -la /nix/var/nix/gcroots/
ls -la /nix/var/nix/profiles/system* | head

Teaching point: deleting a git repo does not free its build products if a profile or result link still roots them.

Theory 3 — Sandbox (Linux)

Why sandbox

Builds should see:

declared inputs only
no network (except fixed-output derivations with hashed output)
clean env, controlled /dev, empty /build

So “works on my laptop” impurities become hard errors.

Mechanisms (Linux)

Rough stack (details evolve; concepts stable):

Mechanism	Role
User namespaces / uid remap	Build user isolation
Mount namespaces	Private `/`, bind store inputs
PID / IPC / UTS namespaces	Process isolation
Network namespace	No ambient net
Seccomp / syscall filters	Reduce kernel attack surface (where enabled)
`sandbox-paths`	Extra allowed paths (dangerous if wide)

Config knobs

# NixOS
nix.settings.sandbox = true;          # true | false | relaxed
nix.settings.extra-sandbox-paths = [ ];
# nix.settings.sandbox-fallback = false;

Value	Meaning
`true`	Full sandbox (preferred)
`relaxed`	Some impurities allowed (debug)
`false`	Off — only for local debugging, never as silent prod default

nix show-config | grep -i sandbox
# or
nix config show | grep -i sandbox

Fixed-output derivations (FOD)

Fetchers (fetchurl, etc.) may have network if the output hash is fixed. Wrong hash → fail. This is intentional purity for source fetching.

Theory 4 — Substituters and signatures

Flow

Eval produces derivation goals.
Daemon checks local store.
Queries substituters (HTTP binary caches).
Verifies signatures against trusted public keys.
Registers paths if valid.

Trust policy

nix.settings = {
  substituters = [
    "https://cache.nixos.org"
    "https://myorg.cachix.org"
  ];
  trusted-public-keys = [
    "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
    "myorg.cachix.org-1:…"
  ];
};

Misconfiguration	Outcome
Substituter without key	Paths refused (or require trust elevation)
Key without care	You accept binaries from that signer as “as good as build”
`require-sigs = false`	Dangerous; only for isolated experiments

On NixOS, nix.settings.require-sigs should stay true in normal life.

Signing your own cache (awareness)

CI often builds → signs → pushes to Cachix/Attic/Harmonia. Day 85 will wire this into Capstone; today only understand why the public key is in your flake/host config.

Theory 5 — Reading build logs

nix build nixpkgs#hello --rebuild   # force local rebuild if substitutes exist
nix log /nix/store/…-hello-….drv
# or after failure:
nix log .#packages.x86_64-linux.default

Sandbox-related failure fingerprints:

Log signal	Likely cause
`Network access disabled` / connection errors in non-FOD	Impure net need
`No such file or directory` for `/usr/...`	Hardcoded FHS path
Permission denied under `/nix/store`	Trying to mutate inputs
Hash mismatch	FOD impurity or wrong expected hash
`cannot build … because … is not valid`	Missing input; corrupted store

strace on daemon builds is advanced and rarely needed; prefer log literacy first.

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day73
cd ~/lab/90daysofx/02-nixos/day73

Work on a machine with multi-user Nix (NixOS lab VM ideal).

Lab 1 — Map live roots

nix-store --gc --print-roots 2>/dev/null | tee roots.txt | head -80
readlink -f /run/current-system || true
ls -la /nix/var/nix/profiles/ | tee profiles.txt

In ROOTS.md, classify 10 roots into: system generation, user profile, result link, gcroots explicit, other. Explain what would free a path you care about.

Lab 2 — Daemon & trust surface

systemctl cat nix-daemon.service | tee daemon-unit.txt
nix config show | tee nix-config.txt
grep -E 'trusted|substitut|sandbox|require-sigs|allowed' nix-config.txt

Answer in TRUST.md:

Who is trusted?
Which substituters and keys?
Is sandbox on?
What would you change before exposing a shared builder?

Lab 3 — Sandbox log literacy

Force a small local build and capture the log:

nix build nixpkgs#hello --rebuild -L 2>&1 | tee hello-rebuild.log
# find drv / log:
nix log nixpkgs#hello | tee hello-nix-log.txt

If substitutes always win and --rebuild isn’t available in your version, build a tiny local derivation:

# tiny.nix
{ pkgs ? import <nixpkgs> { } }:
pkgs.runCommand "day73-tiny" { } ''
  echo hello > $out
''

nix build -f tiny.nix -L 2>&1 | tee tiny.log

Annotate 5 lines in the log (phases, sandbox mentions, store paths).

Lab 4 — Optional impurity demo (VM only)

Warning

Do this only on a disposable VM. Re-enable sandbox afterward.

Note current sandbox setting.
Temporarily set sandbox = false via nix.settings or a one-off documented experiment.
Observe that impure references may start “working.”
Restore sandbox = true and rebuild the host.
Record the moral in TRUST.md.

Prefer reading docs + logs over leaving sandbox off.

Lab 5 — Signature refusal mental model

Write a short scenario (no need to actually serve malware):

Cache C serves path P signed by key K_bad. Your host only trusts K_official.

What should the daemon do? What if you are a trusted-user and force import? Document answers—tie to Capstone CI signing later.

Common gotchas

Symptom	What to do
`cannot connect to daemon`	`nix-daemon` down; socket perms; single-user mismatch
GC deleted something “needed”	Missing root; reinstall profile / rebuild system
Build needs network mid-compile	Package not pure; fix derivation, don’t disable sandbox forever
`untrusted substituter`	Key missing or not trusted for your user
Rootful Docker “fixed” Nix	Different problem; don’t conflate
Filling disk	Roots + old generations; Day 67 hygiene

Checkpoint

Multi-user vs single-user explained
ROOTS.md with classified roots
TRUST.md with trusted-users, keys, sandbox
One build log annotated
Sandbox left enabled on lab host
Three personal gotchas

Commit

git add .
git commit -m "day73: store daemon sandbox and substituter trust"

Write three personal gotchas before continuing.

Tomorrow

Day 74 — Performance: --max-jobs, cores, eval cache, and timing a full rebuild before/after tuning.

Day 24 — Evaluator Internals, Module System & Store Daemon

Day 71 — Evaluator internals

Why this day exists

Theory 1 — Evaluation is a graph of thunks

Values vs thunks

Why modules love laziness

Eager edges you already know

Theory 2 — Infinite recursion patterns

Pattern A — Direct self-reference

Pattern B — Mutual recursion without a base case

Pattern C — Attrset self-reference (legal shape, dangerous force)

Pattern D — Module / option cycles (production shape)

Pattern E — with + shadowing confusion

Theory 3 — Debugging toolkit

builtins.trace

builtins.tryEval

seq vs deepSeq

lib.debug.traceVal / traceSeq / traceValSeq (nixpkgs)

NIX_SHOW_STATS and evaluator metrics

nix-instantiate --eval / nix eval

Theory 4 — When to force (and when not to)

Worked example — Construct, diagnose, fix

Broken

Fixed

Module-shaped fix

Lab 0 — Workspace

Lab 1 — Watch memoization

Lab 2 — Infinite recursion zoo

Lab 3 — seq vs deepSeq

Lab 4 — Trace a tricky real eval (your flake)

Lab 5 — Intentional soft failure with tryEval

Common gotchas

Checkpoint

Commit

Tomorrow

Day 72 — Module system deep

Why this day exists

Theory 1 — Anatomy of a module (recall with precision)

Theory 2 — Type catalog (practical reference)

Scalars & paths

Collections

Structure

Function-ish / paths to callables

Useful helpers

Theory 3 — Definitions: mkIf, mkMerge, priorities

mkIf

mkMerge

Priority ladder (high level)

Example conflict

Theory 4 — Assertions & warnings

Theory 5 — Submodules & attrsOf services

Worked example — Redesign sketch

Before (weak)

After (typed)

Lab 0 — Workspace

Lab 1 — Type catalog cheat sheet (you write it)

Lab 2 — Redesign one module

Lab 3 — Priority drill

Lab 4 — Intentional bad config

Common gotchas

Checkpoint

Commit

Tomorrow

Day 73 — Store & daemon

Why this day exists

Theory 1 — Single-user vs multi-user

Trusted users

Allowed users

Theory 2 — Store paths, validity, and roots

Path anatomy (recall)

Validity

GC roots

Theory 3 — Sandbox (Linux)

Why sandbox

Mechanisms (Linux)

Config knobs

Fixed-output derivations (FOD)

Theory 4 — Substituters and signatures

Flow

Trust policy

Pattern E — `with` + shadowing confusion

`builtins.trace`

`builtins.tryEval`

`seq` vs `deepSeq`

`lib.debug.traceVal` / `traceSeq` / `traceValSeq` (nixpkgs)

`NIX_SHOW_STATS` and evaluator metrics

`nix-instantiate --eval` / `nix eval`

Lab 3 — `seq` vs `deepSeq`

Lab 5 — Intentional soft failure with `tryEval`

Theory 3 — Definitions: `mkIf`, `mkMerge`, priorities

`mkIf`

`mkMerge`

Theory 5 — Submodules & `attrsOf` services