Day 73 — Store & daemon

Updated

July 30, 2026

Day 73 — Store & daemon

Stage VII · ~4h
Goal: Explain multi-user Nix trust boundaries, GC roots, sandbox isolation on Linux, and substituter signature policy—then prove it by reading a real sandbox build log and mapping live roots on your machine.

Why this day exists

“It came from the cache” and “sandbox off fixed my build” are not strategies. Capstone CI, remote builders, and disaster recovery all assume you understand who can write the store, what a root pins, and what isolation actually guarantees.

Theory 1 — Single-user vs multi-user

Mode Builder identity Typical install Risk profile
Single-user Your UID owns /nix/store Rare on servers; some dev installs User compromise ≈ store compromise
Multi-user nix-daemon (root) builds; clients speak to daemon Default multi-user installer / NixOS Trusted users can influence builds; untrusted cannot freely poison store

On NixOS you almost always run multi-user with nix-daemon.service.

systemctl status nix-daemon
ps aux | grep '[n]ix-daemon'

Clients (nix build, nixos-rebuild) send build goals to the daemon over a Unix socket (/nix/var/nix/daemon-socket/socket).

Trusted users

nix.settings.trusted-users = [ "root" "@wheel" ];
# or classic nix.conf: trusted-users = root @wheel

Trusted clients may set extra substituters, import unsigned paths (depending on config), and change certain build parameters. Do not put random CI runners in trusted-users without understanding that.

Allowed users

allowed-users gates who may talk to the daemon at all. Default is often *.

Theory 2 — Store paths, validity, and roots

Path anatomy (recall)

/nix/store/<hash>-<name>

The hash fingerprints the derivation inputs (input-addressed world—the default you live in). Day 75 contrasts content-addressed experimental paths.

Validity

A path is valid if the daemon registered it (built or substituted). Query:

nix path-info /nix/store/…-hello-…
nix path-info -r /nix/store/…-hello-…   # closure
nix path-info -S /nix/store/…           # size

GC roots

Garbage collection deletes unreferenced valid paths. Roots include:

Root class Examples
Profiles /nix/var/nix/profiles/system, per-user profiles
Generations System generation symlinks
Result symlinks ./result, ./result-1 from nix build
Runtime roots Running processes’ mapped files (when gcroots scanning enabled)
Explicit roots nix-store --add-root, GC root dirs under /nix/var/nix/gcroots 
nix-store --gc --print-roots | head -50
# modern:
nix store gc --dry-run   # careful; prefer print-roots first
ls -la /nix/var/nix/gcroots/
ls -la /nix/var/nix/profiles/system* | head

Teaching point: deleting a git repo does not free its build products if a profile or result link still roots them.

Theory 3 — Sandbox (Linux)

Why sandbox

Builds should see:

  • declared inputs only
  • no network (except fixed-output derivations with hashed output)
  • clean env, controlled /dev, empty /build

So “works on my laptop” impurities become hard errors.

Mechanisms (Linux)

Rough stack (details evolve; concepts stable):

Mechanism Role
User namespaces / uid remap Build user isolation
Mount namespaces Private /, bind store inputs
PID / IPC / UTS namespaces Process isolation
Network namespace No ambient net
Seccomp / syscall filters Reduce kernel attack surface (where enabled)
sandbox-paths Extra allowed paths (dangerous if wide)

Config knobs

# NixOS
nix.settings.sandbox = true;          # true | false | relaxed
nix.settings.extra-sandbox-paths = [ ];
# nix.settings.sandbox-fallback = false;
Value Meaning
true Full sandbox (preferred)
relaxed Some impurities allowed (debug)
false Off — only for local debugging, never as silent prod default 
nix show-config | grep -i sandbox
# or
nix config show | grep -i sandbox

Fixed-output derivations (FOD)

Fetchers (fetchurl, etc.) may have network if the output hash is fixed. Wrong hash → fail. This is intentional purity for source fetching.

Theory 4 — Substituters and signatures

Flow

  1. Eval produces derivation goals.
  2. Daemon checks local store.
  3. Queries substituters (HTTP binary caches).
  4. Verifies signatures against trusted public keys.
  5. Registers paths if valid.

Trust policy

nix.settings = {
  substituters = [
    "https://cache.nixos.org"
    "https://myorg.cachix.org"
  ];
  trusted-public-keys = [
    "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
    "myorg.cachix.org-1:…"
  ];
};
Misconfiguration Outcome
Substituter without key Paths refused (or require trust elevation)
Key without care You accept binaries from that signer as “as good as build”
require-sigs = false Dangerous; only for isolated experiments

On NixOS, nix.settings.require-sigs should stay true in normal life.

Signing your own cache (awareness)

CI often builds → signs → pushes to Cachix/Attic/Harmonia. Day 85 will wire this into Capstone; today only understand why the public key is in your flake/host config.

Theory 5 — Reading build logs

nix build nixpkgs#hello --rebuild   # force local rebuild if substitutes exist
nix log /nix/store/…-hello-….drv
# or after failure:
nix log .#packages.x86_64-linux.default

Sandbox-related failure fingerprints:

Log signal Likely cause
Network access disabled / connection errors in non-FOD Impure net need
No such file or directory for /usr/... Hardcoded FHS path
Permission denied under /nix/store Trying to mutate inputs
Hash mismatch FOD impurity or wrong expected hash
cannot build … because … is not valid Missing input; corrupted store

strace on daemon builds is advanced and rarely needed; prefer log literacy first.

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day73
cd ~/lab/90daysofx/02-nixos/day73

Work on a machine with multi-user Nix (NixOS lab VM ideal).

Lab 1 — Map live roots

nix-store --gc --print-roots 2>/dev/null | tee roots.txt | head -80
readlink -f /run/current-system || true
ls -la /nix/var/nix/profiles/ | tee profiles.txt

In ROOTS.md, classify 10 roots into: system generation, user profile, result link, gcroots explicit, other. Explain what would free a path you care about.

Lab 2 — Daemon & trust surface

systemctl cat nix-daemon.service | tee daemon-unit.txt
nix config show | tee nix-config.txt
grep -E 'trusted|substitut|sandbox|require-sigs|allowed' nix-config.txt

Answer in TRUST.md:

  1. Who is trusted?
  2. Which substituters and keys?
  3. Is sandbox on?
  4. What would you change before exposing a shared builder?

Lab 3 — Sandbox log literacy

Force a small local build and capture the log:

nix build nixpkgs#hello --rebuild -L 2>&1 | tee hello-rebuild.log
# find drv / log:
nix log nixpkgs#hello | tee hello-nix-log.txt

If substitutes always win and --rebuild isn’t available in your version, build a tiny local derivation:

# tiny.nix
{ pkgs ? import <nixpkgs> { } }:
pkgs.runCommand "day73-tiny" { } ''
  echo hello > $out
''
nix build -f tiny.nix -L 2>&1 | tee tiny.log

Annotate 5 lines in the log (phases, sandbox mentions, store paths).

Lab 4 — Optional impurity demo (VM only)

Warning

Do this only on a disposable VM. Re-enable sandbox afterward.

  1. Note current sandbox setting.
  2. Temporarily set sandbox = false via nix.settings or a one-off documented experiment.
  3. Observe that impure references may start “working.”
  4. Restore sandbox = true and rebuild the host.
  5. Record the moral in TRUST.md.

Prefer reading docs + logs over leaving sandbox off.

Lab 5 — Signature refusal mental model

Write a short scenario (no need to actually serve malware):

Cache C serves path P signed by key K_bad. Your host only trusts K_official.

What should the daemon do? What if you are a trusted-user and force import? Document answers—tie to Capstone CI signing later.

Common gotchas

Symptom What to do
cannot connect to daemon nix-daemon down; socket perms; single-user mismatch
GC deleted something “needed” Missing root; reinstall profile / rebuild system
Build needs network mid-compile Package not pure; fix derivation, don’t disable sandbox forever
untrusted substituter Key missing or not trusted for your user
Rootful Docker “fixed” Nix Different problem; don’t conflate
Filling disk Roots + old generations; Day 67 hygiene

Checkpoint

  • Multi-user vs single-user explained
  • ROOTS.md with classified roots
  • TRUST.md with trusted-users, keys, sandbox
  • One build log annotated
  • Sandbox left enabled on lab host
  • Three personal gotchas

Commit

git add .
git commit -m "day73: store daemon sandbox and substituter trust"

Write three personal gotchas before continuing.

Tomorrow

Day 74 — Performance: --max-jobs, cores, eval cache, and timing a full rebuild before/after tuning.