Day 19 — Remote Builders, Reproducibility & Gate IV

Updated

July 30, 2025

Day 19 — Remote Builders, Reproducibility & Gate IV

Day 54 — Remote builders

Stage V · ~4h
Goal: Offload realization to a remote Nix machine (Linux builder VM is ideal), understand builders / remote-systems.conf style config, and know when remote build beats “just use a cache.” Align lab with nixos-26.05 clients and builders when possible.

Note

Remote builders move work; binary caches move results. Confusing them wastes weeks. You usually want both: build once on a beefy box, push to Cachix/Attic, everyone substitutes.

Why this day exists

Laptops should not compile Chromium. CI runners should not all rebuild the same uncached leaf. Remote builders are how Darwin talks to Linux, how small VMs ship large closures, and how fleets keep eval local while realization lives elsewhere. Stage VI deploys assume this vocabulary.

Theory 1 — Cache vs remote builder

	Binary cache	Remote builder
Needs existing path?	Yes	No
Uses your flake inputs	Downloads NARs	Evaluates plan, builds drv remotely
Good for	Repeat installs	First build, exotic systems
Trust	Signing keys	SSH + Nix auth + who runs builder
Network pattern	Pull results	Upload inputs + download outputs

Often: build once on a beefy builder → push to Cachix/Attic → everyone’s substituters hit.

[dev] --build--> [builder] --push--> [cache] --substitute--> [fleet]

Theory 2 — Configuration surfaces (2026)

NixOS (`nix.buildMachines`)

{
  nix.distributedBuilds = true;
  nix.buildMachines = [
    {
      hostName = "builder.lab.internal";
      system = "x86_64-linux";
      protocol = "ssh-ng"; # preferred modern protocol when available
      sshUser = "nixbuilder";
      sshKey = "/root/.ssh/id_build"; # or agent patterns
      maxJobs = 8;
      speedFactor = 4;
      supportedFeatures = [ "nixos-test" "big-parallel" "kvm" ];
      mandatoryFeatures = [ ];
    }
  ];
  nix.settings = {
    builders-use-substitutes = true; # remote may fetch deps itself
  };
}

Ad-hoc one-shot

nix build .#pkg --builders 'ssh://nixbuilder@builder.lab.internal x86_64-linux'
# classic:
# --option builders 'ssh://…'

Features

Feature	Typical meaning
`big-parallel`	Allow heavy parallel builds
`kvm`	Nested VM tests (`nixosTest`)
`nixos-test`	OS tests scheduling
`benchmark`	Specialization (rare)

If a drv requires kvm and the builder lacks it, scheduling skips that machine—often looking like “mysterious local build” or hard failure depending on flags.

Theory 3 — SSH and store talk

Remote build roughly:

Local eval (usually) produces derivations
Upload input paths the remote lacks
Remote builds
Download result paths back (unless you only needed remote side effects)

# Test SSH as the Nix daemon user (often root on NixOS)
sudo ssh -i /root/.ssh/id_build nixbuilder@builder.lab.internal nix-store --version

Daemon context: On multi-user Nix, root (nix-daemon) must reach the builder—not only your user ssh config—unless using remote build modes that run as you (know your setup).

Check	Why
Non-interactive SSH	No password prompts in daemon
`known_hosts` stable	Automation vs MITM prompts
Dedicated build key	Least privilege vs your personal key
Builder disk free	Full disk mid-build is common

Theory 4 — Cross and multi-system

Goal	Approach
Build `aarch64-linux` from `x86_64` laptop	aarch64 builder or qemu-binfmt (slower)
Build Linux from Darwin	Linux remote builder almost always
Same-arch offload only	Simplest lab
`nixosTest` needing kvm	Builder with kvm feature + hardware

# Prefer real hardware/VM matching system
system = "aarch64-linux";

Honest lab goal: one successful remote same-arch build. Cross is stretch, not gate.

Theory 5 — Security notes

Risk	Mitigation
Builder can see sources	Treat builder as trusted environment
SSH key sprawl	Dedicated build key; restrict `authorized_keys` if hardened
Untrusted builder returns malicious paths	Only use builders you control; CI policies
World-open builder	VPN/Tailscale; no public SSH
Builder as root playground	Separate user; trusted-users carefully

Remote builders are not a sandbox that makes untrusted code safe. A compromised builder is a supply-chain incident.

Theory 6 — Scheduling knobs that matter

Knob	Effect
`max-jobs 0` (client)	Force offload when supported
`speedFactor`	Prefer faster builders
`builders-use-substitutes`	Remote fetches deps itself
`supportedFeatures`	Eligibility filter
Local `max-jobs` > 0	Local may steal work

# force remote in many setups
nix build .#cooltool -L \
  --builders "ssh-ng://nixbuilder@builder x86_64-linux - 8 1 big-parallel" \
  --max-jobs 0

Worked example — two-VM lab topology

[dev laptop / host A] --ssh-ng--> [builder VM B: NixOS 26.05, multi-user Nix]
        |                                |
        +--------> Cachix/Attic <--------+  (optional push from B or A)

Builder VM minimal extras:

# on builder
users.users.nixbuilder = {
  isNormalUser = true;
  # group membership per current wiki; keep tight
};
nix.settings.trusted-users = [ "root" "nixbuilder" ];
# SSH key auth only; open firewall only to admin net

Exact nix.sshServe / store write permissions vary by version—verify against 26.05 docs when wiring production.

Lab — multi-step

Suggested workspace: ~/lab/90daysofx/02-nixos/day54

Step 1 — Inventory local capacity

nix show-config 2>/dev/null | rg 'max-jobs|cores|builders' || nix config show | rg 'max-jobs|cores|builders'
nproc
df -h /nix

Step 2 — Provision builder

Options:

Second NixOS VM on LAN/VPN
Same machine via ssh://localhost (limited learning value but tests plumbing)
Cloud NixOS box

Install Nix multi-user; enable flakes; ensure free disk (tens of GB).

Step 3 — SSH from daemon-relevant user

ssh -i ~/.ssh/id_build nixbuilder@builder 'nix-store --version'
# and as root if daemon is root:
sudo ssh -i /root/.ssh/id_build nixbuilder@builder 'nix-store --version'

Document which user the daemon uses on your OS.

Step 4 — One-shot remote build

nix build .#cooltool -L \
  --builders "ssh-ng://nixbuilder@builder x86_64-linux - 8 1 big-parallel" \
  --max-jobs 0

Capture log proving remote participation (builder hostname in log lines).

Step 5 — Persist config

Add nix.buildMachines on the client host flake or user remote builders file. Rebuild; run without CLI flags.

Step 6 — Feature flag experiment

Request a bogus mandatory feature; observe scheduling failure; remove it. Write one sentence on what you saw.

Step 7 — Combine with cache (optional)

On success, push the result (Day 53). Then disable builder and rebuild from substitute-only.

Step 8 — Write topology diagram

ASCII in NOTES.md: who evals, who builds, who signs, who deploys, trust boundaries.

Step 9 — Prove which machine built

# after remote build, inspect path info / logs for builder host
nix path-info -Sh .#cooltool
# keep a snippet of the build log that mentions the remote host

If you cannot show evidence of remote work, you may have silently built local—re-check --max-jobs 0 and builder SSH as daemon user.

Step 10 — Failure injection

Stop sshd on builder (or block with firewall).
Retry build; capture the error class (SSH vs scheduling).
Restore builder; confirm recovery.

Document: “client symptoms when builder is down.”

Decision card — do I need a builder today?

Situation	Prefer
Package already on cache	Substituter only
First build of large leaf on weak laptop	Remote builder
Darwin needing Linux closure	Linux builder
CI rebuilds identical drv every job	Push cache after one builder run
Untrusted contributor PR	Do not give them your builder

Common gotchas

Symptom / mistake	What to do
`cannot build on …`	Features/system mismatch; check `supportedFeatures`
SSH works as user, not as root	Configure root’s key or daemon SSH
Huge upload times	Enable `builders-use-substitutes`; share cache with builder
Darwin → Linux confusion	Need Linux builder; not just `--system`
Stale known_hosts / MITM prompts	Non-interactive SSH hardening for automation
Builder disk full	GC on builder; monitor
Nested flake eval differences	Same Nix version band; same inputs
Assuming cache = builder	Re-read Theory 1

Checkpoint

Distinguished cache vs remote builder
Completed at least one remote (or localhost SSH) build
Knew which SSH identity the builder used
Documented persistent config approach
Topology notes include trust

Commit

git add .
git commit -m "day54: remote builders lab and topology"

Write three personal gotchas before continuing.

Tomorrow

Day 55 — Reproducibility honesty: what is and is not bit-for-bit.

Day 55 — Reproducibility honesty

Stage V · ~4h
Goal: Measure what “reproducible” means for your packages—bit-for-bit where possible, rebuild-equivalent where not—and document limits without marketing language.

Note

Baseline: flakes + nixpkgs 26.05-class pins. Today is measurement and claims hygiene—not a promise that every package is bitwise identical forever.

Why this day exists

Nix marketing slides say “reproducible builds.” Production engineers say “same flake lock gave the same store paths on two Linux boxes last Tuesday.” Both can be true at different strengths. Stage V ends with honesty, or Stage VI/VIII false confidence will hurt—especially Capstone rebuild (Day 88) and cache trust.

Theory 1 — Strength ladder (reprise from Day 1, sharper)

Level	Claim	How to test
L0	Builds on my laptop once	`nix build`
L1	Rebuilds same path on same machine	`nix build --rebuild` → identical out path
L2	Same lock → same paths on two similar Linux hosts	Compare `nix build --print-out-paths`
L3	Bit-for-bit identical NAR contents	`diffoscope` / hash NARs / `nix-diff`
L4	Cross-OS / cross-vendor identical	Often false for complex pkgs
L5	Reproducible data and uptime state	Not Nix’s job alone

Most flake apps aim for L1–L2. Many packages are not fully L3 yet (timestamps, capture order, non-deterministic linking). L5 is backups, migrations, and ops—not stdenv.

How to talk about this in a runbook

Weak claim	Stronger claim
“Fully reproducible”	“With lock X on x86_64-linux, attr Y yields path P”
“Same everywhere”	“Same path on two Linux builders; Darwin not claimed”
“Immutable forever”	“Unchanged until inputs change; updates are deliberate”

Theory 2 — Input-addressed sameness

In the default model, the output path hash is a function of the derivation inputs (names, other paths, builder, system, env). If two machines produce the same out path, Nix agrees the build plan matched—not always that every byte was independently audited by you.

Content-addressed derivations (Stage VII, Day 75) change the story: output hash can track content. Know they exist; do not depend on CA for today’s gate unless you already enabled it deliberately.

input-addressed:  out path = f(derivation inputs…)
content-addressed: out path can reflect output hash (advanced)

Theory 3 — Why bit-for-bit fails

Source	Example
Embedded timestamps	gzip headers, archives, man pages
Unordered maps / readdir	Parallel compression, zip member order
CPU-specific codegen	`-march=native` (avoid in shared nixpkgs packages)
Download nondeterminism	Unpinned floating tags (Day 46 FODs)
Test data with time	Snapshot tests embedding dates
Build-id / path capture	Binaries recording build paths poorly
RNG in build	Unseeded UUIDs in artifacts

# Bad idea in shared packages:
NIX_CFLAGS_COMPILE = "-march=native";

Prefer portable flags; put optimized overlays in explicit local overlays, not silent defaults.

Fixed-output derivations (FODs)

FODs pin output hash of fetched content. They improve input stability but do not magically make compilers deterministic. A wrong hash fails the build; a floating main branch fetch is still an honesty bug at the pin layer.

Theory 4 — Tools for comparison

# Same path string?
nix build .#cooltool --print-out-paths | tee path-a.txt
# on machine B with same lock:
# nix build .#cooltool --print-out-paths | tee path-b.txt
diff path-a.txt path-b.txt

# Force rebuild and compare path (avoid silent substitute hiding local nondeterminism)
nix build .#cooltool --rebuild --print-out-paths

# Diff two store paths if they differ
nix store diff-closures /nix/store/…-a /nix/store/…-b

# Detailed binary diff
nix shell nixpkgs#diffoscope -c diffoscope ./result-a ./result-b

# Hash runtime closure (rough fingerprint)
nix path-info -rS ./result | sort | tee closure.txt
# single path hash
nix hash path ./result

# When you must ignore caches to test local build determinism
nix build .#cooltool --rebuild --option substitute false -L

Theory 5 — Operational reproducibility vs academic

Ops need	Nix contribution
Rebuild server from git	Flake lock + Disko + secrets keys
Same CI binary as prod	Shared cache + same installable attr
Legal/compliance bit-identical	Extra tooling; not free with Nix alone
Debug “works on my machine”	Locks + sandboxed builds + L1/L2 tests
Trust binary cache	Signatures + trusted keys (Day 53)

Write claims in runbooks as testable sentences, not adjectives.

Theory 6 — What Nix does not make reproducible

Domain	Why
Database contents	State; backups (Day 69)
TLS certificates from ACME	Time and CA interaction
VM disk drift	Imperative writes outside store
Hardware firmware	Out of band
“Same UI screenshot”	Fonts, GPU, timing

Capstone honesty: system closure rebuildable ≠ product data immortal.

Worked example — measurement script

#!/usr/bin/env bash
# scripts/measure-repro.sh
set -euo pipefail
ATTR="${1:-.#cooltool}"
OUT1=$(nix build "$ATTR" --print-out-paths --no-link)
OUT2=$(nix build "$ATTR" --rebuild --print-out-paths --no-link)
echo "first:   $OUT1"
echo "rebuild: $OUT2"
if [[ "$OUT1" == "$OUT2" ]]; then
  echo "L1 path match: YES"
else
  echo "L1 path match: NO"
  nix store diff-closures "$OUT1" "$OUT2" || true
fi

chmod +x scripts/measure-repro.sh
./scripts/measure-repro.sh .#cooltool | tee ~/lab/90daysofx/02-nixos/day55/l1-cooltool.txt

Worked example — deliberate nondeterminism

# pkgs/bad-stamp.nix — lab only
{ stdenv }:
stdenv.mkDerivation {
  name = "bad-stamp";
  phases = [ "buildPhase" "installPhase" ];
  buildPhase = ''
    date > stamped.txt
  '';
  installPhase = ''
    mkdir -p $out
    cp stamped.txt $out/
    echo ok > $out/ok
  '';
}

Rebuild twice; discuss whether out path changed (input-addressed plan may still match if recipe identical while contents differ—observe carefully with --rebuild and content hashes). Then remove date and remeasure.

Lab — multi-step

Suggested workspace: ~/lab/90daysofx/02-nixos/day55

Step 1 — Environment record

mkdir -p ~/lab/90daysofx/02-nixos/day55
cd /path/to/your/flake
nix flake metadata | tee ~/lab/90daysofx/02-nixos/day55/metadata.txt
nix --version | tee ~/lab/90daysofx/02-nixos/day55/nix-version.txt
uname -m | tee ~/lab/90daysofx/02-nixos/day55/uname.txt

Step 2 — Pick three artifacts

Your Stage V primary package (e.g. .#cooltool)
nixpkgs#hello (or nixpkgs/nixos-26.05#hello via registry/flake)
One “fat” but sane package you use (e.g. git—not a browser)

Step 3 — L1 measurements

For each, run rebuild comparison; record path equality in a table:

Attr	Path1	Path2	L1 match?

Step 4 — L2 if you have two environments

Same flake.lock, two machines or two VMs. Compare printed out paths. If only one machine, use a clean VM from Stage II or a second builder (Day 54).

Step 5 — Substitute false control

For your package, once with substitutes allowed and once with --option substitute false --rebuild. Note time and path. Confirm you understand cache hits vs local rebuild.

Step 6 — Deliberate nondeterminism (lab only)

Add bad-stamp package; rebuild; fix by removing date; note findings.

Step 7 — Closure drift story

Change a dependency pin (or overlay bump) while keeping app source identical. Show out path change. One paragraph on reproducible inputs.

Step 8 — Honesty statement

REPRO.md template:

## Claims we make
- With flake.lock revision … on x86_64-linux, `nix build .#cooltool` yields path P.
- We push P to cache C signed by key K (if applicable).

## Claims we do not make
- Bit-for-bit identity on Darwin vs Linux
- Reproducible database contents
- Forever stability without lock updates
- L3 for all transitive nixpkgs deps

## Measured on
- date, nix version, systems, results table

## Residual risks
- …

Step 9 — Optional diffoscope

If two outputs differ, run diffoscope and paste the top finding into notes—not a 10k-line dump.

Step 10 — Cache trust paragraph

Write five lines linking Day 53–54 to honesty:

## Cache vs reproducibility
- Same out path from cache ≠ I audited the build locally.
- Signatures answer “who published this path?” not “is the universe bit-identical?”
- Our claim: trusted substituters may supply P if signatures verify.
- Our non-claim: every path was built twice under our eyes.

Step 11 — README claim scrub

If your Stage V package README says “fully reproducible,” rewrite that sentence using L1/L2 language from REPRO.md. Paste before/after into day55 notes.

Worked example — two-machine log template

## L2 attempt
Lock: flake.lock nixpkgs rev …
Attr: .#cooltool
Machine A: hostname, nix version, out path
Machine B: hostname, nix version, out path
Match: YES/NO
If NO: first nix store diff-closures lines / system difference notes

Even a failed L2 (no second machine) earns credit if you document the blocker and a next attempt plan.

Common gotchas

Symptom / mistake	What to do
Assumed same path across Darwin/Linux	Different `system` → different paths normal
Compared without same lock	Update discipline first (Day 24)
Trusted cache hit hiding local nondeterminism	`--rebuild` / `--option substitute false`
Marketing “fully reproducible” in README	Replace with measured claims
Floating `main` input	Pin; FODs with correct hashes
Compared `result` symlinks without realizing GC	Paths may still exist; note roots
L3 required for every package	Prefer honest L1–L2 for apps

Checkpoint

Ladder L0–L5 explained in own words
L1 measured for own package
Environment metadata captured
Deliberate nondeterminism demo + fix
REPRO.md honesty statement written
Cache trust paragraph written
README claim scrubbed if needed
Know path-sameness vs byte-sameness
Three personal gotchas

Commit

git add .
git commit -m "day55: reproducibility measurements and honesty notes"

Write three personal gotchas before continuing.

Tomorrow

Day 56 — Stage V gate: package + cache story consumable by host or CI.

Day 56 — Stage V gate

Stage V · ~4h (integration)
Goal: Prove packaging craft end-to-end: a flake-exported package you use, built from lock alone, with a cache/substitute story and notes another engineer could follow—on a nixos-26.05 pin.

Important

Gates are not ceremonies. If G1–G8 fail, stay in Stage V. Tourism into Disko while your package only builds on a dirty laptop store is how fleets inherit shame.

Why this day exists

Stages without gates rot into tutorial tourism. Exit Stage V only when someone else (or future you on a clean VM) can realize your package without tribal knowledge. Stage VI multiplies packaging mistakes across hosts and images.

Gate bar (explicit)

#	Requirement	Evidence
G1	Package expression is yours (or substantially maintained by you)	Path under your flake
G2	Language helper or clean stdenv—not a mystery binary copy	Source + builder
G3	FODs pinned (src + vendor/cargo/npm as needed)	Hashes in git
G4	`nix build .#…` works from clean checkout + lock	Transcript / CI
G5	Consumed by host or CI or documented `nix run` entry	Config snippet
G6	Cache story: push, file-store, or public cache hit path	Day 53 artifact
G7	Debug notes + REPRO honesty linked	Days 51 & 55
G8	Meta: license, mainProgram, description	`nix eval …meta`

Optional stretch: overlay integration on lab NixOS; multi-output library; nix flake check includes a run test.

Theory 1 — Integration layout

flake.nix
  packages.<system>.cooltool
  checks.<system>.*
  overlays.default
  apps.<system>.default
  nixosConfigurations.lab → environment.systemPackages = [ cooltool ]
  # or just apps/run for non-NixOS

# modules/cooltool-host.nix
{ pkgs, ... }:
{
  environment.systemPackages = [ pkgs.cooltool ];
}

# in flake, when composing host pkgs:
pkgs = import nixpkgs {
  inherit system;
  overlays = [ self.overlays.default ];
};

One pkgs instance rule

Overlays that exist only on packages.* but not on nixosConfigurations produce “works in nix run / fails on host” ghosts. Wire one overlay path for all consumers.

Theory 2 — What “yours” means (G1 honesty)

Acceptable	Reject
Fork of a nixpkgs expr you maintain	Unmodified `pkgs.hello`
Thin wrapper around upstream you understand	Copy-paste you cannot explain
`buildGoModule` / `buildRustPackage` you authored	Prebuilt blob with no source story
Vendored FODs you can re-hash	Network fetch without hash

If the package is trivial, make the process excellent: meta, checks, cache, docs.

Theory 3 — Clean-checkout proof

dirty laptop store  →  false confidence
clean VM + flake.lock only  →  gate truth

Proof quality	Method
Best	Second machine / fresh VM
Good	CI from clean runner
Weak	Same machine after `git clean` only
Fail	“It builds here” with untracked files

Worked example — gate flake skeleton

{
  description = "Stage V gate — cooltool";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

  outputs = { self, nixpkgs }:
    let
      systems = [ "x86_64-linux" "aarch64-linux" ];
      forAllSystems = nixpkgs.lib.genAttrs systems;
      overlay = final: prev: {
        cooltool = final.callPackage ./pkgs/by-name/co/cooltool/package.nix { };
      };
    in {
      overlays.default = overlay;

      packages = forAllSystems (system:
        let pkgs = import nixpkgs { inherit system; overlays = [ overlay ]; };
        in {
          default = pkgs.cooltool;
          cooltool = pkgs.cooltool;
        });

      apps = forAllSystems (system: {
        default = {
          type = "app";
          program = "${self.packages.${system}.cooltool}/bin/cooltool";
        };
      });

      checks = forAllSystems (system:
        let pkgs = import nixpkgs { inherit system; overlays = [ overlay ]; };
        in {
          cooltool-runs = pkgs.runCommand "cooltool-runs" {
            nativeBuildInputs = [ pkgs.cooltool ];
          } ''
            cooltool --help >/dev/null || cooltool >/dev/null
            touch $out
          '';
        });

      # Optional host wiring if you maintain a lab configuration in-repo
      # nixosConfigurations.lab = …;
    };
}

Meta sanity

nix eval .#cooltool.meta.license.spdxId
nix eval .#cooltool.meta.mainProgram
nix eval .#cooltool.meta.description

Missing meta is a G8 fail even if the binary runs.

Lab — gate protocol (multi-step)

Suggested workspace: ~/lab/90daysofx/02-nixos/day56
(or promote your best day47–52 tree and freeze it here)

Step 1 — Select the candidate

One primary package. Delete half-finished side quests from the gate flake.

Step 2 — Clean tree check

git status
# commit or stash noise
nix flake metadata
nix flake check -L
nix build .#cooltool -L
nix run .#

Step 3 — Fresh environment proof

On a second user/VM:

git clone <your-repo> gate-verify && cd gate-verify
nix build .#cooltool -L
nix run .#

If second machine missing, use a disposable VM snapshot or CI. Prefer VM over heroic same-store tricks.

Step 4 — Cache path

Demonstrate one of:

Cachix/Attic push + pull on clean side
nix copy --to file://… + pull
Written exception if network policy blocks, with file-store proof

# example file-store
nix copy --to file:///tmp/gate-cache .#cooltool
# on clean side:
# nix copy --from file:///tmp/gate-cache .#cooltool

Step 5 — Host or CI consumption

Host: add package to environment.systemPackages via overlay; rebuild; which cooltool.
CI: minimal workflow file that builds the attr (full CI polish is Day 66—skeleton enough).

Step 6 — Gate dossier

Write GATE-V.md:

# Stage V gate dossier

## Package
- attr:
- source:
- builder:

## Build
- nix version:
- system:
- out path:
- nixpkgs pin (26.05 lock rev):

## Cache
- method:
- how to consume:

## Repro claim
- link to REPRO.md

## Debug
- link to runbook

## Consumer
- host snippet / CI snippet

## Exit criteria self-score
- G1–G8 table with pass/fail

Step 7 — Self-score ruthlessly

Any fail → fix same day or explicitly defer with reason (not silent). Deferred items become Stage VI risks—list them.

Step 8 — Tag

git tag -a stage-v-gate -m "Stage V packaging gate"
git push origin stage-v-gate  # if remote exists

Step 9 — Teach-back (optional but strong)

Explain FOD pins and overlay wiring out loud in under three minutes. If you cannot, the dossier is incomplete.

Step 10 — Consumer matrix

Consumer	Wired?	Command proof
`packages.<sys>.cooltool`		`nix build .#cooltool`
`apps` / `nix run`		`nix run .#`
Overlay → host		`which cooltool` after rebuild
CI job		URL or log path
`checks`		`nix flake check`

At least two consumer rows must pass for a strong gate; G5 requires one, excellence requires more.

Red flags that fail the gate (automatic)

Red flag	Why
Source path is `/home/you/...` without FOD	Not portable
Binary copied from internet without hash	Not a package, a ritual
Only builds with `--impure`	Lock story broken
Overlay defined but never imported	Host lies about availability
License unknown for redistributed code	Meta/compliance hole

Common gotchas

Symptom / mistake	What to do
Only works with dirty local store paths	Use FODs; no `/home/…` src without copying into store
Gate package is `pkgs.hello`	Reject—must be your maintenance surface
Cache not documented	Fails G6
Checks not green	Fix or remove flaky checks honestly
Overlay not connected to host pkgs	Same `pkgs` instance everywhere
Secrets in expression	Remove; rebuild trust
`mainProgram` wrong	`nix run` breaks; fix meta
Multi-system attr missing	Gate at least your lab system; note aarch64 stretch

Checkpoint

Commit

git add .
git commit -m "day56: stage V gate dossier and package freeze"

Write three personal gotchas before continuing.

Tomorrow

Day 57 — Disko for real: declarative disks for reinstall and cloud nodes (Stage VI begins).

Day 19 — Remote Builders, Reproducibility & Gate IV

Day 54 — Remote builders

Why this day exists

Theory 1 — Cache vs remote builder

Theory 2 — Configuration surfaces (2026)

NixOS (nix.buildMachines)

Ad-hoc one-shot

Features

Theory 3 — SSH and store talk

Theory 4 — Cross and multi-system

Theory 5 — Security notes

Theory 6 — Scheduling knobs that matter

Worked example — two-VM lab topology

Lab — multi-step

Step 1 — Inventory local capacity

Step 2 — Provision builder

Step 3 — SSH from daemon-relevant user

Step 4 — One-shot remote build

Step 5 — Persist config

Step 6 — Feature flag experiment

Step 7 — Combine with cache (optional)

Step 8 — Write topology diagram

Step 9 — Prove which machine built

Step 10 — Failure injection

Decision card — do I need a builder today?

Common gotchas

Checkpoint

Commit

Tomorrow

Day 55 — Reproducibility honesty

Why this day exists

Theory 1 — Strength ladder (reprise from Day 1, sharper)

How to talk about this in a runbook

Theory 2 — Input-addressed sameness

Theory 3 — Why bit-for-bit fails

Fixed-output derivations (FODs)

Theory 4 — Tools for comparison

Theory 5 — Operational reproducibility vs academic

Theory 6 — What Nix does not make reproducible

Worked example — measurement script

Worked example — deliberate nondeterminism

Lab — multi-step

Step 1 — Environment record

Step 2 — Pick three artifacts

Step 3 — L1 measurements

Step 4 — L2 if you have two environments

Step 5 — Substitute false control

Step 6 — Deliberate nondeterminism (lab only)

Step 7 — Closure drift story

Step 8 — Honesty statement

Step 9 — Optional diffoscope

Step 10 — Cache trust paragraph

Step 11 — README claim scrub

Worked example — two-machine log template

Common gotchas

Checkpoint

Commit

Tomorrow

Day 56 — Stage V gate

Why this day exists

Gate bar (explicit)

Theory 1 — Integration layout

One pkgs instance rule

Theory 2 — What “yours” means (G1 honesty)

Theory 3 — Clean-checkout proof

Worked example — gate flake skeleton

Meta sanity

Lab — gate protocol (multi-step)

Step 1 — Select the candidate

Step 2 — Clean tree check

Step 3 — Fresh environment proof

Step 4 — Cache path

Step 5 — Host or CI consumption

Step 6 — Gate dossier

Step 7 — Self-score ruthlessly

Step 8 — Tag

Step 9 — Teach-back (optional but strong)

Step 10 — Consumer matrix

Red flags that fail the gate (automatic)

Common gotchas

NixOS (`nix.buildMachines`)