Day 55 — Reproducibility honesty

Updated

July 30, 2026

Day 55 — Reproducibility honesty

Stage V · ~4h
Goal: Measure what “reproducible” means for your packages—bit-for-bit where possible, rebuild-equivalent where not—and document limits without marketing language.

Note

Baseline: flakes + nixpkgs 26.05-class pins. Today is measurement and claims hygiene—not a promise that every package is bitwise identical forever.

Why this day exists

Nix marketing slides say “reproducible builds.” Production engineers say “same flake lock gave the same store paths on two Linux boxes last Tuesday.” Both can be true at different strengths. Stage V ends with honesty, or Stage VI/VIII false confidence will hurt—especially Capstone rebuild (Day 88) and cache trust.

Theory 1 — Strength ladder (reprise from Day 1, sharper)

Level Claim How to test
L0 Builds on my laptop once nix build
L1 Rebuilds same path on same machine nix build --rebuild → identical out path
L2 Same lock → same paths on two similar Linux hosts Compare nix build --print-out-paths
L3 Bit-for-bit identical NAR contents diffoscope / hash NARs / nix-diff
L4 Cross-OS / cross-vendor identical Often false for complex pkgs
L5 Reproducible data and uptime state Not Nix’s job alone

Most flake apps aim for L1–L2. Many packages are not fully L3 yet (timestamps, capture order, non-deterministic linking). L5 is backups, migrations, and ops—not stdenv.

How to talk about this in a runbook

Weak claim Stronger claim
“Fully reproducible” “With lock X on x86_64-linux, attr Y yields path P”
“Same everywhere” “Same path on two Linux builders; Darwin not claimed”
“Immutable forever” “Unchanged until inputs change; updates are deliberate”

Theory 2 — Input-addressed sameness

In the default model, the output path hash is a function of the derivation inputs (names, other paths, builder, system, env). If two machines produce the same out path, Nix agrees the build plan matched—not always that every byte was independently audited by you.

Content-addressed derivations (Stage VII, Day 75) change the story: output hash can track content. Know they exist; do not depend on CA for today’s gate unless you already enabled it deliberately.

input-addressed:  out path = f(derivation inputs…)
content-addressed: out path can reflect output hash (advanced)

Theory 3 — Why bit-for-bit fails

Source Example
Embedded timestamps gzip headers, archives, man pages
Unordered maps / readdir Parallel compression, zip member order
CPU-specific codegen -march=native (avoid in shared nixpkgs packages)
Download nondeterminism Unpinned floating tags (Day 46 FODs)
Test data with time Snapshot tests embedding dates
Build-id / path capture Binaries recording build paths poorly
RNG in build Unseeded UUIDs in artifacts 
# Bad idea in shared packages:
NIX_CFLAGS_COMPILE = "-march=native";

Prefer portable flags; put optimized overlays in explicit local overlays, not silent defaults.

Fixed-output derivations (FODs)

FODs pin output hash of fetched content. They improve input stability but do not magically make compilers deterministic. A wrong hash fails the build; a floating main branch fetch is still an honesty bug at the pin layer.

Theory 4 — Tools for comparison

# Same path string?
nix build .#cooltool --print-out-paths | tee path-a.txt
# on machine B with same lock:
# nix build .#cooltool --print-out-paths | tee path-b.txt
diff path-a.txt path-b.txt

# Force rebuild and compare path (avoid silent substitute hiding local nondeterminism)
nix build .#cooltool --rebuild --print-out-paths

# Diff two store paths if they differ
nix store diff-closures /nix/store/…-a /nix/store/…-b

# Detailed binary diff
nix shell nixpkgs#diffoscope -c diffoscope ./result-a ./result-b
# Hash runtime closure (rough fingerprint)
nix path-info -rS ./result | sort | tee closure.txt
# single path hash
nix hash path ./result
# When you must ignore caches to test local build determinism
nix build .#cooltool --rebuild --option substitute false -L

Theory 5 — Operational reproducibility vs academic

Ops need Nix contribution
Rebuild server from git Flake lock + Disko + secrets keys
Same CI binary as prod Shared cache + same installable attr
Legal/compliance bit-identical Extra tooling; not free with Nix alone
Debug “works on my machine” Locks + sandboxed builds + L1/L2 tests
Trust binary cache Signatures + trusted keys (Day 53)

Write claims in runbooks as testable sentences, not adjectives.

Theory 6 — What Nix does not make reproducible

Domain Why
Database contents State; backups (Day 69)
TLS certificates from ACME Time and CA interaction
VM disk drift Imperative writes outside store
Hardware firmware Out of band
“Same UI screenshot” Fonts, GPU, timing

Capstone honesty: system closure rebuildable ≠ product data immortal.

Worked example — measurement script

#!/usr/bin/env bash
# scripts/measure-repro.sh
set -euo pipefail
ATTR="${1:-.#cooltool}"
OUT1=$(nix build "$ATTR" --print-out-paths --no-link)
OUT2=$(nix build "$ATTR" --rebuild --print-out-paths --no-link)
echo "first:   $OUT1"
echo "rebuild: $OUT2"
if [[ "$OUT1" == "$OUT2" ]]; then
  echo "L1 path match: YES"
else
  echo "L1 path match: NO"
  nix store diff-closures "$OUT1" "$OUT2" || true
fi
chmod +x scripts/measure-repro.sh
./scripts/measure-repro.sh .#cooltool | tee ~/lab/90daysofx/02-nixos/day55/l1-cooltool.txt

Worked example — deliberate nondeterminism

# pkgs/bad-stamp.nix — lab only
{ stdenv }:
stdenv.mkDerivation {
  name = "bad-stamp";
  phases = [ "buildPhase" "installPhase" ];
  buildPhase = ''
    date > stamped.txt
  '';
  installPhase = ''
    mkdir -p $out
    cp stamped.txt $out/
    echo ok > $out/ok
  '';
}

Rebuild twice; discuss whether out path changed (input-addressed plan may still match if recipe identical while contents differ—observe carefully with --rebuild and content hashes). Then remove date and remeasure.

Lab — multi-step

Suggested workspace: ~/lab/90daysofx/02-nixos/day55

Step 1 — Environment record

mkdir -p ~/lab/90daysofx/02-nixos/day55
cd /path/to/your/flake
nix flake metadata | tee ~/lab/90daysofx/02-nixos/day55/metadata.txt
nix --version | tee ~/lab/90daysofx/02-nixos/day55/nix-version.txt
uname -m | tee ~/lab/90daysofx/02-nixos/day55/uname.txt

Step 2 — Pick three artifacts

  1. Your Stage V primary package (e.g. .#cooltool)
  2. nixpkgs#hello (or nixpkgs/nixos-26.05#hello via registry/flake)
  3. One “fat” but sane package you use (e.g. git—not a browser)

Step 3 — L1 measurements

For each, run rebuild comparison; record path equality in a table:

Attr Path1 Path2 L1 match?

Step 4 — L2 if you have two environments

Same flake.lock, two machines or two VMs. Compare printed out paths. If only one machine, use a clean VM from Stage II or a second builder (Day 54).

Step 5 — Substitute false control

For your package, once with substitutes allowed and once with --option substitute false --rebuild. Note time and path. Confirm you understand cache hits vs local rebuild.

Step 6 — Deliberate nondeterminism (lab only)

Add bad-stamp package; rebuild; fix by removing date; note findings.

Step 7 — Closure drift story

Change a dependency pin (or overlay bump) while keeping app source identical. Show out path change. One paragraph on reproducible inputs.

Step 8 — Honesty statement

REPRO.md template:

## Claims we make
- With flake.lock revision … on x86_64-linux, `nix build .#cooltool` yields path P.
- We push P to cache C signed by key K (if applicable).

## Claims we do not make
- Bit-for-bit identity on Darwin vs Linux
- Reproducible database contents
- Forever stability without lock updates
- L3 for all transitive nixpkgs deps

## Measured on
- date, nix version, systems, results table

## Residual risks
- …

Step 9 — Optional diffoscope

If two outputs differ, run diffoscope and paste the top finding into notes—not a 10k-line dump.

Step 10 — Cache trust paragraph

Write five lines linking Day 53–54 to honesty:

## Cache vs reproducibility
- Same out path from cache ≠ I audited the build locally.
- Signatures answer “who published this path?” not “is the universe bit-identical?”
- Our claim: trusted substituters may supply P if signatures verify.
- Our non-claim: every path was built twice under our eyes.

Step 11 — README claim scrub

If your Stage V package README says “fully reproducible,” rewrite that sentence using L1/L2 language from REPRO.md. Paste before/after into day55 notes.

Worked example — two-machine log template

## L2 attempt
Lock: flake.lock nixpkgs rev …
Attr: .#cooltool
Machine A: hostname, nix version, out path
Machine B: hostname, nix version, out path
Match: YES/NO
If NO: first nix store diff-closures lines / system difference notes

Even a failed L2 (no second machine) earns credit if you document the blocker and a next attempt plan.

Common gotchas

Symptom / mistake What to do
Assumed same path across Darwin/Linux Different system → different paths normal
Compared without same lock Update discipline first (Day 24)
Trusted cache hit hiding local nondeterminism --rebuild / --option substitute false
Marketing “fully reproducible” in README Replace with measured claims
Floating main input Pin; FODs with correct hashes
Compared result symlinks without realizing GC Paths may still exist; note roots
L3 required for every package Prefer honest L1–L2 for apps

Checkpoint

  • Ladder L0–L5 explained in own words
  • L1 measured for own package
  • Environment metadata captured
  • Deliberate nondeterminism demo + fix
  • REPRO.md honesty statement written
  • Cache trust paragraph written
  • README claim scrubbed if needed
  • Know path-sameness vs byte-sameness
  • Three personal gotchas

Commit

git add .
git commit -m "day55: reproducibility measurements and honesty notes"

Write three personal gotchas before continuing.

Tomorrow

Day 56 — Stage V gate: package + cache story consumable by host or CI.