Reproducibility (honest view)

Updated

July 30, 2026

Reproducibility (honest view)

Goal: Measure what “reproducible” means for your packages—bit-for-bit where possible, rebuild-equivalent where not—and document limits without marketing language.

Note

Baseline: flakes + nixpkgs 26.05-class pins. This chapter is measurement and claims hygiene—not a promise that every package is bitwise identical forever.

Why it matters

Nix marketing slides say “reproducible builds.” Production engineers say “same flake lock gave the same store paths on two Linux boxes last Tuesday.” Both can be true at different strengths. Packaging craft ends with honesty, or fleet ops and disaster recovery false confidence will hurt.

Theory 1 — Strength ladder

Level Claim How to test
L0 Builds on my laptop once nix build
L1 Rebuilds same path on same machine nix build --rebuild → identical out path
L2 Same lock → same paths on two similar Linux hosts Compare nix build --print-out-paths
L3 Bit-for-bit identical NAR contents diffoscope / hash NARs
L4 Cross-OS / cross-vendor identical Often false for complex pkgs
L5 Reproducible data and uptime state Not Nix’s job alone

Most flake apps aim for L1–L2. Many packages are not fully L3 yet (timestamps, capture order, non-deterministic linking). L5 is backups, migrations, and ops—not stdenv.

How to talk about this in a runbook

Weak claim Stronger claim
“Fully reproducible” “With lock X on x86_64-linux, attr Y yields path P”
“Same everywhere” “Same path on two Linux builders; Darwin not claimed”
“Immutable forever” “Unchanged until inputs change; updates are deliberate”

Theory 2 — Input-addressed sameness

In the default model, the output path hash is a function of the derivation inputs (names, other paths, builder, system, env). If two machines produce the same out path, Nix agrees the build plan matched—not always that every byte was independently audited by you.

Content-addressed derivations (internals part) change the story: output hash can track content. Know they exist; do not depend on CA here unless you already enabled it deliberately.

input-addressed:  out path = f(derivation inputs…)
content-addressed: out path can reflect output hash (advanced)

Path sameness vs byte sameness

Observation Means
Same out path Same build plan (input-addressed)
Same NAR hash Same bytes
Cache hit of path P Someone trusted published P; not necessarily you built it twice

Theory 3 — Why bit-for-bit fails

Source Example
Embedded timestamps gzip headers, archives, man pages
Unordered maps / readdir Parallel compression, zip member order
CPU-specific codegen -march=native (avoid in shared packages)
Download nondeterminism Unpinned floating tags (FODs)
Test data with time Snapshot tests embedding dates
Build-id / path capture Binaries recording build paths poorly
RNG in build Unseeded UUIDs in artifacts 
# Bad idea in shared packages:
NIX_CFLAGS_COMPILE = "-march=native";

Prefer portable flags; put optimized overlays in explicit local overlays, not silent defaults.

Fixed-output derivations (FODs)

FODs pin output hash of fetched content. They improve input stability but do not magically make compilers deterministic. A wrong hash fails the build; a floating main branch fetch is still an honesty bug at the pin layer.

Theory 4 — Tools for comparison

# Same path string?
nix build .#cooltool --print-out-paths | tee path-a.txt
# on machine B with same lock:
# nix build .#cooltool --print-out-paths | tee path-b.txt
diff path-a.txt path-b.txt

# Force rebuild and compare path (avoid silent substitute hiding local nondeterminism)
nix build .#cooltool --rebuild --print-out-paths

# Diff two store paths if they differ
nix store diff-closures /nix/store/…-a /nix/store/…-b

# Detailed binary diff
nix shell nixpkgs#diffoscope -c diffoscope ./result-a ./result-b
# Hash runtime closure (rough fingerprint)
nix path-info -rS ./result | sort | tee closure.txt
nix hash path ./result
# When you must ignore caches to test local build determinism
nix build .#cooltool --rebuild --option substitute false -L

Theory 5 — Operational reproducibility vs academic

Ops need Nix contribution
Rebuild server from git Flake lock + Disko + secrets keys
Same CI binary as prod Shared cache + same installable attr
Legal/compliance bit-identical Extra tooling; not free with Nix alone
Debug “works on my machine” Locks + sandboxed builds + L1/L2 tests
Trust binary cache Signatures + trusted keys

Write claims in runbooks as testable sentences, not adjectives.

Theory 6 — What Nix does not make reproducible

Domain Why
Database contents State; backups
TLS certificates from ACME Time and CA interaction
VM disk drift Imperative writes outside store
Hardware firmware Out of band
“Same UI screenshot” Fonts, GPU, timing
User home clutter Not in the flake

Honesty: system closure rebuildable ≠ product data immortal.

Theory 7 — Cache vs reproducibility

## Cache vs reproducibility
- Same out path from cache ≠ I audited the build locally.
- Signatures answer “who published this path?” not “is the universe bit-identical?”
- Our claim: trusted substituters may supply P if signatures verify.
- Our non-claim: every path was built twice under our eyes.

Substitutes accelerate L2 distribution; they do not replace measurement.

Worked example — measurement script

#!/usr/bin/env bash
# scripts/measure-repro.sh
set -euo pipefail
ATTR="${1:-.#cooltool}"
OUT1=$(nix build "$ATTR" --print-out-paths --no-link)
OUT2=$(nix build "$ATTR" --rebuild --print-out-paths --no-link)
echo "first:   $OUT1"
echo "rebuild: $OUT2"
if [[ "$OUT1" == "$OUT2" ]]; then
  echo "L1 path match: YES"
else
  echo "L1 path match: NO"
  nix store diff-closures "$OUT1" "$OUT2" || true
fi
chmod +x scripts/measure-repro.sh
./scripts/measure-repro.sh .#cooltool | tee ~/lab/nixos/packaging/repro/l1-cooltool.txt

Worked example — deliberate nondeterminism

# pkgs/bad-stamp.nix — lab only
{ stdenv }:
stdenv.mkDerivation {
  name = "bad-stamp";
  phases = [ "buildPhase" "installPhase" ];
  buildPhase = ''
    date > stamped.txt
  '';
  installPhase = ''
    mkdir -p $out
    cp stamped.txt $out/
    echo ok > $out/ok
  '';
}

Rebuild twice; discuss whether out path changed (input-addressed plan may still match if recipe identical while contents differ—observe carefully with --rebuild and content hashes). Then remove date and remeasure.

Worked example — REPRO.md template

## Claims we make
- With flake.lock revision … on x86_64-linux, `nix build .#cooltool` yields path P.
- We push P to cache C signed by key K (if applicable).

## Claims we do not make
- Bit-for-bit identity on Darwin vs Linux
- Reproducible database contents
- Forever stability without lock updates
- L3 for all transitive nixpkgs deps

## Measured on
- date, nix version, systems, results table

## Residual risks
- …

Two-machine log template

## L2 attempt
Lock: flake.lock nixpkgs rev …
Attr: .#cooltool
Machine A: hostname, nix version, out path
Machine B: hostname, nix version, out path
Match: YES/NO
If NO: first nix store diff-closures lines / system difference notes

Even a failed L2 (no second machine) earns credit if you document the blocker and a next attempt plan.

Lab — multi-step

Suggested workspace: ~/lab/nixos/packaging/repro

Step 1 — Environment record

mkdir -p ~/lab/nixos/packaging/repro
cd /path/to/your/flake
nix flake metadata | tee ~/lab/nixos/packaging/repro/metadata.txt
nix --version | tee ~/lab/nixos/packaging/repro/nix-version.txt
uname -m | tee ~/lab/nixos/packaging/repro/uname.txt

Step 2 — Pick three artifacts

  1. Your primary packaging package (e.g. .#cooltool)
  2. nixpkgs#hello
  3. One “fat” but sane package you use (e.g. git—not a browser)

Step 3 — L1 measurements

For each, run rebuild comparison; record path equality in a table:

Attr Path1 Path2 L1 match?

Step 4 — L2 if you have two environments

Same flake.lock, two machines or two VMs. Compare printed out paths. If only one machine, document the blocker.

Step 5 — Substitute false control

For your package, once with substitutes allowed and once with --option substitute false --rebuild. Note time and path.

Step 6 — Deliberate nondeterminism (lab only)

Add bad-stamp package; rebuild; fix by removing date; note findings.

Step 7 — Closure drift story

Change a dependency pin (or overlay bump) while keeping app source identical. Show out path change. One paragraph on reproducible inputs.

Step 8 — Honesty statement

Write REPRO.md from the template.

Step 9 — Optional diffoscope

If two outputs differ, run diffoscope and paste the top finding into notes—not a 10k-line dump.

Step 10 — Cache trust paragraph

Link binary cache trust to honesty using the Theory 7 bullets.

Step 11 — README claim scrub

If your package README says “fully reproducible,” rewrite that sentence using L1/L2 language from REPRO.md. Paste before/after into notes.

Exercises

  1. Classify five real packages you use into L0–L3 ambition.
  2. Argue for or against requiring L3 for an internal Go CLI (one page max).
  3. Show that floating inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable" without lock discipline breaks L2.
  4. Measure L1 for a multi-output package’s out vs dev separately.
  5. Write an executive one-liner that is true and useful about Nix reproducibility.

Common gotchas

Symptom / mistake What to do
Assumed same path across Darwin/Linux Different system → different paths normal
Compared without same lock Update discipline first
Trusted cache hit hiding local nondeterminism --rebuild / --option substitute false
Marketing “fully reproducible” in README Replace with measured claims
Floating main input Pin; FODs with correct hashes
Compared result symlinks without realizing GC Paths may still exist; note roots
L3 required for every package Prefer honest L1–L2 for apps
Confused data durability with package repro Backups chapter / ops

Checkpoint

  • Ladder L0–L5 explained in own words
  • L1 measured for own package
  • Environment metadata captured
  • Deliberate nondeterminism demo + fix
  • REPRO.md honesty statement written
  • Cache trust paragraph written
  • README claim scrubbed if needed
  • Know path-sameness vs byte-sameness
  • Three personal gotchas

Commit

git add .
git commit -m "packaging: reproducibility measurements and honesty notes"

You now have the packaging spine: phases, FODs, ecosystems, overrides, multi-out, debug, nixpkgs shape, caches, builders, honesty. Ops and fleet chapters ship the closures you learned to build.