The Reproducibility Crisis

Updated

September 12, 2026

The Reproducibility Crisis

Software engineering has spent decades tolerating ambient state: hidden dependencies, global machine mutations, and “works on my machine” excuses. The boring default of this book is: every development environment, dependency, and production system must be a pure, declarative, reproducible expression from zero.

Mental model

Traditional operations rely on imperative mutation. You provision a virtual machine or container, run a sequence of apt-get, curl | bash, and pip install commands, and hope that: 1. Upstream package repositories haven’t updated or removed files. 2. Network connections remain uninterrupted mid-install. 3. System state matches the exact sequence of commands previously tested.

When any of these fail, you get Heisenbugs: intermittent build failures, subtle ABI mismatches, and production outages that disappear when you attempt to isolate them.

Traditional Imperative Model:
  Base OS -> [Step 1: apt-get] -> [Step 2: pip install] -> [Step 3: git clone] -> ??? (Unknown state)

Nix Declarative Model:
  Inputs (Source + Hashes) -> Pure Derivation -> Immutable /nix/store Path (Deterministic)

Nix replaces stateful mutation with pure mathematical evaluation: given the exact same source code and inputs, it always yields the exact same byte-for-byte system closure.

Worked examples

Case 1: Detecting Ambient Host Contamination

Save as test_environment.sh. Inspect how standard shell scripts unknowingly depend on host binaries and paths.

# test_environment.sh
#!/usr/bin/env bash
set -euo pipefail

echo "Inspecting system path dependencies:"
which python3 || echo "python3 missing"
which node || echo "node missing"
which gcc || echo "gcc missing"

Run:

bash test_environment.sh

Output (on a typical developer workstation):

Inspecting system path dependencies:
/usr/bin/python3
/home/user/.nvm/versions/node/v20.0.0/bin/node
/usr/bin/gcc

Notice how three completely different tools live in three different unpinned locations (/usr/bin, user home directories, global OS paths). If a colleague runs the exact same script on a fresh laptop, it fails immediately.

Case 2: Purity and Isolated Evaluation

Save as pure_check.nix. A minimal Nix expression evaluated in isolation.

# pure_check.nix
let
  system = builtins.currentSystem;
  message = "No ambient host contamination allowed";
in
{
  inherit system message;
}

Run:

nix-instantiate --eval pure_check.nix

Output:

{ message = "No ambient host contamination allowed"; system = "x86_64-linux"; }

Evaluating this expression does not query the internet, look at /usr/local, or depend on environment variables.

Case 3: Hermetic Sandboxing in Action

Nix builds run in sandboxed namespaces where network access and non-declared filesystem paths are inaccessible.

Save as sandbox_check.nix:

# sandbox_check.nix
derivation {
  name = "sandbox-proof";
  system = builtins.currentSystem;
  builder = "/bin/sh";
  args = [
    "-c"
    "echo 'Built inside isolated namespace' > $out"
  ];
}

Run:

nix-build sandbox_check.nix

Output:

these 1 derivations will be built:
  /nix/store/10h9v...-sandbox-proof.drv
building '/nix/store/10h9v...-sandbox-proof.drv'...
/nix/store/v73ka...-sandbox-proof

Inspect output:

cat result

Output:

Built inside isolated namespace

The output was generated purely from the declared inputs.

The trap

Relying on “Golden Images” (pre-baked AMIs, Docker images without lockfiles, or manual server setup steps documented in internal wikis).

Wiki page: "Run these 14 commands to set up the desk service..."

By week three, one PPA repository goes offline, a patch release bumps a minor dependency, and onboarding is broken for every new hire.

The fix: Check in code that defines the environment completely down to cryptographic hashes. If it builds once, it builds forever.

The boring rule

Never mutate system state in-place with ad-hoc shell commands.
Pin all toolchain and library dependencies via cryptographic hashes.
Eliminate dependency on the host’s ambient /usr/bin and $PATH.
Every environment must be reproducible from a fresh Git checkout.

Try this

Run env in your current shell and note how many variables point to user home directories or mutable local paths.
In pure_check.nix, add a field timestamp = builtins.currentTime; and observe how pure evaluation restricts non-deterministic built-ins.