Debugging builds

Updated

July 30, 2026

Debugging builds

Goal: Recover from failed package builds using nix log, -L, --keep-failed, breakpointHook, and structured isolation of phase failures—without disabling the sandbox as a first resort.

Why it matters

Packaging skill is half writing expressions and half reading sandboxes under stress. Teams lose days because someone only knows “it failed” and rebuilds with random flags. This chapter builds a deliberate debug playbook.

Theory 1 — Where information lives

Tool	Shows	When
`nix build -L` / `--print-build-logs`	Live phase output	First failure
`nix log <drv\\|path>`	Stored build log	After failure/success
`nix derivation show`	JSON-ish drv env	Input/env inspection
`--keep-failed`	Leaves `tmp` build dir	Interactive inspect
`breakpointHook`	Drops into shell on failure	Deep phase debug
`nix why-depends`	Why a dep is in a closure	Bloat / unexpected refs
`--rebuild`	Force re-realization	Cache hiding issues

nix build .#broken -L
nix log .#broken
nix derivation show .#broken | less

Modern CLI accepts installables (.#pkg); classic nix-store -l /nix/store/...drv still works.

Theory 2 — Eval failure vs build failure

Separate these immediately:

nix eval .#packages.x86_64-linux.foo.drvPath
# works? → build issue. fails? → expression / module issue.

Class	Signals	Look at
Eval	`error:` before builder starts	Nix syntax, missing attrs, infinite recursion
Build	Phase logs, compiler errors	Sources, deps, phases
FOD	`hash mismatch`	URL/rev/hash trust
Substitute	download errors	Network, cache keys

Spending an hour in config.log for an eval error is pure waste.

Theory 3 — Keep failed trees

nix build .#broken --keep-failed -L

On failure, Nix prints a path under /tmp/nix-build-… (or configured temp). You can:

cd /tmp/nix-build-foo-*
ls -la
# inspect source tree after patchPhase, object files, config.log, etc.

Warning

The kept tree is not the sandbox re-entered with the same env. It is a corpse for forensics. Re-running make there may work or lie—prefer breakpointHook for a live env.

Never commit keep-failed trees. They are for diagnosis only.

Theory 4 — `breakpointHook`

{ stdenv, breakpointHook }:
stdenv.mkDerivation {
  pname = "explode";
  version = "0.0.1";
  dontUnpack = true;
  nativeBuildInputs = [ breakpointHook ];
  buildPhase = ''
    echo "about to fail"
    false
  '';
}

When the build fails, you get a shell inside the build environment (with caveats on platforms and interactive settings). Fix hypotheses, then exit and fix the Nix expression permanently.

Also related: temporary extra tools in nativeBuildInputs for inspection (strace, file, busybox)—remove before merge.

Theory 5 — Common failure classes

Class	Signals	First moves
Hash / FOD	`hash mismatch`	Fetchers chapter workflow
Configure	`config.log`, missing headers	`nativeBuildInputs`, pkg-config
Compile	`error:`, missing symbols	deps, flags, patches
Link	`undefined reference`	`buildInputs`, order, static/shared
Install	empty `$out`, wrong paths	`installPhase`, multi-out
Check	test suite	`doCheck`, sandbox network, golden files
Patch	rejects	refresh patch
Purity	banned network/path	FOD or remove impurity
Eval	before build	fix Nix syntax/attrs

Reading a log quickly

Jump to the last phase banner before the error
Read the first error: line, not the last warning
Check whether the failing command is yours or a setup hook
Re-run with -L if the stored log is truncated in your UI

Theory 6 — Sandbox: don’t disable casually

Flag / knob	Effect	Use?
`--option sandbox false`	Allows host impurities	Last resort local diagnosis only
`--no-sandbox`	Same family	Never as CI policy
`require-sigs` / substituters	Cache trust	Unrelated to compile fails
`__noChroot` (legacy patterns)	Avoid	Prefer proper FODs

If it only builds with sandbox off, your package is not done. Undeclared /usr deps will break on the next machine.

nix show-config | rg -i sandbox
# or
nix config show | rg -i sandbox

Theory 7 — Reproducing CI failures locally

# Clean-ish rebuild of one attr
nix build .#foo --rebuild -L

# Ignore binary cache for this attr (force local compile)
nix build .#foo --option substitute false -L

# Show why a dep is present
nix why-depends .#foo nixpkgs#openssl

Match CI: same flake lock, same system, same Nix version band (book baseline: 2.34.x, nixpkgs 26.05).

Bisect mental model

old lock (green) ── update input ──> new lock (red)
                 narrow: which input?
                 then: which package rebuild?
                 then: which phase?

Lock discipline + phase logs = faster bisect.

Theory 8 — Save the drv path

Always save the drv path from a failure before GC:

nix eval .#packages.x86_64-linux.foo.drvPath
nix log /nix/store/….drv

Without it, debugging after garbage collection becomes archaeology.

Worked example — scripted break/fix series

# pkgs/debug-lab.nix
{ stdenv, lib }:
stdenv.mkDerivation {
  pname = "debug-lab";
  version = "0.1.0";
  src = ../src/debug-lab;

  buildPhase = ''
    runHook preBuild
    $CC broken.c -o debug-lab
    runHook postBuild
  '';

  installPhase = ''
    runHook preInstall
    install -Dm755 debug-lab $out/bin/debug-lab
    runHook postInstall
  '';

  meta = {
    mainProgram = "debug-lab";
    license = lib.licenses.mit;
  };
}

/* broken.c — missing semicolon */
#include <stdio.h>
int main(void) {
  puts("debug-lab")
  return 0;
}

Fix: add the semicolon. Re-run with -L until green.

Failure dossier template

## Failure dossier
- Command:
- Eval or build?
- Phase (if build):
- First error line:
- Fix:
- Verification command:

Worked example — personal runbook skeleton

# DEBUG-RUNBOOK

1. Eval or build? (`nix eval …drvPath`)
2. `-L` first failure phase
3. `nix log`
4. Classify: hash vs compile vs link vs install vs check vs purity
5. `--keep-failed` if need tree
6. `breakpointHook` / extra tools if need live env
7. Only then consider sandbox experiments
8. Port fix into Nix sources; never only /tmp
9. Add a check so it cannot regress silently

Lab — multi-step

Suggested workspace: ~/lab/nixos/packaging/debug

Step 1 — Baseline success package

Start from stdenv addone or ecosystem A tool. Confirm green build + log:

nix build .#addone -L
nix log .#addone | tail -n 50

Step 2 — Inject a compile failure

Break a source file. Capture:

nix build .#addone -L 2>&1 | tee fail-compile.txt
nix log .#addone | tee fail-compile-log.txt

Step 3 — `--keep-failed`

nix build .#addone --keep-failed -L || true
# locate printed /tmp/nix-build-* path; list files; find .o / sources

Document what you could see vs not see.

Step 4 — Inject an install failure

Fix compile; break installPhase (mkdir missing, wrong path). Repeat log + keep-failed.

Step 5 — FOD failure recall

Temporarily wrong src.hash; confirm you recognize the error class in under 10 seconds.

Step 6 — `breakpointHook` (Linux recommended)

Add hook; force false in buildPhase; enter the breakpoint if your environment supports it. Type env | sort | head, inspect $CC, exit.

Step 7 — Write a personal runbook

DEBUG-RUNBOOK.md with your ordered steps (max one page).

Step 8 — Force local rebuild

nix build .#addone --rebuild --option substitute false -L

Note time vs cached build.

Step 9 — Complete failure dossier

For one intentional break, fill the dossier template under notes/debug-dossier.md.

Step 10 — Sandbox proof note

nix show-config 2>/dev/null | rg -i sandbox || nix config show | rg -i sandbox

State whether you ever disabled sandbox and why that is dangerous (undeclared deps “work on my machine”).

Exercises

Given only a CI log snippet ending mid-compile, list three next commands you would run locally.
Break doCheck with a failing test; identify checkPhase in logs.
Use nix derivation show to find an env var you did not know was set.
Compare --keep-failed corpse vs breakpointHook live shell for the same failure—write pros/cons.
Time how long it takes you to classify five pasted error messages (FOD/eval/compile/link/install).

Common gotchas

Symptom / mistake	What to do
`nix log` says no build log	Build never started (eval) or GC’d; rebuild with `-L`
Kept temp already deleted	Re-run with `--keep-failed` immediately
breakpointHook “does nothing”	Platform/interactive limits; use keep-failed + better logs
Fixed tree by hand in `/tmp` only	Always port fix into Nix sources
Disabled sandbox “to move on”	Track impurity; fix properly
CI-only failure	Align system, lock, substituters, case-sensitive paths
Silent `doCheck` fails	Watch for checkPhase in log; set `doCheck` explicitly
Scrolled past first error	Search for `error:` from top of failure region

Checkpoint

Used -L and nix log on a real failure
Used --keep-failed and inspected the tree
Classified at least three failure kinds
Tried breakpointHook or documented why not
Personal debug runbook committed
Eval vs build separation practiced
Sandbox policy note written

Commit

git add .
git commit -m "packaging: debug builds runbook and failure drills"

Write three personal gotchas before navigating nixpkgs.

Debugging builds

Why it matters

Theory 1 — Where information lives

Theory 2 — Eval failure vs build failure

Theory 3 — Keep failed trees

Theory 4 — breakpointHook

Theory 5 — Common failure classes

Reading a log quickly

Theory 6 — Sandbox: don’t disable casually

Theory 7 — Reproducing CI failures locally

Bisect mental model

Theory 8 — Save the drv path

Worked example — scripted break/fix series

Failure dossier template

Worked example — personal runbook skeleton

Lab — multi-step

Step 1 — Baseline success package

Step 2 — Inject a compile failure

Step 3 — --keep-failed

Step 4 — Inject an install failure

Step 5 — FOD failure recall

Step 6 — breakpointHook (Linux recommended)

Step 7 — Write a personal runbook

Step 8 — Force local rebuild

Step 9 — Complete failure dossier

Step 10 — Sandbox proof note

Exercises

Common gotchas

Checkpoint

Commit

Theory 4 — `breakpointHook`

Step 3 — `--keep-failed`

Step 6 — `breakpointHook` (Linux recommended)