Day 51 — Debug builds

Day 51 — Debug builds

Stage V · ~4h (lab-heavy)
Goal: Recover from failed package builds using nix log, -L, --keep-failed, breakpointHook, and structured isolation of phase failures—without disabling the sandbox as a first resort.

Why this day exists

Packaging skill is half writing expressions and half reading sandboxes under stress. Teams lose days because someone only knows “it failed” and rebuilds with random flags. Today you build a deliberate debug playbook.

---

Theory 1 — Where information lives

Tool	Shows	When
nix build -L / --print-build-logs	Live phase output	First failure
nix log <drv\|path>	Stored build log	After failure/success
nix show-derivation	JSON-ish drv env	Input/env inspection
--keep-failed	Leaves tmp build dir	Interactive inspect
breakpointHook	Drops into shell on failure	Deep phase debug
NIX_LOG_FD / verbose	Extra noise	Rare

nix build .#broken -L
nix log .#broken
nix derivation show .#broken | less

Modern CLI accepts installables (.#pkg); classic nix-store -l /nix/store/...drv still works.

---

Theory 2 — Keep failed trees

nix build .#broken --keep-failed -L

On failure, Nix prints a path under /tmp/nix-build-… (or configured temp). You can:

cd /tmp/nix-build-foo-*
# inspect source tree after patchPhase, object files, config.log, etc.

Warning

The kept tree is not the sandbox re-entered with the same env. It is a corpse for forensics. Re-running make there may work or lie—prefer breakpointHook for a live env.

---

Theory 3 — breakpointHook

{ stdenv, breakpointHook }:
stdenv.mkDerivation {
  pname = "explode";
  version = "0.0.1";
  dontUnpack = true;
  nativeBuildInputs = [ breakpointHook ];
  buildPhase = ''
    echo "about to fail"
    false
  '';
}

When the build fails, you get a shell inside the build environment (with caveats on platforms and interactive settings). Fix hypotheses, then exit and fix the Nix expression permanently.

Also related: pkgs.busybox / extra tools in nativeBuildInputs temporarily for inspection (strace, file)—remove before merge.

---

Theory 4 — Common failure classes

Class	Signals	First moves
Hash / FOD	hash mismatch	Day 46 workflow
Configure	config.log, missing headers	nativeBuildInputs, PKG_CONFIG_PATH
Compile	error:, missing symbols	deps, flags, patches
Link	undefined reference	buildInputs, order, static/shared
Install	empty $out, wrong paths	installPhase, multi-out
Check	test suite	doCheck, sandbox network, golden files
Patch	rejects	refresh patch
Purity	banned network/path	FOD or remove impurity
Eval	before build	fix Nix syntax/attrs—not nix log of build

Separate eval failures from build failures immediately:

nix eval .#packages.x86_64-linux.foo.drvPath
# works? then build issue. fails? expression issue.

---

Theory 5 — Sandbox: don’t disable casually

Flag / knob	Effect	Use?
--option sandbox false	Allows host impurities	Last resort local diagnosis only
--no-sandbox	Same family	Never in CI policy
require-sigs / substituters	Cache trust	Unrelated to compile fails
__noChroot (legacy patterns)	Avoid	Prefer proper FODs

If it only builds with sandbox off, your package is not done.

---

Theory 6 — Reproducing CI failures locally

# Clean-ish rebuild of one attr
nix build .#foo --rebuild -L

# Ignore binary cache for this attr (force local compile)
nix build .#foo --option substitute false -L

# Show why a dep is present
nix why-depends .#foo .#bar

Match CI: same flake lock, same system, same Nix version band (book baseline: 2.34.x, nixpkgs 26.05).

---

Worked example — scripted break/fix series

# pkgs/debug-lab.nix
{ stdenv, lib, breakpointHook }:
stdenv.mkDerivation {
  pname = "debug-lab";
  version = "0.1.0";
  src = ../src/debug-lab; # contains broken.c / fixed later

  # Toggle while learning:
  # nativeBuildInputs = [ breakpointHook ];

  buildPhase = ''
    runHook preBuild
    $CC broken.c -o debug-lab
    runHook postBuild
  '';

  installPhase = ''
    runHook preInstall
    install -Dm755 debug-lab $out/bin/debug-lab
    runHook postInstall
  '';

  meta = {
    mainProgram = "debug-lab";
    license = lib.licenses.mit;
  };
}

/* broken.c — missing semicolon */
#include <stdio.h>
int main(void) {
  puts("debug-lab")
  return 0;
}

Fix:

  puts("debug-lab");

---

Lab — multi-step

Suggested workspace: ~/lab/90daysofx/02-nixos/day51

Step 1 — Baseline success package

Start from Day 45 addone or Day 47 tool. Confirm green build + log:

nix build .#addone -L
nix log .#addone | tail -n 50

Step 2 — Inject a compile failure

Break a source file. Capture:

nix build .#addone -L 2>&1 | tee fail-compile.txt
nix log .#addone | tee fail-compile-log.txt

Step 3 — --keep-failed

nix build .#addone --keep-failed -L || true
# locate printed /tmp/nix-build-* path; list files; find .o / sources

Document what you could see vs not see.

Step 4 — Inject an install failure

Fix compile; break installPhase (mkdir missing, wrong path). Repeat log + keep-failed.

Step 5 — FOD failure recall

Temporarily wrong src.hash; confirm you recognize the error class in under 10 seconds.

Step 6 — breakpointHook (Linux recommended)

Add hook; force false in buildPhase; enter the breakpoint if your environment supports it. Type env | sort | head, inspect $CC, exit.

Step 7 — Write a personal runbook

DEBUG-RUNBOOK.md with your ordered steps (max one page). Example skeleton:

1. Eval or build?
   
2. -L first failure phase
   
3. nix log
   
4. hash vs compile vs install
   
5. --keep-failed
   
6. hook / extra tools
   
7. only then consider sandbox experiments

Step 8 — Force local rebuild

nix build .#addone --rebuild --option substitute false -L

Note time vs cached build.

---

---

Theory 7 — Reading nix log paths

nix log /nix/store/….drv 2>/dev/null || true
# or from failed build output: link to log under /nix/var/log/nix/drvs

Always save the drv path from a failure before GC. Without it, debugging becomes archaeology.

---

Theory 8 — --keep-failed and dirty trees

nix build -L --keep-failed .#brokenPkg || true
# inspect /tmp/nix-build-… when printed

Compare with pure rebuilds—never commit keep-failed trees. They are for diagnosis only.

---

Worked example — bisect mental model

When a pin update breaks a package:

old lock (green) ── update input ──> new lock (red)
                 narrow: which input?
                 then: which package rebuild?
                 then: which phase?

Day 24 lock discipline + today’s phase logs = faster bisect.

---

Lab — capture a complete failure dossier

For one intentional break, write:

1. Command
   
2. Eval or build?
   
3. Phase (if build)
   
4. First error line
   
5. Fix
   
6. Verification command

Store under notes/debug-dossier.md.

---

Lab — sandbox proof

# show sandbox setting
nix show-config | rg -i sandbox

State whether you ever disabled sandbox (you should not for normal work) and why that is dangerous (undeclared deps “work on my machine”).

Common gotchas

Symptom / mistake	What to do
nix log says no build log	Build never started (eval) or GC’d; rebuild with -L
Kept temp already deleted	Re-run with --keep-failed immediately
breakpointHook “does nothing”	Platform/interactive limits; use keep-failed + better logs
Fixed tree by hand in /tmp only	Always port fix into Nix sources
Disabled sandbox “to move on”	Track impurity; fix properly before Day 56 gate
CI-only failure	Align system, lock, substituters, case-sensitive paths
Silent doCheck fails	Watch for checkPhase in log; set doCheck explicitly

---

Checkpoint

• Used -L and nix log on a real failure
  
• Used --keep-failed and inspected the tree
  
• Classified at least three failure kinds
  
• Tried breakpointHook or documented why not
  
• Personal debug runbook committed

Commit

git add .
git commit -m "day51: debug builds runbook and failure drills"

Write three personal gotchas before continuing.

Tomorrow

Day 52 — nixpkgs layout & by-name: conventions for PR-shaped packages.