Troubleshooting: Build Infrastructure Issues

Troubleshooting: Build Infrastructure Issues

Operating a distributed build farm introduces failure modes distinct from standard software bugs: storage exhaustion, zombie containers, and lock contention.

1. Out-of-Disk Space During Large C++ Builds (LLVM / WebKit)

Symptom: Worker nodes crash mid-build with:

fatal error: error writing to /tmp/ccXXXXXX.s: No space left on device

Yet df -h shows plenty of disk space on the root filesystem!

Root Cause: - Many modern Linux systems mount /tmp as a tmpfs (RAM disk) restricted to 50% of physical RAM. - Compiling massive projects like LLVM or Chromium with debug symbols enabled (-g) can generate over 40 gigabytes of intermediate object files and compiler temporary scratch space, filling /tmp in minutes.

Remedy: In your build runner configuration, explicitly redirect TMPDIR to a physical NVMe scratch volume:

export TMPDIR="/var/cache/build-scratch"
mkdir -p "$TMPDIR"

2. Dynamic Memory Exhaustion by Parallel Linkers (ld OOM)

Symptom: During the final linking step of an executable, the build node freezes or the process terminates with:

collect2: fatal error: ld terminated with signal 9 [Killed]

Root Cause: - GNU ld uses significant RAM when linking massive C++ binaries with full symbols. - If you run make -j32 and 4 linkers fire simultaneously, they can consume over 64 GB of RAM, triggering the kernel’s Out-Of-Memory (OOM) Killer.

Remedy: - Replace the legacy GNU bfd linker with modern high-performance linkers: lld (LLVM) or mold (High-Performance Modern Linker): bash export LDFLAGS="-fuse-ld=mold" mold is up to 10x faster than GNU ld and consumes a fraction of the resident memory.

3. Orphaned Mounts & Leaked Loop Devices in Containers

Symptom: systemd-nspawn or chroot teardown fails with device is busy, leaving dozens of abandoned /dev/loopX devices in /dev.

Remedy: - Before unmounting, ensure all lingering processes inside the rootfs namespace are terminated: bash # fuser -k -m /mnt/buildroot # umount -R /mnt/buildroot - Clean up dangling loopback attachments: bash # losetup -D