Chapter 11: Troubleshooting and Diagnostics
Chapter 11: Troubleshooting and Diagnostics
When complex distributed systems fail, haphazard troubleshooting—rebooting machines, guessing kernel parameters, or blindly pasting forum snippets—compounds outages and destroys forensic evidence.
In Ubuntu 26.04 LTS, troubleshooting is a disciplined, falsifiable engineering science. From analyzing early Dracut initramfs boot stalls and extracting memory post-mortems via kdump, to isolating socket latency using ss and rescuing damaged root partitions via chroot, engineers must understand how each subsystem signals distress.
This chapter details the diagnostic methodology, tooling, and emergency rescue procedures required to identify and resolve critical system failures.
The Troubleshooting Mindset
Effective troubleshooting adheres to a structured, hypothesis-driven cycle:
┌────────────────────────────────────────────────────────────────────────┐
│ STRUCTURED TROUBLESHOOTING CYCLE │
│ │
│ 1. Define the Symptom: What broke? When? Who is affected? │
│ 2. Collect Non-Destructive Evidence: Logs, metrics, dmesg, state │
│ 3. Formulate a Falsifiable Hypothesis: "X is failing because of Y" │
│ 4. Execute a Bounded Test: Verify or falsify the hypothesis │
│ 5. Implement Single Controlled Remediation: Change one variable │
│ 6. Verify Resolution: Test service functionality under load │
│ 7. Prevent Recurrence: Document post-mortem and automate checks │
└────────────────────────────────────────────────────────────────────────┘Boot Failures and Dracut Initramfs
Ubuntu 26.04 utilizes Dracut as the modern, modular initramfs generation framework.
┌────────────────────────────────────────────────────────────────────────┐
│ DRACUT BOOT HANDOFF FLOW │
│ │
│ UEFI Firmware ──► GRUB / systemd-boot ──► Linux Kernel (vmlinuz) │
│ │ │
│ ▼ │
│ Dracut Initramfs (in RAM) │
│ ├─ Load NVMe/SATA storage modules│
│ ├─ Query TPM 2.0 / Decrypt LUKS │
│ ├─ Mount real root filesystem │
│ └─ switch_root to /sbin/init! │
└────────────────────────────────────────────────────────────────────────┘The Dracut Emergency Shell
If the root device UUID changes or the storage controller driver is missing, boot execution drops into the Dracut emergency shell:
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
dracut:/# Diagnostic commands inside the initramfs shell:
# 1. View initialization failures
journalctl -xb
# 2. Inspect block devices recognized by kernel
ls -la /dev/disk/by-uuid/
blkid
# 3. Manually unlock encrypted partition if TPM failed
cryptsetup luksOpen /dev/nvme0n1p3 root_crypt
# 4. Mount root manually and test handoff
mount /dev/mapper/root_crypt /sysroot
exitRegenerating Initramfs
Once booted, regenerate Dracut images after updating drivers or storage configurations:
sudo dracut --force --kver $(uname -r)Kernel Crash Dumps with kdump and kexec
When a fatal kernel panic or hardware machine-check exception (MCE) occurs, the running kernel cannot be trusted to write logs.
kdump uses kexec to boot a completely isolated, pre-reserved secondary capture kernel that dumps physical memory into /var/crash/:
# Install and configure kdump
sudo apt install -y linux-crashdump kexec-tools
# Verify crash kernel memory reservation in boot parameters
cat /proc/cmdline | grep crashkernel
# crashkernel=512M
# Check kdump service readiness
sudo kdump-config status
# current state: ready to kdumpWhen a crash occurs, inspect the resulting vmcore post-mortem using the interactive crash utility:
sudo crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcoreDeep Log Forensics with journalctl
Binary logs in systemd-journald provide rapid filtering across boot sessions and units:
# Filter logs from previous boot session (-b -1) for fatal errors
sudo journalctl -b -1 -p err..emerg -o short-precise
# Search logs for out-of-memory events
sudo journalctl -g "Out of memory"
# Correlate logs between kernel and specific service during outage window
sudo journalctl -k -u postgresql.service --since "2026-10-30 08:30" --until "2026-10-30 09:00"Network Debugging: ip, ss, and Packet Capture
When network connectivity fails, isolate the failure layer systematically:
# Layer 1 & 2: Link State & Carrier
ip link show
ethtool enp3s0 | grep "Link detected"
# Layer 3: Addressing and Routing
ip -4 addr show
ip route get 8.8.8.8
# Layer 4: Sockets and TCP Listeners (modern replacement for netstat)
# -t: TCP, -u: UDP, -l: Listening, -p: Process, -n: Numeric
sudo ss -tulpn
# Diagnostic Trace: Combined traceroute and ping with packet loss per hop
mtr --report --report-cycles 10 1.1.1.1
# Layer 7: Real-Time Packet Capture
sudo tcpdump -nn -i enp3s0 port 443 -w capture.pcapFilesystem Corruption and fsck
If an ungraceful power loss or disk hardware fault causes filesystem metadata corruption, ext4 marks the filesystem with an error flag (s_state & EXT4_ERROR_FS).
# Check if root filesystem requires repair (must unmount or mount read-only first!)
sudo fsck.ext4 -fn /dev/nvme0n1p2
# Automated repair of damaged block groups, extents, and directory HTrees
sudo fsck.ext4 -fy /dev/nvme0n1p2Offline Emergency Rescue with Live USB and chroot
If a misconfigured /etc/fstab, broken PAM configuration, or corrupted kernel image renders the machine unbootable, rescue the system using a Live USB environment:
┌────────────────────────────────────────────────────────────────────────┐
│ OFFLINE CHROOT RESCUE PIPELINE │
│ │
│ 1. Boot machine using Ubuntu 26.04 Live USB │
│ 2. Open terminal; unlock encrypted partition: │
│ sudo cryptsetup luksOpen /dev/nvme0n1p3 target_root │
│ 3. Mount root filesystem: │
│ sudo mount /dev/mapper/target_root /mnt │
│ 4. Mount EFI System Partition: │
│ sudo mount /dev/nvme0n1p1 /mnt/boot/efi │
│ 5. Bind mount essential pseudo-filesystems: │
│ for dir in /dev /dev/pts /proc /sys /run; do │
│ sudo mount --bind $dir /mnt$dir │
│ done │
│ 6. Enter target root environment: │
│ sudo chroot /mnt │
│ 7. Repair system (reinstall GRUB, fix passwords, update packages): │
│ passwd adminuser │
│ update-grub │
│ exit │
│ 8. Cleanly unmount and reboot into host OS! │
└────────────────────────────────────────────────────────────────────────┘Chapter Summary
- Structured troubleshooting relies on falsifiable hypotheses and non-destructive evidence gathering.
- Dracut manages the early boot transition; emergency shells allow manual storage and encryption repair.
kdumppreserves memory crash dumps invmcorefor post-mortem panic analysis.ss,ip, andmtrprovide Layer 2 through Layer 4 network diagnostic clarity.- Offline rescue using Live USB and
chrootallows recovery from broken kernels, bootloaders, and configuration files.