Chapter 13: Troubleshooting Guide

Updated

October 31, 2026

Chapter 13: Troubleshooting Guide

Operating multi-kernel bare-metal systems introduces complex failure modes spanning firmware handoffs, IOMMU translation failures, PCIe reset issues, and cross-kernel resource contention. This chapter provides an evidence-based diagnostic guide to resolve the most common and challenging failure scenarios.

Boot Failures and Recovery

Symptom: Secondary Kernel Freezes on Early Boot

  • Diagnosis: The secondary kernel hangs immediately after decompression with no output on the console.
  • Root Cause: Console output is directed to a virtual VGA or serial port that is unmapped in the VMM configuration, or ACPI power tables conflict with guest direct boot.
  • Fix: Add earlyprintk=serial,ttyS0,115200 console=ttyS0 keep_bootcon to the guest kernel command line. Ensure QEMU includes -serial mon:stdio -nographic. If ACPI hangs, test with acpi=off to isolate ACPI table parsing bugs.

Symptom: Host Panics During Kernel Boot with isolcpus

  • Diagnosis: Host kernel enters a panic or hangs during SMP bringup when isolcpus or nohz_full is specified.
  • Root Cause: Core 0 (the primary bootstrap processor - BSP) was inadvertently included in the isolation mask.
  • Fix: Core 0 must always remain a housekeeping core. Ensure isolation ranges begin at Core 1 or higher (e.g., isolcpus=domain,managed_irq,16-31).

Device Assignment and VFIO Issues

Symptom: Device is ineligible for IOMMU domain attach

  • Diagnosis: QEMU fails to start with error: Failed to assign device: Device is ineligible for IOMMU domain attach due to other devices in group.
  • Root Cause: The PCIe device belongs to an IOMMU group containing other devices (e.g., PCIe root ports or onboard audio) that are still bound to host drivers.
  • Fix: Check ls /sys/bus/pci/devices/<pci_id>/iommu_group/devices/. Either bind every device in that group to vfio-pci, install the device into an ACS-isolated PCIe slot directly connected to the CPU root complex, or enable PCIe ACS override patches (for lab/testing environments only).

Symptom: PCIe Device Fails Function-Level Reset (FLR)

  • Diagnosis: Guest reboots fail with Device failed to reset after FLR; device will be marked dead.
  • Root Cause: The physical PCIe controller does not implement standard PCIe FLR, or retains internal state across resets (common on certain AMD GPUs and RAID controllers).
  • Fix: Use vendor-specific reset quirks (e.g., vendor-reset kernel module for AMD GPUs) or configure secondary bus reset (SBR) via setpci.

Networking Problems

Symptom: Asymmetric Routing or Dropped Packets on Inter-Kernel Bridge

  • Diagnosis: Secondary kernel can ping host, but host cannot ping secondary kernel; or high packet drop rate under load.

  • Root Cause: Netfilter/iptables rules on the host are filtering bridge traffic via the br_netfilter module, or MTU mismatches exist between tap interface and bridge.

  • Fix:

    # Disable bridge netfilter packet mangling
sudo sysctl -w net.bridge.bridge-nf-call-iptables=0
sudo sysctl -w net.bridge.bridge-nf-call-ip6tables=0

# Ensure matching MTUs across bridge and tap
sudo ip link set dev br-internal mtu 9000
sudo ip link set dev tap0 mtu 9000

Performance Degradation Diagnosis

Symptom: Unexplained High Latency Jitter (P99 Spikes)

  • Diagnosis: Real-time tasks running in the secondary kernel experience 50µs+ latency spikes.
  • Investigation Steps:
    1. Check for unexpected VM-exits using perf kvm stat live. Look for excessive CR3_LOAD (TLB flushes) or IO_INSTRUCTION exits.
    2. Verify that SMT sibling threads are not running unpinned host workloads.
    3. Verify that irqbalance is not sending physical interrupts to isolated cores.
    4. Check for Intel Turbo Boost frequency throttling or CPU C-state sleep transitions. Force C-state limiting with intel_idle.max_cstate=1 processor.max_cstate=1.

Kernel Panics and Crashes

Symptom: Host Freezes During Secondary Kernel Shutdown

  • Diagnosis: The entire physical server locks up when stopping a secondary kernel instance.
  • Root Cause: A passed-through PCIe device issued a DMA transaction to an unmapped host physical address immediately after the guest tore down its memory tables.
  • Fix: Update motherboard BIOS/firmware to resolve IOMMU DMAR bugs. Verify that the host kernel command line includes iommu=pt to ensure physical translation bypass is enabled for host devices.

Resource Exhaustion

Symptom: Cannot allocate memory When Allocating Hugepages

  • Diagnosis: Attempting to allocate static 1GB hugepages fails even though free -m reports sufficient free RAM.
  • Root Cause: Memory fragmentation has broken contiguous physical memory blocks required for 1GB allocations.
  • Fix: Pre-allocate hugepages at boot time via the kernel command line (hugepagesz=1G default_hugepagesz=1G hugepages=64) before physical memory becomes fragmented.