Troubleshooting: Diagnosing Kernel Panics

Updated

October 30, 2026

Troubleshooting: Diagnosing Kernel Panics

A kernel panic occurs when the Linux kernel encounters an unrecoverable internal condition or fatal hardware error from which it cannot safely recover. Unlike a userspace segmentation fault (which simply terminates the offending process), a kernel panic halts the entire operating system to prevent disk corruption.

[    1.248102] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

Anatomy of the Panic Message

When a panic occurs, the kernel dumps diagnostic state directly to the console framebuffer or serial port:

[    1.120340] Call Trace:
[    1.121002]  <TASK>
[    1.121500]  dump_stack_lvl+0x44/0x60
[    1.122100]  panic+0x140/0x320
[    1.122700]  mount_block_root+0x1a8/0x220
[    1.123300]  mount_root+0x9c/0x130
[    1.123900]  prepare_namespace+0x13c/0x170
[    1.124500]  kernel_init+0x18/0x140
[    1.125100]  ret_from_fork+0x28/0x40
[    1.125700]  </TASK>

The stack trace reveals the exact sequence of functions leading up to the failure. In the example above, prepare_namespace called mount_root, which failed and triggered panic().

Common Panic Scenarios for Distro Builders

1. VFS: Unable to mount root fs on unknown-block(0,0)

This is by far the most common panic encountered by operating system builders. It means the kernel was instructed to mount a root partition (root=/dev/sda1 or root=UUID=...), but could not locate or open the device.

Root Causes & Solutions: - Missing disk controller driver: The SATA (ahci), NVMe (nvme), or Virtual VirtIO (virtio_pci, virtio_blk) driver was compiled as a module (=m) instead of built-in (=y), and no initramfs was loaded. Fix: Recompile driver with =y or pack it into the initramfs. - Missing filesystem driver: The root filesystem is formatted as ext4, but CONFIG_EXT4_FS is =m or not set. Fix: Set CONFIG_EXT4_FS=y. - Incorrect root= kernel parameter: The UUID or device node specified in the bootloader does not match the actual disk geometry.

2. Kernel panic - not syncing: Attempted to kill init!

The kernel successfully mounted the root filesystem, found the binary at /sbin/init (or /bin/sh), and executed it. However, PID 1 died or returned. PID 1 is never allowed to exit in Linux. If PID 1 terminates, the kernel panics immediately.

Root Causes & Solutions: - Missing dynamic linker: The init binary was dynamically linked against /lib64/ld-linux-x86-64.so.2, but the linker does not exist at that path in the target rootfs. - Missing shared libraries: The init binary links against libc.so.6, but it is missing from /lib or /usr/lib. - Wrong CPU architecture: Running an x86_64 binary on a 32-bit kernel, or an aarch64 binary without an emulator. - Execution permissions: The init binary lacks execute permissions (chmod +x /sbin/init).

Diagnostic Tools

  1. Early Printk: Add earlyprintk=vga or earlycon and console=ttyS0,115200 to the kernel command line in your bootloader. This prints kernel messages even if the graphics framebuffer crashes early.

  2. QEMU Serial Redirection: When testing your kernel in QEMU, redirect serial output to the terminal:

    $ qemu-system-x86_64 -kernel bzImage -append "console=ttyS0 root=/dev/vda" -nographic

  3. Panic Timeout: Add panic=10 to reboot automatically 10 seconds after a panic, or panic=-1 to freeze the screen for manual inspection.