Chapter 4: kexec and Kernel-to-Kernel Transitions

Updated

October 31, 2026

Chapter 4: kexec and Kernel-to-Kernel Transitions

While hardware-assisted virtualization enables concurrent multi-kernel execution, many production operations require sequential kernel transitions on bare-metal hardware. Performing a standard server reboot triggers prolonged firmware POST memory training, UEFI device enumeration, and IPMI handshakes that can take 10 to 30 minutes on large multi-socket systems. The Linux kexec subsystem bypasses hardware firmware entirely, directly booting a new kernel from memory in under two seconds.

How kexec Works

The kexec mechanism operates in two distinct phases: load and execute.

[Running Kernel A]
        │
        ▼ (kexec_load syscall)
[Stage Kernel Image, Initramfs & Cmdline into Reserved Memory Pages]
        │
        ▼ (reboot syscall with LINUX_REBOOT_CMD_KEXEC)
[Quiesce Hardware & Drain Device DMA]
        │
        ▼ (Disable Interrupts / Reset APIC)
[Jump to Sub-Trampoline Code]
        │
        ▼ (Overwrites Memory with Staged Image)
[Entry Point of Kernel B (_start)]
  1. System Call kexec_load(): The user space utility loads the new kernel (vmlinuz), initramfs, and kernel command-line strings into memory pages marked safe by the running kernel.
  2. Device Quiescence: When execution is triggered, the kernel invokes device_shutdown() on all registered bus drivers, cleanly stopping DMA engines, clearing PCI device registers, and masking interrupts.
  3. Execution Jump: The CPU transitions to real-mode or identity-mapped assembly code, disables the MMU, copies the staged kernel into its final physical load address, and jumps directly to the entry point of the new kernel.

kexec-tools Configuration and Usage

The kexec-tools package provides userspace control over kexec operations:

# 1. Install kexec-tools
sudo apt-get update && sudo apt-get install -y kexec-tools

# 2. Stage the new kernel into memory
sudo kexec -l /boot/vmlinuz-6.8.0-generic \
  --initrd=/boot/initrd.img-6.8.0-generic \
  --command-line="$(cat /proc/cmdline)"

# 3. Verify that the kernel is loaded
cat /sys/kernel/kexec_loaded
# Output: 1

# 4. Trigger instantaneous transition to the new kernel
sudo kexec -e

When systemd is configured with systemctl kexec, it unmounts filesystems, terminates services cleanly, and issues the kexec execute call automatically.

kdump for Crash Dumping

kexec serves as the foundation for enterprise Linux crash analysis (kdump):

  • Memory Reservation: The host kernel reserves a contiguous block of physical memory at boot using the crashkernel= parameter (e.g., crashkernel=512M).
  • Crash Kernel Staging: Early in userspace boot, the kdump-tools service loads a minimal capture kernel into this reserved memory region via kexec -p.
  • NMI / Panic Trigger: When a fatal kernel panic or hardware Machine Check Exception occurs, the panicked kernel immediately executes crash_kexec(), jumping to the pre-loaded capture kernel without touching the primary memory image.
  • Dumping /proc/vmcore: The capture kernel boots using only its reserved 512MB RAM, exposes the panicked kernel’s entire physical memory at /proc/vmcore, and uses makedumpfile to compress and write the crash dump to disk or network storage.

Limitations of kexec for Multi-Kernel Goals

While indispensable for fast lifecycle transitions and crash analysis, kexec has fundamental architectural limitations:

  • Sequential, Not Concurrent: kexec replaces the currently running kernel. It cannot execute two kernels side-by-side simultaneously.
  • Hardware State Leakage: Bypassing hardware firmware initialization means peripheral controllers (especially complex PCIe cards, RAID HBAs, and GPUs) may retain dirty state, requiring driver-level reset routines (FLR - Function Level Reset) to avoid driver crashes in the new kernel.
  • In-Flight DMA Risks: If an ill-behaved driver fails to cancel DMA mappings before transition, physical memory corruption can occur during the handoff.

Production Use Cases for kexec

  1. Zero-POST Kernel Upgrades: Upgrading the production kernel on 4-socket, 4TB RAM bare-metal database nodes without paying the 20-minute firmware reboot penalty.
  2. Crash Analytics Pipelines: Automated generation and network forwarding of diagnostic core dumps upon kernel panic across large bare-metal fleets.
  3. Staged Provisioning Transitions: Booting an ephemeral discovery/provisioning OS in RAM, and immediately transitioning via kexec into the deployed target OS on disk.