Chapter 5: Hardware-Assisted Multi-Kernel with KVM/QEMU

Updated

October 31, 2026

Chapter 5: Hardware-Assisted Multi-Kernel with KVM/QEMU

When true concurrent multi-kernel execution is required on commodity x86_64 or ARM64 servers, hardware-assisted virtualization through KVM and QEMU provides the most robust, battle-tested foundation. However, running a secondary kernel with bare-metal performance guarantees requires abandoning default cloud-style overcommit behaviors. Every vCPU must be strictly pinned, memory statically backed by hugepages, and NUMA locality rigorously preserved.

KVM Setup and Kernel Module Configuration

To prepare an Ubuntu 26.04 host for low-latency multi-kernel operations:

  1. Verify Hardware Virtualization Extensions:

    egrep -c '(vmx|svm)' /proc/cpuinfo
# Must return core count > 0

  2. Load and Configure KVM Modules: Create /etc/modprobe.d/kvm.conf:

    # For Intel platforms
options kvm_intel nested=1
options kvm_intel preemption_timer=0
options kvm_intel enable_shadow_vmcs=1
options kvm ignore_msrs=1

  3. Verify Module Initialization:

    sudo modprobe kvm_intel
ls -la /dev/kvm
# Verify crw-rw----+ root kvm

Guest Kernel Installation and Configuration

Secondary kernels running alongside the host should be purpose-built to eliminate unnecessary drivers, emulated device probing, and ACPI bloat:

  • Minimalist .config: Compile the guest kernel with only CONFIG_VIRTIO, CONFIG_VFIO, and specific hardware drivers needed for passed-through PCIe devices.
  • Direct Kernel Boot: Rather than emulating a virtual disk containing a virtual bootloader (GRUB), QEMU can boot the guest kernel image directly from the host filesystem using the -kernel, -initrd, and -append arguments. This eliminates seconds of virtual firmware negotiation.

CPU Pinning and NUMA-Aware VM Placement

To eliminate scheduling latency, context-switching overhead, and cross-socket bus contention:

  1. Identify Host NUMA Topology:

    numactl --hardware
# Node 0: CPUs 0-15, 64GB RAM
# Node 1: CPUs 16-31, 64GB RAM

  2. Isolate Physical Cores for the Secondary Kernel: Host boot command line: isolcpus=domain,managed_irq,16-31 nohz_full=16-31 rcu_nocbs=16-31.

  3. Pin QEMU vCPUs to Dedicated Physical Cores: Each guest vCPU thread is mapped 1:1 to an isolated physical core. In libvirt XML:

    <vcpu placement='static'>16</vcpu>
<cputune>
  <vcpupin vcpu='0' cpuset='16'/>
  <vcpupin vcpu='1' cpuset='17'/>
  <vcpupin vcpu='2' cpuset='18'/>
  <!-- ... through core 31 ... -->
  <emulatorpin cpuset='0-1'/>
</cputune>
<numatune>
  <memory mode='strict' nodeset='1'/>
</numatune>

Memory Ballooning vs Static Allocation

Default virtualization relies on memory ballooning (virtio-balloon) to dynamically reclaim memory from idle guests, and KSM (Kernel Samepage Merging) to deduplicate shared memory pages. Both must be disabled for multi-kernel bare-metal workloads:

  • Memory Ballooning Penalties: Causes random guest kernel page allocation latency spikes when the balloon driver inflates.

  • KSM Overhead: Periodically scans physical memory, invalidating page table entries and triggering TLB shootdowns.

  • Static Hugepages Allocation: Back the secondary kernel’s entire memory footprint with 1GB static hugepages allocated exclusively from NUMA Node 1:

    echo 64 | sudo tee /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages

In QEMU/libvirt configuration:

<memoryBacking>
  <hugepages>
    <page size='1' unit='G' nodeset='1'/>
  </hugepages>
  <locked/>
</memoryBacking>

The <locked/> directive executes mlock(), guaranteeing the secondary kernel’s pages are permanently locked in physical RAM and cannot be swapped out under any circumstance.

Complete Working Deployment Example

Here is a standalone bash script launching an independent Linux kernel on NUMA Node 1:

#!/usr/bin/env bash
set -euo pipefail

GUEST_KERNEL="/opt/kernels/vmlinuz-6.8.0-rt"
GUEST_INITRD="/opt/kernels/initrd.img-6.8.0-rt"
GUEST_CMDLINE="console=ttyS0 root=/dev/nvme0n1p1 rw isolcpus=1-15 nohz_full=1-15"

taskset -c 16-31 qemu-system-x86_64 \
  -name worker-kernel-node1,debug-threads=on \
  -enable-kvm \
  -cpu host,migratable=off,+invtsc \
  -smp 16,sockets=1,cores=16,threads=1 \
  -m 64G \
  -object memory-backend-memfd,id=mem1,size=64G,hugetlb=on,hugetlbsize=1G,host-nodes=1,policy=strict \
  -numa node,memdev=mem1,nodeid=0 \
  -kernel "$GUEST_KERNEL" \
  -initrd "$GUEST_INITRD" \
  -append "$GUEST_CMDLINE" \
  -nographic \
  -serial mon:stdio

This configuration delivers deterministic microsecond execution directly on bare metal alongside the primary management kernel.