Chapter 3: Virtualization vs True Bare-Metal Multi-Kernel

Updated

October 31, 2026

Chapter 3: Virtualization vs True Bare-Metal Multi-Kernel

Achieving multi-kernel execution on physical hardware requires choosing between hardware-assisted virtualization, bare-metal hypervisors, lightweight microVMs, and true physical hardware partitioning. This chapter breaks down the architectural trade-offs, execution models, and isolation mechanics of each paradigm.

Type-1 Hypervisors: ESXi, Xen, KVM in Bare-Metal Mode

Hypervisors are classically categorized by their relationship to the underlying physical hardware:

  • Type-1 (Bare-Metal) Hypervisors: Run directly on bare-metal hardware without an underlying general-purpose host OS.
    • VMware ESXi: Proprietary, closed-source bare-metal hypervisor (VMkernel). Tight hardware compatibility list (HCL), monolithic VM management.
    • Xen: Open-source microkernel hypervisor running in CPU Ring -1 (VMX root mode). Distinguishes between the privileged management domain (dom0) and unprivileged guest domains (domU). Hardware drivers run primarily in dom0.
  • KVM in Bare-Metal Mode: Although technically a Linux kernel module (kvm.ko), KVM transforms a stripped-down Linux host into a Type-1 hypervisor. The Linux kernel runs directly on physical hardware in VMX root mode; guest kernels execute in VMX non-root mode. When tuned with CPU pinning, static memory allocation, and PCIe passthrough, KVM provides bare-metal performance with the maturity of the Linux driver ecosystem.

KVM Architecture Deep Dive

KVM (Kernel-based Virtual Machine) operates as a collaboration between a kernel-space module and a userspace Virtual Machine Monitor (VMM, typically QEMU, Cloud-Hypervisor, or Firecracker):

┌─────────────────────────────────────────────────────────────┐
│                       USERSPACE                             │
│   ┌─────────────────────────┐   ┌────────────────────────┐  │
│   │ QEMU / VMM Instance 1   │   │ QEMU / VMM Instance 2  │  │
│   │ [vCPU 0]      [vCPU 1]  │   │ [vCPU 0]      [vCPU 1] │  │
│   └────┬──────────────┬─────┘   └────┬──────────────┬────┘  │
│        │ ioctl()      │ ioctl()      │ ioctl()      │       │
├────────┼──────────────┼──────────────┼──────────────┼───────┤
│        ▼              ▼              ▼              ▼       │
│   ┌──────────────────────────────────────────────────────┐  │
│   │                KVM Kernel Subsystem                  │  │
│   │               /dev/kvm Character Device              │  │
│   └──────────────────────────┬───────────────────────────┘  │
│                              │                              │
│                              ▼                              │
│                     VMX / SVM Hardware                      │
│                  Extended Page Tables (EPT)                 │
└─────────────────────────────────────────────────────────────┘
  1. The /dev/kvm Interface: The VMM opens /dev/kvm and issues KVM_CREATE_VM to instantiate a virtual machine descriptor.
  2. vCPU Creation: KVM_CREATE_VCPU creates execution threads. Each vCPU is an ordinary POSIX thread in userspace scheduled onto physical CPU cores.
  3. Execution Loop: The VMM enters KVM_RUN. The processor executes a VMLAUNCH/VMRESUME instruction, transitioning the core into VMX non-root mode to execute guest kernel code natively at full clock speed.
  4. VM-Exits: When the guest kernel executes privileged instructions (e.g., programming I/O ports, modifying CR3 page directory base register without shadowing, or triggering hardware interrupts), the hardware triggers a VM-Exit, returning control to KVM.

Linux Containers and Shared-Kernel Architectures

Linux containers (Docker, Podman, LXC/LXD) isolate processes using kernel software boundaries:

  • Namespaces: Seven distinct namespaces (pid, net, mnt, ipc, uts, user, cgroup) partition the process’s view of global system resources.
  • cgroups v2: Enforce resource limits, accounting, and bandwidth throttling across CPU, memory, I/O, and pids.

Why Containers Fail the Multi-Kernel Requirement: - All containers share the single host kernel binary and runtime memory. - A kernel panic crashes every container simultaneously. - Different kernel versions, custom module requirements, and kernel command-line isolation parameters cannot be applied per-container. - Containers share the kernel syscall attack surface.

MicroVMs and Lightweight Virtualization

MicroVMs (such as AWS Firecracker and Cloud-Hypervisor) represent a convergence of container density and hypervisor-grade isolation:

  • Stripped-Down Device Model: They completely eliminate legacy PC emulation (no IDE controllers, no floppy drives, no ACPI power states, no PCI buses).
  • Virtio-Over-MMIO: Device I/O is handled strictly through memory-mapped I/O (MMIO) virtio queues.
  • Sub-50ms Boot Time: Lightweight guest kernels boot from memory in tens of milliseconds with minimal memory footprint (~5MB overhead per instance).
  • Rust-Based Memory Safety: The VMM implementation in Rust eliminates entire classes of buffer overflow and memory corruption vulnerabilities.

True Bare-Metal Partitioning Concepts

True bare-metal hardware partitioning achieves multi-kernel execution without any hypervisor software:

  • Physical Board-Level Partitioning: Enterprise hardware (such as IBM Power Systems LPAR, HP Superdome nPars) uses physical hardware switches and crossbar fabric routing to split a chassis into electrically isolated hardware partitions.
  • Firmware-Partitioned Multikernels: Modern systems use customized firmware (or early bootloader partitions) to carve non-overlapping physical CPU ranges and memory addresses, booting two distinct kernels side-by-side on bare silicon.
  • PCIe Fabric Partitioning: Modern PCIe switches (such as Microchip Pax and Broadcom ExpressFabric) partition physical PCIe lanes and root complexes into distinct virtual hierarchies routed to separate physical hosts or processing clusters.