Chapter 11: Performance Engineering

Updated

October 31, 2026

Chapter 11: Performance Engineering

Running multiple kernels on bare metal is often driven by stringent performance requirements. While hardware-assisted virtualization provides near-native execution speed, unoptimized configurations introduce subtle latencies, cache pollution, and I/O bottlenecks. This chapter quantifies the “virtualization tax,” evaluates cache and NUMA effects, benchmarks storage and networking datapaths, and presents a rigorous methodology for measuring multi-kernel performance.

Boot Time Analysis

In ephemeral or fast-recovery architectures, kernel boot time is a critical operational SLA:

  • Firmware Overhead: Physical UEFI initialization takes 60–600 seconds on server hardware.
  • Direct Kernel Boot vs. Virtual Firmware: Direct kernel booting via QEMU (-kernel) reduces guest boot time from ~12 seconds (with SeaBIOS or OVMF) to <1.2 seconds.
  • Kernel Image Trimming: Disabling unneeded drivers, modules, and early splash screens reduces kernel decompression and hardware probing time. Using uncompressed vmlinux with modern fast NVMe storage can yield boot times under 150ms.
  • Initramfs Optimization: Constructing minimal static initramfs binaries (e.g., using BusyBox or custom Go/Rust binaries) cuts early userspace startup to tens of milliseconds.

CPU Overhead and Virtualization Tax

When CPU cores are strictly pinned (1:1 vCPU-to-pCPU mapping), where does overhead come from?

  1. VM-Exits: A context switch between VMX non-root mode (guest execution) and VMX root mode (KVM handler) costs approximately 800 to 1,500 CPU cycles. Minimizing VM-exits is the primary goal of low-latency virtualization.
  2. Two-Dimensional Paging (EPT/NPT): Translating guest virtual addresses (GVA) to guest physical addresses (GPA), and then to host physical addresses (HPA), requires traversing nested page tables. In the worst case, a TLB miss costs up to 24 memory accesses.
    • Mitigation: Backing memory with 1GB hugepages reduces page table depth to 2 levels and maximizes TLB coverage.
  3. Timer Virtualization: Modern processors feature hardware TSC scaling and Invariant TSC (+invtsc). Enabling this flag exposes the physical RDTSC instruction directly to the guest without hypervisor interception.

Cache Effects and NUMA Locality

Cross-node memory access severely degrades throughput:

Direct Local Memory Access (Node 1)    : ~75 ns latency, 150 GB/s bandwidth
Cross-Socket UPI/Fabric Access (Node 0): ~145 ns latency, 45 GB/s bandwidth
  • NUMA Misplacement Penalties: If a secondary kernel running on Socket 1’s cores allocates memory from Socket 0, every memory read traverses the inter-socket bus, halving bandwidth and doubling latency.
  • L3 Cache Slicing: On multi-die architectures (AMD EPYC CCDs), tasks must be pinned within the same Core Complex (CCX) to avoid intra-socket L3 cache miss penalties.

I/O Performance: Virtio vs Passthrough

Empirical throughput and latency comparisons for 100GbE networking and NVMe storage:

I/O Datapath Throughput (100GbE / NVMe) P99 Latency CPU Utilization
Emulated (e1000 / IDE) 1.8 Gbps / 45 MB/s 850 µs High (100% core saturation)
Virtio (virtio-net / virtio-blk) 38 Gbps / 1,800 MB/s 35 µs Moderate (vhost-net kernel thread)
SR-IOV Virtual Function (VFIO) 94 Gbps / N/A 3.2 µs Minimal (zero hypervisor overhead)
PCI Passthrough (Direct NVMe) N/A / 6,800 MB/s 8.5 µs Native (exact bare-metal equivalence)

PCI passthrough and SR-IOV eliminate the hypervisor from the I/O datapath entirely, delivering 98–99% of raw bare-metal hardware capacity.

Benchmarking Methodology for Multi-Kernel Systems

Designing statistically valid benchmarks to evaluate multi-kernel isolation:

  1. Establish Single-Kernel Baseline: Benchmark the physical server under single-kernel native Linux using standard tools (fio, iperf3, wrk, stress-ng).
  2. Co-Located Idle Test: Deploy the secondary kernel in an idle state and re-run host benchmarks to measure baseline hypervisor noise.
  3. Noisy Neighbour Stress Test: Generate intense cache-thrashing, memory bus saturation, and I/O pressure on Kernel A while recording the latency distribution of deterministic tasks on Kernel B.
  4. Latency Percentile Profiling: Focus on high percentiles (P99, P99.9, P99.99) rather than averages. Multi-kernel isolation proves its value at the tail.