Containers share a host kernel. Traditional virtual machines emulate a 1990s motherboard—BIOS, PCI bridges, IDE controllers, PS/2 keyboards, floppy drives—just to run a Linux server. MicroVMs sit in the sweet spot between the two: hardware-enforced kernel isolation via KVM, paired with a minimalist, purpose-built Virtual Machine Monitor (VMM) that boots a Linux kernel in ~100 milliseconds with under 5 MiB of memory overhead.

This guide builds the entire mental model from silicon up to production, using NixOS as our reproducible host and guest platform. By the end, you will understand what the CPU actually does on a VM-Exit, how to drive /dev/kvm yourself in Rust, how a VMM boots Linux without a BIOS, how virtio moves packets and disk blocks without burning CPU cycles, and how Firecracker, Cloud Hypervisor, and microvm.nix put it all together in production.

Table of Contents⌗

  1. Trap-and-emulate to VT-x, VMCS and EPT
  2. KVM from user space in Rust
  3. Booting Linux
  4. virtio devices
  5. Production VMMs: Firecracker and Cloud Hypervisor

NixOS Host Prerequisites⌗

Before diving into CPU internals and Rust code, let’s declare a clean NixOS host environment. One of the best parts of doing virtualization work on NixOS is that kernel modules, device permissions, and toolchains live in declarative files rather than scattered shell history.

In your /etc/nixos/configuration.nix (or your host flake module), enable KVM and add your user to the kvm group:

{ config, pkgs, ... }:

{
  # Load Intel VT-x or AMD-V KVM module
  boot.kernelModules = [
    "kvm-intel" # use "kvm-amd" on AMD CPUs
    "tun"
    "vhost_net"
  ];

  # Optional: enable nested virtualization if running inside a VM
  boot.extraModprobeConfig = ''
    options kvm_intel nested=1
  '';

  # Allow your non-root user to open /dev/kvm
  users.users.king.extraGroups = [ "kvm" ];

  # Enable IP forwarding if you plan to route traffic from microVM TAP devices
  boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
}

Apply it with sudo nixos-rebuild switch (and re-login for the kvm group), then verify hardware virtualization support:

# Check CPU flags (vmx for Intel VT-x, svm for AMD-V)
grep -E -c '(vmx|svm)' /proc/cpuinfo

# Confirm /dev/kvm is accessible
ls -l /dev/kvm

Throughout Sections 2–5, we will use a single flake.nix dev shell so every tool (rustc, cargo, firecracker, cloud-hypervisor, firectl, iproute2, e2fsprogs, jq, curl) is pinned and reproducible:

# flake.nix
{
  description = "MicroVMs with KVM, Rust, Firecracker, and Cloud Hypervisor";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";

  outputs = { self, nixpkgs }:
    let
      system = "x86_64-linux";
      pkgs = nixpkgs.legacyPackages.${system};
    in {
      devShells.${system}.default = pkgs.mkShell {
        packages = with pkgs; [
          rustc
          cargo
          rust-analyzer
          pkg-config
          firecracker
          firectl
          cloud-hypervisor
          curl
          jq
          iproute2
          e2fsprogs
          cpio
          busybox
          screen
        ];
      };
    };
}

Enter it anytime with nix develop.


1. Trap-and-emulate to VT-x, VMCS and EPT⌗

Popek & Goldberg and Why Classic x86 Broke Trap-and-Emulate⌗

In 1974, Gerald Popek and Robert Goldberg formalized the conditions for classical virtualization. Suppose you want to run an unmodified guest OS directly on the CPU without letting it take over the machine:

  • Privileged instructions trap (raise an exception into the hypervisor) when executed outside Ring 0.
  • Sensitive instructions read or modify privileged hardware state (interrupt flags, page tables, descriptor tables, privilege level).

Trap-and-emulate works cleanly if and only if every sensitive instruction is also a privileged instruction. You demote the guest kernel from Ring 0 to Ring 1 (or Ring 3), let it run at native speed, and whenever it tries to touch sensitive state, the CPU traps into the Ring 0 hypervisor, which emulates the effect on virtual CPU state and resumes the guest.

Classic x86 violated this theorem. Roughly 17 x86 instructions were sensitive but unprivileged—when executed in Ring 1 or Ring 3, they silently failed or leaked host state instead of trapping:

InstructionWhat happens in Ring 1/3 on classic x86
POPFSilently ignores changes to the Interrupt Flag (IF) instead of trapping
PUSHFReads the host’s IF rather than the guest’s virtual IF
SGDT / SIDT / SLDTLeaks the host’s Global/Interrupt/Local Descriptor Table base addresses
SMSWReads machine status word without trapping
CS register readGuest kernel inspects CS & 3 and discovers it is not in Ring 0

For a decade, hypervisors worked around this in two painful ways:

  1. Binary translation (early VMware): scan guest kernel code blocks at runtime and rewrite unsafe instructions into hypervisor calls.
  2. Paravirtualization (early Xen): modify the guest OS kernel source code so it makes explicit hypercalls instead of executing unsafe instructions.

Intel VT-x (VMX) and VMCS⌗

In 2005–2006, Intel (VT-x / VMX) and AMD (AMD-V / SVM) fixed this in silicon by adding an orthogonal privilege dimension: VMX Root Operation (for the hypervisor) and VMX Non-Root Operation (for the guest). Both modes have their own Rings 0–3, so an unmodified Linux guest runs happily in Non-Root Ring 0 while the host kernel stays in control in Root Ring 0.

+-------------------------------------------------------------------+
|                  VMX ROOT OPERATION (Host / KVM)                  |
|                                                                   |
|  Ring 3: VMM User Space (Rust / Firecracker / Cloud Hypervisor)   |
|  Ring 0: Host Linux Kernel + KVM (kvm.ko / kvm_intel.ko)          |
|                                                                   |
|          VMLAUNCH / VMRESUME              VM-Exit (Trap)          |
|         (loads Guest State)            (saves Guest State,        |
|                  |                      loads Host State)         |
|                  v                             ^                  |
+------------------|-----------------------------|------------------+
                   |     +-----------------+     |
                   +---->|      VMCS       |-----+
                         |  (4 KiB region) |
                         +-----------------+
                   |                             ^
                   v                             |
+------------------|-----------------------------|------------------+
|               VMX NON-ROOT OPERATION (Guest VM)                   |
|                                                                   |
|  Ring 0: Unmodified Guest Linux Kernel                            |
|  Ring 3: Guest Applications / Containers                          |
|                                                                   |
|  Executes natively until: I/O port, MMIO, HLT, CPUID, EPT fault,  |
|  external interrupt, MSR access, etc. -> triggers VM-Exit         |
+-------------------------------------------------------------------+

Every virtual CPU (vCPU) is backed by a 4 KiB hardware-managed structure in memory called the VMCS (Virtual Machine Control Structure), accessed via VMREAD and VMWRITE:

+-------------------------------------------------------------------+
|               VMCS (Virtual Machine Control Structure)            |
+-------------------------------------------------------------------+
| 1. Guest-State Area     : RIP, RSP, RFLAGS, CR0/3/4, CS/DS/SS/TR, |
|                           GDTR, IDTR, EFER, debug & activity state|
+-------------------------------------------------------------------+
| 2. Host-State Area      : Host RIP (KVM exit handler), RSP, CR3,  |
|                           segment selectors, MSRs                 |
+-------------------------------------------------------------------+
| 3. VM-Execution Controls: Which events cause VM-Exit? (I/O bitmap,|
|                           MSR bitmap, HLT, CPUID, EPT pointer,    |
|                           VPID, preemption timer)                 |
+-------------------------------------------------------------------+
| 4. VM-Exit Controls     : Save/load MSRs on exit, host addr-space |
|                           size (64-bit), ack interrupt on exit    |
+-------------------------------------------------------------------+
| 5. VM-Entry Controls    : Event/interrupt injection into guest,   |
|                           IA-32e (64-bit) guest mode              |
+-------------------------------------------------------------------+
| 6. VM-Exit Information  : Exit Reason (e.g., IO_INSTRUCTION=30,   |
|                           EPT_VIOLATION=48), Exit Qualification,  |
|                           Guest Physical Address                  |
+-------------------------------------------------------------------+

The hardware transition is atomic:

  • VM-Entry (VMLAUNCH on first run, VMRESUME thereafter): the CPU snapshots host state into the Host-State Area, loads guest registers from the Guest-State Area, switches to VMX Non-Root mode, and starts executing guest instructions at native speed.
  • VM-Exit: when the guest hits a configured intercept (like IN/OUT, HLT, CPUID, an unmapped MMIO address, or a host hardware interrupt), the CPU saves guest state back into the VMCS, records the Exit Reason, restores host state, and jumps straight into KVM’s exit handler in Root Ring 0.

Memory Virtualization: From Shadow Page Tables to EPT⌗

CPU virtualization solves instruction execution, but memory is harder. A guest OS thinks it owns physical RAM starting at address 0x0, and programs its own page tables (CR3) to map Guest Virtual Addresses (GVA) to Guest Physical Addresses (GPA). The host hardware, however, only understands Host Physical Addresses (HPA).

Before hardware support, hypervisors maintained Shadow Page Tables in software, trapping every guest CR3 write and page-table modification—enormous overhead.

Intel EPT (Extended Page Tables) (and AMD NPT / Nested Page Tables) adds a second hardware page-table walk managed by the hypervisor via the EPTP field in the VMCS:

                 Guest Virtual Address (GVA)
                             |
                             v
            +---------------------------------+
            | Stage 1: Guest Page Tables      |  (Managed by Guest OS,
            | (Guest CR3: PML4->PDPT->PD->PT) |   each table pointer is
            +---------------------------------+   itself a GPA!)
                             |
                             v
                 Guest Physical Address (GPA)
                             |
                             v
            +---------------------------------+
            | Stage 2: Extended Page Tables   |  (Managed by KVM via
            | (VMCS EPTP: PML4->PDPT->PD->PT) |   KVM_SET_USER_MEMORY)
            +---------------------------------+
                             |
                             v
                 Host Physical Address (HPA)

Notice the subtlety: every pointer inside the guest’s 4-level page table is itself a GPA, so on a full TLB miss, the hardware MMU performs a 2D page walk—walking EPT to translate each level of the guest page table, plus a final EPT walk for the data page (up to $4 \times 4 + 4 + 4 = 24$ memory references in the worst case!). To keep this fast:

  • VPID (Virtual Processor Identifier) tags TLB entries per vCPU so TLBs do not have to be flushed on every VM-Entry/VM-Exit.
  • Huge pages (2 MiB / 1 GiB) in both guest and EPT dramatically collapse the walk depth.
  • EPT Violations: when a guest accesses a GPA that is not mapped in EPT (for example, a virtual device’s MMIO register range), the CPU triggers an EPT_VIOLATION VM-Exit and hands the faulting GPA to KVM, which forwards it to our user-space VMM as an MMIO exit.

2. KVM from user space in Rust⌗

Linux turns the host kernel into a Type-1-class hypervisor via KVM (/dev/kvm). KVM itself does not emulate disks, NICs, or serial consoles—it handles VT-x/SVM, VMCS, EPT, and vCPU scheduling, and exposes a clean ioctl-based API to a user-space Virtual Machine Monitor (VMM).

The KVM ioctl Lifecycle⌗

Every VMM—whether QEMU, Firecracker, Cloud Hypervisor, or a 100-line Rust binary—follows the exact same file-descriptor hierarchy:

User-Space VMM (Rust)                       Host Kernel (KVM)
=====================                       =================

1. open("/dev/kvm") ----------------------+-> System fd (kvm_fd)
                                          |
2. ioctl(KVM_CREATE_VM) ------------------+-> VM fd (vm_fd)
                                          |
3. mmap(anonymous RAM)                    |
   ioctl(KVM_SET_USER_MEMORY_REGION) -----+-> Maps HVA -> GPA in EPT
                                          |
4. ioctl(KVM_CREATE_VCPU, id=0) ----------+-> vCPU fd (vcpu_fd)
   mmap(vcpu_fd, kvm_run_size) -----------+-> Shared struct kvm_run
                                          |
5. ioctl(KVM_SET_SREGS / KVM_SET_REGS) ---+-> Writes initial VMCS state
                                          |
6. loop {                                 |
     ioctl(KVM_RUN) ----------------------+-> VMLAUNCH / VMRESUME
                                          |     [Guest executes natively]
     <-- returns on VM-Exit --------------+<- VM-Exit (fills kvm_run)
     match exit_reason {                  |
       IoOut  => write to stdout          |
       Mmio   => handle virtio device     |
       Hlt    => break                    |
     }                                    |
   }

Memory Model: HVA, GPA, and mmap⌗

When you allocate guest RAM in user space with mmap(MAP_ANONYMOUS | MAP_PRIVATE) and register it via KVM_SET_USER_MEMORY_REGION, the same physical pages are accessible from both sides:

Host User Space (VMM)                  Guest Physical Address Space
=====================                  ============================
HVA: 0x7f8a_0000_0000  <-- writes -->  GPA: 0x0000_0000 (RAM start)
  (returned by mmap)       kernel,       ...
                           code,         ...
                           virtqueues    ...
HVA: +16 MiB                           GPA: 0x0100_0000 (16 MiB)
                                       ----------------------------
                                       GPA: 0xd000_0000 (Unmapped!)
                                         -> Access triggers EPT
                                            violation -> VcpuExit::Mmio

This zero-copy shared memory model is why our VMM can simply memcpy a kernel image into the mmap’d slice before calling KVM_RUN, and later read/write guest virtio rings directly from user space.

A Minimal VMM in Rust⌗

Let’s build a real, working VMM in Rust inside our NixOS nix develop shell. It will create a VM, map 64 KiB of guest RAM, write raw x86 machine code that adds 2 + 3, converts the result to ASCII '5', prints it to serial port 0x3f8, and halts.

Create a new Cargo project:

nix develop
cargo new mini-vmm && cd mini-vmm

Add kvm-ioctls, kvm-bindings, and libc to Cargo.toml:

[package]
name = "mini-vmm"
version = "0.1.0"
edition = "2021"

[dependencies]
kvm-ioctls = "0.19"
kvm-bindings = "0.10"
libc = "0.2"

Now write src/main.rs:

use kvm_bindings::kvm_userspace_memory_region;
use kvm_ioctls::{Kvm, VcpuExit};
use std::ptr;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 1. Open /dev/kvm and create a VM
    let kvm = Kvm::new()?;
    let vm = kvm.create_vm()?;

    // 2. Allocate 64 KiB of host memory for guest RAM
    let mem_size: usize = 0x10000;
    let guest_addr: u64 = 0x1000;
    let load_addr = unsafe {
        libc::mmap(
            ptr::null_mut(),
            mem_size,
            libc::PROT_READ | libc::PROT_WRITE,
            libc::MAP_ANONYMOUS | libc::MAP_SHARED | libc::MAP_NORESERVE,
            -1,
            0,
        )
    };
    assert_ne!(load_addr, libc::MAP_FAILED, "mmap failed");

    // 3. Tiny 16-bit x86 guest program:
    //    mov dx, 0x3f8      ; COM1 serial port
    //    mov al, 2
    //    add al, 3          ; al = 5
    //    add al, '0'        ; al = '5' (0x35)
    //    out dx, al         ; trigger VcpuExit::IoOut
    //    mov al, '\n'
    //    out dx, al         ; trigger VcpuExit::IoOut
    //    hlt                ; trigger VcpuExit::Hlt
    let guest_code: &[u8] = &[
        0xba, 0xf8, 0x03, // mov dx, 0x3f8
        0xb0, 0x02,       // mov al, 2
        0x04, 0x03,       // add al, 3
        0x04, 0x30,       // add al, '0'
        0xee,             // out dx, al
        0xb0, 0x0a,       // mov al, '\n'
        0xee,             // out dx, al
        0xf4,             // hlt
    ];

    unsafe {
        ptr::copy_nonoverlapping(guest_code.as_ptr(), load_addr as *mut u8, guest_code.len());
    }

    // 4. Register host memory into the guest's EPT at GPA 0x1000
    let mem_region = kvm_userspace_memory_region {
        slot: 0,
        guest_phys_addr: guest_addr,
        memory_size: mem_size as u64,
        userspace_addr: load_addr as u64,
        flags: 0,
    };
    unsafe { vm.set_user_memory_region(mem_region)? };

    // 5. Create vCPU 0 and configure initial segment & general registers
    let vcpu = vm.create_vcpu(0)?;

    let mut sregs = vcpu.get_sregs()?;
    sregs.cs.base = 0;
    sregs.cs.selector = 0;
    vcpu.set_sregs(&sregs)?;

    let mut regs = vcpu.get_regs()?;
    regs.rip = guest_addr;
    regs.rflags = 0x2; // Reserved bit 1 must always be set on x86
    vcpu.set_regs(&regs)?;

    // 6. Enter the KVM_RUN VM-Exit loop
    loop {
        match vcpu.run()? {
            VcpuExit::IoOut(port, data) => {
                if port == 0x3f8 {
                    print!("{}", String::from_utf8_lossy(data));
                }
            }
            VcpuExit::Hlt => {
                eprintln!("[vmm] Guest halted cleanly.");
                break;
            }
            other => {
                eprintln!("[vmm] Unexpected VM-Exit: {:?}", other);
                break;
            }
        }
    }

    unsafe {
        libc::munmap(load_addr, mem_size);
    }
    Ok(())
}

Run it:

cargo run --quiet

Output:

5
[vmm] Guest halted cleanly.

In under 90 lines of Rust, you just executed native x86 instructions inside VT-x Non-Root mode, intercepted two OUT instructions via VM-Exit, and cleanly handled HLT. Notice that kvm-ioctls and kvm-bindings are the exact same rust-vmm crates used inside Firecracker and Cloud Hypervisor.


3. Booting Linux⌗

Running 13 bytes of 16-bit real-mode code is a great sanity check, but booting a real 64-bit Linux kernel requires setting up the CPU and memory state that a firmware/bootloader (BIOS + GRUB) would normally prepare. MicroVMs skip BIOS, UEFI, and GRUB completely using Direct Kernel Boot.

Guest Physical Memory Layout (x86_64 Direct Boot)⌗

To drop straight into a 64-bit uncompressed ELF kernel (vmlinux), the VMM constructs a clean, deterministic layout in guest RAM before the very first KVM_RUN:

Guest Physical Address (GPA)
0x0000_0000 +---------------------------------------------------+
            | Low Memory / GDT (0x500) & IDT                    |
0x0000_7000 +---------------------------------------------------+
            | Zero Page (struct boot_params, 4 KiB)             |
            |  - e820 RAM map (usable RAM + reserved regions)   |
            |  - cmd_line_ptr -> 0x20000                        |
            |  - ramdisk_image / ramdisk_size (if initramfs)    |
0x0000_9000 +---------------------------------------------------+
            | Identity Page Tables (64-bit Long Mode)           |
            |  - 0x9000: PML4 (Page Map Level 4)                |
            |  - 0xa000: PDPT (Page Directory Pointer Table)    |
            |  - 0xb000: PD   (2 MiB huge pages, 0..1 GiB)      |
0x0002_0000 +---------------------------------------------------+
            | Kernel Command Line (null-terminated ASCII)       |
            | "console=ttyS0 reboot=k panic=1 pci=off ..."      |
0x0010_0000 +---------------------------------------------------+
            | Uncompressed Linux Kernel (vmlinux ELF PT_LOAD)   |
            |   Entry point -> RIP                              |
            +---------------------------------------------------+
            | ... Free Guest RAM ...                            |
            +---------------------------------------------------+
            | Optional initramfs (near top of RAM)              |
0x1000_0000 +---------------------------------------------------+ (256 MiB)
            |                                                   |
0xd000_0000 +---------------------------------------------------+
            | Unmapped MMIO Hole (virtio-mmio devices)          |
            |  - 0xd000_0000: virtio-blk                        |
            |  - 0xd000_1000: virtio-net                        |
            +---------------------------------------------------+

Putting the vCPU Straight into 64-Bit Long Mode⌗

Rather than letting Linux crawl up from 16-bit real mode to 32-bit protected mode to 64-bit long mode, the VMM configures KVM_SET_SREGS and KVM_SET_REGS directly:

  1. In-kernel IRQ chip & PIT: Call vm.create_irq_chip() and vm.create_pit2() so KVM emulates the LAPIC, IOAPIC, and legacy timer in the kernel (avoiding expensive user-space exits for timer ticks).
  2. CPUID: Call kvm.get_supported_cpuid() and pass it to vcpu.set_cpuid2() so the guest kernel knows what CPU features (SSE, AVX, x2APIC, TSC) exist.
  3. Identity-mapped 4-level paging: Write a minimal PML4 -> PDPT -> PD table at 0x9000 using 2 MiB huge pages (PRESENT | RW | PS) mapping GPA 0..1 GiB 1:1.
  4. Control registers (sregs):
    • CR3 = 0x9000 (PML4 base)
    • CR4 = CR4_PAE (Physical Address Extension)
    • EFER = EFER_LME | EFER_LMA (Long Mode Enable + Active)
    • CR0 = CR0_PE | CR0_MP | CR0_ET | CR0_NE | CR0_WP | CR0_AM | CR0_PG
    • CS: 64-bit code segment (l = 1, db = 0, present = 1, g = 1)
    • DS, ES, FS, GS, SS: flat 64-bit data segments
  5. General registers (regs):
    • RIP = ELF entry point of vmlinux (parsed from the ELF header, typically 0x0100_0000 or 0x0010_0000)
    • RSI = 0x7000 (pointer to boot_params, as required by the Linux x86 64-bit boot protocol)
    • RFLAGS = 0x2
  6. Minimal Serial Device (UART 16550A): Intercept VcpuExit::IoIn / VcpuExit::IoOut on ports 0x3f8..=0x3ff so earlyprintk=ttyS0 and console=ttyS0 print kernel logs straight to your terminal.

In practice, the rust-vmm ecosystem provides linux-loader to parse vmlinux ELF/bzImage/PVH headers and populate boot_params, and vm-superio to emulate the 16550A serial UART cleanly.

Building a Minimal vmlinux and Rootfs Declaratively on NixOS⌗

MicroVMs prefer an uncompressed vmlinux (skipping self-decompression shaves tens of milliseconds off boot time) stripped of desktop/hardware drivers (no USB, GPU, Bluetooth, or sound).

On NixOS, you can build a minimal Firecracker-ready kernel declaratively with pkgs.linuxKernel or fetch/customize one in a Nix expression:

# kernel.nix — Build a minimal uncompressed vmlinux for microVMs
{ pkgs ? import <nixpkgs> {} }:

let
  minimalKernel = pkgs.linux_6_6.override {
    structuredExtraConfig = with pkgs.lib.kernel; {
      # Core virtio & microVM essentials
      VIRTIO = yes;
      VIRTIO_MMIO = yes;
      VIRTIO_MMIO_CMDLINE_DEVICES = yes;
      VIRTIO_BLK = yes;
      VIRTIO_NET = yes;
      VIRTIO_VSOCKETS = yes;
      VIRTIO_BALLOON = yes;
      VIRTIO_CONSOLE = yes;
      SERIAL_8250 = yes;
      SERIAL_8250_CONSOLE = yes;
      EXT4_FS = yes;
      TMPFS = yes;
      DEVTMPFS = yes;
      DEVTMPFS_MOUNT = yes;

      # Trim bloat for fast boot
      MODULES = no;
      SOUND = no;
      DRM = no;
      USB_SUPPORT = no;
      WIRELESS = no;
      WLAN = no;
    };
    ignoreConfigErrors = true;
  };
in
pkgs.runCommand "microvm-vmlinux" {} ''
  mkdir -p $out
  # NixOS kernel derivations put uncompressed vmlinux in .dev/vmlinux
  cp ${minimalKernel.dev}/vmlinux $out/vmlinux
''

And for a tiny BusyBox ext4 rootfs (no Docker or debootstrap needed—pure Nix):

# rootfs.nix — Minimal ext4 rootfs with static BusyBox
{ pkgs ? import <nixpkgs> {} }:

let
  busybox = pkgs.pkgsStatic.busybox;
in
pkgs.runCommand "microvm-rootfs" {
  nativeBuildInputs = [ pkgs.e2fsprogs ];
} ''
  mkdir -p root/{bin,sbin,dev,proc,sys,etc,tmp,root}
  cp ${busybox}/bin/busybox root/bin/busybox
  for cmd in sh mount ls cat ip ping ps top poweroff reboot; do
    ln -s busybox root/bin/$cmd
  done

  cat > root/sbin/init <<'EOF'
#!/bin/sh
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t devtmpfs devtmpfs /dev
echo "=== Booted Nix-built MicroVM rootfs ==="
exec /bin/sh
EOF
  chmod +x root/sbin/init

  mkdir -p $out
  # Create a 32 MiB ext4 image populated directly from root/
  mke2fs -d root -t ext4 -L rootfs $out/rootfs.ext4 32M
''

Build both with nix-build kernel.nix and nix-build rootfs.nix. You now have a bit-for-bit reproducible vmlinux and rootfs.ext4 sitting in /nix/store.


4. virtio devices⌗

Emulating a real hardware device (like an Intel e1000 NIC or an AHCI SATA controller) is brutally slow: every register read/write causes a VM-Exit, and the guest driver was designed assuming hardware DMA latency, not hypervisor traps.

virtio (OASIS standard) solves this with paravirtualized I/O: the guest kernel knows it is inside a VM and communicates with the VMM over shared-memory ring buffers called virtqueues.

Split Virtqueue Architecture⌗

A split virtqueue lives entirely in guest RAM (which the VMM already has mmap’d!) and consists of three lock-free structures:

           GUEST KERNEL (virtio driver)        HOST VMM (virtio backend)
           ============================        =========================
                 |                                      ^
   1. Populate   |                                      | 3. Read descriptors
      buffers    v                                      |    & process I/O
      +------------------------------------------------------+
      |               Descriptor Table (Guest RAM)           |
      |  [0] addr=0x40000, len=16,   flags=NEXT,   next=1    | (req header)
      |  [1] addr=0x41000, len=4096, flags=WRITE|NEXT, next=2| (data buffer)
      |  [2] addr=0x42000, len=1,    flags=WRITE,  next=0    | (status byte)
      +------------------------------------------------------+
                 |                                      |
   2. Write head |                                      | 4. Write head &
      index (0)  v                                      v    bytes written
      +-----------------------+          +-----------------------+
      |   Available Ring      |          |      Used Ring        |
      |  (Driver -> Device)   |          |  (Device -> Driver)   |
      |  idx=1, ring[0]=0     |          |  idx=1, ring[0]=(0,1) |
      +-----------------------+          +-----------------------+
                 |                                      |
   2b. Doorbell  | (MMIO write -> ioeventfd)            | 5. Inject IRQ
       kick -----+------------------------------------->|    (irqfd)
                                                        v
                                              Guest interrupt handler
                                              wakes blocked process

Why is this so fast?

  1. Batching: The guest can enqueue dozens of disk reads or network packets into the Available Ring before ringing the doorbell once.
  2. Zero-copy: The VMM reads/writes the guest’s data buffers directly at their host virtual address (HVA = ram_base + gpa).
  3. Asynchronous ioeventfd + irqfd:
    • KVM_IOEVENTFD: The VMM registers a Linux eventfd for the virtqueue’s QueueNotify MMIO address. When the guest kicks the doorbell, KVM increments the eventfd in-kernel and immediately resumes the vCPU without exiting to user space—an epoll loop on a dedicated VMM I/O thread wakes up asynchronously.
    • KVM_IRQFD: When the VMM finishes the disk/network operation and updates the Used Ring, it writes to an irqfd, and KVM injects the virtual interrupt straight into the guest’s LAPIC/IOAPIC.

Transport: virtio-mmio vs virtio-pci⌗

Virtqueues are transport-agnostic. MicroVMs generally pick one of two transports:

Aspectvirtio-mmiovirtio-pci
Used byFirecrackerCloud Hypervisor, QEMU
DiscoveryPassed explicitly via kernel cmdline (virtio_mmio.device=4K@0xd0000000:5)Dynamic PCI bus enumeration (ECAM / BARs)
Attack surfaceMinimal (no PCI host bridge or config space emulation)Slightly larger (emulates PCI config space)
InterruptsLegacy GSI / IRQ linesMSI-X (per-virtqueue interrupts, better multi-core scaling)
Hotplug / VFIONot supportedSupported (hotplug NICs/disks, GPU/NIC passthrough)

Essential MicroVM virtio Devices⌗

  • virtio-blk: Exposes a host file, block device, or LVM volume as /dev/vda. Supports rate limiting (IOPS & bandwidth token buckets).
  • virtio-net: Pairs a virtqueue with a host Linux TAP device (tap0). Guest sends Ethernet frames -> VMM writes to TAP fd -> host kernel routes/NATs them.
  • virtio-vsock: Host-guest socket communication (AF_VSOCK, CID-based) with zero network stack configuration. Ideal for guest agents, metadata services, and SSH-over-vsock.
  • virtio-balloon: Dynamic memory reclamation—host asks guest to “inflate” the balloon (allocate & surrender unused pages via MADV_DONTNEED) so memory returns to the host OS.
  • virtio-fs: High-performance shared filesystem (FUSE over virtio + DAX) used heavily by Cloud Hypervisor and NixOS microVMs to share /nix/store read-only across dozens of VMs without duplicating gigabytes of packages.

5. Production VMMs: Firecracker and Cloud Hypervisor⌗

Both Firecracker (created by AWS for Lambda and Fargate) and Cloud Hypervisor (originally Intel, now Linux Foundation) are written in Rust and share core rust-vmm crates (kvm-ioctls, vm-memory, virtio-queue, linux-loader). Yet they optimize for distinct workloads.

Firecracker Architecture⌗

Firecracker is ruthlessly minimal: one process per microVM, zero BIOS, zero PCI, five emulated devices (virtio-net, virtio-blk, virtio-vsock, virtio-balloon, virtio-rng), a serial console, and a single keyboard reset controller.

+-----------------------------------------------------------------------+
|                       NixOS Host (Linux + KVM)                        |
|                                                                       |
|  +-----------------------------------------------------------------+  |
|  | Jailer Sandbox (chroot, PID/net/mount namespaces, cgroups v2,   |  |
|  |                 non-root UID/GID, strict seccomp-bpf filter)    |  |
|  |                                                                 |  |
|  |   +---------------------------------------------------------+   |  |
|  |   | Firecracker VMM Process                                 |   |  |
|  |   |                                                         |   |  |
|  |   |  [API Thread] <--- HTTP/1.1 over Unix Socket (.sock)    |   |  |
|  |   |  [vCPU 0..N ] <--- KVM_RUN threads                      |   |  |
|  |   |  [virtio-mmio]                                          |   |  |
|  |   |    |-- virtio-blk  ---> rootfs.ext4 (with rate limiter) |   |  |
|  |   |    |-- virtio-net  ---> Host TAP (tap0)                 |   |  |
|  |   |    +-- virtio-vsock --> Host Unix socket                |   |  |
|  |   +---------------------------------------------------------+   |  |
|  +-----------------------------------------------------------------+  |
+-----------------------------------------------------------------------+

Hands-on: Booting Firecracker on NixOS⌗

Inside your nix develop shell, start Firecracker listening on a Unix domain socket in Terminal 1:

rm -f /tmp/firecracker.sock
firecracker --api-sock /tmp/firecracker.sock

In Terminal 2, configure and boot the microVM via its REST API (using the vmlinux and rootfs.ext4 we built in Section 3):

SOCK="/tmp/firecracker.sock"
KERNEL="$(nix-build kernel.nix --no-out-link)/vmlinux"
ROOTFS="./rootfs.ext4"

# Copy rootfs out of read-only /nix/store so the VM can mount it rw
cp "$(nix-build rootfs.nix --no-out-link)/rootfs.ext4" "$ROOTFS"
chmod u+w "$ROOTFS"

# 1. Machine size: 1 vCPU, 128 MiB RAM
curl --unix-socket "$SOCK" -i -X PUT "http://localhost/machine-config" \
  -H "Content-Type: application/json" \
  -d '{
    "vcpu_count": 1,
    "mem_size_mib": 128,
    "smt": false
  }'

# 2. Direct Linux boot source
curl --unix-socket "$SOCK" -i -X PUT "http://localhost/boot-source" \
  -H "Content-Type: application/json" \
  -d "{
    \"kernel_image_path\": \"${KERNEL}\",
    \"boot_args\": \"console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw\"
  }"

# 3. Root block device (virtio-blk)
curl --unix-socket "$SOCK" -i -X PUT "http://localhost/drives/rootfs" \
  -H "Content-Type: application/json" \
  -d "{
    \"drive_id\": \"rootfs\",
    \"path_on_host\": \"${ROOTFS}\",
    \"is_root_device\": true,
    \"is_read_only\": false
  }"

# 4. Boot the microVM!
curl --unix-socket "$SOCK" -i -X PUT "http://localhost/actions" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "InstanceStart"}'

Switch back to Terminal 1—within ~120 milliseconds, you are dropped straight into the BusyBox shell inside a hardware-isolated Linux kernel.

Firecracker’s Secret Weapons: Jailer & Snapshots⌗

  1. Jailer: In production, you never run firecracker directly. You invoke jailer --id <vm-id> --exec-file $(which firecracker) --uid 1000 --gid 1000, which places the VMM inside a chroot, drops capabilities, isolates namespaces, applies cgroups v2 limits, and lets Firecracker install a seccomp-bpf allowlist that permits only ~35 syscalls. Even if a guest escaped KVM into the VMM process, the process is trapped in an empty jail.
  2. Snapshots & MicroVM Cloning: Firecracker can pause a warmed-up VM (PATCH /vm {"state": "Paused"}) and dump its CPU/device state (vmstate) and guest RAM (mem_file) to disk (PUT /snapshot/create). Restoring from a snapshot (PUT /snapshot/load) uses mmap (or userfaultfd for lazy page-in), resuming a fully booted runtime (Python/Node/JVM already initialized!) in under 10 milliseconds.

Cloud Hypervisor⌗

While Firecracker refuses to add PCI, hotplug, or GPU passthrough to keep its attack surface tiny, Cloud Hypervisor targets modern, general-purpose cloud workloads (like Kata Containers, Kubernetes nodes, and NixOS service VMs) while still avoiding QEMU’s legacy baggage.

Key capabilities Cloud Hypervisor adds over Firecracker:

  • virtio-pci with MSI-X: Scales cleanly to dozens of vCPUs and high-throughput NICs.
  • virtio-fs: Share host directories (especially /nix/store!) with near-native speed.
  • vCPU, Memory, Disk, and NIC Hotplug: Resize running microVMs dynamically.
  • VFIO Device Passthrough: Pass physical GPUs or SR-IOV NICs directly into a microVM.
  • Confidential Computing: Intel TDX and AMD SEV-SNP support.

Booting our same Nix-built kernel and rootfs with Cloud Hypervisor in nix develop:

cloud-hypervisor \
  --kernel "$(nix-build kernel.nix --no-out-link)/vmlinux" \
  --cmdline "console=ttyS0 root=/dev/vda rw" \
  --cpus boot=2 \
  --memory size=256M \
  --disk path=./rootfs.ext4 \
  --serial tty \
  --console off

Firecracker vs Cloud Hypervisor vs QEMU⌗

DimensionFirecrackerCloud HypervisorQEMU
LanguageRust (rust-vmm)Rust (rust-vmm)C
Primary Use CaseMulti-tenant serverless, FaaS, sandboxesCloud-native VMs, Kata, NixOS servicesGeneral-purpose, legacy OSes, Windows, desktops
Boot Time~100–125 ms (<10 ms from snapshot)~150–300 ms~500 ms–2 s+
Memory Overhead< 5 MiB / VM~10–15 MiB / VM~30–100+ MiB / VM
Device Transportvirtio-mmio onlyvirtio-pci (MSI-X)PCI, ISA, USB, virtio, legacy
Shared Host FSNo (block / vsock only)Yes (virtio-fs / DAX)Yes (virtio-fs, 9p)
Hotplug & VFIONoYes (vCPU, RAM, PCI, VFIO)Yes
Snapshot / RestoreFirst-class (with UFFD)SupportedSupported (live migration)
Security IsolationBuilt-in jailer + strict seccompSeccomp + external sandboxExternal (libvirt / AppArmor / SELinux)

The NixOS Superpower: Declarative MicroVMs with microvm.nix⌗

Manually scripting curl --unix-socket calls and copying rootfs.ext4 files is great for learning, but on NixOS we can do something far better: declare entire NixOS microVMs in a flake using microvm.nix.

The killer feature of NixOS microVMs is sharing the host’s /nix/store read-only (via virtio-fs on Cloud Hypervisor, or a read-only squashfs/erofs image on Firecracker). Because every binary, library, and systemd unit in NixOS is immutable and content-addressed in /nix/store, a full NixOS microVM requires zero disk duplication for its OS packages and boots in under a second.

+-------------------------------------------------------------------+
|                         NixOS Host                                |
|                                                                   |
|   /nix/store  (Immutable, shared read-only across all microVMs)   |
|       |                                                           |
|       +--- virtiofs / squashfs ---> [ MicroVM 1: web-api   ]      |
|       +--- virtiofs / squashfs ---> [ MicroVM 2: postgres  ]      |
|       +--- virtiofs / squashfs ---> [ MicroVM 3: ci-runner ]      |
|                                                                   |
|   Each VM gets: own Linux kernel, own KVM isolation, tiny rw      |
|   overlay/var volume, and dedicated TAP interface!                |
+-------------------------------------------------------------------+

Here is a complete flake.nix that defines a self-contained NixOS microVM. Notice how you can switch between firecracker and cloud-hypervisor with a single line:

# flake.nix — Declarative NixOS MicroVM
{
  description = "Declarative NixOS MicroVM with Firecracker / Cloud Hypervisor";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    microvm = {
      url = "github:astro/microvm.nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, microvm }:
    let
      system = "x86_64-linux";
    in {
      nixosConfigurations.demo-vm = nixpkgs.lib.nixosSystem {
        inherit system;
        modules = [
          microvm.nixosModules.microvm
          ({ pkgs, ... }: {
            networking.hostName = "demo-vm";
            users.users.root.password = ""; # Dev only! Use SSH keys in prod

            # Switch between "firecracker" and "cloud-hypervisor" here!
            microvm = {
              hypervisor = "firecracker";
              vcpu = 2;
              mem = 512;

              # Share /nix/store read-only (erofs/squashfs for Firecracker;
              # for Cloud Hypervisor you can also use proto = "virtiofs")
              shares = [
                {
                  source = "/nix/store";
                  mountPoint = "/nix/.ro-store";
                  tag = "ro-store";
                  proto = "virtiofs"; # Automatically uses erofs on Firecracker
                }
              ];

              # Virtual network interface backed by a host TAP device
              interfaces = [
                {
                  type = "tap";
                  id = "vm-demo0";
                  mac = "02:00:00:00:00:01";
                }
              ];

              # Optional persistent writable volume for /var
              volumes = [
                {
                  mountPoint = "/var";
                  image = "var.img";
                  size = 256; # MiB
                }
              ];
            };

            services.openssh.enable = true;
            environment.systemPackages = [ pkgs.htop pkgs.curl ];
            system.stateVersion = "24.11";
          })
        ];
      };

      packages.${system}.default =
        self.nixosConfigurations.demo-vm.config.microvm.declaredRunner;
    };
}

Build and launch the microVM directly from the flake:

nix run .#

Or integrate it directly into your NixOS host’s configuration.nix via microvm.nixosModules.host so systemd manages each microVM declaratively (systemctl start microvm@demo-vm) alongside your host bridges and firewall rules.


Wrapping Up⌗

When you strip away the marketing, a microVM is a remarkably clean stack of five layers:

  1. Silicon (VT-x / VMCS / EPT) lets an unmodified guest kernel run natively in Non-Root Ring 0 while hardware traps sensitive operations and translates guest physical memory in two dimensions.
  2. KVM (/dev/kvm) turns those hardware primitives into a file-descriptor ioctl API controllable from user space.
  3. Direct Linux Boot bypasses 40 years of PC firmware legacy by loading vmlinux, boot_params, and 64-bit page tables straight into mmap’d guest RAM.
  4. virtio replaces chatty hardware emulation with lock-free shared-memory ring buffers, ioeventfd, and irqfd.
  5. Firecracker & Cloud Hypervisor (paired with NixOS and microvm.nix) package those exact primitives into fast, auditable, reproducible, hardware-isolated workloads.

If you want to go deeper, the best next step is reading the source of rust-vmm/vmm-reference alongside the Firecracker design doc—everything in both repositories maps 1-to-1 to the five layers above.