Chapter 6: Device Assignment and I/O Isolation

Updated

October 31, 2026

Chapter 6: Device Assignment and I/O Isolation

Emulating virtual I/O devices through software models introduces CPU traps, memory copies, and latency jitter. To achieve true bare-metal throughput and deterministic latency across concurrent kernels, physical peripheral devices must be decoupled from the host and assigned directly to secondary kernel instances. This chapter covers the mechanics of VFIO (Virtual Function I/O), IOMMU group validation, SR-IOV network slicing, storage controller dedication, and accelerator assignment.

PCI Passthrough (VFIO) Fundamentals

The Virtual Function I/O (VFIO) framework is the standard Linux subsystem for exposing direct device access to userspace drivers and virtual machine instances in a secure, memory-protected manner:

  • IOMMU Protection: VFIO relies on hardware IOMMU (Intel VT-d or AMD-Vi) units to program DMA remapping tables. A device passed to a guest kernel can only perform DMA transfers into physical memory pages explicitly mapped to that guest.
  • IOMMU Groups: The hardware enforces isolation at the granularity of IOMMU groups. An IOMMU group represents the smallest set of physical devices that cannot be differentiated from one another by the PCIe root complex (often due to PCIe bridges lacking Access Control Services - ACS).
  • Rule of VFIO Assignment: Every device in an IOMMU group must either be assigned to the same guest or bound to the vfio-pci stub driver on the host.

To inspect IOMMU groups:

for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done

SR-IOV and Virtual Functions

Single Root I/O Virtualization (SR-IOV) is a PCIe standard that allows a single physical PCIe endpoint to partition itself into multiple distinct hardware interfaces:

  • Physical Function (PF): The primary PCIe device containing full configuration space, configuration registers, and administrative control.
  • Virtual Functions (VF): Lightweight PCIe functions carved out of the PF, each with its own independent configuration space, Base Address Registers (BARs), and transmit/receive queue pairs.
┌────────────────────────────────────────────────────────┐
│             PHYSICAL NIC (SR-IOV ENDPOINT)             │
│                                                        │
│   ┌───────────────────┐    ┌──────────────────────┐    │
│   │ Physical Function │    │ Virtual Functions    │    │
│   │ (PF: enp4s0f0)    │    │ (VF 0, VF 1, ... VF n│    │
│   └─────────┬─────────┘    └──────────┬───────────┘    │
└─────────────┼─────────────────────────┼────────────────┘
              │                         │
     Host Driver (mlx5_core)     Assigned to Kernel 2 (VFIO)

Enabling VFs on an enterprise NIC (e.g., Mellanox ConnectX-6 or Intel E810):

# Provision 4 Virtual Functions on physical interface enp4s0f0
echo 4 | sudo tee /sys/class/net/enp4s0f0/device/sriov_numvfs

# Verify new PCIe addresses
lspci | grep -i "Virtual Function"

Network Interface Assignment

Once VFs are spawned, bind a target VF to vfio-pci:

# 1. Unbind VF from default kernel driver
echo "0000:04:00.1" | sudo tee /sys/bus/pci/devices/0000:04:00.1/driver/unbind

# 2. Bind to vfio-pci
echo "15b3 101e" | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:04:00.1" | sudo tee /sys/bus/pci/drivers/vfio-pci/bind

The secondary kernel now owns the virtual network interface directly. Packets bypass the host operating system entirely, moving directly between physical wire and the secondary kernel’s memory rings with sub-microsecond latency.

Storage Device Partitioning

Multi-kernel storage assignment balances capacity and I/O isolation:

  1. Direct NVMe Controller Passthrough: Pass an entire physical PCIe NVMe M.2 or U.2 drive to the secondary kernel via VFIO. The guest kernel loads its native NVMe driver, programs submission/completion queues directly, and achieves full drive IOPS (e.g., 1.5M IOPS).
  2. NVMe Namespace Delegation: Enterprise NVMe drives support hardware namespaces. One controller exposes multiple distinct namespaces (/dev/nvme0n1, /dev/nvme0n2). Individual namespaces can be formatted and mapped to secondary kernels via virtio-blk or raw block passthrough.
  3. Raw Block Delegation (virtio-scsi / virtio-blk): Pass a dedicated physical disk partition directly with cache disabled (cache=none,aio=native), bypassing the host page cache.

Specialized Hardware: GPUs, FPGAs, Accelerators

Modern AI and low-latency financial systems dedicate accelerators directly to specific kernels:

  • GPU Passthrough: Isolate NVIDIA/AMD PCIe GPUs and their corresponding HDMI/audio endpoints into VFIO, preventing host nouveau or amdgpu drivers from claiming them.
  • PCIe Bifurcation: Configure BIOS bifurcation settings (e.g., splitting a 16x PCIe slot into 4x/4x/4x/4x) to route physical lanes to independent riser cards.
  • Interrupt Routing (MSI-X): Ensure physical interrupt lines are delivered directly to the target CPU cores via hardware interrupt remapping without host intervention.