Chapter 7: Networking Between Kernel Instances

Updated

October 31, 2026

Chapter 7: Networking Between Kernel Instances

When multiple operating system kernels execute concurrently on a single bare-metal chassis, they rarely operate in total isolation. Control planes, storage synchronization, telemetry aggregation, and distributed service communication demand high-bandwidth, ultra-low-latency inter-kernel connectivity. This chapter covers internal virtual network architectures, external network topologies, performance optimization, and memory-mapped inter-kernel communication.

Internal Networking Architectures

Connecting co-located kernel instances can be achieved through multiple internal mechanisms:

  1. Linux Bridge with Tap Interfaces:
    • Standard, simple bridge (br0) bridging host tap devices.
    • Low complexity, but packets traverse host software bridge code, incurring latency overhead (~15–25µs).
  2. Macvtap (Direct Bridge Mode):
    • Replaces the software bridge by attaching guest tap endpoints directly to a physical host network interface.
    • In bridge mode, co-located endpoints communicate directly within the network driver stack without external switch reflection.
  3. Open vSwitch (OVS) with DPDK Datapath:
    • Kernel-bypass software switching using polling threads.
    • Ideal for complex VLAN, VXLAN, or flow-table routing between multiple instances, but consumes dedicated CPU polling cores.

External Network Configuration

For production multi-kernel servers connected to data center fabrics:

  • Dedicated Physical Uplinks: Server Motherboards with multiple 25G/100G interfaces dedicate physical Port 0 to the primary management kernel and Port 1 to the secondary worker kernel.

  • VLAN Trunking over SR-IOV: A single physical 100G uplink carries 802.1Q tagged traffic. The host administrator configures the physical function (PF) to assign VLAN tags automatically to specific Virtual Functions (VFs):

    sudo ip link set enp4s0f0 vf 0 vlan 100 qos 5
sudo ip link set enp4s0f0 vf 1 vlan 200 qos 3

    The guest kernels receive clean, untagged traffic natively isolated at the switch/NIC silicon layer.

Network Performance Optimization

Where virtual devices (virtio-net) must be used instead of SR-IOV:

  • vhost-net Acceleration: Move packet processing from QEMU userspace directly into a dedicated kernel thread (vhost-<pid>).
  • Multi-Queue Virtio: Match the number of virtio transmit/receive queues to the number of allocated vCPUs (queues=16), distributing packet processing across all guest cores via Receive Side Scaling (RSS).
  • Offload Configuration: Enable TCP Segmentation Offload (TSO), Generic Segmentation Offload (GSO), and Generic Receive Offload (GRO) to maximize single-stream throughput up to 40Gbps+.

Inter-Kernel Communication Patterns

For microsecond-scale inter-kernel coordination:

  • Inter-VM Shared Memory (IVSHMEM): Exposes a shared POSIX shared memory region (/dev/shm) directly as a PCI MMIO device inside secondary kernels. Instances read and write to the same physical RAM pages without any TCP/IP or network stack overhead (<1µs latency).

  • Virtio-VSOCK: Zero-configuration socket family (AF_VSOCK) enabling point-to-point stream and datagram communication between host and guest kernels:

    struct sockaddr_vm sa = {
    .svm_family = AF_VSOCK,
    .svm_cid = VMADDR_CID_HOST,
    .svm_port = 1234
};
connect(fd, (struct sockaddr*)&sa, sizeof(sa));

    Bypasses IP routing, firewall tables, and packet serialization.

Complete Networking Configuration Example

The following script configures an isolated low-latency internal network bridge and a secondary high-speed vsock channel:

#!/usr/bin/env bash
set -euo pipefail

# 1. Create isolated internal bridge
sudo ip link add br-internal type bridge
sudo ip addr add 192.168.100.1/24 dev br-internal
sudo ip link set br-internal up

# 2. Spawn tap device with multi-queue support
sudo ip tuntap add dev tap0 mode tap vnet_hdr multi_queue
sudo ip link set tap0 master br-internal up

# 3. Launch secondary kernel with vhost-net and vsock
qemu-system-x86_64 \
  -enable-kvm \
  -smp 8 \
  -m 16G \
  -netdev tap,id=net0,ifname=tap0,script=no,downscript=no,vhost=on,queues=8 \
  -device virtio-net-pci,netdev=net0,mq=on,vectors=18 \
  -device vhost-vsock-pci,id=vhost-vsock-pci0,guest-cid=3 \
  -kernel /opt/kernels/vmlinuz-worker \
  -append "console=ttyS0 ip=192.168.100.2::192.168.100.1:255.255.255.0::eth0:none"

This configuration yields full internal network routing alongside high-speed programmatic vsock channels.