Chapter 22: Containers, Virtualization and Noisy Neighbors

Updated

October 31, 2026

Chapter 22: Containers, Virtualization and Noisy Neighbors

In modern infrastructure, workloads rarely run directly on dedicated bare metal. Instead, they execute inside Docker/OCI containers or Virtual Machines (VMs) managed by hypervisors like KVM. While multi-tenancy maximizes hardware utilization, it introduces isolation overhead and exposes workloads to the “noisy neighbor” problem—where an adjacent tenant monopolizes shared caches, interconnect bandwidth, or disk I/O queues.

This chapter quantifies container versus virtualization overhead, analyzes noisy neighbor mechanisms, and establishes isolation best practices.

Container Performance: Isolation and Overhead

Containers are not virtual machines; they are standard Linux processes executed inside kernel Namespaces and governed by cgroups:

  • Compute & Memory Overhead: Effectively zero. Container processes execute machine instructions directly on physical CPU cores with native hardware MMU translation. There is no hypervisor trap, no instruction emulation, and no guest kernel layer.
  • Networking Overhead: If containers use traditional veth pairs and Linux bridge devices, packets traverse the kernel network stack twice, introducing 10%–20% throughput degradation and microsecond latency penalties.
  • Filesystem Overhead: Overlay filesystems (overlay2) add path resolution and inode lookups across upper and lower directory layers, degrading heavy directory traversal operations.
┌────────────────────────────────────────────────────────────────────────┐
│                   CONTAINER vs VIRTUAL MACHINE OVERHEAD                │
│                                                                        │
│   OCI Container                                Virtual Machine (KVM)   │
│   ┌─────────────────────────────┐              ┌─────────────────────┐ │
│   │ Application Process         │              │ Application Process │ │
│   ├─────────────────────────────┤              ├─────────────────────┤ │
│   │ Namespaces & cgroups v2     │              │ Guest OS & Kernel   │ │
│   ├─────────────────────────────┤              ├─────────────────────┤ │
│   │ Host Linux Kernel           │              │ QEMU Device Emul    │ │
│   ├─────────────────────────────┤              ├─────────────────────┤ │
│   │ Bare-Metal Hardware         │              │ KVM (VMX/EPT Trap)  │ │
│   └─────────────────────────────┘              ├─────────────────────┤ │
│                                                │ Host Linux Kernel   │ │
│   CPU Tax: ~0%                                 ├─────────────────────┤ │
│   Memory Tax: ~0%                              │ Bare-Metal Hardware │ │
│   I/O Tax: 1-5% (overlayfs/veth)               └─────────────────────┘ │
│                                                CPU Tax: 2-5%           │
│                                                Memory Tax: 3-8% (EPT)  │
│                                                I/O Tax: 5-20% (virtio) │
└────────────────────────────────────────────────────────────────────────┘

cgroups and Namespaces in Containers

  • Namespaces provide illusion of isolation (PID, NET, MNT, IPC, UTS, USER).
  • cgroups enforce resource quotas.
  • A critical failure in containerized environments is runtime awareness of cgroup limits. Older versions of runtimes (e.g., Java 8 prior to update 191) read /proc/cpuinfo and /proc/meminfo instead of cgroup limits, detecting all 128 host cores and 512GB host RAM, leading to massive thread over-allocation and instant OOM termination.

Virtualization Overhead: Hypervisors and Emulation

Virtual machines run a complete guest kernel using hardware virtualization extensions (Intel VT-x / AMD-V):

  1. VM Exits (vmexit): When the guest kernel executes privileged instructions (modifying CR3, accessing I/O ports, halting), the hardware triggers an exit to the host KVM hypervisor. Each vmexit costs 500ns to 2\(\mu\)s.
  2. Two-Dimensional Paging (EPT / NPT): Memory accesses require walking two sets of page tables: Guest Virtual \(\to\) Guest Physical, then Guest Physical \(\to\) Host Physical. A TLB miss can require up to 24 DRAM memory references, increasing memory latency by 10%–20%.
  3. vCPU Steal Time (%steal): The percentage of time a guest vCPU was ready to run but the host hypervisor was busy scheduling another guest or host task.
# Monitor CPU steal time in VM (st column)
sar -u 1 5

Detecting and Mitigating Noisy Neighbors

A noisy neighbor degrades performance without exceeding their own CPU or memory quotas by saturating unmetered shared hardware resources:

  • Last-Level Cache (LLC) Thrashing: A streaming batch process constantly evicts latency-sensitive database cache lines from the shared L3 cache.
  • Memory Bus Saturation: High memory throughput saturates memory controller channels, delaying memory access for adjacent cores.
  • Interconnect Saturation: Heavy cross-NUMA traffic chokes inter-socket UPI buses.

Hardware-Level Mitigation: Intel RDT and AMD MBA

Modern enterprise CPUs support Intel Resource Director Technology (RDT) and AMD Memory Bandwidth Allocation (MBA): - Cache Allocation Technology (CAT): Partitions the shared L3 cache into hardware bitmasks, reserving dedicated L3 cache slices for critical VMs or containers. - Memory Bandwidth Allocation (MBA): Enforces memory controller bandwidth throttling on noisy batch workloads.

# Manage Intel RDT allocations via resctrl filesystem
sudo mount -t resctrl resctrl /sys/fs/resctrl

Cloud vs Bare Metal: Performance Implications

Factor Cloud Multi-Tenant VM Dedicated Bare-Metal Server
Tail Latency (p99/p99.9) High jitter due to hypervisor steal, shared LLC, and virtualized network interrupts. Deterministic microsecond execution; zero hypervisor exits.
I/O Latency Floor ~500\(\mu\)s–1ms (virtualized network block storage EBS/PD). ~10\(\mu\)s–25\(\mu\)s (direct PCIe Gen5 NVMe).
Hardware Control No control over NUMA topology, CPU stepping, or firmware power states. Full control over CPU governors, isolcpus, huge pages, and BIOS tuning.

Best Practices for Containerized Workloads

  1. Size Thread Pools to CPU Quota, Not Host Cores: Set Go GOMAXPROCS or JVM active processor count to match cpu.max quota to avoid catastrophic thread contention and CFS throttling.
  2. Avoid Host Network Sharing Unless Required: Use host networking (--net=host) for ultra-high-throughput networking services to bypass veth bridges.
  3. Use Dedicated Host Storage Volumes for I/O: Mount host block devices or fast XFS directories directly into containers via bind mounts, bypassing overlay2 storage drivers for database and logging paths.
  4. Enforce Memory Limits with Sufficient Headroom: Set memory.high to trigger early graceful garbage collection before memory.max invokes the destructive OOM killer.