Chapter 10: Observability and Operations

Updated

October 31, 2026

Chapter 10: Observability and Operations

Operating multiple independent kernels on one physical server complicates telemetry, logging, crash diagnostics, and performance profiling. Because guest kernels execute in their own isolated address spaces, standard host monitoring tools cannot directly inspect their processes, memory allocations, or internal kernel states. This chapter establishes an end-to-end observability and telemetry architecture for multi-kernel environments.

Metrics and Monitoring Architecture

A robust telemetry pipeline combines hardware, host, and guest metric layers:

┌─────────────────────────────────────────────────────────────┐
│                 CENTRAL OBSERVABILITY PLATFORM              │
│                 (Prometheus / Grafana / OTel)               │
└──────────────────────────────┬──────────────────────────────┘
                               ▲
         Prometheus Remote Write (Encrypted Link)
                               │
┌──────────────────────────────┴──────────────────────────────┐
│                    PHYSICAL BARE-METAL HOST                 │
│                                                             │
│   HOST MANAGEMENT DOMAIN          SECONDARY WORKER KERNEL   │
│   ┌───────────────────────────┐   ┌───────────────────────┐ │
│   │ Node Exporter             │   │ Node Exporter         │ │
│   │ IPMI / BMC Exporter       │   │ Workload Exporter     │ │
│   │ perf kvm monitoring       │   │ In-guest telemetry    │ │
│   └─────────────┬─────────────┘   └───────────┬───────────┘ │
│                 │                             │             │
│                 └────── Out-of-Band / Vsock ──┘             │
└─────────────────────────────────────────────────────────────┘
  1. Host-Level Metrics: IPMI hardware sensors (chassis temperatures, fan speeds, power draw per socket), host CPU frequencies, and KVM VM-exit rates.
  2. Guest-Level Metrics: In-guest node_exporter or OpenTelemetry collectors emitting OS counters (load averages, memory fragmentation, disk queue depths) over internal network interfaces or dedicated vsock sockets.
  3. Hardware PMU Counters: Monitoring Performance Monitoring Unit (PMU) counters across sockets to detect memory bus saturation and cache line contention.

Logging Strategy Across Kernel Instances

Centralizing log streams without creating network loops or cross-kernel dependency stalls:

  • Isolated Journald Forwarding: Secondary kernels forward systemd journal records to the host using systemd-journal-remote over the dedicated internal link (br-internal or vsock).
  • Serial Console Multiplexing: QEMU instances output serial consoles to local Unix domain sockets or ptys (-serial chardev:char0), allowing the host system to capture early boot messages and panic logs even when network stacks are uninitialized.
  • Out-of-Band IPMI Serial-over-LAN (SoL): Captures host kernel output in the event of complete physical node lockups.

Crash Diagnostics and kdump Integration

Managing crashes across independent kernel instances:

  • Guest Kernel Panics: Configure secondary kernels with independent kdump configurations. When a secondary kernel panics, it triggers a warm jump to its own local crash kernel, writing the dump to its dedicated NVMe or network storage volume without affecting the host.
  • Host Panics with Running Guests: If the host kernel panics, it executes host-level kdump. Host kdump scripts should attempt to send graceful ACPI shutdown signals or freeze guest vCPUs before capturing the host memory image.

Performance Profiling Methodology

Investigating latency spikes and overhead in multi-kernel systems:

  1. perf kvm: Profile guest and host cycles simultaneously:

    # Record host and guest cycles with symbols
sudo perf kvm --host --guest record -a -F 99 sleep 10

# Analyze vm-exit causes and execution hotspots
sudo perf kvm stat report

  2. eBPF Tracing: Trace hardware interrupt delays and vCPU wake-up latencies using BCC and bpftrace scripts targeting kvm:kvm_exit and sched:sched_switch.

  3. Hardware PMU Profiling: Track Last-Level Cache (LLC) misses with perf stat -e LLC-load-misses,LLC-store-misses.

Health Checks and Alerting Design

Automated failure detection and recovery policies:

  • Hardware Watchdogs: Modern motherboards feature IPMI hardware watchdogs. If the host kernel hangs, the watchdog resets the chassis after a configurable timeout (e.g., 60 seconds).
  • Software Watchdogs (virtio-watchdog): Guests configure virtual watchdog devices. If a secondary kernel locks up, QEMU receives an event (WATCHDOG_ACTION_RESET) and automatically reboots the secondary kernel instance.
  • Heartbeat Daemon over Vsock: A lightweight watchdog daemon continuously exchanges heartbeats between host and guest. Missing three consecutive heartbeats initiates automated remediation.