systemd-nspawn Test Backend

Updated

October 31, 2026

systemd-nspawn Test Backend

NixOS 26.05 “Yarara” added systemd-nspawn as an alternative backend to QEMU for the integration test driver (nixpkgs #470248, #478109, #479968). Most service and network tests do not need a full guest kernel. The boring default is: declare those machines under containers instead of nodes, keep the same testScript API, configure the host Nix daemon for uid-range / auto-allocate-uids / cgroups once, and reserve QEMU (nodes) for kernel, graphics, specialisations, SUID, and restrictive systemd hardening.

Containers share the host kernel, boot faster, use less memory, run on nested VM builders without KVM, and can bind-mount host devices (GPU/CUDA). That is a different isolation story than QEMU — not a drop-in replacement for every test.

Mental model

                    NixOS integration test driver
                              │
              ┌───────────────┴───────────────┐
              ▼                               ▼
         nodes = {…}                    containers = {…}
         QEMU / KVM                     systemd-nspawn
         own guest kernel               host kernel + namespaces
         needs kvm feature              needs uid-range + auto-allocate-uids
              │                               │
              └──────── testScript ───────────┘
                   (same Python API)

Piece	Role
`nodes`	QEMU VMs — full hardware model, guest kernel
`containers`	systemd-nspawn — fast, light, host kernel
`testScript`	Unchanged Python driver API (`start_all`, `.succeed`, …)
`nix.settings` on the builder host	Daemon knobs so nspawn can allocate UIDs and cgroups
`/dev/net` in the sandbox	Only when containers and VMs share a VLAN

Flip the declarative attrset. Do not rewrite the script. Prefer containers for “does this unit listen / can these hosts ping / does NAT work.” Prefer QEMU when the claim under test is about the kernel, a display, a specialisation, a setuid wrapper, or hardening options the sandbox will not honor inside nspawn.

Worked examples

Case 1: Host daemon settings for container tests

On the builder (desk laptop, CI runner, or remote Nix machine that will evaluate and run the test), enable the features the nspawn backend requires. Save as nspawn-builder.nix and import it on that host:

# nspawn-builder.nix
{
  # Required so systemd-nspawn tests can allocate UIDs and manage cgroups.
  nix.settings = {
    auto-allocate-uids = true;
    experimental-features = [ "auto-allocate-uids" "cgroups" ];
    extra-system-features = [ "uid-range" ];

    # Only when a single test mixes containers + VMs that share a VLAN:
    # extra-sandbox-paths = [ "/dev/net" ];
  };
}

Apply and confirm the daemon sees the features (exact list may include other features you already advertise):

sudo nixos-rebuild switch --flake .#builder
nix show-config | rg -e 'auto-allocate-uids|system-features|experimental-features|sandbox-paths'

Representative snippets:

auto-allocate-uids = true
experimental-features = auto-allocate-uids cgroups nix-command flakes
system-features = … uid-range …

Without uid-range / auto-allocate-uids / cgroups, container tests fail at build time or when nspawn tries to start — not with a friendly “flip this option” banner. Fix the host once; every containers = { … } test on that builder benefits.

Case 2: Minimal two-machine ping with `containers`

Migrate the classic two-host ping by renaming nodes → containers. The testScript stays the same:

# ping-containers.nix
{ pkgs ? import <nixpkgs> { } }:

pkgs.testers.runNixOSTest {
  name = "ping-containers";

  # systemd-nspawn backend — not QEMU
  containers = {
    machine1 = { };
    machine2 = { };
  };

  testScript = ''
    start_all()

    machine1.succeed("ping -c 1 machine2")
    machine2.succeed("ping -c 1 machine1")
  '';
}

Build it (host must already have Case 1 settings):

nix-build ping-containers.nix -L

Representative success shape (timings vary; container boots are typically a few seconds for a trivial ping):

machine1 # ping -c 1 machine2
… 1 packets transmitted, 1 received …
machine2 # ping -c 1 machine1
… 1 packets transmitted, 1 received …
/nix/store/…-vm-test-run-ping-containers

Or as a flake check against the 26.05 pin:

# flake.nix fragment
{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

  outputs = { self, nixpkgs }: {
    checks.x86_64-linux.ping-containers =
      nixpkgs.legacyPackages.x86_64-linux.testers.runNixOSTest {
        name = "ping-containers";
        containers = {
          machine1 = { };
          machine2 = { };
        };
        testScript = ''
          start_all()
          machine1.succeed("ping -c 1 machine2")
          machine2.succeed("ping -c 1 machine1")
        '';
      };
  };
}

nix flake check -L
# or:
nix build .#checks.x86_64-linux.ping-containers -L

No /dev/kvm required. Nested builders that cannot offer KVM can still run this check.

Case 3: Keep QEMU `nodes` when the claim needs a real VM

Containers are wrong when the test asserts something about a guest kernel, a graphical session, a specialisation, setuid in the sandbox, or systemd hardening the sandbox strips. Keep (or restore) nodes:

# kernel-module.nix — must stay on QEMU
{ pkgs ? import <nixpkgs> { } }:

pkgs.testers.runNixOSTest {
  name = "desk-kernel-module";

  nodes.machine = { pkgs, ... }: {
    boot.kernelModules = [ "dummy" ];
    # … load / unload a module, assert lsmod, etc.
  };

  testScript = ''
    machine.wait_for_unit("multi-user.target")
    machine.succeed("lsmod | grep -q '^dummy'")
  '';
}

Prefer `containers`	Prefer `nodes` (QEMU)
Unit start / listen / curl	Kernel modules, custom kernels
Multi-host IP / NAT / DNS	Graphical apps / screenshots
Cheap CI without KVM	Specialisations
GPU bind-mount experiments	SUID wrappers under the Nix sandbox
Fast feedback loops	Restrictive `ProtectSystem=`-style hardening many modules enable

# Still needs kvm on the builder:
nix-build kernel-module.nix -L

If a PR “migrates everything to containers” and green-washes a kernel or GUI claim, the suite got faster by testing less. Split the suite: container jobs for services, QEMU jobs for the narrow kernel/graphics set.

Case 4: Mixed containers + VMs sharing a VLAN need `/dev/net`

One test can declare both attrsets. Shared VLAN bridging needs /dev/net inside the sandbox:

# mixed-vlan.nix
{ pkgs ? import <nixpkgs> { } }:

pkgs.testers.runNixOSTest {
  name = "mixed-vlan";

  containers.edge = { ... }: {
    virtualisation.vlans = [ 1 ];
  };

  nodes.core = { ... }: {
    virtualisation.vlans = [ 1 ];
  };

  testScript = ''
    start_all()
    edge.wait_for_unit("network.target")
    core.wait_for_unit("network.target")
    edge.succeed("ping -c 1 core")
  '';
}

On the builder, uncomment the sandbox path from Case 1:

# nspawn-builder-mixed.nix
{
  nix.settings = {
    auto-allocate-uids = true;
    experimental-features = [ "auto-allocate-uids" "cgroups" ];
    extra-system-features = [ "uid-range" ];
    # Bridge nspawn ↔ QEMU over TAP
    extra-sandbox-paths = [ "/dev/net" ];
  };
}

One-shot without editing the host config:

nix-build mixed-vlan.nix -L --option extra-sandbox-paths /dev/net

Representative failure without the path: the driver cannot set up the TAP bridge between backends. Fix the sandbox, not the testScript. Pure-container tests (Case 2) do not need /dev/net.

Case 5: Bind-mount a host device (GPU sketch) — still not a kernel test

Containers can pass host device nodes with nspawn options when those paths are visible in the sandbox. Sketch only — adjust device paths and sandbox mounts for your desk:

# cuda-sketch.nix
{ pkgs ? import <nixpkgs> { } }:

pkgs.testers.runNixOSTest {
  name = "cuda-sketch";

  containers.worker = { ... }: {
    virtualisation.systemd-nspawn.options = [
      "--bind=/dev/nvidia0"
      "--bind=/dev/nvidiactl"
    ];
    # environment.systemPackages = [ pkgs.cudaPackages.cudatoolkit ];
  };

  testScript = ''
    start_all()
    worker.wait_for_unit("multi-user.target")
    worker.succeed("test -e /dev/nvidia0")
  '';
}

Host still needs Case 1 settings, plus whatever sandbox-paths / programs.nix-required-mounts make /dev/nvidia* visible to the build. This is why containers help GPU/CUDA service tests: exclusive PCI passthrough into QEMU is a different, heavier path. It does not mean you can validate a custom kernel module inside nspawn — you are still on the host kernel.

The trap

Trap A — rename nodes → containers for a kernel, graphics, specialisation, SUID, or hardening test. The suite gets faster and quietly stops testing the claim. Keep those on QEMU.

Trap B — run container tests without host uid-range / auto-allocate-uids / cgroups. Failures look like driver or nspawn bugs. Configure the builder first (Case 1).

Trap C — mix containers and VMs on one VLAN without /dev/net in the sandbox. Bridging fails; pure-container tests did not need that path, so the “it worked yesterday” story is usually “we added a QEMU node.”

Trap D — assume container isolation equals VM isolation. Same host kernel, shared device bind-mounts, sandbox limits on SUID and many Protect* settings. Use containers for speed and hardware sharing; use QEMU when the threat model or the assertion needs a guest kernel.

# Anti-pattern — “containers everywhere”
{
  containers.machine = { pkgs, ... }: {
    boot.kernelPackages = pkgs.linuxPackages_latest; # does not give you that kernel
    services.xserver.enable = true;                 # no screenshots / GUI driver path
  };
}

# Boring fix — split by claim
{
  containers.api = { services.nginx.enable = true; };  # fast service check
  nodes.gui = { services.xserver.enable = true; };     # still QEMU when needed
}

The boring rule

On NixOS 26.05+, prefer containers for service and network integration tests; keep testScript unchanged.
Configure the builder once: auto-allocate-uids, experimental auto-allocate-uids + cgroups, extra-system-features = [ "uid-range" ].
Add extra-sandbox-paths = [ "/dev/net" ] only when containers and VMs share a VLAN in one test.
Keep nodes for kernel, graphics, specialisations, SUID-under-sandbox, and restrictive hardening cases.
Use container bind-mounts for host devices (e.g. GPU) when that is the point; never confuse that with testing a guest kernel.
Pin nixpkgs to nixos-26.05 (or newer) before relying on the backend in CI.

Try this

On a disposable 26.05 builder, apply Case 1 and confirm uid-range appears in nix show-config.
nix-build Case 2’s ping test. Note wall time. Re-run the same script with nodes instead of containers and compare (needs KVM).
Break Case 2 by commenting out extra-system-features on the builder; capture the failure mode; restore it.
Add a throwaway QEMU nodes machine sharing a VLAN with a container; run once without /dev/net, once with Case 4’s sandbox path.
Grep the desk flake for runNixOSTest / nodes = and list which checks could move to containers vs which must stay on QEMU. Open tickets; do not mass-rename blindly.

systemd-nspawn Test Backend

Mental model

Worked examples

Case 1: Host daemon settings for container tests

Case 2: Minimal two-machine ping with containers

Case 3: Keep QEMU nodes when the claim needs a real VM

Case 4: Mixed containers + VMs sharing a VLAN need /dev/net

Case 5: Bind-mount a host device (GPU sketch) — still not a kernel test

The trap

The boring rule

Try this

Case 2: Minimal two-machine ping with `containers`

Case 3: Keep QEMU `nodes` when the claim needs a real VM

Case 4: Mixed containers + VMs sharing a VLAN need `/dev/net`