Overlay Lab

Updated

September 4, 2026

Overlay Lab

This chapter is a portfolio lab: build a small multi-site / multi-leaf overlay on free tools, break it on purpose, and prove each plane. Choose track A (GRE sites), track B (VXLAN static), or track C (EVPN if your FRR/SR Linux build supports it). Mid-2026 defaults: Containerlab + FRR + Alpine.

Learning goals

By the end of this chapter you can:

  • Deliver one complete overlay lab folder that rebuilds from git
  • Show captures proving encapsulation
  • Run a failure matrix (underlay, tunnel endpoint, VNI/policy)
  • Document MTU and security stance
  • Meet the book’s definition of done for overlay work

Pick a track

Track Focus Difficulty
A — Dual-site GRE/WG Site tunnels + routing ★★☆
B — VXLAN HER two-leaf L2 overlay data plane ★★★
C — EVPN two-leaf + RR Control + data ★★★★

Do A then B if new to overlays. Attempt C only after B host pings work.

Definition of done (all tracks)

  • README.md with diagram + addressing
  • *.clab.yml + configs/scripts in git
  • Clean containerlab deploy on cold start
  • verify.sh green
  • Capture artifact (pcap or logged tcpdump) showing outer header
  • ≥ 3 failure drills with notes
  • MTU test recorded
  • Retrospective: what was underlay vs overlay

Track A — Dual-site GRE (reference build)

Reuse topology from Site Tunnels; package it cleanly.

labs/overlay-a/
  site-tun.clab.yml
  addressing.md
  scripts/bringup.sh
  verify.sh
  drills.md
  captures/.gitkeep

Success metrics

  • h1 ↔︎ h2 continuous ping
  • inet tcpdump shows GRE
  • Underlay link down stops overlay; restore recovers
  • DF large ping behavior documented

verify.sh

#!/usr/bin/env bash
set -euo pipefail
docker exec clab-site-tun-h1 ping -c2 -W1 192.168.20.10
docker exec clab-site-tun-h2 ping -c2 -W1 192.168.10.10
docker exec clab-site-tun-sitea ip link show gre1 | grep -q UP
echo OK-A

Failure matrix A

# Action Expect
A1 inet eth2 down overlay down
A2 delete gre on sitea blackhole
A3 remove LAN route gre peer lives, hosts die
A4 lower mtu wrongly large packet fail

Track B — VXLAN two-leaf (reference build)

Package VXLAN data plane chapter topology as labs/overlay-b/.

Success metrics

  • Underlay VTEP lo ping
  • h1 ↔︎ h2 same subnet across leaves
  • Spine capture UDP/4789
  • Wrong VNI experiment isolates
  • FDB shows remote MAC after traffic

bringup excerpt

#!/usr/bin/env bash
set -euo pipefail
# called after containerlab deploy
sudo containerlab deploy -t vxlan-dp.clab.yml
# apply addresses + vxlan/bridge (idempotent scripts per node)
./scripts/addr.sh
./scripts/vxlan.sh
./verify.sh

verify.sh B

#!/usr/bin/env bash
set -euo pipefail
fail() { echo FAIL:$* >&2; exit 1; }
docker exec clab-vxlan-dp-leaf1 ping -c1 -W1 10.0.0.2 || fail underlay
docker exec clab-vxlan-dp-h1 ping -c2 -W1 10.10.10.20 || fail overlay
echo OK-B

Failure matrix B

# Action Expect
B1 spine forward off underlay fail → overlay fail
B2 remove FDB flood entry BUM/unknown fail
B3 VNI mismatch isolation
B4 host unbridge eth1 local access fail

Capture helper

docker exec clab-vxlan-dp-spine sh -c \
  'tcpdump -ni eth1 -c 20 udp port 4789 -w /tmp/vx.pcap'
# docker cp out to captures/vx.pcap

Track C — EVPN intro (stretch)

Package EVPN chapter with explicit feature probe:

docker exec clab-evpn-intro-leaf1 vtysh -c 'show bgp l2vpn evpn summary' \
  || { echo "EVPN not available—document fallback to Track B"; exit 0; }

Success metrics

  • BGP EVPN Established
  • EVPN MAC routes visible
  • Host ping works
  • Neighbor shutdown kills overlay learning path
  • Underlay remains pingable

Failure matrix C

# Action Expect
C1 RR down leaves lose routes (no alternate)
C2 advertise-all-vni removed control stops updates
C3 underlay lo route removed BGP over loopback dies
C4 RT mismatch experiment import empty

Integrated “mini fabric snack” (optional merge)

If ambitious, combine track B underlay style with dual spines from the fabrics part later. Do not expand scope until verify is green on two leaves.

     s1
    /  \
  l1    l2
  |      |
  h1     h2

Underlay ECMP + VXLAN HER to both directions is enough complexity for one week.

Observability expectations

Tool Use
ping / traceroute Inner vs outer endpoints
tcpdump / tshark Prove encap
bridge fdb / wg show / ip xfrm State tables
vtysh show bgp l2vpn evpn Control plane
ip route get Which path outer takes

Hardening lite (overlay)

  • Underlay allows only needed outer ports between VTEPs
  • No accidental route leaking inner prefixes to “inet”
  • Keys for WireGuard/IPsec not in public git
  • VNI inventory file (vnis.md)

Lab report template

# Overlay lab report
Track: B
Date: 2026-...
Underlay: static loopbacks via spine
Overlay: VXLAN VNI 100 HER
Evidence:
- captures/vx.pcap
- verify.sh OK
Drills: B1–B4 notes...
What surprised me:
Next: EVPN or dual-spine

Predict → observe → fix (capstone habit)

For every drill row, write three lines in drills.md. If observe ≠ predict, you learned something—that is the point.

Time-box guidance

Block Time
Deploy + address 30–60 min
Overlay up 1–2 h
Captures + verify 30 min
Failure matrix 1–2 h
Writeup 30 min

If blocked >1 h on EVPN syntax, drop to Track B and file an issue note—do not thrash.

Common mistakes

Mistake Symptom
Mixing all tracks half-done Nothing verifies
No cold-start test “Works on my dirty ns”
Capturing on wrong node Never see UDP 4789
Skipping underlay prove Overlay debug theater
Giant jumbo scope No finish

Summary

  • One finished overlay lab beats three partial ones
  • Tracks A→B→C escalate control-plane complexity
  • Always prove encap with packets and failures with a matrix
  • Keep VNI/tunnel inventory and MTU notes in git
  • Next part scales underlays into fabrics and multi-area design

When this lab is green, you are ready for Clos-style underlays and multi-implementation leaves.