L2 Failure Drills

Updated

September 4, 2026

L2 Failure Drills

This chapter is a practice arena. You combine Ethernet, VLANs, loop prevention, and aggregation under intentional failure. Competence is measured by evidence and recovery, not by green first pings.

Learning goals

By the end of this chapter you can:

  • Run a structured failure catalog against an L2 lab
  • Collect the minimum evidence set for each class of break
  • Recover cleanly and re-verify
  • Write short incident notes a peer can follow
  • Know when an “L2 issue” is actually L3 or host firewall

Baseline lab

Use a multi-switch VLAN lab (from the VLANs chapter) with:

  • Two switches, trunk between them
  • VLAN 10 and VLAN 20
  • Hosts h1/h3 in VLAN 10, h2/h4 in VLAN 20
  • Optional inter-VLAN router for a subset of drills

Document addressing in addressing.md. Ensure ./verify.sh is green before any inject.

Evidence kit

# Host
ip -br a
ip neigh
ping -c 3 -W 1 <target>
tcpdump -ni eth1 -e -c 30 arp or icmp

# Switch / bridge
bridge vlan show
bridge fdb show
ip -s link
# STP port states if enabled

Save outputs under journal.md with timestamps.

Drill catalog

For each drill: predict → inject → observe → restore → harden.

Drill 1 — Access port wrong VLAN

Inject: Move h1’s switch port from VLAN 10 to VLAN 20 (PVID change).

Predict: h1 cannot ARP h3; may see wrong subnet peers if any; verify.sh fails.

Observe: bridge vlan show, tcpdump for ARP who-has never answered by h3.

Restore: Correct PVID/untagged VLAN.

Harden: Label ports in diagrams; negative test in verify (optional).

Drill 2 — Trunk missing VLAN

Inject: Remove VLAN 10 from trunk allow list on one switch only.

Predict: Local VLAN 10 still works on that switch; cross-switch VLAN 10 dies; VLAN 20 may still work.

Observe: Asymmetric “works on same switch only.”

Restore: Re-add VID on both ends.

Harden: Checklist “both ends allow list.”

Drill 3 — Native VLAN mismatch

Inject: Native/PVID untagged mapping differs on trunk ends.

Predict: Untagged frames land in different broadcast domains; confusing intermittent leaks.

Observe: Captures show untagged where you expected tags; wrong FDB.

Restore: Align native VLAN; prefer tagged-only user VLANs.

Harden: Explicit lab policy: native VLAN unused for data.

Drill 4 — Cable / interface down

Inject:

docker exec clab-l2-vlan-sw1 ip link set eth3 down

Predict: Cross-switch traffic dies; FDB ages; local switching may live.

Observe: ping loss; interface counters; STP reconvergence if redundant path exists.

Restore: ip link set eth3 up; wait for learning/STP.

Harden: If no redundant path, document single point of failure.

Drill 5 — Broadcast storm (controlled)

Inject: Temporary bridging loop without STP on an isolated lab (very short).

Predict: CPU spikes, loss of management responsiveness, exploding counters.

Observe: ip -s link, host load.

Restore: containerlab destroy --cleanup immediately if unstable.

Harden: STP on; never lab this on a shared prod-like bridge.

Drill 6 — STP blocked port confusion

Inject: With STP triangle, shut the active root port path.

Predict: Alternate port activates; brief loss; new tree.

Observe: Port state sysfs or NOS show; ping -c 50 loss burst.

Restore: Bring link up; tree may revert depending on costs.

Harden: Note expected reconvergence budget.

Drill 7 — MAC move

Inject: Move host container link from sw1 to sw2 (redeploy link or migrate veth).

Predict: FDB updates; short blackhole until relearn; ARP may refresh.

Observe: bridge fdb show before/after; neigh on peers.

Restore: Stable attachment.

Harden: In virtualization, understand port channels / bond moves.

Drill 8 — Duplicate IP

Inject: Configure h4 temporarily with h3’s IP on VLAN 10 (careful).

Predict: Flapping ARP, intermittent delivery, wrong neigh.

Observe: ip neigh oscillating; tcpdump conflicting ARP replies.

Restore: Unique IPs.

Harden: Addressing plan ownership; IPAM habit.

Drill 9 — Duplicate MAC (advanced)

Inject: Spoof MAC on a second host (lab only).

ip link set eth1 down
ip link set eth1 address aa:bb:cc:dd:ee:ff
ip link set eth1 up

Predict: FDB flap; unstable delivery.

Restore: Original MAC; destroy/recreate if messy.

Harden: Understand L2 trust boundaries.

Drill 10 — LAG member down

Inject: On a bond/LAG lab, shut one member.

Predict: Traffic continues if min-links satisfied; single-flow rate unchanged.

Observe: /proc/net/bonding/bond0; ping continuity.

Restore: Member up.

Harden: Alert on member-down even if bond stays up.

Drill 11 — Inter-VLAN path mistaken for L2

Inject: Delete router inter-VLAN route or shut subinterface.

Predict: Same-VLAN ok; cross-VLAN fails—looks like “VLAN broken” if you misread.

Observe: traceroute never leaves gateway; ARP to gateway fails or gateway unreachable.

Restore: Router config.

Harden: Always check same-VLAN baseline before blaming trunk.

Drill 12 — Host firewall

Inject: On Linux host, drop ICMP.

# if nft/iptables available
iptables -A INPUT -p icmp -j DROP

Predict: Others cannot ping in; egress may still work.

Observe: Asymmetric ping stories.

Restore: Flush rule.

Harden: Include host endpoint in the failure picture.

Incident note template

## Incident: trunk missing VLAN 10 on sw2
Date:
Lab: l2-vlan

### User symptom
h1 cannot reach h3; h1 reaches local same-switch hosts.

### Prediction before fix
Trunk allow list asymmetry.

### Evidence
- bridge vlan show sw1/sw2 (paste)
- tcpdump trunk: no VLAN 10 tags toward sw2
- ping results

### Root cause
vid 10 deleted from sw2 eth3

### Fix
bridge vlan add vid 10 dev eth3

### Prevent
verify.sh checks cross-switch VLAN 10 and 20
README allow-list both ends

Scorecard

Run at least six drills across different classes. Check:

Class Drills done
VLAN membership
Trunk tagging
Link down
STP / loop
Aggregation
Host/IP mistakes

Combined verify.sh ideas

#!/usr/bin/env bash
set -euo pipefail
# positive
docker exec clab-l2-vlan-h1 ping -c 2 -W 1 10.10.10.3
docker exec clab-l2-vlan-h2 ping -c 2 -W 1 10.10.20.4
# negative isolation
if docker exec clab-l2-vlan-h1 ping -c 1 -W 1 10.10.20.4 2>/dev/null; then
  echo "unexpected inter-VLAN without router" >&2
  exit 1
fi
echo OK

After each restore, re-run.

Checkpoint: L2 competence

You pass the part when you can:

  1. Deploy multi-switch VLANs from code
  2. Produce evidence for tag/allow-list bugs
  3. Explain and demo loop risk
  4. Fail over a link or LAG member deliberately
  5. Write an incident note with root cause and prevention

Summary

  • Failure drills convert L2 theory into ops judgment
  • Always baseline with verify.sh
  • Predict before inject; restore and re-verify after
  • Separate L2, L3, and host causes with evidence
  • Incident notes are part of the curriculum artifact

Next part: layer 3—addressing, routing fundamentals, static labs, ICMP and path MTU.