VXLAN Data Plane

Updated

September 4, 2026

VXLAN Data Plane

VXLAN encapsulates Ethernet frames in UDP/IP so you can stretch L2 segments over an L3 underlay. This chapter focuses on the data plane: VNI, VTEP, flood behavior, and a Linux VXLAN lab in Containerlab (mid-2026). Control planes (EVPN) come next.

Learning goals

By the end of this chapter you can:

  • Define VTEP, VNI, and underlay vs overlay roles for VXLAN
  • Configure Linux ip link add type vxlan with static flood lists or bridge
  • Capture VXLAN UDP (port 4789) and relate outer/inner headers
  • Predict MAC learning and flooding behavior without EVPN
  • Handle MTU for VXLAN overhead

Concepts

Roles

Term Meaning
VTEP VXLAN Tunnel End Point—encap/decap device
VNI VXLAN Network Identifier—segment ID (~24-bit space)
Underlay IP network between VTEP loopbacks/addresses
Overlay Ethernet segments identified by VNI 
  VM/host -- L2 -- [ VTEP ]==== underlay IP ====[ VTEP ] -- L2 -- VM/host
                      VNI 100                              VNI 100

Header (mental)

Outer Eth | Outer IP | UDP dport 4789 | VXLAN (VNI) | Inner Eth | Inner payload

UDP source port often encodes entropy for ECMP.

Data-plane only learning

Without EVPN (or a controller):

  1. Unknown unicast / broadcast / multicast may flood to all VTEPs in the flood list (head-end replication) or underlay multicast group.
  2. Source MAC learning binds MAC → remote VTEP.
  3. Scale limits and wasteful flood—why EVPN exists.

Multicast vs head-end replication

Mode Idea
Underlay multicast BUM to group; needs multicast routing
Head-end replication (HER) Ingress VTEP unicasts BUM to each peer VTEP

Labs often use HER / static remote VTEP list for simplicity.

When VXLAN is appropriate

  • DC fabric tenant segments
  • Lab simulation of multi-leaf L2
  • Separating many logical L2s without many VLANs on underlay

Not always needed for simple L3-only campuses.

Topology — two VTEPs + underlay + hosts

name: vxlan-dp

topology:
  nodes:
    leaf1:
      kind: linux
      image: alpine:3.20
      exec:
        - apk add --no-cache iproute2 bridge iputils tcpdump
    leaf2:
      kind: linux
      image: alpine:3.20
      exec:
        - apk add --no-cache iproute2 bridge iputils tcpdump
    spine:
      kind: linux
      image: alpine:3.20
      exec:
        - apk add --no-cache iproute2 iputils tcpdump
        - sysctl -w net.ipv4.ip_forward=1
    h1:
      kind: linux
      image: alpine:3.20
      exec:
        - apk add --no-cache iproute2 iputils
    h2:
      kind: linux
      image: alpine:3.20
      exec:
        - apk add --no-cache iproute2 iputils

  links:
    - endpoints: ["h1:eth1", "leaf1:eth1"]
    - endpoints: ["h2:eth1", "leaf2:eth1"]
    - endpoints: ["leaf1:eth2", "spine:eth1"]
    - endpoints: ["leaf2:eth2", "spine:eth2"]

Addressing

Node Underlay Access
leaf1 lo 10.0.0.1/32 —
leaf2 lo 10.0.0.2/32 —
leaf1 eth2 10.1.1.1/30 —
spine eth1 10.1.1.2/30 —
leaf2 eth2 10.1.2.1/30 —
spine eth2 10.1.2.2/30 —
h1 — 10.10.10.10/24
h2 — 10.10.10.20/24
VNI 100 same L2 segment

Underlay routes: leaf1 reaches 10.0.0.2 via spine; advertise loopbacks with statics:

docker exec clab-vxlan-dp-leaf1 sh -c '
  ip addr add 10.0.0.1/32 dev lo
  ip addr add 10.1.1.1/30 dev eth2; ip link set eth2 up
  ip route add 10.0.0.2/32 via 10.1.1.2
  ip route add 10.1.2.0/30 via 10.1.1.2
'
docker exec clab-vxlan-dp-spine sh -c '
  ip addr add 10.1.1.2/30 dev eth1; ip link set eth1 up
  ip addr add 10.1.2.2/30 dev eth2; ip link set eth2 up
  ip route add 10.0.0.1/32 via 10.1.1.1
  ip route add 10.0.0.2/32 via 10.1.2.1
'
# leaf2 similar with 10.0.0.2 and 10.1.2.1

VXLAN + bridge on leaf1

docker exec clab-vxlan-dp-leaf1 sh -c '
  ip link add br100 type bridge
  ip link set br100 up
  ip link add vxlan100 type vxlan id 100 \
    local 10.0.0.1 dev eth2 dstport 4789
  # static remote VTEP (HER)
  bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.0.2
  ip link set vxlan100 up
  ip link set eth1 up
  ip link set eth1 master br100
  ip link set vxlan100 master br100
'

leaf2:

docker exec clab-vxlan-dp-leaf2 sh -c '
  ip link add br100 type bridge
  ip link set br100 up
  ip link add vxlan100 type vxlan id 100 \
    local 10.0.0.2 dev eth2 dstport 4789
  bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.0.1
  ip link set vxlan100 up
  ip link set eth1 up
  ip link set eth1 master br100
  ip link set vxlan100 master br100
'

Hosts (same subnet, no default needed for local L2 test):

docker exec clab-vxlan-dp-h1 sh -c '
  ip addr add 10.10.10.10/24 dev eth1; ip link set eth1 up
'
docker exec clab-vxlan-dp-h2 sh -c '
  ip addr add 10.10.10.20/24 dev eth1; ip link set eth1 up
'

Syntax note: bridge fdb / ip link add vxlan flags vary slightly by iproute2 version; consult man ip-link on your image. Some use vxlan ... remote 10.0.0.2 for point-to-point single peer.

Deploy checks

docker exec clab-vxlan-dp-leaf1 ping -c 2 10.0.0.2
docker exec clab-vxlan-dp-h1 ping -c 3 10.10.10.20
docker exec clab-vxlan-dp-leaf1 bridge fdb show
docker exec clab-vxlan-dp-spine tcpdump -ni eth1 -c 10 udp port 4789

Predict

  • Underlay loopback ping works
  • Host ping works over VNI 100
  • Spine sees UDP/4789, not bare inner ARP as local LAN

Observe

docker exec clab-vxlan-dp-spine tcpdump -ni eth1 -vv -c 5 udp port 4789
docker exec clab-vxlan-dp-leaf1 ip -d link show vxlan100

Fix

Failure Checks
Underlay down static routes, spine forward
No FDB flood entry remote VTEP missing
VNI mismatch id 100 both sides
Host not in bridge master br100

Drill — MTU

VXLAN overhead commonly ~50 bytes class. Set underlay MTU 1550 or overlay-aware 1450:

docker exec clab-vxlan-dp-leaf1 ip link set vxlan100 mtu 1450
docker exec clab-vxlan-dp-leaf2 ip link set vxlan100 mtu 1450
docker exec clab-vxlan-dp-h1 ping -c 2 -M do -s 1472 10.10.10.20
docker exec clab-vxlan-dp-h1 ping -c 2 -M do -s 1400 10.10.10.20

Predict

Too-large DF fails; tuned sizes pass.

Drill — wrong VNI isolation

Create VNI 200 on leaf2 only for a third host experiment—or change leaf2 to id 200 temporarily:

Predict

h1 cannot reach h2 (different segments).
### Fix

Align VNI; document allocation.

Drill — MAC move / learning

docker exec clab-vxlan-dp-leaf1 bridge fdb show br100
docker exec clab-vxlan-dp-h1 ping -c 1 10.10.10.20
docker exec clab-vxlan-dp-leaf1 bridge fdb show br100

Predict

After traffic, remote MAC points at vxlan device / dst VTEP.

Flood scope risk

Static 00:00:00:00:00:00 FDB means BUM replicates to listed VTEPs. More leaves → more replication. This is why control planes advertise who needs which VNI.

FRR / SR Linux awareness

NOS VTEPs bind VNI to VLAN/IRB and underlay loopbacks. Data-plane counters and show vxlan equivalents matter operationally. Linux lab teaches headers; production uses integrated L2/L3.

Verification checklist

Check Intent
Underlay VTEP reachability Outer IP works
UDP 4789 on path Encap happening
Same VNI Segment match
FDB remote Mapping exists
Host ARP/ping Inner L2
MTU tests Size tax handled

Common mistakes

Mistake Symptom
VTEP IP not loopback stable Flaps when link IP changes
No underlay route to remote VTEP Silent blackhole
Bridging mis-order Loops or no pass
Expecting EVPN routes without config Only static works
One huge flood list forever BUM explosion

Summary

  • VXLAN is Ethernet-in-UDP/IP with VNI segment IDs
  • VTEPs encap/decap; underlay only sees outer IP/UDP
  • Static HER labs teach data plane before EVPN
  • Verify with underlay ping, host ping, and spine captures
  • Plan MTU and flood scope deliberately

Next: EVPN control-plane intro—distributing MAC/IP and VNI membership without pure flood-and-learn.