Observability for Networks

Updated

September 4, 2026

Observability for Networks

If you cannot see it, you cannot operate it. This chapter turns lab verification habits into a minimal observability stack: interface signals, routing state, captures, logs, and a path toward metrics/telemetry—without requiring paid NMS.

Learning goals

By the end of this chapter you can:

  • Build a layered observability checklist for any lab or small network
  • Use Linux and FRR show/counters for incident triage
  • Capture and filter packets productively
  • Sketch metrics that matter (link, session, error)
  • Write a short runbook entry from an observed failure

Observability layers

Layer Questions Tools
L1/L2 link Up? errors? flaps? ip -s link, ethtool, MAC/FDB
L3 reachability Path? loss? ping, mtr, traceroute
Control plane Neighbors/sessions? vtysh shows, logs
Data plane path Which hop? route get, capture
Services App timeouts? ss, curl, synthetic checks
Continuity When did it start? logs, metrics, journals

Planes reminder for where to look

Planes

Baseline: what “healthy” looks like

Before incidents, save baselines:

mkdir -p baseline
docker exec clab-ospf-tri-r1 vtysh -c 'show ip ospf neighbor' > baseline/r1-ospf-nbr.txt
docker exec clab-ospf-tri-r1 vtysh -c 'show ip route' > baseline/r1-route.txt
docker exec clab-ospf-tri-r1 ip -s link > baseline/r1-link.txt

Diff during incidents:

diff -u baseline/r1-ospf-nbr.txt <(docker exec clab-ospf-tri-r1 vtysh -c 'show ip ospf neighbor')

Interface and error signals

ip -s -s link show eth1
# Look at RX/TX errors, drops, carrier changes

Drill: bounce a link; confirm counters and oper state track reality.

ip monitor link

Control-plane signals

OSPF

vtysh -c 'show ip ospf neighbor'
vtysh -c 'show ip ospf interface'
vtysh -c 'show log'   # if logging configured

BGP

vtysh -c 'show bgp summary'
vtysh -c 'show bgp neighbors 10.0.0.2'
vtysh -c 'show bgp neighbors 10.0.0.2 routes'

Watch for: session flaps, prefix count drops to zero, unexpected AS paths.

Path quality

ping -c 50 -i 0.2 192.168.2.10
mtr -rwzc 100 192.168.2.10
traceroute -n 192.168.2.10

Interpret:

Pattern Suspect
Loss from first hop Local segment
Loss starting mid-path Transit / congestion / ACL
High latency one hop jump That hop or link serialization
Jittery wireless-like Lab CPU contention sometimes!

On shared lab hosts, CPU starvation fakes network loss—check docker stats.

Packet capture practice

tcpdump -ni eth1 -w /tmp/inc.pcap not port 22
# reproduce
tcpdump -r /tmp/inc.pcap -nn
tshark -r /tmp/inc.pcap -Y 'icmp || bgp || arp' -V | less

Capture points

Question Where to capture
Did host send? Host interface
Did router rewrite L2? Router egress
Did peer receive? Peer ingress
Policy drop? Both sides + counters

Logging

FRR

Enable meaningful logging to file or syslog inside the node (image-dependent):

log file /var/log/frr/frr.log
log timestamp precision 3
docker exec clab-ospf-tri-r1 tail -f /var/log/frr/frr.log

Kernel

dmesg -T | tail
# martians, neighbor table full, etc.

Metrics sketch (Prometheus-shaped thinking)

Even if you do not run Prometheus yet, define signals:

Metric idea Type Labels
link_up gauge node, iface
interface_errors_total counter node, iface, dir
bgp_session_up gauge node, peer
bgp_prefixes_received gauge node, peer
ospf_neighbors_full gauge node
ping_loss_ratio gauge src, dst

Synthetic ping exporter + node exporter covers a surprising amount for home/lab NetOps.

Minimal blackbox idea

#!/usr/bin/env bash
# probe.sh — exit 1 on failure for CI/cron
ping -c 3 -W 1 192.168.2.10 >/dev/null

Cron or systemd timer + alert on fail is valid ops.

Telemetry awareness (2026)

Modern NOS (e.g. SR Linux community) emphasize streaming telemetry (gNMI). FRR/Linux labs may stick to show + exporters. Skill transfer:

  • Subscribe to interface + protocol state
  • Prefer push metrics over SSH scraping when scale grows
  • Still keep packet capture for truth

Incident timeline template

## Incident
Start (UTC):
Detect signal:
Impact:
## Timeline
- T0: 
- T+2m: 
## Hypotheses
1.
2.
## Checks performed
## Root cause
## Fix
## Follow-ups
- monitor:
- prevent:

Lab drill: observe a BGP flap

  1. Baseline show bgp summary
  2. neighbor shutdown or link down
  3. Record timestamps of state change and route withdrawal
  4. Restore; measure time to Established and ping recovery
  5. Write incident note with detection idea (“alert if bgp_session_up==0 for 60s”)

Lab drill: silent MTU blackhole

From the MTU chapter: large ping fails, small works.

Observability angle: which metric would catch this? (not link_down). Need synthetic large-packet probe or TCP service check.

Runbook snippet library

Keep short runbooks next to labs:

## Runbook: OSPF neighbors down
1. Check link state ip link
2. Check addressing same subnet
3. show ip ospf interface MTU
4. capture hello packets
5. check area / auth / passive

Privacy and safety

  • Captures may contain payloads—store carefully
  • Do not tcpdump production spans without authorization
  • Redact journals before sharing

Summary

  • Observability is layered: link → path → control → packet → service
  • Baselines make incidents measurable
  • Counters + mtr + captures beat vibes
  • Define metrics before you buy platforms
  • Runbooks convert personal heroics into repeatable ops

Next: automation / lab-as-code—pushing configs, smoke deploys, and CI-shaped checks.