Name, Time, and Log Hygiene

Updated

September 4, 2026

Name, Time, and Log Hygiene

Broken DNS, skewed clocks, and silent logs turn simple outages into multi-hour mysteries. This chapter treats name resolution, time sync, and logging as first-class edge services—still on free Linux tooling in Containerlab (mid-2026).

Learning goals

By the end of this chapter you can:

  • Design a minimal lab DNS path (stub resolver → recursive or authoritative)
  • Explain why certificate and log correlation depend on clock sync
  • Run chrony/ntp-style time sync in a lab and detect skew
  • Centralize or at least structure logs so incidents are replayable
  • Apply a hygiene checklist to any edge or fabric lab

Why this chapter exists

Failure Symptom people mis-blame
DNS “The network is down” (TCP works, names fail)
Time skew TLS fails, Kerberos fails, log order nonsense
No logs Every fix is guesswork; no postmortem

NetOps competence includes service dependencies of the control and management planes.

Concepts — naming

Layers of DNS use

Application
    → stub resolver (/etc/resolv.conf)
        → recursive resolver (lab or ISP)
            → authoritative servers

In labs you often run:

Role Example tool
Authoritative for lab.example dnsmasq, unbound, knot (as available)
Recursive unbound, dnsmasq
Stub on nodes nameserver in resolv.conf

Records you actually need

Type Lab use
A/AAAA Host and VIP names
PTR Reverse for traceroute readability (optional)
SRV Only when practicing real apps
CNAME Alias hygiene—avoid chains

Split horizon awareness

Internal names may not match external. Document which view a resolver serves. Mis-split horizon looks like “works on my jump host.”

Concepts — time

  • UTC everywhere in configs and logs when possible
  • NTP/chrony/ptp family: clients step or slew toward sources
  • Auth and TLS validate notBefore/notAfter against local clock
  • Distributed tracing and log merge need comparable timestamps

Skew of minutes is enough to break modern auth. Skew of seconds confuses incident timelines.

Concepts — logs

Plane What to capture
Device/OS link up/down, daemon start, OOM
Routing neighbor flaps, policy denies (where logged)
Security filter drops if logged
Lab harness deploy/destroy, verify.sh results

Structured fields beat walls of free text: timestamp host facility message.

Addressing plan (hygiene lab)

  client ---- edge ---- dns-ntp
                 \
                  pe
Node IP Role
edge eth1 10.40.0.1/24 client GW
client 10.40.0.10/24 stub resolver → edge or dns
dns-ntp 10.40.0.53/24 dnsmasq + chrony server
edge eth2 198.51.100.1/30 optional upstream

Topology YAML

name: hygiene-edge

topology:
  nodes:
    edge:
      kind: linux
      image: alpine:3.20
      exec:
        - apk add --no-cache iproute2 iptables iputils bind-tools chrony
    dnsntp:
      kind: linux
      image: alpine:3.20
      exec:
        - apk add --no-cache iproute2 dnsmasq chrony iputils bind-tools
    client:
      kind: linux
      image: alpine:3.20
      exec:
        - apk add --no-cache iproute2 iputils bind-tools
    pe:
      kind: linux
      image: alpine:3.20
      exec:
        - apk add --no-cache iproute2 iputils

  links:
    - endpoints: ["client:eth1", "edge:eth1"]
    - endpoints: ["dnsntp:eth1", "edge:eth2"]
    - endpoints: ["edge:eth3", "pe:eth1"]

Use a small bridge if you want client and dnsntp on one LAN with edge as L3 gateway only—either design is fine if documented.

Bring-up addressing (example)

# edge: two LANs + pe
docker exec clab-hygiene-edge-edge sh -c '
  ip addr add 10.40.0.1/24 dev eth1; ip link set eth1 up
  ip addr add 10.40.1.1/24 dev eth2; ip link set eth2 up
  ip addr add 198.51.100.1/30 dev eth3; ip link set eth3 up
  sysctl -w net.ipv4.ip_forward=1
'
docker exec clab-hygiene-edge-dnsntp sh -c '
  ip addr add 10.40.1.53/24 dev eth1; ip link set eth1 up
  ip route add default via 10.40.1.1
'
docker exec clab-hygiene-edge-client sh -c '
  ip addr add 10.40.0.10/24 dev eth1; ip link set eth1 up
  ip route add default via 10.40.0.1
  echo nameserver 10.40.1.53 > /etc/resolv.conf
'
# edge route to dns LAN is connected; client needs path to 10.40.1.53
docker exec clab-hygiene-edge-edge ip route add 10.40.1.0/24 dev eth2 2>/dev/null || true

Ensure client can reach 10.40.1.53 (forwarding + reverse path).

DNS lab — dnsmasq authoritative snack

docker exec clab-hygiene-edge-dnsntp sh -c 'cat >/etc/dnsmasq.conf <<EOF
interface=eth1
no-resolv
address=/vip.lab.example/10.40.0.254
address=/edge.lab.example/10.40.0.1
address=/client.lab.example/10.40.0.10
log-queries
EOF'
docker exec clab-hygiene-edge-dnsntp dnsmasq

Predict

docker exec clab-hygiene-edge-client nslookup vip.lab.example 10.40.1.53
# or
docker exec clab-hygiene-edge-client dig +short vip.lab.example @10.40.1.53

Returns 10.40.0.254.

Observe / fix

Symptom Fix
timeout routing, firewall, dnsmasq not listening
NXDOMAIN typo in address= or wrong server
uses wrong resolver resolv.conf

Drill — DNS outage looks like network outage

docker exec clab-hygiene-edge-dnsntp pkill dnsmasq || true
docker exec clab-hygiene-edge-client ping -c 2 vip.lab.example   # may fail resolve
docker exec clab-hygiene-edge-client ping -c 2 10.40.0.1         # IP still works

Predict → observe → fix

  • Name fails, IP works → name plane, not forwarding
  • Restart dnsmasq; retest dig
  • Harden: secondary resolver IP in resolv.conf (even if both lab-local)
nameserver 10.40.1.53
nameserver 10.40.1.54

Time sync — chrony sketch

On dnsntp as a simple lab time source (isolated labs often use local orphan mode or host chrony):

docker exec clab-hygiene-edge-dnsntp sh -c 'cat >/etc/chrony/chrony.conf <<EOF
local stratum 8
allow 10.40.0.0/16
makestep 1.0 3
EOF'
docker exec clab-hygiene-edge-dnsntp chronyd

Client/edge:

docker exec clab-hygiene-edge-client sh -c '
  apk add --no-cache chrony
  echo "server 10.40.1.53 iburst" > /etc/chrony/chrony.conf
  chronyd
'
docker exec clab-hygiene-edge-client chronyc tracking
docker exec clab-hygiene-edge-client chronyc sources -v

Package paths vary (/etc/chrony.conf vs /etc/chrony/chrony.conf)—adjust to the image.

Skew drill

# induce skew on client (lab only)
docker exec clab-hygiene-edge-client date -s "2020-01-01 00:00:00"
docker exec clab-hygiene-edge-client chronyc -a makestep
docker exec clab-hygiene-edge-client date
docker exec clab-hygiene-edge-dnsntp date

Predict

After makestep/slew, client approaches server time; chronyc tracking shows low offset.

Observe

Record offset before/after. Correlate with a fake “TLS” check: if you generate certs with short validity in lab tools, wrong dates fail verification.

Logging hygiene

Local journal pattern

# example: log dnsmasq and link events to a file on edge
docker exec clab-hygiene-edge-edge sh -c '
  mkdir -p /var/log/lab
  echo "$(date -Is) lab start" >> /var/log/lab/edge.log
'

syslog-style forward (concept)

device/app → rsyslog/syslog-ng → central lab collector

In Containerlab, mounting a host directory for /var/log/lab keeps evidence after destroy:

    edge:
      kind: linux
      image: alpine:3.20
      binds:
        - ./logs/edge:/var/log/lab

What to log for network incidents

Event Why
Link down/up Correlates with neighbor loss
Daemon restart Control plane blip
DHCP ACK / VRRP transition Edge role changes
Filter deny (sampled) Policy vs outage
verify.sh pass/fail Automation truth

FRR logging (when routers are FRR)

log file /var/log/frr/frr.log
log timestamp precision 3
docker exec clab-...-r1 vtysh -c 'show log'
# or tail the bound log file

Bind /var/log/frr into the lab folder for postmortems.

Hygiene checklist (use every serious lab)

  • Addressing plan names every important VIP and service IP
  • DNS A records for VIP, routers, servers (even if only lab zone)
  • Clients have ≥1 working resolver; document fallback
  • Clocks within seconds of a known source (or explicitly isolated)
  • Logs land on durable storage (bind mount / central)
  • Timezone policy stated (UTC recommended)
  • Incident notes include timestamps from the same clock domain

Predict → observe → fix mini-scenarios

Scenario A — “Git clone fails” in a lab VM

  • Predict: DNS or proxy
  • Observe: ping 1.1.1.1 vs ping github.com
  • Fix: resolv.conf / reachability to recursive resolver

Scenario B — “Certificate expired” on brand-new lab CA

  • Predict: node year is wrong
  • Observe: date -u on client and server
  • Fix: chrony; disable fake date experiments

Scenario C — “It failed an hour ago” with no trace

  • Predict: no logs retained
  • Observe: empty /var/log/lab, container destroyed
  • Fix: binds + verify.sh archival

Optional: name the lab graph

Containerlab graph exports help humans; DNS helps apps. Keep hostname = inventory name = DNS label when you can:

clab-hygiene-edge-edge  ↔  edge.lab.example

Common mistakes

Mistake Symptom
resolv.conf points at host DNS unreachable from namespace all names fail
No forward path to DNS subnet intermittent based on which node
Logging only to container overlay evidence vanishes on destroy
Mixing local timezones in notes impossible timelines
NTP blocked by ACL you added “for hardening” chronic skew

Summary

  • Name, time, logs are edge/management dependencies, not optional polish
  • Practice DNS outage vs forwarding outage until the difference is reflex
  • Sync clocks before debugging TLS or distributed logs
  • Bind-mount lab logs; structure enough fields to postmortem
  • Add this hygiene checklist to dual-home and fabric labs later

Next: edge dual-home lab—tie FHRP, routing preference, and these services into one site edge workout.