Alerting and Runbooks

Updated

September 8, 2026

Alerting and Runbooks

Overview

Alerts should be actionable, rare, and tied to user pain. Every page should have a short runbook: what it means, what to check, how to mitigate.

Alert design

Prefer Avoid
SLO burn (error budget) Raw CPU > 80% forever
Symptom (errors, latency) Cause-only (GC pause) without user impact
Multi-window burn rates Noisy single-sample thresholds

Example intent:

Alert if 5m error rate > 5% AND 1h error rate > 1%
→ catches both spikes and slow burns

Severity

Level Response
page Human now
ticket Next business hours
log-only Not an alert

If nobody will act at 3 a.m., it is not a page.

Runbook template

# HighErrorRate

## Symptom
5xx rate above SLO burn threshold.

## Impact
Users fail to load checkout / API clients retry.

## Quick checks
1. Deploy recently? (`kubectl rollout history`)
2. Dependency errors in logs (filter `dependency=`)
3. Saturation: CPU, threads, DB connections

## Mitigations
1. Rollback last deploy if correlated
2. Shed non-critical traffic / enable rate limit
3. Scale if clearly capacity-bound

## Escalation
Platform on-call if cluster-wide

Linking code to ops

Log a stable error code and metric label (code=db_timeout) so runbooks can search.

slog.Error("checkout_failed", "code", "db_timeout", "err", err)

Rules of thumb

Do Don’t
One runbook per paging alert Page without owner
Test alerts in staging Alert on untested promql
Review flappy alerts weekly Mute forever without fixing

Try next

  1. Write runbooks for two real alerts.
  2. Delete or downgrade one noisy alert.
  3. Add code= to top error logs.