039 Project 39: Prometheus Alert Rule Tester

039 Build a Prometheus Alert Rule Tester

Evaluate synthetic metric streams against threshold rules.

Full main.go

package main

import "fmt"

func shouldAlert(v float64, threshold float64, forSteps int, streak int) bool {
    if v > threshold {
        streak++
    } else {
        streak = 0
    }
    return streak >= forSteps
}

func main() {
    vals := []float64{70, 82, 90, 88, 91, 72, 95}
    threshold := 85.0
    forSteps := 3
    streak := 0

    for i, v := range vals {
        if v > threshold {
            streak++
        } else {
            streak = 0
        }
        alert := streak >= forSteps
        fmt.Printf("t=%d val=%.1f streak=%d alert=%v\n", i, v, streak, alert)
    }
}

Step-by-Step Explanation

1. Define metrics types for totals, current values, and distributions.
2. Register metrics and expose a scrape endpoint.
3. Update metrics in business flow.
4. Validate with manual requests and Prometheus scrape.
5. Tune labels to avoid high-cardinality cardinality issues.

Code Anatomy

• Metric definitions at startup.
• Request/probe path updates counters, gauges, or histograms.
• /metrics endpoint exposes state for observability stack.

Learning Goals

• Design practical service metrics.
• Connect application behavior to SLO monitoring.
• Build observability by default.