Day 80 — Load test

Updated

July 30, 2026

Day 80 — Load test

Stage VIII · ~3h
Goal: Run a controlled load test (vegeta, k6, or hey) against your service, record latency/throughput/error rate, correlate with metrics/pprof, and name one bottleneck with evidence. Go 1.26 service recommended.

Why this day exists

Unit benchmarks (Day 66) miss:

  • Real HTTP stack
  • Pool exhaustion
  • Lock contention under concurrency
  • GC under multi-core load
  • Proxy/middleware overhead

Load tests validate Stage VII observability investments (metrics, pprof) and feed Day 81–89 hardening. Capstone demos that never saw concurrent clients are fiction.

Important

Only attack local or systems you own. Do not load-test third-party production APIs. Cap duration; watch machine temperature/throttling.

Theory 1 — What to measure

Metric Why
Throughput (RPS / RPS achieved) Capacity
Latency p50 / p95 / p99 User experience
Error rate Correctness under load
Saturation CPU, memory, goroutines, DB conns
Tail behavior Cliffs when queues form

Stop conditions

Stop or reduce rate when:

  • Error budget exceeded (e.g. >1% non-2xx)
  • Latency SLO miss (define a lab SLO, even if informal)
  • Resource limit (CPU pegged, OOM, DB too many connections)
  • Machine thermal throttle (note it—laptop science is noisy)

Theory 2 — Tools (pick one)

vegeta (simple HTTP, constant rate)

go install github.com/tsenart/vegeta@latest
echo "GET http://127.0.0.1:8080/v1/hello" \
  | vegeta attack -duration=30s -rate=100 \
  | vegeta report

echo "GET http://127.0.0.1:8080/v1/hello" \
  | vegeta attack -duration=30s -rate=100 \
  | vegeta report -type=json > load.json

Targets file for headers/auth:

GET http://127.0.0.1:8080/v1/items/1
Authorization: Bearer <token>
vegeta attack -targets=targets.txt -duration=30s -rate=50 | vegeta report

hey

go install github.com/rakyll/hey@latest
hey -z 30s -c 50 http://127.0.0.1:8080/v1/hello

Good for quick concurrency knobs (-c workers). Compare carefully to vegeta’s constant rate.

k6 (scripted scenarios)

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = { vus: 20, duration: '30s' };

export default function () {
  const res = http.get('http://127.0.0.1:8080/v1/hello');
  check(res, { 'status 200': (r) => r.status === 200 });
  sleep(0.1);
}
k6 run load.js

Any one tool is enough. Prefer constant rate then step up.

Theory 3 — Experiment design

  1. Establish baseline at low RPS (or low VUs).
  2. Increase rate until p99 or errors break.
  3. Hold steady; capture pprof + /metrics.
  4. Change one variable (pool size, GOMAXPROCS, query, cache flag).
  5. Re-measure; write the story.
# terminal 1
./service

# terminal 2 — during attack
curl -o cpu.pb.gz "http://127.0.0.1:6060/debug/pprof/cpu?seconds=20"
curl -s localhost:8080/metrics > metrics-under-load.txt

Ambient conditions (record them)

Factor Your note
Bare metal vs Docker
DB local vs remote
Laptop power / thermal
GOMAXPROCS
Other heavy processes

Without this, day-over-day comparisons lie.

Theory 4 — Common bottlenecks in Go services

Symptom Suspect
High CPU in json / encoding Payload size; buffer reuse
Goroutines climb without bound Leak or blocking without timeout
DB too many connections SetMaxOpenConns; pool vs load
Lock contention in pprof mutex profile Shared map without sharding
GC CPU high Allocation rate (Days 66–67)
Low CPU, high latency External dependency / lock / sleep
Errors only at high RPS Queue overflow, timeouts, pool starvation
Fine on /healthz, dies on /v1/work Health not representative—good

DB pool knobs (reminder)

db.SetMaxOpenConns(10)
db.SetMaxIdleConns(5)
db.SetConnMaxLifetime(30 * time.Minute)

Change one knob per experiment.

Theory 5 — Choosing the target endpoint

Endpoint Use?
/healthz Smoke only—not primary load target
Authenticated read Good baseline
Authenticated write Stronger; watch disk/DB
Cheap CPU burn /v1/work Isolates app CPU
Cacheable GET Watch hit-rate interaction (Day 58)

Prefer a realistic path your users hit. Document auth setup in LOAD.md.

Worked example — vegeta rate matrix

for r in 50 100 200 400; do
  echo "=== rate $r ==="
  echo "GET http://127.0.0.1:8080/v1/work" \
    | vegeta attack -duration=20s -rate=$r \
    | vegeta report \
    | tee "load-rate-$r.txt"
done

LOAD.md table:

Rate Success % p50 p95 p99 Notes
50 baseline
100
200 cliff?
400

Worked example — step load + metrics correlation

curl -s localhost:8080/metrics > /tmp/m0.txt

echo "GET http://127.0.0.1:8080/v1/work" \
  | vegeta attack -duration=15s -rate=50 \
  | vegeta report | tee /tmp/r50.txt
curl -s localhost:8080/metrics > /tmp/m50.txt

echo "GET http://127.0.0.1:8080/v1/work" \
  | vegeta attack -duration=15s -rate=200 \
  | vegeta report | tee /tmp/r200.txt
curl -s localhost:8080/metrics > /tmp/m200.txt

grep demo_request /tmp/m50.txt /tmp/m200.txt || true

While rate=200 runs (separate terminal):

curl -o /tmp/cpu-load.pb.gz "http://127.0.0.1:6060/debug/pprof/cpu?seconds=15"
go tool pprof -top /tmp/cpu-load.pb.gz | head -20

Story template (required prose)

At 50 RPS p99 was 20ms with 0% errors.
At 200 RPS p99 jumped to 180ms and error rate 2%.
pprof showed time in encoding/json.Marshal and database/sql.(*DB).conn.
Metrics: demo_request_errors_total +N; db pool wait (if exposed).
Hypothesis: MaxOpenConns=2 saturates under concurrency.
Next experiment: raise MaxOpenConns from 2 to 10 and retest once.
Result after change: ...

Theory 6 — Mitigations worth one experiment

Lever How
DB pool SetMaxOpenConns
Timeouts Server/client deadlines reduce pileups
Caching Cache-aside on hot GET (if correct)
Payload Smaller JSON; avoid chatty fields
GOMAXPROCS Usually leave default; note if container CPU limit
Reduce allocs Only after pprof says so

One mitigation + retest beats five unmeasured “optimizations.”

Lab 1 — Tooling and smoke

Suggested workspace: instrumented service from Days 69–78 (or Stage VI gate service).

  1. Pick vegeta or hey or k6; install; record version.
  2. Start service; confirm one manual curl succeeds.
  3. 5s smoke attack at low rate.
echo "GET http://127.0.0.1:8080/healthz" \
  | vegeta attack -duration=5s -rate=10 \
  | vegeta report

Lab 2 — Rate matrix on a real endpoint

  1. Target a non-health route.
  2. Run ≥3 rates × ≥15–20s each.
  3. Save reports under load/ or paste into LOAD.md.
  4. Mark the cliff rate if any.

Lab 3 — Correlate observability

  1. Under the highest “interesting” rate, capture CPU pprof or mutex profile.
  2. Snapshot /metrics before/after.
  3. Optional: curl /debug/pprof/goroutine?debug=1 | head for leak signals.
curl -s "http://127.0.0.1:6060/debug/pprof/goroutine?debug=1" | head -40

Ensure pprof is localhost-bound (Day 65/81)—do not expose it to the world for this lab.

Lab 4 — One mitigation experiment

  1. Form a hypothesis from pprof/metrics.
  2. Change one variable.
  3. Re-run the cliff rate.
  4. Record before/after in LOAD.md.
  5. If no improvement, that is still a valid result—document it.

Lab 5 — Write LOAD.md evidence pack

# Load test — day80

Date:
Service:
Go: go1.26.x
Tool: vegeta|hey|k6 version:
Target URL:
Auth: none|bearer ...

## Environment
- Host:
- DB:
- GOMAXPROCS:

## Results table
| Rate | Success % | p95 | p99 | Notes |
|------|-----------|-----|-----|-------|

## Profiles / metrics paths
- 

## Bottleneck (one sentence)
- 

## Experiment
- Change:
- Result:

Hour-by-hour suggestion

Time Focus
0:00–0:20 Tool install + smoke one request
0:20–1:20 Rate matrix + reports
1:20–2:10 pprof/metrics under load
2:10–2:40 One mitigation experiment
2:40–3:00 LOAD.md write-up

Common gotchas

Gotcha Fix
Load tester on same starved laptop Note limits; lower rate
Keep-alive differences across tools Compare tools fairly; stick to one
Warm vs cold caches Run twice; discard first
Hitting readiness during deploy Steady-state only
Optimizing without profile Capture pprof first
/healthz only Not representative
Attacking shared staging Only local/owned systems
Auth failures counted as “app slow” Fix tokens first
Forgetting to pin duration Always set -duration / -z
pprof on public interface Bind 127.0.0.1:6060

Checkpoint

  • Tool installed and command recorded
  • ≥1 sustained attack (≥20–30s) on non-health route
  • Report numbers saved
  • Metrics or pprof under load
  • Bottleneck named in LOAD.md
  • Optional mitigation + retest documented
  • Safety: only local/owned target

Commit

git add .
git commit -m "day80: load test bottleneck evidence"

Write three personal gotchas before continuing.

Tomorrow

Day 81 — Security checklist: a structured pass over auth, input, deps, and runtime exposure before design-doc day—load evidence informs which timeouts and limits matter most.