Distributed Infrastructure Overview

Updated

July 30, 2026

Distributed Infrastructure Overview

This part introduces distributed reliability patterns used in production Go systems. The objective is to reason about failure as a normal operating condition, not an exception path you only sketch in design docs.

Why / Overview

A single-process Go service is relatively easy. A system of services is not:

  • Instances appear and disappear
  • Networks delay, drop, and duplicate
  • Dependencies degrade partially
  • Retries turn brownouts into outages
  • Queues hide problems until they explode
clients
   |
   v
[ discovery / LB ] -> service A -> service B
                         |            |
                      queue/workers   cache/db

This part focuses on three control planes you will reimplement (or configure) forever:

  1. Where is healthy capacity? (discovery + health)
  2. How do we call others safely? (retry + circuit breaking)
  3. How do we absorb load spikes? (queues + backpressure)

Learning Path

Chapter Focus Outcome
151 Discovery & health Registration, TTL, readiness Route only to safe instances
152 Retry & circuit breaker Control loops Avoid amplification cascades
153 Queues & workers Buffering, concurrency, poison messages Degrade gracefully under overload

Core Mental Models

Partial failure

In distributed systems, “up” is not boolean. An instance can:

  • Accept TCP but fail application readiness
  • Pass health while serving 10× latency
  • Succeed writes but fail reads

Design for degraded modes: serve stale cache, disable non-critical features, fail fast on optional deps.

Amplification

1 user request
  -> 1 API
    -> 3 retries
      -> each fans out to 5 backends
        = 15 backend calls (and more if each retries)

Always multiply retry policy × fan-out × client count.

Backpressure vs buffering

Approach Effect Risk
Drop Protect system Data loss
Queue Smooth spikes Latency + memory
Reject (503) Signal clients Client must handle
Slow down producers True backpressure Needs cooperation

Go channels are in-process queues; they do not replace durable external queues when you need restart survival.

Go Building Blocks

  • context for deadlines across hops
  • net/http clients with tuned transports
  • golang.org/x/sync/errgroup for bounded fan-out
  • sync.Mutex / atomics for in-memory breakers
  • Workers: goroutines + channels or external brokers (NATS, SQS, Kafka, RabbitMQ)
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(8) // Go 1.20+
for _, id := range ids {
    id := id
    g.Go(func() error {
        return fetch(ctx, id)
    })
}
if err := g.Wait(); err != nil {
    return err
}

Production Principles

  1. Timeouts everywhere — no unbounded waits.
  2. Idempotency — safe retries need safe operations.
  3. Explicit queues — size, DLQ, visibility timeout documented.
  4. Health is routing data — not a vanity endpoint.
  5. Observable control state — breaker open, queue depth, retry counts as first-class metrics.
  6. Load test failure — chaos and dependency brownouts in staging.

Relationship to Other Parts

  • Network systems (13) — transport budgets feed retry decisions.
  • Observability (16) — you cannot tune breakers without metrics/traces.
  • Security (17) — identity and mTLS between discovered peers.
  • Performance (18) — worker counts and contention under load.

Literacy Checklist

  • Distinguish liveness vs readiness
  • Explain TTL registration and stale endpoint risk
  • Design retry with jitter and attempt caps
  • Draw circuit breaker states and transitions
  • Size a worker pool against dependency capacity
  • Define poison-message handling and DLQ
  • Compute amplification for a proposed retry policy

Part Warm-Up Exercises

  1. Inventory outbound calls in a service; list timeout and retry for each.
  2. Compute worst-case fan-out for your hottest API.
  3. Find your health endpoints; do they check dependencies?
  4. Measure queue depth (if any) during a past incident.
  5. Write a one-paragraph policy: “when dependency X is down, we …”

Worked Example: Amplification Math

Suppose:

  • 1,000 RPS at the edge
  • Each request fans out to 4 dependencies
  • Each hop retries up to 3 times on failure
  • Failure rate at one dependency is 10%

Rough upper bound if every layer retries naively:

edge RPS × fan-out × attempts
= 1000 × 4 × 3
= 12,000 dependency calls/sec peak intent

During a brownout, effective load can climb further as queues rebuild. This is why attempt caps, breakers, and single-owner retry policy matter more than micro-optimizing JSON.

Decision Guide

Problem First pattern to consider
Clients hardcode hosts Discovery + health
Blips become user errors Retry with jitter (idempotent)
Dependency melt under retries Circuit breaker
Spikes OOM the API process Queue + backpressure / 503
Duplicate side effects Idempotency keys / dedupe store
Poison payload spins forever DLQ + max receive count

What This Part Will Not Cover

  • Consensus algorithms (Raft/Paxos) in depth
  • Exactly-once distributed transactions
  • Full service-mesh product configuration

Those matter, but the everyday Go engineer wins more reliability from timeouts, health, retries, and queues done correctly.

Incident Starter Questions

When pages fire, ask in order:

  1. Is the dependency down, slow, or wrong?
  2. Are we amplifying with retries?
  3. Is discovery pointing at zombies?
  4. Is a queue lagging or a worker stuck?
  5. What do breakers and depths show right now?

If you cannot answer from dashboards, instrument before rewriting architecture.

Reference Architecture (Mental Model)

                  +---------------- discovery / endpoints ---------------+
                  |                                                      |
 clients -> edge LB/proxy -> service instances -> outbound clients        |
                  |                |                    |                |
                  |           readiness             breakers/retries     |
                  |                |                    |                |
                  +------------ metrics/logs/traces (part 16) -----------+
                                       |
                                  async workers <--> queue <--> DLQ

You do not need every box on day one. Add boxes when incidents demand them — but know which box is missing when they do.

Configuration Knobs Inventory

Keep a living document per service:

Knob Example Owner
outbound timeout 300ms service
retry max 2 service
breaker threshold 5 fails / 10s service
worker count 32 service
queue depth alert 10k platform
probe interval 5s platform

Unknown knobs default to dangerous framework defaults.

Testing Strategy

Layer Technique
Unit classify errors; pure policy functions
Integration fake dependency with flaky latency
Chaos kill pods; network delay; certificate expiry
Load sustain + spike; watch depth and p99

Policy without tests regresses the first time someone “simplifies” retries.

Glossary

  • Readiness — safe to receive traffic.
  • TTL registration — auto-expire stale instances.
  • Idempotent — safe to apply twice.
  • Half-open — breaker probing recovery.
  • Backpressure — slowing or rejecting producers.
  • DLQ — dead-letter queue for poison messages.
  • Amplification — retries × fan-out × clients multiplying load.
  • Visibility timeout — lease duration before a queue redelivers.

More examples

Tour: health gate + retry budget

mkdir -p /tmp/go-dist-tour && cd /tmp/go-dist-tour
go mod init example.com/dist-tour

Save as main.go:

package main

import (
    "context"
    "fmt"
    "time"
)

func healthy() bool { return true }

func call(ctx context.Context, n *int) error {
    *n++
    if *n < 2 {
        return fmt.Errorf("transient")
    }
    return ctx.Err()
}

func main() {
    if !healthy() {
        fmt.Println("skip call: not ready")
        return
    }
    ctx, cancel := context.WithTimeout(context.Background(), time.Second)
    defer cancel()
    var n int
    var err error
    for attempt := 1; attempt <= 3; attempt++ {
        err = call(ctx, &n)
        if err == nil {
            break
        }
        time.Sleep(5 * time.Millisecond)
    }
    fmt.Println("attempts:", n, "err:", err)
}
go run .

Expected output:

attempts: 2 err: <nil>

Runnable example

Tour of this part: registry of instances with TTL expiry, a failed call that trips a mini breaker, and a bounded queue reject—three control planes in one process.

mkdir -p /tmp/go-dist-tour && cd /tmp/go-dist-tour
go mod init example.com/dist-tour

Save as main.go:

package main

import (
    "fmt"
    "sync"
    "time"
)

type instance struct {
    addr    string
    expires time.Time
}

type registry struct {
    mu   sync.Mutex
    inst map[string]instance
}

func (r *registry) heartbeat(addr string, ttl time.Duration) {
    r.mu.Lock()
    r.inst[addr] = instance{addr: addr, expires: time.Now().Add(ttl)}
    r.mu.Unlock()
}

func (r *registry) healthy() []string {
    r.mu.Lock()
    defer r.mu.Unlock()
    now := time.Now()
    var out []string
    for a, i := range r.inst {
        if now.Before(i.expires) {
            out = append(out, a)
        } else {
            delete(r.inst, a)
        }
    }
    return out
}

type breaker struct {
    fails int
    open  bool
}

func (b *breaker) call(fn func() error) error {
    if b.open {
        return fmt.Errorf("breaker open")
    }
    if err := fn(); err != nil {
        b.fails++
        if b.fails >= 3 {
            b.open = true
        }
        return err
    }
    b.fails = 0
    return nil
}

func main() {
    reg := &registry{inst: map[string]instance{}}
    reg.heartbeat("10.0.0.1:8080", 50*time.Millisecond)
    reg.heartbeat("10.0.0.2:8080", time.Second)
    time.Sleep(60 * time.Millisecond)
    fmt.Println("healthy:", reg.healthy())

    var b breaker
    for i := 0; i < 4; i++ {
        err := b.call(func() error { return fmt.Errorf("dep down") })
        fmt.Printf("call %d: %v open=%v\n", i+1, err, b.open)
    }

    q := make(chan string, 2)
    send := func(s string) error {
        select {
        case q <- s:
            return nil
        default:
            return fmt.Errorf("queue full")
        }
    }
    fmt.Println("enq a:", send("a"))
    fmt.Println("enq b:", send("b"))
    fmt.Println("enq c:", send("c"))
    fmt.Println("tour: discovery + breaker + backpressure")
}
go run .

Expected output:

healthy: [10.0.0.2:8080]
call 1: dep down open=false
call 2: dep down open=false
call 3: dep down open=true
call 4: breaker open open=true
enq a: <nil>
enq b: <nil>
enq c: queue full
tour: discovery + breaker + backpressure

What to notice

  • TTL discovery expires zombies; breakers stop retry storms; bounded queues apply backpressure.
  • Chapters 151–153 deepen each control plane.

Try next

  • Add half-open probe recovery to the breaker.
  • Drain the queue with N workers and measure lag.

Next Chapter

Service Discovery and Health Checks — knowing who is safe to call.