Project: Resilient Worker

Project: Resilient Worker

Overview

Build a worker that pulls jobs from an in-memory (then Redis/SQS) queue with: retries, backoff, poison handling, concurrency cap, and graceful shutdown.

Features

Feature	Spec
Concurrency	-w 8 workers
Retry	max 5, exponential backoff
Poison	after max → dead-letter slice/log
Shutdown	finish current job or requeue
Metrics	processed, failed, dlq counters

Interface

type Job struct {
    ID      string
    Payload []byte
    Attempt int
}

type Queue interface {
    Receive(ctx context.Context) (Job, error)
    Ack(ctx context.Context, id string) error
    Retry(ctx context.Context, j Job, after time.Duration) error
    DeadLetter(ctx context.Context, j Job, reason string) error
}

Loop

for i := 0; i < workers; i++ {
    go func() {
        for {
            j, err := q.Receive(ctx)
            if err != nil { return }
            if err := handle(ctx, j); err != nil {
                if j.Attempt+1 >= max {
                    _ = q.DeadLetter(ctx, j, err.Error())
                } else {
                    j.Attempt++
                    _ = q.Retry(ctx, j, backoff(j.Attempt))
                }
                continue
            }
            _ = q.Ack(ctx, j.ID)
        }
    }()
}

Acceptance

• Inject fail-N-times job; succeeds after retries
  
• Poison goes to DLQ
  
• SIGTERM stops receiving; no new jobs
  
• Race test clean

Connects chapters 152–153 + 155.