Idempotency and Deduplication

Idempotency and Deduplication

Overview

Networks retry. Clients double-submit. Without idempotency, you charge twice or create duplicate rows. Key idea: same logical request → same effect.

Idempotency-Key header

POST /payments
Idempotency-Key: 9f3c...

Server stores key → response (or in-flight marker) with TTL.

type store interface {
    Get(ctx context.Context, key string) (status int, body []byte, ok bool, err error)
    Put(ctx context.Context, key string, status int, body []byte, ttl time.Duration) error
}

func withIdempotency(s store, next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if r.Method != http.MethodPost {
            next.ServeHTTP(w, r)
            return
        }
        key := r.Header.Get("Idempotency-Key")
        if key == "" {
            http.Error(w, "missing Idempotency-Key", http.StatusBadRequest)
            return
        }
        if st, body, ok, err := s.Get(r.Context(), key); err == nil && ok {
            w.WriteHeader(st)
            _, _ = w.Write(body)
            return
        }
        // capture response → Put (use response recorder middleware)
        next.ServeHTTP(w, r)
    })
}

DB unique constraints

Natural idempotency: UNIQUE(idempotency_key) or UNIQUE(user_id, client_request_id).

INSERT ... ON CONFLICT (idempotency_key) DO NOTHING

At-least-once consumers

Queue workers: store processed message_id; skip duplicates.

Rules of thumb

Do	Don’t
Require keys on money POSTs	Retry POSTs blindly without keys
TTL the key store	Infinite growth of keys
Prefer DB uniqueness	Only in-memory dedupe multi-instance

Try next

1. Memory idempotency store + double POST test.
   
2. Unique index migration for order client IDs.
   
3. Worker skips duplicate delivery.