Leader Election Basics

Leader Election Basics

Overview

Sometimes exactly one instance should run a loop (compactions, cron, singleton consumer). Leader election picks that instance. Keep it simple: lease in a database/etcd/K8s lock—not custom Paxos in app code.

Lease pattern (DB)

UPDATE leases SET owner=?, expires=?
 WHERE name=? AND (expires < now() OR owner=?)

Winner renews before expiry; losers idle and retry.

type LeaseStore interface {
    TryAcquire(ctx context.Context, name, owner string, ttl time.Duration) (bool, error)
    Renew(ctx context.Context, name, owner string, ttl time.Duration) (bool, error)
    Release(ctx context.Context, name, owner string) error
}

func RunAsLeader(ctx context.Context, s LeaseStore, name, owner string, ttl time.Duration, work func(context.Context) error) error {
    t := time.NewTicker(ttl / 2)
    defer t.Stop()
    for {
        ok, err := s.TryAcquire(ctx, name, owner, ttl)
        if err != nil {
            return err
        }
        if !ok {
            select {
            case <-ctx.Done():
                return ctx.Err()
            case <-time.After(ttl / 2):
                continue
            }
        }
        // lead
        wctx, cancel := context.WithCancel(ctx)
        errCh := make(chan error, 1)
        go func() { errCh <- work(wctx) }()
        for {
            select {
            case err := <-errCh:
                cancel()
                _ = s.Release(ctx, name, owner)
                return err
            case <-t.C:
                ok, err := s.Renew(ctx, name, owner, ttl)
                if err != nil || !ok {
                    cancel()
                    <-errCh
                    break
                }
            case <-ctx.Done():
                cancel()
                _ = s.Release(context.Background(), name, owner)
                return ctx.Err()
            }
        }
    }
}

Kubernetes

Lease objects / leader-election libraries (client-go tools) — prefer platform primitives in K8s.

Rules

Do	Don’t
TTL + renew	Infinite leadership without heartbeat
Work cancel on loss	Continue writes after lease loss
Idempotent leader work	Assume exactly-once without more design

Try next

1. Memory lease store + two goroutines racing.
   
2. Kill leader; follower takes over within TTL.
   
3. Ensure work stops on lease loss.