Secrets Management and Rotation

Updated

July 30, 2026

Secrets Management and Rotation

Secrets are dynamic credentials with a lifecycle, not static configuration strings. Most secret-related outages come from rotation choreography errors, not from encryption algorithm choice.

Why / Overview

Secrets include:

  • Database passwords
  • API tokens and HMAC keys
  • TLS private keys
  • OAuth client secrets
  • Encryption keys (DEK/KEK hierarchy)
issue new secret -> readers accept both -> writers switch -> revoke old

If you flip both ends at once without overlap, you race yourself into downtime.

Never Do This

Anti-pattern Why
Secrets in git History is forever; bots scan public repos
Secrets in Docker layers Recoverable from images
Secrets in client apps Extractable
Long-lived shared passwords Huge blast radius
Logs printing env CI and support tools leak

Loading Secrets at Runtime

Environment variables

Common in 12-factor apps; simple but:

  • Visible in /proc and some debug endpoints
  • Easy to dump accidentally
  • Awkward dual-read of two values (use DB_PASSWORD + DB_PASSWORD_OLD)
func mustSecret(key string) string {
    v := os.Getenv(key)
    if v == "" {
        slog.Error("missing secret", "key", key)
        os.Exit(1)
    }
    return v
}

Files (Kubernetes secrets mounts)

func readSecretFile(path string) (string, error) {
    b, err := os.ReadFile(path)
    if err != nil {
        return "", err
    }
    return strings.TrimSpace(string(b)), nil
}

Many platforms rotate by updating the file; processes must re-read or restart.

Secret managers

AWS SM, GCP SM, Azure Key Vault, HashiCorp Vault, etc.:

  • Short-lived credentials via IAM roles
  • Central audit
  • API-driven rotation

Wrap the vendor SDK behind a small interface:

type Source interface {
    Get(ctx context.Context, name string) (string, error)
}

Fail Closed at Startup

func run() error {
    cfg, err := loadConfig()
    if err != nil {
        return err
    }
    if cfg.DBPassword == "" {
        return errors.New("db password required")
    }
    // open DB, then listen
    return serve(cfg)
}

Do not listen on :8080 and return 500 for all traffic because secrets failed — fail before ready.

Dual-Read Rotation (Passwords / HMAC)

Time ->
DB password:   [ v1 active ][ v1+v2 accepted ][ v2 active ]
App readers:   [ v1        ][ try v2 then v1 ][ v2        ]
App writers:   [ v1        ][ switch to v2   ][ v2        ]
type RotatingToken struct {
    mu   sync.RWMutex
    cur  []byte
    prev []byte
}

func (r *RotatingToken) Verify(mac, msg []byte) bool {
    r.mu.RLock()
    defer r.mu.RUnlock()
    if hmac.Equal(mac, sum(r.cur, msg)) {
        return true
    }
    if len(r.prev) > 0 && hmac.Equal(mac, sum(r.prev, msg)) {
        return true
    }
    return false
}

func (r *RotatingToken) Rotate(newKey []byte) {
    r.mu.Lock()
    defer r.mu.Unlock()
    r.prev = r.cur
    r.cur = newKey
}

For DB users, prefer two users or two passwords supported by the server during cutover.

Runtime Refresh Without Restart

func (a *App) watchSecrets(ctx context.Context, every time.Duration) {
    t := time.NewTicker(every)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case <-t.C:
            sec, err := a.src.Get(ctx, "db-password")
            if err != nil {
                slog.Error("secret refresh failed", "err", err)
                continue // keep old; alert on repeated failure
            }
            if err := a.db.UpdatePassword(sec); err != nil {
                slog.Error("db rebind failed", "err", err)
            }
        }
    }
}

Some drivers need a new pool to pick up password changes — test the path.

Encryption Keys vs Passwords

For application-level encryption:

  • Use a KEK in a KMS to wrap DEKs
  • Store wrapped DEKs next to ciphertext
  • Rotate KEK via rewrap; rotate DEK by re-encrypting data in batches

Do not invent AES modes; use crypto/cipher with GCM properly (unique nonces) or higher-level libraries.

func encrypt(key, plaintext []byte) ([]byte, error) {
    block, err := aes.NewCipher(key)
    if err != nil {
        return nil, err
    }
    gcm, err := cipher.NewGCM(block)
    if err != nil {
        return nil, err
    }
    nonce := make([]byte, gcm.NonceSize())
    if _, err := rand.Read(nonce); err != nil {
        return nil, err
    }
    return gcm.Seal(nonce, nonce, plaintext, nil), nil
}

CI/CD and Build-Time Leaks

  • Inject secrets at deploy time, not build time, when possible
  • Mask secrets in CI logs
  • Prefer OIDC federated login to clouds over long-lived CI keys
  • Scan releases with secret scanners

Redaction Everywhere

type RedactedString string

func (r RedactedString) String() string { return "***" }
func (r RedactedString) GoString() string { return "***" }
func (r RedactedString) MarshalJSON() ([]byte, error) { return []byte(`"***"`), nil }

Use carefully for types that must not appear in logs.

Production Checklist

  • No secrets in git or images
  • Required secrets fail startup (not silent empty)
  • Dual-read/dual-trust windows for rotation
  • Automated rotation where platform supports it
  • Refresh or restart path tested quarterly
  • Access audit on secret manager
  • Least-privilege IAM for secret read
  • Alert on refresh failures
  • Revocation plan for leaked credentials
  • TLS keys use restricted file perms / KMS

Common Pitfalls

  1. Single global static secret for all environments.
  2. Rotation without dual-read → full outage.
  3. Copy-paste secrets into tickets and Slack.
  4. Same DB password for migrate job and app with different cutover times.
  5. Not revoking after suspected leak (only rotating sometimes).
  6. Base64 ≠ encryption for “hiding” secrets in config.
  7. Long-lived JWTs as a substitute for session server state without revoke list.

Exercises

  1. Refactor a sample app to fail closed when API_TOKEN is missing.
  2. Implement dual-key HMAC verify; rotate; prove old and new both verify during window.
  3. Mount a secret file; update it; implement re-read; confirm new value used.
  4. Add RedactedString; log a config struct; ensure secret not visible.
  5. Simulate leak: document revoke + rotate steps as a runbook.
  6. Compare env vs file vs Vault for your deployment model (table of tradeoffs).
  7. Create two DB users; rotate app from A to B without downtime (local docker).
  8. Scan a docker image history for accidental .env copy; fix Dockerfile.
  9. Instrument secret_refresh_failures_total; alert on >0 for 15m.
  10. Design DEK/KEK scheme for storing personal tokens at rest; review nonce handling.

More examples

Dual-secret accept during rotation

mkdir -p /tmp/go-secret-dual && cd /tmp/go-secret-dual
go mod init example.com/secret-dual

Save as main.go:

package main

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
)

func sign(secret, msg string) string {
    m := hmac.New(sha256.New, []byte(secret))
    _, _ = m.Write([]byte(msg))
    return hex.EncodeToString(m.Sum(nil))
}

func valid(secrets []string, msg, sig string) bool {
    for _, s := range secrets {
        want, err1 := hex.DecodeString(sign(s, msg))
        got, err2 := hex.DecodeString(sig)
        if err1 != nil || err2 != nil {
            continue
        }
        if hmac.Equal(want, got) {
            return true
        }
    }
    return false
}

func main() {
    old, neu := "old-secret", "new-secret"
    msg := "payload"
    // After rotation: sign with new, accept both for a window.
    sig := sign(neu, msg)
    fmt.Println("new only:", valid([]string{neu}, msg, sig))
    fmt.Println("dual window:", valid([]string{neu, old}, msg, sig))
    fmt.Println("old sig still ok in window:", valid([]string{neu, old}, msg, sign(old, msg)))
    fmt.Println("after cutover old rejected:", valid([]string{neu}, msg, sign(old, msg)))
}
go run .

Expected output:

new only: true
dual window: true
old sig still ok in window: true
after cutover old rejected: false

Load secret from file with restrictive mode check

mkdir -p /tmp/go-secret-file && cd /tmp/go-secret-file
go mod init example.com/secret-file

Save as main.go:

package main

import (
    "fmt"
    "os"
)

func loadSecret(path string) (string, error) {
    st, err := os.Stat(path)
    if err != nil {
        return "", err
    }
    if st.Mode().Perm()&0o077 != 0 {
        return "", fmt.Errorf("secret %s mode too open: %v", path, st.Mode().Perm())
    }
    b, err := os.ReadFile(path)
    if err != nil {
        return "", err
    }
    return string(b), nil
}

func main() {
    path := "secret.txt"
    _ = os.WriteFile(path, []byte("s3cr3t"), 0o600)
    defer os.Remove(path)
    s, err := loadSecret(path)
    fmt.Println("ok:", s, err)

    _ = os.Chmod(path, 0o644)
    _, err = loadSecret(path)
    fmt.Println("open mode rejected:", err != nil)
}
go run .

Expected output:

ok: s3cr3t <nil>
open mode rejected: true

Runnable example

Fail-closed env loading, dual-key HMAC verify during rotation, and a RedactedString that never prints the secret.

mkdir -p /tmp/go-secrets && cd /tmp/go-secrets
go mod init example.com/secrets

Save as main.go:

package main

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "os"
)

type RedactedString string

func (r RedactedString) String() string { return "***" }
func (r RedactedString) GoString() string { return "RedactedString(***)" }
func (r RedactedString) Reveal() string   { return string(r) }

func mustEnv(k string) string {
    v := os.Getenv(k)
    if v == "" {
        panic("missing required secret: " + k)
    }
    return v
}

func sign(key, msg string) string {
    m := hmac.New(sha256.New, []byte(key))
    _, _ = m.Write([]byte(msg))
    return hex.EncodeToString(m.Sum(nil))
}

func verify(keys []string, msg, sig string) bool {
    raw, err := hex.DecodeString(sig)
    if err != nil {
        return false
    }
    for _, k := range keys {
        mac := hmac.New(sha256.New, []byte(k))
        _, _ = mac.Write([]byte(msg))
        if hmac.Equal(raw, mac.Sum(nil)) {
            return true
        }
    }
    return false
}

func main() {
    // Demo defaults when unset (production: mustEnv only).
    primary := os.Getenv("API_TOKEN_CURRENT")
    if primary == "" {
        primary = "current-secret"
    }
    previous := os.Getenv("API_TOKEN_PREVIOUS") // may be empty after rotation window

    secret := RedactedString(primary)
    fmt.Printf("config log: token=%s\n", secret) // *** 

    msg := "payload"
    sigOld := sign("previous-secret", msg)
    sigNew := sign(primary, msg)

    keys := []string{primary}
    if previous != "" {
        keys = append(keys, previous)
    } else {
        keys = append(keys, "previous-secret") // demo dual-read window
    }
    fmt.Println("accept new:", verify(keys, msg, sigNew))
    fmt.Println("accept old during dual-read:", verify(keys, msg, sigOld))
    fmt.Println("reject other:", verify(keys, msg, sign("other", msg)))

    // Fail-closed illustration
    if os.Getenv("REQUIRE_STRICT") == "1" {
        _ = mustEnv("API_TOKEN_CURRENT")
    } else {
        fmt.Println("strict mode off (set REQUIRE_STRICT=1 to exercise mustEnv)")
    }
}
go run .
REQUIRE_STRICT=1 API_TOKEN_CURRENT=abc go run .

Expected output:

config log: token=***
accept new: true
accept old during dual-read: true
reject other: false
strict mode off (set REQUIRE_STRICT=1 to exercise mustEnv)

What to notice

  • Dual-read (current + previous) avoids total outage at rotation cutover.
  • Custom String() prevents accidental secret leakage via fmt and slog.
  • Missing required secrets should prevent process start, not run with empty keys.

Try next

  • Re-read a secret file on SIGHUP or timer; atomic swap of the active key slice.
  • Metric secret_refresh_failures_total when reload fails.

Further Reading

  • NIST guidance on secrets management (high level)
  • Previous: authn/authz that use these secrets
  • Next part: Performance Engineering — speed with evidence, not secrets of folklore