Production Emergency Scenarios

Real Production Emergency Scenarios

After this page you have a short playbook for “production is wrong and Git is how we talk about it.”

Assume Linux, ~/desk/desk-api, and a main that is already on a remote. You are in a terminal, not a GUI.

 published on origin/main?
    │
    ├─ no   ──►  reset / amend / rebase     (rewrite)
    │
    └─ yes  ──►  revert or cherry-pick fix  (forward)
                    │
                    └─ secret in history ──► rotate now; filter-repo later

1. Bad commit is HEAD on main, already pulled by others

git switch main
git pull --ff-only
git revert HEAD --no-edit
# run tests
git push

2. Bad commit is in the middle of main

git log --oneline -20
git revert --no-edit <hash>
# tests
git push

If several commits are one logical change, revert the range newest-first, or revert the merge with -m 1.

3. Fix exists on a feature branch, production cannot wait

git log feat/whatever --oneline
git switch main
git cherry-pick -x <fix-hash>
# tests
git push

Leave the feature branch as-is. The pick is a new hash on main.

4. You committed a secret

git revert does not remove the secret from history; it only removes it from HEAD. Anyone who fetched still has the blob. Rotate the secret at the provider. History cleanup is chapter 20 (git filter-repo). For the next five minutes: revert or a follow-up commit that deletes the file, rotate, tell the team.

5. Merge to main was wrong, merge commit is HEAD, not pushed

git reset --hard HEAD~1

If it was pushed, revert with -m 1 instead.

6. You are not sure what main is

git status
git log --oneline --graph --decorate -15
git reflog -10

Status, graph, reflog. Then pick revert (shared) or reset (local only).

Write the team a one-line message: hash reverted, why, who is rotating secrets if any. Git is the record; the pager is the people.