Day 86 — Conditional probability & Bayes

Updated

July 30, 2026

Day 86 — Conditional probability, chain rule, and Bayes

Stage VIII · concept day
Goal: Define \(P(A\mid B)\); chain rule; Bayes theorem; total probability; base-rate fallacy examples; independence tests via conditionals.

Why this matters

Conditional probability is how evidence updates beliefs. Spam filters, medical tests, debugging with logs, and “given that the hash collided…” are conditional statements. Base-rate fallacies are among the most common quantitative mistakes in technical life.

Note

No labs. Hand computations with fractions; word problems as math, not software.

Theory

Conditional probability

Definition. If \(P(B)>0\), the conditional probability of \(A\) given \(B\) is \[ P(A\mid B)=\frac{P(A\cap B)}{P(B)}. \]

Interpretation. Restrict the sample space to \(B\), renormalize.

Immediate: \(P(B\mid B)=1\); \(P(A\cap B)=P(A\mid B)P(B)\).

Chain rule

\[ P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A). \] For three events: \[ P(A\cap B\cap C)=P(A)\,P(B\mid A)\,P(C\mid A\cap B) \] (when the intermediate conditionings are defined).

Law of total probability

If \(B_1,\ldots,B_k\) partition \(\Omega\) (pairwise disjoint, union \(\Omega\)) and \(P(B_i)>0\), then for any event \(A\), \[ P(A)=\sum_{i=1}^k P(A\mid B_i)P(B_i). \] Tree diagrams: branch on partition, multiply along paths, sum paths into \(A\).

Bayes’ theorem

Theorem (Bayes). If \(P(A)>0\) and \(P(B)>0\), \[ P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}. \] With total probability in the denominator for a partition \(\{A_i\}\): \[ P(A_j\mid B)=\frac{P(B\mid A_j)P(A_j)}{\sum_i P(B\mid A_i)P(A_i)}. \]

Language. \(P(A)\) prior; \(P(A\mid B)\) posterior; \(P(B\mid A)\) likelihood.

Independence via conditionals

Events \(A,B\) with \(P(B)>0\) are independent iff \(P(A\mid B)=P(A)\) (equivalently \(P(B\mid A)=P(B)\) when \(P(A)>0\)).
This matches \(P(A\cap B)=P(A)P(B)\).

Base-rate fallacy

People confuse \(P(\text{disease}\mid \text{positive test})\) with \(P(\text{positive}\mid \text{disease})\), ignoring the base rate \(P(\text{disease})\).

Canonical setup. Rare disease with prevalence \(p\) small; test sensitivity \(P(+|D)=s\) high; false positive rate \(P(+|D^c)=f\). Then \[ P(D\mid +)=\frac{s p}{s p + f(1-p)}, \] which can be modest even when \(s\approx 1\) if \(p\ll f\).

Multiple updates (idea)

Posterior becomes prior for the next independent piece of evidence (when conditional independence structures allow). Full graphical models are out of scope; the algebra of one Bayes update is enough.

Confusion matrix literacy (tests)

\(D\) \(D^c\)
\(+\) TP FP
\(-\) FN TN

Sensitivity \(=TP/(TP+FN)\); specificity \(=TN/(TN+FP)\); false positive rate \(=1-\text{specificity}\). Bayes uses these with prevalence.

Worked examples

Example 1 — Cards.
Draw one card. \(P(\text{Ace}\mid \text{spade})=\frac{1/52}{13/52}=1/13\). Same as unconditional Ace rate — suit and rank independent in a fair deck for these events.

Example 2 — Two children.
Classic care needed: define \(\Omega\) carefully. “At least one boy” vs “first is boy” change conditionals.

Example 3 — Total probability.
Factory: lines \(B_1,B_2\) make \(60\%,40\%\) of items; defect rates \(1\%,3\%\).
\(P(\text{defect})=0.01\cdot 0.6+0.03\cdot 0.4=0.018\).

Example 4 — Bayes on defects.
\(P(B_1\mid\text{defect})=\frac{0.01\cdot 0.6}{0.018}=\frac{0.006}{0.018}=\frac13\).

Example 5 — Medical base rate.
Prevalence \(p=0.001\), sensitivity \(0.99\), false positive rate \(0.01\).
\(P(D\mid +)=\frac{0.99\cdot 0.001}{0.99\cdot 0.001+0.01\cdot 0.999}=\frac{0.00099}{0.00099+0.00999}=\frac{0.00099}{0.01098}\approx 0.09\).
Only about \(9\%\) despite a “\(99\%\) accurate” slogan (ambiguous slogan!).

Example 6 — Independence check.
Two fair coins. \(A=\) first H, \(B=\) second H. \(P(A\mid B)=1/2=P(A)\).

Example 7 — Die.
\(P(\text{six}\mid \text{even})=P(\{6\})/P(\{2,4,6\})=(1/6)/(1/2)=1/3\).

Example 8 — Chain rule.
\(P(A\cap B\cap C)\) for sequential draws without replacement: multiply successive conditionals.

Example 9 — Spam literacy.
Prior \(P(\text{spam})\), likelihood of word given spam vs ham — Bayes filter idea (not building one).

Example 10 — False intuition.
“Test is \(95\%\) accurate” is ambiguous: does it mean sensitivity, specificity, or PPV \(P(D\mid +)\)? Always expand into conditionals.

Exercises

A. Definition and chain rule

  1. From the definition, prove \(P(A\cap B)=P(A\mid B)P(B)\).
  2. Die: compute \(P(\text{odd}\mid \ge 4)\).
  3. Two fair dice: \(P(\text{sum}=7\mid \text{first die shows }3)\).
  4. Urn: \(3\) red, \(2\) blue; draw two without replacement. \(P(\text{second red}\mid \text{first red})\).
  5. Write the three-event chain rule and use it on cards drawn without replacement for three aces in a row (setup).

B. Total probability

  1. Rework Example 3 with defect rates \(2\%\) and \(5\%\), production \(70\%/30\%\).
  2. Three machines equal output, defect rates \(1\%,2\%,3\%\). Find \(P(\text{defect})\).
  3. Prove the law of total probability from additivity and definition of conditional prob.
  4. Tree diagram: first coin, then if H roll die “success on 6”, if T success automatic — find \(P(\text{success})\).

C. Bayes

  1. Complete Example 5 with \(p=0.01\) instead of \(0.001\); compare.
  2. Factory Example 4 variant: find \(P(B_2\mid\text{defect})\).
  3. Two types of cabs: \(85\%\) of cabs black, \(15\%\) blue. Witness says blue, \(80\%\) reliable both ways. Find \(P(\text{cab blue}\mid \text{witness says blue})\).
  4. State Bayes and identify prior, likelihood, posterior in problem 12.
  5. Prove Bayes from the chain rule.

D. Independence and fallacies

  1. Show \(A\perp B\) iff \(P(A\mid B)=P(A)\) when \(P(B)>0\).
  2. Explain the base-rate fallacy in 5–8 sentences with numbers from Example 5.
  3. Invent a CS false-positive scenario (intrusion alerts) with base rates; compute PPV.
  4. Are “sum is even” and “first die is \(1\)” independent for two dice? Check.

E. Stretch

  1. Screening: sensitivity \(0.95\), specificity \(0.95\), prevalence \(0.005\). Compute \(P(D\mid +)\) and \(P(D^c\mid -)\).
  2. Show that if \(P(A\mid B)>P(A)\) then \(P(B\mid A)>P(B)\) (when defined) — evidence is symmetric in a sense.
  3. Conditional independence idea: \(P(A\cap B\mid C)=P(A\mid C)P(B\mid C)\). Give a verbal example.
  4. Simpson’s paradox awareness: one paragraph that aggregation can reverse conditionals — no full proof required.

Deep dive — Bayes computational pattern

1. Name hypotheses H1..Hk (partition) and evidence E
2. Write priors P(Hi)
3. Write likelihoods P(E|Hi)
4. Compute P(E) = sum_i P(E|Hi)P(Hi)
5. Posterior P(Hj|E) = P(E|Hj)P(Hj) / P(E)
6. Sanity: posteriors sum to 1

Natural frequencies (optional pedagogy): “1000 people; 1 diseased; …” often reduces base-rate errors versus pure decimals.

Deep dive — medical numbers template

Symbol Meaning
\(p\) prevalence \(P(D)\)
\(s\) sensitivity \(P(+|D)\)
\(f\) false positive rate \(P(+|D^c)\)
PPV \(P(D|+)=sp/(sp+f(1-p))\)
NPV \(P(D^c|-)\) via similar algebra

Additional worked examples

Example 11 — Natural frequencies.
\(1000\) people, \(p=0.001\) ⇒ \(1\) diseased. Sensitivity \(0.99\) ⇒ \(\approx 1\) true positive. \(999\) healthy, \(f=0.01\) ⇒ \(\approx 10\) false positives. PPV \(\approx 1/11\approx 9\%\).

Example 12 — Two-die conditional.
\(P(\text{sum}=7\mid \text{first}=2)=P(\text{second}=5)=1/6\).

Example 13 — Without replacement.
Urn \(5\) red \(5\) blue; \(P(\text{2nd red}\mid \text{1st blue})=5/9\).

Example 14 — Independence failure.
Die: \(A=\{\le 2\}\), \(B=\{\text{even}\}\). \(P(A\cap B)=P(\{2\})=1/6\), \(P(A)P(B)=(1/3)(1/2)=1/6\) — independent. Change \(B\) to \(\{\le 3\}\) and recompute.

Example 15 — Spam idea.
Prior \(P(S)=0.4\), \(P(W|S)=0.3\), \(P(W|H)=0.05\).
\(P(S|W)=\frac{0.3\cdot 0.4}{0.3\cdot 0.4+0.05\cdot 0.6}=\frac{0.12}{0.12+0.03}=0.8\).

More exercises

  1. Prevalence \(0.02\), sensitivity \(0.9\), FPR \(0.05\): compute PPV.
  2. Prove \(P(A\cap B\cap C)=P(A)P(B|A)P(C|A\cap B)\).
  3. Cab problem variant: \(70\%\) green cabs; witness \(75\%\) accurate; posterior green given “green.”
  4. Show \(P(A|B)>P(A)\) implies \(P(B|A)>P(B)\) when all defined.
  5. CS: IDS alerts with base rate of true attacks \(10^{-4}\); FPR \(10^{-3}\); compute PPV order of magnitude.
  6. Total probability with \(4\) machines equal share, defect rates \(1,2,3,4\%\).
  7. Conditional of complement: express \(P(A^c|B)\) in terms of \(P(A|B)\).
  8. Explain in prose why \(P(\text{positive}|\text{disease})\) is not “the chance I’m sick if positive.”

Selected mini-solutions

  1. \(P(\text{odd}\mid\ge 4)=P(\{5\})/P(\{4,5,6\})=(1/6)/(1/2)=1/3\).
  2. \(p=0.01\): PPV \(=\frac{0.99\cdot 0.01}{0.99\cdot 0.01+0.01\cdot 0.99}=0.5\).
  3. Classic: posterior blue often \(\approx 0.41\) with standard numbers — recompute carefully.
  4. Direct from definition and \(P(A\cap B)=P(A)P(B)\).

Common pitfalls

Pitfall Fix
Swapping \(P(A\mid B)\) and \(P(B\mid A)\) Write definitions; Bayes to convert
Ignoring base rates Always include \(P(A)\) in Bayes
Conditioning on probability-zero events Require \(P(B)>0\)
Assuming independence to multiply Check or justify
“\(99\%\) accurate” without defining Expand to sensitivity/specificity/PPV
Forgetting partition completeness Priors must sum to 1

Study notes — conditional essentials

Name Formula
Conditional \(P(A\mid B)=P(A\cap B)/P(B)\)
Chain \(P(A\cap B)=P(A\mid B)P(B)\)
Total prob \(P(A)=\sum P(A\mid B_i)P(B_i)\)
Bayes \(P(A\mid B)=P(B\mid A)P(A)/P(B)\)
PPV \(sp/(sp+f(1-p))\)

Fallacy: do not confuse \(P(+|D)\) with \(P(D|+)\).

Sanity: posteriors over a partition given evidence \(E\) must sum to \(1\).

Synthesis — Bayes workout

S1. Definition drill: compute \(P(A\mid B)\) for die events \(A=\{\le 2\}\), \(B=\{\text{even}\}\).

S2. Total probability: two lines \(70\%/30\%\), defect \(1\%/4\%\) — \(P(\text{defect})\).

S3. Bayes on S2: \(P(\text{line 1}\mid\text{defect})\).

S4. Medical: \(p=0.005\), \(s=0.98\), \(f=0.02\) — PPV.

S5. Natural frequency rewrite of S4 with \(10{,}000\) people.

S6. Prove Bayes from chain rule in three lines.

S7. Base-rate fallacy: write a wrong “intuitive” answer then correct it.

S8. CS IDS: attack rate \(10^{-5}\), detection \(0.9\), FPR \(10^{-4}\) — order of PPV.

Checkpoint

  • Can compute \(P(A\mid B)\) from definition
  • Can use total probability (partition)
  • Can apply Bayes with a clear prior
  • Can explain base-rate fallacy with numbers
  • Can test independence via conditionals
  • Exercises A–D done; synthesis attempted

Two personal takeaways:

  1. …
  2. …

Deeper theory — Bayes as reweighting priors

Start with prior masses \(P(H_i)\) on a partition. Evidence \(E\) multiplies each hypothesis by its likelihood \(P(E\mid H_i)\) and renormalizes: \[ P(H_j\mid E)=\frac{P(E\mid H_j)P(H_j)}{\sum_i P(E\mid H_i)P(H_i)}. \] No mysticism: this is definition of conditional probability plus total probability in the denominator. The “update” language is interpretation.

Odds form (optional).
\[ \frac{P(H\mid E)}{P(H^c\mid E)}=\frac{P(E\mid H)}{P(E\mid H^c)}\cdot\frac{P(H)}{P(H^c)}. \] Posterior odds \(=\) likelihood ratio \(\times\) prior odds. Useful for quick mental Bayes.

Deeper theory — base rates with natural frequencies

Population \(N\) Diseased \(Np\) Healthy \(N(1-p)\)
Test \(+\) \(\approx Nps\) true \(+\) \(\approx N(1-p)f\) false \(+\)
Test \(-\) false \(-\) true \(-\)

PPV \(\approx\) (true \(+\))/(true \(+\) + false \(+\)). Same algebra as Bayes; fewer decimal slips.

Worked examples — full Bayes numbers

Example 16 — Cab problem standard numbers.
Prior blue \(15\%\), black \(85\%\). Witness says blue; accuracy \(80\%\) both ways: \(P(\text{says blue}\mid\text{blue})=0.8\), \(P(\text{says blue}\mid\text{black})=0.2\).
\(P(\text{says blue})=0.8\cdot 0.15+0.2\cdot 0.85=0.12+0.17=0.29\).
\(P(\text{blue}\mid\text{says blue})=0.12/0.29\approx 0.414\).

Example 17 — Two-step tree.
Machine \(A\) makes \(70\%\) of parts, defect \(2\%\); machine \(B\) \(30\%\), defect \(5\%\).
\(P(D)=0.014+0.015=0.029\).
\(P(A\mid D)=0.014/0.029=14/29\).
\(P(B\mid D)=15/29\).

Example 18 — Without replacement chain.
Deck: \(P(\text{two aces in two draws})=P(A_1)P(A_2\mid A_1)=\frac{4}{52}\cdot\frac{3}{51}=\frac{12}{2652}=\frac{1}{221}\).

Example 19 — Independence via conditional.
Two dice: \(A=\) first is \(6\), \(B=\) sum is \(7\). \(P(B\mid A)=P(\text{second }=1)=1/6=P(B)\). Independent.
\(A=\) first is \(6\), \(B=\) sum is \(12\): \(P(B\mid A)=P(\text{second }=6)=1/6\), \(P(B)=1/36\) — not independent.

Example 20 — IDS order of magnitude.
Attack rate \(p=10^{-4}\), detection \(s=0.9\), FPR \(f=10^{-3}\).
PPV \(\approx (0.9\cdot 10^{-4})/(0.9\cdot 10^{-4}+10^{-3})\approx 0.00009/0.00109\approx 0.083\) — about \(8\%\) of alerts are real.

Extra exercises — Bayes mastery

  1. Prevalence \(0.002\), sensitivity \(0.95\), FPR \(0.01\): PPV and natural-frequency version with \(N=10000\).
  2. Prove the odds form of Bayes for binary \(H\).
  3. Three hypotheses equal prior, likelihoods of \(E\) are \(0.1,0.2,0.3\) — posteriors?
  4. Show \(P(A\mid B)+P(A^c\mid B)=1\).
  5. Medical: compute NPV \(P(D^c\mid -)\) with \(p=0.01\), \(s=0.9\), specificity \(0.95\).
  6. CS spam: prior \(0.3\), \(P(W|S)=0.5\), \(P(W|H)=0.02\) — \(P(S|W)\).
  7. Explain in 6 sentences a base-rate failure in hiring tests or security alerts.
  8. Chain rule for four events (write the formula).

Mini-solutions (selected)

  1. \(sp=0.0019\), \(f(1-p)\approx 0.01\), PPV \(\approx 0.0019/0.0119\approx 0.16\).
  2. Unnormalized \(0.1,0.2,0.3\) sum \(0.6\); posteriors \(1/6,1/3,1/2\).
  3. \(0.5\cdot 0.3/(0.15+0.02\cdot 0.7)=0.15/0.164\approx 0.915\).

Closing synthesis card

Name Role
\(P(A\mid B)\) renormalize to \(B\)
Total probability law of alternatives
Bayes invert likelihoods with priors
PPV \(P(D\mid +)\) — not sensitivity
Natural frequencies base-rate guardrail

Never confuse \(P(+|D)\) with \(P(D|+)\).

Tomorrow

Day 87 — Expectation, variance, coupon collector, concentration inequalities.