Statistical Inference: Confidence and Hypothesis Tests

Updated

July 30, 2026

1. Goal of Inference

Use sample data to make probabilistic statements about unknown population parameters.

2. Confidence Intervals

For sample mean (large n):

xbar +- z_{alpha/2} * s/sqrt(n).

Interpretation: long-run procedure coverage, not probability statement about fixed parameter after observing data.

3. Hypothesis Testing Workflow

  1. State H0 and H1
  2. Choose statistic
  3. Compute p-value
  4. Compare against alpha
  5. Report effect size and uncertainty

4. Type I and Type II Errors

  • Type I: false alarm (alpha)
  • Type II: miss
  • Power: 1-beta

5. Multiple Testing

Repeated tests inflate false positive rate. Use controls like Bonferroni or FDR methods.

6. Worked A/B Test Example

Difference in conversion rates with two-proportion z-test. Include confidence interval and practical effect interpretation.

7. Common Pitfalls

  • p-value misinterpretation
  • ignoring practical significance
  • peeking without correction
  • underpowered experiments

Exercises

  1. Compute 95% CI from sample mean/std/size.
  2. Run one-sample t-test on toy data.
  3. Explain difference between confidence interval and prediction interval.
  4. Calculate required sample size for target detectable effect.