Effect Size Measures

Why This Matters

Statistical inference isn't just about determining whether an effect exists—it's about understanding how big that effect actually is. While p-values tell you if results are statistically significant, effect size measures tell you if those results are practically meaningful. This distinction is critical: a study with thousands of participants might find a "significant" difference that's so tiny it has no real-world importance. You're being tested on your ability to choose the right effect size measure for different study designs, interpret what the values mean, and explain why effect sizes matter for evidence-based decision-making, meta-analysis, and replication studies.

Effect size measures fall into distinct families based on what they quantify: standardized differences between groups, variance explained, and measures of association for categorical outcomes. Understanding these categories helps you quickly identify which measure fits which research scenario. Don't just memorize formulas—know what type of data each measure handles and when you'd choose one over another.

---

Standardized Mean Differences

These measures express the difference between group means in standard deviation units, making them ideal for comparing results across studies that use different measurement scales. The core principle: divide the mean difference by a measure of variability to create a unit-free comparison.

Cohen's d

• Most widely used effect size for two-group comparisons—calculated as the difference between means divided by the pooled standard deviation: $d = \frac{\bar{X}_1 - \bar{X}_2}{s_{pooled}}$
• Benchmark interpretations of small (0.2), medium (0.5), and large (0.8) come from Cohen's original guidelines, though context matters
• Best applied when both groups have similar variances and sample sizes; assumes equal standard deviations across groups

Hedges' g

• Corrects Cohen's d for small-sample bias—applies a correction factor that becomes negligible with larger samples (typically $n > 20$)
• Preferred in meta-analyses because it provides unbiased estimates when combining studies with varying sample sizes
• Interpretation identical to Cohen's d—same benchmarks apply, making results directly comparable

Glass's Delta

• Uses only the control group's standard deviation for standardization: $\Delta = \frac{\bar{X}_1 - \bar{X}_2}{s_{control}}$
• Ideal when treatment affects variability—if an intervention changes not just the mean but also the spread of scores, pooling standard deviations would be misleading
• Common in experimental designs where you want to express treatment effects relative to baseline variability

Standardized Mean Difference (SMD)

• Umbrella term encompassing Cohen's d, Hedges' g, and Glass's delta—refers to any effect size that standardizes group differences
• Essential for meta-analysis because it allows combining results from studies using different measurement instruments
• Watch the terminology—some software outputs "SMD" generically, so always check which specific formula was applied

---

Variance-Explained Measures

These measures tell you what proportion of the outcome's variability can be attributed to your predictor(s). The underlying logic: partition total variance into explained and unexplained components.

R-squared (R²)

• Proportion of variance explained by the regression model—ranges from 0 to 1, where $R^2 = 1 - \frac{SS_{residual}}{SS_{total}}$
• Interpretation is context-dependent—an $R^2$ of 0.30 might be excellent in psychology but weak in physics; always consider the field's standards
• Limitation: always increases when you add predictors, even useless ones—this is why adjusted $R^2$ exists for multiple regression

Eta-squared (η²)

• ANOVA's version of $R^2$—represents the proportion of total variance in the dependent variable explained by the factor: $\eta^2 = \frac{SS_{between}}{SS_{total}}$
• Tends to overestimate population effect sizes, especially with small samples or multiple factors
• Quick benchmarks: small (0.01), medium (0.06), large (0.14)—but these are rough guidelines, not rigid cutoffs

Partial Eta-squared

• Controls for other factors in the model—shows the unique variance explained by one factor after removing variance explained by others
• Standard output in factorial ANOVA—most statistical software reports partial $\eta^2$ by default rather than eta-squared
• Not directly comparable to eta-squared because denominators differ; partial values are typically larger for the same effect

---

Correlation-Based Measures

Correlation coefficients quantify the strength and direction of relationships between variables. The key insight: these measures are already standardized, making them natural effect sizes.

Pearson's Correlation Coefficient (r)

• Measures linear association between two continuous variables—ranges from $-1$ (perfect negative) to $+1$ (perfect positive), with 0 indicating no linear relationship
• Effect size benchmarks: small (0.10), medium (0.30), large (0.50)—note these differ from Cohen's d benchmarks
• Squaring gives variance explained—$r^2$ tells you the proportion of variance shared between variables, directly connecting correlation to regression

---

Measures for Categorical Outcomes

When your outcome is binary (yes/no, disease/no disease), you need effect sizes designed for proportions and odds. These measures compare event rates or odds between groups rather than means.

Odds Ratio

• Compares odds between groups—calculated as $OR = \frac{odds_{group1}}{odds_{group2}}$, where odds = probability of event / probability of no event
• Interpretation anchored at 1.0—values above 1 indicate higher odds in the numerator group; values below 1 indicate lower odds
• Standard in logistic regression and case-control studies—when you can't calculate risk directly (retrospective designs), odds ratios are your go-to measure

Risk Ratio (Relative Risk)

• Compares probabilities directly—calculated as $RR = \frac{P(event|exposed)}{P(event|unexposed)}$
• More intuitive than odds ratios for most audiences—"twice the risk" is easier to grasp than "twice the odds"
• Requires prospective data—only valid in cohort studies or RCTs where you can calculate actual incidence rates

---

Quick Reference Table

---

Self-Check Questions

1. You're comparing treatment effects across three studies that used different depression scales. Which effect size measure would allow valid comparisons, and why?

2. A researcher reports $\eta^2 = 0.08$ from a one-way ANOVA. How would you interpret this value, and what benchmark category does it fall into?

3. Compare and contrast the odds ratio and risk ratio: In what study designs is each appropriate, and when do their values converge?

4. Why might a researcher choose Glass's delta over Cohen's d when evaluating an educational intervention? What assumption about the data motivates this choice?

5. An FRQ presents a multiple regression with $R^2 = 0.45$ and asks whether adding another predictor improved the model. Why is $R^2$ alone insufficient to answer this question, and what would you need to know?