Key Model Selection Criteria

Why This Matters

In linear modeling, building a model that fits your training data perfectly is actually easy—just throw in every predictor you can find. The real challenge? Building a model that generalizes to new data without overfitting. This is where model selection criteria become essential. You're being tested on your ability to understand the fundamental tradeoff between model complexity and predictive accuracy, and knowing when to use each criterion can make or break an FRQ response.

These criteria embody core statistical principles: parsimony, bias-variance tradeoff, out-of-sample performance, and hypothesis testing. Don't just memorize that "lower AIC is better"—understand why each criterion penalizes complexity and when one criterion outperforms another. The best exam answers demonstrate that you grasp the underlying logic, not just the formulas.

---

Information-Theoretic Criteria

These criteria use information theory to quantify model quality, balancing goodness of fit against model complexity through explicit penalty terms. The core idea: a model that explains the data well but uses fewer parameters is preferred because it's less likely to be capturing noise.

Akaike Information Criterion (AIC)

• Formula: $AIC = 2k - 2\ln(\hat{L})$—where $k$ is the number of parameters and $\hat{L}$ is the maximized likelihood
• Penalty structure adds $2k$ to discourage overfitting; derived from minimizing expected Kullback-Leibler divergence
• Best use case is comparing multiple non-nested models when prediction accuracy is the primary goal

Bayesian Information Criterion (BIC)

• Formula: $BIC = k\ln(n) - 2\ln(\hat{L})$—the $\ln(n)$ term creates a stronger penalty as sample size grows
• More conservative than AIC because the penalty scales with sample size, favoring simpler models in large datasets
• Derived from Bayesian model selection, approximating the log of the marginal likelihood under certain conditions

Mallow's Cp

• Formula: $C_p = \frac{RSS_p}{\hat{\sigma}^2} - n + 2p$—where $p$ is the number of predictors including the intercept
• Target value is $C_p \approx p$; values much larger than $p$ suggest important predictors are missing
• Requires an estimate of $\sigma^2$ from a full model, making it useful for comparing subsets of a larger predictor set

---

Variance-Based Measures

These criteria focus on how well the model explains variation in the response variable, directly measuring the gap between predicted and observed values. They quantify fit quality but require careful interpretation when comparing models of different complexity.

Adjusted R-squared

• Formula: $R^2_{adj} = 1 - \frac{(1-R^2)(n-1)}{n-k-1}$—penalizes for adding predictors that don't improve fit proportionally
• Can decrease when you add a weak predictor, unlike regular $R^2$ which only increases or stays the same
• Interpretation remains intuitive: proportion of variance explained, adjusted for model complexity

Residual Sum of Squares (RSS)

• Formula: $RSS = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2$—the foundation for most other fit measures
• Always decreases as you add predictors, which is why raw RSS alone cannot guide model selection
• Building block for calculating $R^2$, MSE, AIC, and $F$-tests; know this formula cold

Mean Squared Error (MSE)

• Formula: $MSE = \frac{RSS}{n}$ or $\frac{RSS}{n-k}$ for the unbiased estimator of error variance
• Sensitive to outliers because squaring amplifies large residuals; consider this when data contains extreme values
• Training vs. test MSE distinction is critical—low training MSE with high test MSE signals overfitting

---

Hypothesis Testing Approaches

These methods use formal statistical tests to determine whether additional model complexity is justified. The logic: if adding parameters doesn't significantly improve fit beyond what we'd expect by chance, keep the simpler model.

F-test for Nested Models

• Tests whether the reduction in RSS from adding predictors is statistically significant; requires models to be nested (one is a special case of the other)
• Formula: $F = \frac{(RSS_{reduced} - RSS_{full})/(df_{reduced} - df_{full})}{RSS_{full}/df_{full}}$—compare to $F$-distribution
• Significant result (small p-value) means the additional predictors in the full model provide meaningful improvement

Likelihood Ratio Test

• Test statistic: $\Lambda = -2\ln\left(\frac{L_{reduced}}{L_{full}}\right)$—follows a chi-squared distribution under the null hypothesis
• More general than F-test because it applies to any models estimated via maximum likelihood, not just OLS regression
• Requires nested models just like the F-test; the simpler model must be a restricted version of the complex one

---

Predictive Validation Methods

These approaches directly estimate out-of-sample performance by testing the model on data it hasn't seen. This is the most honest assessment of how your model will perform in practice, avoiding the optimism bias of in-sample measures.

Cross-Validation

• K-fold method splits data into $k$ subsets, trains on $k-1$ folds, tests on the held-out fold, and averages results
• Leave-one-out (LOOCV) is the extreme case where $k = n$; low bias but high variance and computationally expensive
• Gold standard for prediction because it directly estimates generalization error rather than relying on penalty approximations

Prediction Error

• Definition: $\hat{y}_i - y_i$—the raw difference between what the model predicts and what actually occurs
• Expected prediction error is what we're ultimately trying to minimize; it decomposes into bias, variance, and irreducible error
• Test set prediction error is the most reliable metric for model comparison when sufficient data exists for a holdout set

---

Quick Reference Table

---

Self-Check Questions

1. You're comparing three non-nested regression models on a dataset with $n = 500$. Which criterion—AIC or BIC—will tend to select the simpler model, and why?

2. A colleague adds five new predictors to a model and reports that $R^2$ increased. Why is this insufficient evidence that the new model is better, and what metric should they report instead?

3. Compare and contrast the F-test and cross-validation as model selection tools. Under what circumstances would you prefer each?

4. If Mallow's $C_p$ for a model with 6 predictors equals 15, what does this suggest about the model's fit?

5. An FRQ presents training MSE and test MSE for two models: Model A (training: 2.1, test: 8.7) and Model B (training: 3.4, test: 4.2). Which model should you recommend, and what phenomenon explains Model A's performance gap?