Essential Variable Selection Techniques

Why This Matters

Variable selection sits at the heart of building effective linear models. It's the difference between a model that captures true relationships and one that's memorizing noise. When you're tested on this material, you're being evaluated on your understanding of bias-variance tradeoffs, regularization principles, and model parsimony, not just your ability to name techniques.

The techniques below fall into distinct conceptual families: some search through predictor combinations sequentially, others penalize coefficient magnitudes mathematically, and still others transform the predictor space entirely. Focus on why you'd choose one approach over another and what tradeoffs each involves. That comparative understanding is exactly what FRQ prompts target.

---

Sequential Search Methods

These techniques build or trim models one predictor at a time, using statistical significance as the decision rule. The core mechanism is greedy optimization: making the locally optimal choice at each step without guaranteeing a globally optimal solution.

Forward Selection

• Starts with an empty model and adds predictors sequentially based on which one most improves model fit
• Stops when no remaining predictor provides a statistically significant improvement
• Works well when you expect only a few predictors to matter, but it can miss important variable combinations that only help in tandem

Backward Elimination

• Begins with all predictors included and removes the least significant one at each iteration
• Continues until all remaining predictors meet the significance threshold you've specified
• Works well when most predictors are relevant, but it requires enough observations to fit the full model initially (you need $n > p$)

Stepwise Regression

• Combines forward and backward logic, allowing predictors to enter or exit the model at any step
• Offers flexibility for exploratory analysis when you're uncertain which predictors matter most
• Prone to overfitting because of the multiple testing problem: each step involves a significance test, and running many tests inflates the chance of spurious results. Significance thresholds may need adjustment to compensate.

---

Regularization-Based Methods

Rather than testing predictors in or out, these methods keep all predictors but penalize coefficient magnitudes to prevent overfitting. The key mechanism is adding a penalty term to the ordinary least squares loss function, controlled by a tuning parameter $\lambda$. As $\lambda$ increases, the penalty gets stronger and coefficients shrink more aggressively.

Lasso (Least Absolute Shrinkage and Selection Operator)

• Uses L1 regularization by adding $\lambda \sum_{j=1}^{p} |\beta_j|$ to the loss function
• Shrinks some coefficients exactly to zero, performing automatic variable selection alongside estimation. This is a direct consequence of the L1 penalty's geometry: the diamond-shaped constraint region has corners that sit on the axes, making zero-valued solutions likely.
• Ideal for high-dimensional data where you suspect many predictors are irrelevant, since the resulting sparse solutions are highly interpretable

Ridge Regression

• Uses L2 regularization by adding $\lambda \sum_{j=1}^{p} \beta_j^2$ to the loss function
• Shrinks coefficients toward zero but never exactly to zero: all predictors remain in the model. The circular constraint region of the L2 penalty has no corners, so solutions rarely land exactly at zero.
• Excels when predictors are highly correlated. Multicollinearity inflates coefficient variance, and Ridge counteracts this by stabilizing the estimates through shrinkage, even though no variables are dropped.

Elastic Net

• Combines L1 and L2 penalties with two tuning parameters controlling the mix: $\lambda_1 \sum |\beta_j| + \lambda_2 \sum \beta_j^2$
• Handles correlated predictor groups better than Lasso alone. Lasso tends to arbitrarily select just one predictor from a correlated set and zero out the rest; the Ridge component in Elastic Net encourages correlated predictors to share similar coefficients instead.
• Particularly powerful when $p > n$. The Ridge component stabilizes estimation (Lasso alone can select at most $n$ variables), while the Lasso component still provides sparsity.

---

Information Criteria

These aren't selection algorithms themselves but scoring metrics that guide model comparison. The underlying principle is penalized likelihood: rewarding model fit while punishing complexity.

Akaike Information Criterion (AIC)

• Balances fit and complexity using the formula $AIC = 2k - 2\ln(\hat{L})$, where $k$ is the number of parameters and $\hat{L}$ is the maximized likelihood
• Lower values indicate better models, but AIC is a relative measure, only meaningful when comparing models fit to the same data
• Tends toward larger models than BIC, making it preferable when prediction accuracy is the primary goal

Bayesian Information Criterion (BIC)

• Imposes a heavier complexity penalty using $BIC = k\ln(n) - 2\ln(\hat{L})$, where $n$ is sample size
• Favors simpler models than AIC, especially as sample size grows. The penalty per parameter is $\ln(n)$, which exceeds AIC's fixed penalty of 2 once $n > 7$ (since $\ln(8) \approx 2.08$). So for virtually any real dataset, BIC penalizes complexity more heavily.
• Better suited for identifying the "true" generating model when one exists. Use BIC when parsimony matters more than pure predictive performance.

---

Exhaustive and Transformation Methods

These approaches either search comprehensively through model space or transform predictors to reduce dimensionality directly.

Best Subset Selection

• Evaluates all $2^p$ possible predictor combinations to find the optimal subset according to a chosen criterion (such as AIC, BIC, or adjusted $R^2$)
• Guarantees the best model for any given subset size, with no greedy shortcuts that might miss optimal combinations
• Computationally prohibitive for large $p$. With 20 predictors you're fitting about 1 million models; at 30 predictors it's over 1 billion. Practically feasible only up to roughly $p = 20\text{–}25$.

Principal Component Analysis (PCA)

PCA doesn't select original predictors. Instead, it transforms correlated predictors into a new set of uncorrelated principal components, each a linear combination of the originals.

• Reduces dimensionality by retaining only the components that explain the most variance. Typically the first few components capture the bulk of the information.
• Addresses multicollinearity effectively since the components are uncorrelated by construction.
• The tradeoff is interpretability: components don't correspond to original variables, so you can't easily say "predictor X matters."

---

Quick Reference Table

---

Self-Check Questions

1. Which two regularization methods can shrink coefficients exactly to zero, and what penalty term enables this behavior?

2. Compare Forward Selection and Best Subset Selection: what advantage does each have over the other, and when would computational constraints favor one approach?

3. A dataset has 50 observations and 200 predictors. Which variable selection techniques would be appropriate, and which would fail? Explain why.

4. You're comparing two nested models and need to choose between AIC and BIC. If your sample size is very large and you prioritize identifying the simplest adequate model, which criterion should you use and why?

5. Explain why Ridge Regression handles multicollinearity effectively even though it doesn't remove any predictors from the model. How does this differ from Lasso's approach to correlated predictors?