Forecast Accuracy Metrics

Why This Matters

Every forecasting model makes errors—what separates good forecasters from great ones is knowing how to measure those errors and what those measurements reveal. You're being tested on more than just formulas; examiners want to see that you understand when to use MAE versus RMSE, why MAPE can fail spectacularly with near-zero values, and how to determine whether your fancy model actually beats a simple naive forecast. These metrics connect directly to core concepts like model selection, bias detection, outlier sensitivity, and scale independence.

Think of accuracy metrics as diagnostic tools in your forecasting toolkit. Some metrics punish big errors harshly (great for high-stakes decisions), others reveal systematic bias (crucial for model calibration), and still others let you compare across completely different datasets (essential for benchmarking). Don't just memorize the formulas—know what problem each metric solves and when it will mislead you.

---

Absolute Error Metrics: Measuring Raw Magnitude

These metrics measure forecast error in the same units as your original data, making interpretation intuitive. They focus purely on how far off your predictions are, regardless of direction.

Mean Absolute Error (MAE)

• Averages the absolute differences between forecasted and actual values: $MAE = \frac{1}{n}\sum_{i=1}^{n}|A_i - F_i|$
• Treats all errors equally—a 10-unit miss counts the same whether it's your biggest or smallest error
• Best for situations where outliers shouldn't dominate your accuracy assessment and you need interpretable units

Mean Squared Error (MSE)

• Squares each error before averaging, which disproportionately penalizes large errors: $MSE = \frac{1}{n}\sum_{i=1}^{n}(A_i - F_i)^2$
• Highly sensitive to outliers—one massive miss can dominate your entire metric
• Useful when large errors are especially costly in business terms, like inventory stockouts during peak season

Root Mean Squared Error (RMSE)

• Takes the square root of MSE to return the metric to original units: $RMSE = \sqrt{MSE}$
• Maintains the outlier sensitivity of MSE while being directly comparable to your data scale
• Industry standard for model comparison because it balances interpretability with appropriate error weighting

---

Percentage-Based Metrics: Scale-Free Interpretation

These metrics express error as a percentage, enabling comparison across datasets with different scales. The trade-off is sensitivity to small actual values.

Mean Absolute Percentage Error (MAPE)

• Expresses average error as a percentage of actual values: $MAPE = \frac{100\%}{n}\sum_{i=1}^{n}\left|\frac{A_i - F_i}{A_i}\right|$
• Intuitive for stakeholder communication—"our forecasts are off by 8% on average" resonates with executives
• Fails catastrophically when actuals approach zero—division by near-zero values creates infinite or undefined results

Mean Percentage Error (MPE)

• Retains the sign of errors to reveal systematic bias: $MPE = \frac{100\%}{n}\sum_{i=1}^{n}\frac{A_i - F_i}{A_i}$
• Positive MPE indicates under-forecasting; negative MPE indicates over-forecasting on average
• Errors can cancel out, so a low MPE doesn't mean accurate forecasts—it might mean balanced over/under errors

Symmetric Mean Absolute Percentage Error (SMAPE)

• Uses the average of actual and forecast in the denominator: $SMAPE = \frac{100\%}{n}\sum_{i=1}^{n}\frac{|A_i - F_i|}{(|A_i| + |F_i|)/2}$
• Bounded between 0% and 200%, avoiding the infinite values that plague MAPE
• More stable with small actuals but still not immune to issues when both actual and forecast are near zero

---

Relative Performance Metrics: Beating the Baseline

These metrics compare your model against a naive benchmark—typically a random walk or seasonal naive forecast. They answer the critical question: is your model actually adding value?

Theil's U-Statistic

• Compares forecast accuracy to a naive no-change model—values below 1 mean you're beating the baseline
• Values above 1 indicate your model performs worse than simply predicting "tomorrow equals today"
• Essential reality check before deploying complex models that may not justify their computational cost

Mean Absolute Scaled Error (MASE)

• Divides your MAE by the MAE of a naive forecast: $MASE = \frac{MAE}{\frac{1}{n-1}\sum_{i=2}^{n}|A_i - A_{i-1}|}$
• Scale-independent and works across different time series—ideal for comparing forecast accuracy across product lines or regions
• MASE < 1 beats naive; MASE > 1 loses to naive—the clearest benchmark interpretation available

Forecast Skill

• Measures the percentage improvement over a reference forecast, often expressed as $Skill = 1 - \frac{MSE_{model}}{MSE_{reference}}$
• Skill of 1 means perfect forecasts; skill of 0 means no better than the reference; negative skill means worse
• Crucial for justifying model investments—if skill is near zero, simpler methods may be more cost-effective

---

Bias Detection Metrics: Finding Systematic Errors

These metrics help identify whether your forecasts consistently lean in one direction. Detecting bias early prevents compounding errors in operational decisions.

Tracking Signal

• Cumulative sum of errors divided by MAD: $TS = \frac{\sum(A_i - F_i)}{MAD}$
• Values outside ±4 to ±6 typically signal systematic bias requiring model recalibration
• Monitors forecast drift over time—essential for automated forecasting systems that need exception alerts

Mean Percentage Error (MPE)

• Already covered above, but its primary value is bias detection rather than accuracy measurement
• Complements MAE or MAPE by revealing directional tendencies hidden in absolute metrics
• Use alongside tracking signal for comprehensive bias monitoring in rolling forecast systems

---

Quick Reference Table

---

Self-Check Questions

1. Your dataset includes several periods where actual demand was near zero. Which two metrics should you avoid, and what alternatives would you recommend?

2. A colleague reports that their model has an MPE of 0.5% but an MAPE of 15%. What does this combination tell you about the forecast's characteristics?

3. Compare RMSE and MAE: if a model's RMSE is significantly higher than its MAE, what does this indicate about the error distribution, and how might this influence model selection?

4. You need to compare forecast accuracy across three product lines with vastly different sales volumes. Which metric is best suited for this comparison, and why do percentage-based metrics like MAPE fall short here?

5. Your tracking signal has steadily increased from +2 to +7 over the past six months. What action should you take, and what does this trend reveal about your forecasting model?