Glossary

10.1 Measures of forecast accuracy

3 min read•august 9, 2024

Measuring forecast accuracy is crucial in business forecasting. Various metrics help assess how well predictions match actual outcomes. From simple measures like to more complex ones like , each metric offers unique insights.

These accuracy measures serve different purposes. Some, like MAPE, allow comparisons across datasets. Others, like tracking signals, help detect systematic bias. Understanding these metrics is key to evaluating and improving forecasting models in business contexts.

Error Measures

Common Error Measures in Forecasting

Top images from around the web for Common Error Measures in Forecasting

  • Comparision Between Accuracy and MSE,RMSE by Using Proposed Method with Imputation Technique ... View original

    Is this image relevant?

  • 预测问题评价指标：MAE、MSE、R-Square、MAPE和RMSE - 代码先锋网 View original

    Is this image relevant?

  • Comparision Between Accuracy and MSE,RMSE by Using Proposed Method with Imputation Technique ... View original

    Is this image relevant?

1 of 3

Top images from around the web for Common Error Measures in Forecasting

  • Comparision Between Accuracy and MSE,RMSE by Using Proposed Method with Imputation Technique ... View original

    Is this image relevant?

  • 预测问题评价指标：MAE、MSE、R-Square、MAPE和RMSE - 代码先锋网 View original

    Is this image relevant?

  • Comparision Between Accuracy and MSE,RMSE by Using Proposed Method with Imputation Technique ... View original

    Is this image relevant?

1 of 3
  • Mean Absolute Error (MAE) calculates the average of absolute differences between forecasted and actual values
    • Provides a straightforward measure of forecast accuracy
    • Expressed in the same units as the original data
    • Formula: MAE=1n∑t=1n∣Yt−Ft∣MAE = \frac{1}{n} \sum_{t=1}^n |Y_t - F_t|
      • Where YtY_t represents actual values and FtF_t represents forecasted values
  • (MSE) computes the average of squared differences between forecasted and actual values
    • Penalizes larger errors more heavily than smaller ones
    • Expressed in squared units of the original data
    • Formula: MSE=1n∑t=1n(Yt−Ft)2MSE = \frac{1}{n} \sum_{t=1}^n (Y_t - F_t)^2
  • (RMSE) calculates the square root of the Mean Squared Error
    • Provides an error measure in the same units as the original data
    • Useful for comparing different forecasting models
    • Formula: RMSE=1n∑t=1n(Yt−Ft)2RMSE = \sqrt{\frac{1}{n} \sum_{t=1}^n (Y_t - F_t)^2}

Percentage-Based Error Measures

  • (MAPE) computes the average of absolute percentage differences between forecasted and actual values
    • Expresses forecast error as a percentage, making it scale-independent
    • Useful for comparing forecast accuracy across different datasets
    • Formula: MAPE=1n∑t=1n∣Yt−FtYt∣×100%MAPE = \frac{1}{n} \sum_{t=1}^n |\frac{Y_t - F_t}{Y_t}| \times 100\%
  • (SMAPE) addresses limitations of MAPE when actual values are close to zero
    • Ranges from 0% to 200%, providing a more balanced measure
    • Formula: SMAPE=1n∑t=1n∣Yt−Ft∣(∣Yt∣+∣Ft∣)/2×100%SMAPE = \frac{1}{n} \sum_{t=1}^n \frac{|Y_t - F_t|}{(|Y_t| + |F_t|)/2} \times 100\%
  • (MPE) calculates the average of percentage differences between forecasted and actual values
    • Indicates the direction of bias in forecasts (positive or negative)
    • Formula: MPE=1n∑t=1nYt−FtYt×100%MPE = \frac{1}{n} \sum_{t=1}^n \frac{Y_t - F_t}{Y_t} \times 100\%

Forecast Accuracy Metrics

Comparative Accuracy Measures

  • Theil's U-statistic compares the accuracy of a forecasting model to a naive forecast
    • Values less than 1 indicate the model outperforms the naive forecast
    • Values greater than 1 suggest the naive forecast is more accurate
    • Formula: U=∑t=1n(Ft−Yt)2∑t=1n(Yt−Yt−1)2U = \frac{\sqrt{\sum_{t=1}^n (F_t - Y_t)^2}}{\sqrt{\sum_{t=1}^n (Y_t - Y_{t-1})^2}}
  • (RAE) measures the ratio of forecast errors to the errors of a naive forecast
    • Provides a scale-independent measure of forecast accuracy
    • Formula: RAE=∑t=1n∣Yt−Ft∣∑t=1n∣Yt−Yt−1∣RAE = \frac{\sum_{t=1}^n |Y_t - F_t|}{\sum_{t=1}^n |Y_t - Y_{t-1}|}
  • (MASE) scales the errors based on the in-sample MAE from a naive forecast
    • Applicable to both seasonal and non-seasonal time series
    • Formula: MASE=1n∑t=1n∣Yt−Ft∣1n−1∑i=2n∣Yi−Yi−1∣MASE = \frac{1}{n} \sum_{t=1}^n \frac{|Y_t - F_t|}{\frac{1}{n-1} \sum_{i=2}^n |Y_i - Y_{i-1}|}

Bias and Precision Metrics

  • measures the systematic tendency of a forecast to over- or under-predict actual values
    • Calculated as the average of forecast errors
    • Formula: Bias=1n∑t=1n(Yt−Ft)Bias = \frac{1}{n} \sum_{t=1}^n (Y_t - F_t)
  • assesses the consistency or variability of forecast errors
    • Measured by the standard deviation of forecast errors
    • Formula: Precision=1n−1∑t=1n(et−eˉ)2Precision = \sqrt{\frac{1}{n-1} \sum_{t=1}^n (e_t - \bar{e})^2}
      • Where ete_t represents forecast errors and eˉ\bar{e} is the mean forecast error
  • monitors the ratio of cumulative forecast errors to the Mean Absolute Deviation (MAD)
    • Helps detect systematic bias in forecasts over time
    • Formula: TrackingSignal=∑t=1n(Yt−Ft)MADTracking Signal = \frac{\sum_{t=1}^n (Y_t - F_t)}{MAD}
      • Where MAD is the Mean Absolute Deviation

Key Terms to Review (12)

Forecast bias: Forecast bias refers to the systematic error that occurs when predictions consistently overestimate or underestimate actual outcomes. This concept is crucial in assessing the reliability of forecasting methods, as it highlights whether the forecasts are consistently leaning in one direction. Understanding forecast bias helps in selecting appropriate forecasting methods, measuring accuracy, and improving demand predictions for production and service level planning.
Forecast precision: Forecast precision refers to the degree of accuracy in a forecasting model, indicating how closely the predicted values align with the actual values observed over a certain period. High precision means the forecasted data points are tightly clustered around the actual outcomes, while low precision indicates greater dispersion and uncertainty. Understanding forecast precision is essential for evaluating the effectiveness of forecasting methods and improving decision-making processes in business environments.
Mean Absolute Error: Mean Absolute Error (MAE) is a measure of forecast accuracy that calculates the average absolute difference between predicted values and actual values. It helps assess how close forecasts are to the actual outcomes, providing insights into the forecasting process's reliability and effectiveness, as well as supporting improvements in forecasting methodologies.
Mean absolute percentage error: Mean Absolute Percentage Error (MAPE) is a measure used to assess the accuracy of a forecasting method by calculating the average absolute percentage difference between forecasted and actual values. This metric is particularly useful because it expresses accuracy in percentage terms, making it easier to interpret across different scales and contexts. MAPE helps in evaluating forecast performance, allowing comparisons between various forecasting methods and their effectiveness in predicting demand for production planning while also playing a crucial role in creating insightful reports and dashboards.
Mean absolute scaled error: Mean Absolute Scaled Error (MASE) is a measure used to assess the accuracy of forecast models by comparing the absolute errors of forecasts to a baseline model's performance. It helps in understanding how well a forecasting method performs relative to a simple benchmark, often the mean or median of historical data, making it useful for evaluating different forecasting approaches across various datasets.
Mean Percentage Error: Mean Percentage Error (MPE) is a statistical measure used to assess the accuracy of a forecasting method by calculating the average of the percentage errors between forecasted and actual values. This metric helps in evaluating how close the predicted values are to the actual outcomes, allowing businesses to understand the performance of their forecasting models. By analyzing MPE, organizations can make informed decisions on adjusting their forecasting strategies and improving overall accuracy.
Mean Squared Error: Mean Squared Error (MSE) is a statistical measure used to evaluate the accuracy of a forecasting model by calculating the average of the squared differences between predicted values and actual values. This metric emphasizes larger errors more than smaller ones due to the squaring process, making it particularly useful in identifying models that consistently underperform. It connects with various forecasting methods, assessment of forecast accuracy, and is essential in guiding production planning decisions based on demand forecasts.
Relative absolute error: Relative absolute error is a measure used to assess the accuracy of a forecast by comparing the absolute error of the forecast to the actual value. It provides insight into the performance of a forecasting model by expressing the error as a fraction of the actual value, allowing for a standardized way to evaluate and compare forecast accuracy across different datasets or scenarios. This metric is essential in understanding how significant an error is in relation to the size of the actual outcome, helping to identify both overestimation and underestimation in forecasts.
Root mean squared error: Root mean squared error (RMSE) is a widely used measure to assess the accuracy of a forecasting model by calculating the square root of the average of the squares of the errors. It quantifies how well a model's predictions align with actual observed values, giving more weight to larger discrepancies. A lower RMSE indicates better model performance and forecast accuracy, making it a key metric in evaluating various forecasting methods.
Symmetric mean absolute percentage error: The symmetric mean absolute percentage error (SMAPE) is a measure used to evaluate the accuracy of a forecasting model by calculating the average percentage difference between the forecasted and actual values, while providing a symmetric approach to avoid issues with negative values. It addresses the shortcomings of traditional percentage errors by taking the absolute values and averaging them in a way that treats overestimations and underestimations equally, which makes it particularly useful for models where both types of errors are important.
Theil's U-statistic: Theil's U-statistic is a measure of forecast accuracy that compares the performance of a forecasting model to a naïve benchmark model. It helps in assessing whether a particular forecasting method provides better predictions than simply using the last observed value as a forecast. Theil's U-statistic plays an essential role in evaluating forecast accuracy and in comparing different forecasting methods to ensure the most effective approach is chosen.
Tracking Signal: A tracking signal is a measure used in forecasting to assess the accuracy of predictions by comparing the cumulative forecast errors to a predefined threshold. This tool helps identify whether the forecasting model is performing well or needs adjustments, making it an essential part of effective demand management and planning processes.
© 2024 Fiveable Inc. All rights reserved.
AP® and SAT® are trademarks registered by the College Board, which is not affiliated with, and does not endorse this website.
Back
Practice QuizGlossary
Practice QuizGlossary
Next guide