15.3 Regression Analysis and Forecasting

Linear Regression Models for Industrial Engineering

Regression analysis and forecasting help you predict outcomes and understand relationships between variables. In industrial engineering, these techniques drive decisions across production optimization, demand planning, and inventory management.

These models are only as good as the data and assumptions behind them. Data quality issues, violated assumptions, and external disruptions can all undermine accuracy, so you need to interpret results with real-world context in mind.

Simple and Multiple Linear Regression

Simple linear regression models the relationship between two variables using a straight line. Multiple linear regression extends this to handle more than one predictor variable.

The simple linear regression equation:

$Y = \beta_0 + \beta_1 X + \varepsilon$

• $Y$ is the dependent variable (what you're trying to predict)
• $X$ is the independent variable (what you're using to predict)
• $\beta_0$ is the y-intercept (the predicted value of $Y$ when $X$ is zero)
• $\beta_1$ is the slope (how much $Y$ changes for a one-unit increase in $X$)
• $\varepsilon$ is the error term (the difference between predicted and actual values)

Multiple linear regression expands this to include several predictors:

$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_n X_n + \varepsilon$

where $X_1, X_2, \dots, X_n$ are multiple independent variables.

The method of least squares estimates the regression coefficients (the $\beta$ values) by minimizing the sum of squared residuals. It finds the line (or hyperplane, in multiple regression) that makes the total squared distance between predicted and actual values as small as possible.

When interpreting coefficients, pay attention to three things: the magnitude (how large the effect is), the sign (positive or negative relationship), and the statistical significance (whether the effect is likely real or just noise).

Key Assumptions and Applications

Linear regression relies on four key assumptions. If these are violated, your results may be unreliable:

• Linearity: The relationship between dependent and independent variables is linear
• Independence of errors: Residuals are not correlated with each other
• Homoscedasticity: The variance of residuals stays constant across all levels of the independent variable (no "fanning out" pattern)
• Normality of residuals: The residuals follow a roughly normal distribution

Common industrial engineering applications include:

• Demand forecasting: Predicting future product demand based on historical data and factors like price, season, or marketing spend. For example, a beverage company might regress monthly sales on average temperature and advertising budget to plan production.
• Quality control: Analyzing how process parameters (temperature, pressure, speed) relate to product quality metrics like defect rate.
• Process optimization: Identifying the best settings for manufacturing processes to maximize efficiency or minimize costs.

Model Development and Interpretation

You'll typically build regression models using statistical software like R, Python, or Minitab. Here's how to work through the process:

1. Fit the model to your data using the software's regression function.
2. Examine coefficient values: Each coefficient tells you the expected change in the dependent variable for a one-unit change in that predictor, holding all other predictors constant. Positive coefficients mean the variables move together; negative coefficients mean they move in opposite directions.
3. Check p-values: A p-value below your significance level (typically 0.05) means that coefficient is statistically significant, so the relationship is unlikely due to chance alone.
4. Validate assumptions by examining residual plots and diagnostic charts (covered below).
5. Use the model for prediction by plugging in new values of your independent variables.

Assessing Regression Model Fit

Goodness of Fit Metrics

Once you've built a model, you need to know how well it actually fits the data.

R-squared (the coefficient of determination) measures the proportion of variance in the dependent variable that your model explains. It ranges from 0 to 1. An $R^2$ of 0.75 means your model explains 75% of the variance, leaving 25% unexplained.

Adjusted R-squared modifies $R^2$ to account for the number of predictors. This matters because adding any variable to a model will increase $R^2$, even if that variable is useless. Adjusted $R^2$ penalizes unnecessary predictors, making it the better metric when comparing models with different numbers of independent variables.

The F-statistic and its associated p-value test whether your overall model is statistically significant. A large F-statistic with a small p-value (< 0.05) means your model performs significantly better than a model with no predictors at all (just the intercept).

Standard Error of the Estimate (SEE) tells you the average amount that observed values deviate from predicted values. Smaller SEE means tighter predictions. SEE is in the same units as your dependent variable, which makes it easy to interpret practically.

Prediction Error Metrics

These metrics quantify how far off your predictions tend to be:

• Mean Absolute Error (MAE): The average of the absolute differences between predicted and actual values. If your production forecasting model has an MAE of 5 units, your predictions are off by 5 units on average. MAE is intuitive and treats all errors equally.
• Root Mean Square Error (RMSE): Similar to MAE but squares the differences before averaging, then takes the square root. This makes RMSE more sensitive to large errors. If you care more about avoiding big misses than small ones, RMSE is the better metric. RMSE will always be equal to or greater than MAE.

For individual coefficients, the t-statistic and its p-value tell you whether each predictor is statistically significant on its own. A large absolute t-statistic and small p-value (< 0.05) indicate the coefficient is significantly different from zero.

Residual Analysis

Residual plots are your main diagnostic tool for checking whether your model assumptions hold.

• Residuals vs. fitted values plot: Check for linearity and homoscedasticity. You want to see a random scatter with no pattern. A funnel shape (residuals spreading out as fitted values increase) signals heteroscedasticity. A curved pattern signals non-linearity, meaning a straight line isn't capturing the true relationship.
• Q-Q plot: Compares residual distribution to a normal distribution. Points should fall roughly along a straight diagonal line. Deviations at the tails suggest the normality assumption is violated.

Cook's distance measures how much influence a single observation has on the overall regression results. A common rule of thumb is that observations with a Cook's distance greater than $\frac{4}{n}$ (where $n$ is the sample size) deserve closer inspection. Investigate these points to determine if they're legitimate data or errors.

Time Series Analysis for Forecasting

Time Series Components and Basic Techniques

Time series data has four components, and recognizing them is the first step in building a good forecast:

• Trend: The long-term direction of the data (upward, downward, or flat). Think of steadily increasing production volume over several years.
• Seasonality: Regular, repeating fluctuations tied to a known period (daily, weekly, monthly). Retail demand spiking every December is a classic example.
• Cyclical patterns: Longer-term fluctuations not tied to a fixed calendar period, often driven by economic or business cycles. These typically span multiple years.
• Irregular (random) fluctuations: Unpredictable variation that can't be attributed to the other components.

Moving averages smooth out short-term noise to reveal underlying patterns:

• A simple moving average calculates the average of the most recent $n$ observations. For example, a 3-month moving average of demand data averages the last three months at each point. Larger $n$ produces a smoother line but responds more slowly to real changes.
• A weighted moving average assigns different weights to observations, typically giving more importance to recent data.

Exponential smoothing takes this further by giving exponentially decreasing weight to older observations. The smoothing parameter $\alpha$ (between 0 and 1) controls how quickly old data loses influence. There are three main variants, and which one you use depends on your data's characteristics:

• Simple exponential smoothing: For data with no trend or seasonality
• Holt's method (double exponential smoothing): For data with a trend but no seasonality
• Holt-Winters' method (triple exponential smoothing): For data with both trend and seasonality

Advanced Forecasting Models

ARIMA (Autoregressive Integrated Moving Average) models are more flexible and combine three components, specified as ARIMA(p, d, q):

• AR (Autoregressive), order $p$: Uses the relationship between an observation and a set number of lagged observations (past values)
• I (Integrated), order $d$: Applies differencing to make the time series stationary (removing trends so the statistical properties don't change over time)
• MA (Moving Average), order $q$: Uses the relationship between an observation and lagged forecast errors

Seasonal decomposition methods break a time series into its individual components. STL (Seasonal and Trend decomposition using Loess) is a common approach that separates a series into trend, seasonal, and residual components. This is useful both for understanding what's driving your data and for forecasting each component separately.

The Box-Jenkins methodology provides a systematic approach to building ARIMA models:

1. Model identification: Use ACF (autocorrelation function) and PACF (partial autocorrelation function) plots to determine the appropriate AR and MA orders. ACF shows correlation at various lags; PACF shows the direct correlation at each lag after removing the effects of shorter lags.
2. Parameter estimation: Estimate model parameters using maximum likelihood or least squares methods.
3. Model diagnostics: Check whether the fitted model is adequate by examining residuals for remaining patterns. Residuals should resemble white noise (random, with no autocorrelation).

Forecasting Applications and Evaluation

Key industrial engineering applications of time series forecasting:

• Demand forecasting: Predicting future product demand to guide inventory management decisions
• Production planning: Allocating resources (labor, materials, machine time) based on forecasted demand
• Maintenance scheduling: Predicting optimal timing for preventive maintenance to reduce unplanned downtime

To evaluate how well a forecasting model will perform on new data, use time series cross-validation. Unlike standard cross-validation, this uses a rolling-origin approach: you train on data up to a certain point, forecast the next period, then move the origin forward and repeat. This simulates how the model would actually be used in practice, respecting the time ordering of the data.

Common evaluation metrics include:

• MAPE (Mean Absolute Percentage Error): Expresses prediction error as a percentage, making it easy to compare across different scales. A MAPE of 8% means your forecasts are off by 8% on average. Be cautious with MAPE when actual values are near zero, since the percentage can blow up.
• MSE (Mean Squared Error): Squares errors before averaging, penalizing large errors more heavily. RMSE (the square root of MSE) is often preferred because it's in the same units as the original data.

Prediction intervals quantify forecast uncertainty. Wider intervals mean greater uncertainty, and intervals typically widen the further out you forecast. Always report prediction intervals alongside point forecasts so decision-makers understand the range of likely outcomes.

Limitations of Regression and Forecasting Models

Model Assumptions and Stability

The relationships your model captures may not stay constant. In dynamic industrial environments, technology changes, market shifts, and process updates can alter how variables relate to each other. A model linking advertising spend to sales, for example, might become inaccurate when new marketing channels emerge. Periodically retraining your model on recent data helps address this.

Extrapolation is risky. Predicting beyond the range of your observed data (far into the future, or for extreme values of independent variables) increases uncertainty significantly. Forecasting demand for an entirely new product category with limited historical data is a common scenario where this becomes a problem.

Multicollinearity occurs when independent variables are highly correlated with each other. This makes it difficult to isolate the individual effect of each predictor, and it can make coefficient estimates unstable. In manufacturing, temperature and pressure are often correlated, so a regression model may struggle to determine which one is actually driving changes in product quality. You can detect multicollinearity by calculating the Variance Inflation Factor (VIF) for each predictor; VIF values above 5 or 10 (depending on the convention) suggest a problem.

Data Quality and External Factors

• Outliers: A single extreme data point can significantly shift a regression line or skew a time series forecast. An unusual demand spike from a one-time event (like a viral social media post) can distort future predictions if not handled properly. Always investigate outliers before removing them.
• Autocorrelation: In time series data, consecutive observations are often correlated (this month's sales are related to last month's). Standard regression assumes independence of errors, so you need specialized techniques like ARIMA models to handle this. The Durbin-Watson test is a common way to check for autocorrelation in regression residuals.
• Non-linear relationships: Linear regression can't adequately capture relationships that curve or plateau. The relationship between machine speed and product quality, for instance, might have an optimal range with diminishing returns on either side. Polynomial regression or other non-linear models may be needed.
• External factors: Historical data can't account for sudden shifts in economic conditions, new regulations, or technological disruptions. An energy consumption forecast built on past data won't anticipate a sudden change in environmental regulations that forces process modifications. Combining quantitative forecasts with qualitative expert judgment can help address this gap.