5.2 Matrix Formulation of Simple Linear Regression

Linear Regression in Matrix Form

Matrix formulation recasts simple linear regression as a system of matrix equations, giving you a compact way to represent the model, estimate parameters, and derive their properties. This matters because the same matrix framework scales directly to multiple regression and more complex linear models. If you nail the simple case here, the jump to $p$ predictors later is mostly just making the matrices bigger.

Matrix Notation

The simple linear regression model with $n$ observations can be written as:

$y = X\beta + \varepsilon$

Each piece of this equation is a matrix or vector:

• $y$ is an $n \times 1$ vector of response values (the thing you're trying to predict).
• $X$ is an $n \times 2$ design matrix. The first column is all ones (accounting for the intercept), and the second column holds the predictor values $x_1, x_2, \dots, x_n$.
• $\beta$ is a $2 \times 1$ parameter vector containing the intercept $\beta_0$ and the slope $\beta_1$.
• $\varepsilon$ is an $n \times 1$ vector of errors.

The error vector is assumed to follow $\varepsilon \sim N(0,\, \sigma^2 I)$, where $I$ is the $n \times n$ identity matrix. Writing the covariance as $\sigma^2 I$ encodes two things at once: every error has the same variance $\sigma^2$, and errors are uncorrelated with each other.

Model Assumptions

• The true relationship between the predictor and the response is linear in the parameters.
• Errors are independently and identically distributed (i.i.d.) normal: $\varepsilon_i \sim N(0, \sigma^2)$.
• Error variance $\sigma^2$ is constant across all observations (homoscedasticity).
• The predictor values are treated as fixed (non-random) and measured without error.

These are the same assumptions you've seen in scalar form. The matrix version just packages them more efficiently.

Design Matrix and Parameter Vector

Design Matrix Structure

For simple linear regression with $n$ observations and one predictor, $X$ is $n \times 2$:

• Column 1: a vector of ones $(1, 1, \dots, 1)^T$, which multiplies the intercept $\beta_0$.
• Column 2: the predictor values $(x_1, x_2, \dots, x_n)^T$, which multiply the slope $\beta_1$.

Example. With 5 observations and predictor values $(2, 4, 6, 8, 10)$:

$X = \begin{bmatrix} 1 & 2 \\ 1 & 4 \\ 1 & 6 \\ 1 & 8 \\ 1 & 10 \end{bmatrix}$

Each row corresponds to one observation. The matrix product $X\beta$ produces the vector of fitted values $\hat{y}_i = \beta_0 + \beta_1 x_i$ for every observation simultaneously.

Parameter Vector

The parameter vector collects the intercept and slope into a single object:

$\beta = \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix}$

This is what makes the notation compact. Instead of writing out $y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$ for each observation, the single equation $y = X\beta + \varepsilon$ captures all $n$ equations at once.

Least Squares Estimation with Matrices

Objective Function

The goal is the same as in scalar regression: minimize the sum of squared residuals. In matrix form, the residual vector is $(y - X\beta)$, and the sum of squared residuals is:

$S(\beta) = (y - X\beta)^T(y - X\beta)$

This is just the dot product of the residual vector with itself, which equals $\sum_{i=1}^n (y_i - \hat{y}_i)^2$.

To minimize, you differentiate $S(\beta)$ with respect to $\beta$ and set the result to zero. That yields the normal equation:

$X^T(y - X\beta) = 0$

The intuition here: $X^T(y - X\beta) = 0$ says the residual vector is orthogonal to every column of $X$. The residuals have no linear relationship left with the predictors, which is exactly what a good fit should achieve.

Solving the Normal Equations

Rearranging the normal equation gives $X^TX\hat{\beta} = X^Ty$. If $X^TX$ is invertible, you can solve directly:

$\hat{\beta} = (X^TX)^{-1}X^Ty$

Here's the step-by-step process:

1. Compute $X^TX$ (a $2 \times 2$ matrix).
2. Compute $X^Ty$ (a $2 \times 1$ vector).
3. Find $(X^TX)^{-1}$.
4. Multiply: $\hat{\beta} = (X^TX)^{-1}X^Ty$.

Example. Using the design matrix from above and response vector $y = (3, 5, 7, 9, 11)^T$:

$X^TX = \begin{bmatrix} 5 & 30 \\ 30 & 220 \end{bmatrix}, \qquad X^Ty = \begin{bmatrix} 35 \\ 250 \end{bmatrix}$

The determinant of $X^TX$ is $5(220) - 30(30) = 1100 - 900 = 200$, so:

$(X^TX)^{-1} = \frac{1}{200}\begin{bmatrix} 220 & -30 \\ -30 & 5 \end{bmatrix}$

Then:

$\hat{\beta} = \frac{1}{200}\begin{bmatrix} 220 & -30 \\ -30 & 5 \end{bmatrix}\begin{bmatrix} 35 \\ 250 \end{bmatrix} = \frac{1}{200}\begin{bmatrix} 200 \\ 200 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$

So $\hat{\beta}_0 = 1$ and $\hat{\beta}_1 = 1$, giving the fitted line $\hat{y} = 1 + 1 \cdot x$. You can verify: when $x = 2$, $\hat{y} = 3$; when $x = 10$, $\hat{y} = 11$. The fit is perfect here because the data are exactly linear.

Normal Equations Derivation

Expanding the Least Squares Problem

Starting from $X^T(y - X\beta) = 0$ and distributing:

$X^TX\beta = X^Ty$

For simple linear regression, $X^TX$ is $2 \times 2$ and $X^Ty$ is $2 \times 1$. Writing these out element by element:

$\begin{bmatrix} n & \sum_{i=1}^n x_i \\ \sum_{i=1}^n x_i & \sum_{i=1}^n x_i^2 \end{bmatrix} \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix} = \begin{bmatrix} \sum_{i=1}^n y_i \\ \sum_{i=1}^n x_i y_i \end{bmatrix}$

This gives you two simultaneous equations (the classical normal equations you may have seen in scalar form):

1. $n\beta_0 + \beta_1 \sum x_i = \sum y_i$
2. $\beta_0 \sum x_i + \beta_1 \sum x_i^2 = \sum x_i y_i$

The matrix formulation and the scalar normal equations are the same system, just written differently.

Solving for Parameter Estimates

Premultiplying both sides by $(X^TX)^{-1}$:

$\hat{\beta} = (X^TX)^{-1}X^Ty$

The variance-covariance matrix of the parameter estimates is:

$\text{Var}(\hat{\beta}) = \sigma^2(X^TX)^{-1}$

• The diagonal elements of $\sigma^2(X^TX)^{-1}$ give $\text{Var}(\hat{\beta}_0)$ and $\text{Var}(\hat{\beta}_1)$. Taking square roots of these gives you the standard errors you'd use for hypothesis tests and confidence intervals.
• The off-diagonal elements give $\text{Cov}(\hat{\beta}_0, \hat{\beta}_1)$. In simple linear regression, the intercept and slope estimates are generally correlated unless $\bar{x} = 0$.

This is one of the big payoffs of the matrix approach: the formula $\hat{\beta} = (X^TX)^{-1}X^Ty$ and the variance expression $\sigma^2(X^TX)^{-1}$ work regardless of whether you have 1 predictor or 100. The structure stays the same.