Residual Sum of Squares

Residual Sum of Squares, or RSS, is the sum of the squared residuals between observed values and model predictions. In Linear Algebra and Differential Equations, it shows how well a least squares approximation fits the data or system.

What is Residual Sum of Squares?

Residual Sum of Squares, often written as RSS, is the total of the squared residuals in a least squares problem. A residual is the difference between what you actually have and what your model predicts, so RSS measures how far the model misses overall.

In Linear Algebra and Differential Equations, RSS shows up when you solve an overdetermined system, meaning there are more equations than unknowns. Since there may be no exact solution, you look for the coefficient vector that makes the prediction errors as small as possible. RSS is the quantity you minimize to get that best approximating vector.

The squaring matters. Positive and negative errors should not cancel each other out, and larger errors should count more than smaller ones. That is why least squares adds the squared residuals instead of just adding raw differences.

For a data set with observed values y1, y2, y3 and predicted values y hat 1, y hat 2, y hat 3, RSS is (y1 - y hat 1)^2 + (y2 - y hat 2)^2 + (y3 - y hat 3)^2. If the model fits better, those squared gaps shrink and RSS goes down.

This idea is tied to projection. In the least squares setup, the prediction vector is the projection of your target vector onto the column space of the design matrix. The residual vector is what is left over, and RSS is just the squared length of that leftover vector, so it matches the geometry of distance in Euclidean space.

Why Residual Sum of Squares matters in Linear Algebra and Differential Equations

RSS is the number that tells you whether a least squares model is actually close to the data or just technically a solution. In this course, that matters when you are turning a messy system into a matrix problem and then checking whether the approximation is good enough.

It also connects algebra to geometry. When you study projections, orthogonality, and inner products, RSS is the distance measure behind the scenes. A smaller RSS means the predicted vector sits closer to the target vector in the subspace spanned by the columns of the matrix.

You will also see RSS when comparing different models or different choices of basis functions in a continuous least squares problem. If one model gives a smaller RSS on the same data, it captures the pattern better, even if neither model fits perfectly.

A lot of common mistakes come from treating RSS like an average error or a signed error. It is neither. It is a squared total, so the scale of the data matters, and big misses can dominate the result. That is why you often interpret RSS alongside the structure of the problem, not by itself.

How Residual Sum of Squares connects across the course

Residuals

Residuals are the individual differences between observed values and predicted values. RSS is built from those residuals by squaring each one and adding them up, so if you do not know the residuals, you cannot compute RSS. In a problem set, you usually find residuals first, then use them to judge the fit.

Least Squares Method

The least squares method is the process that chooses the model or coefficient vector with the smallest RSS. RSS is the quantity being minimized, so it is the objective function of the method. When a system has no exact solution, least squares gives the best approximation instead of forcing an impossible answer.

Best Approximating Vector

The best approximating vector is the projection of your target vector onto the relevant subspace. Its distance from the target vector determines the residual vector, and the squared length of that residual is the RSS. This is the geometric version of the same idea you use in matrix equations.

Design Matrix

The design matrix organizes the variables or basis functions in a least squares model. Its columns determine the subspace you are projecting onto, so the design matrix controls what kind of predictions are possible and what RSS you can achieve. Changing the matrix changes the fit.

Is Residual Sum of Squares on the Linear Algebra and Differential Equations exam?

A quiz or problem set usually asks you to compute RSS from a table of observed and predicted values, or to explain why one least squares fit is better than another. You may also be given a matrix system and asked to find the coefficient vector that minimizes RSS by solving the normal equations.

If the question is conceptual, you should connect RSS to residuals, projection, and distance. If it is computational, write out each residual, square it, and add them carefully. A common trap is forgetting to square before adding, or mixing up RSS with a single residual or an average error. In a linear algebra setting, your final answer should usually make clear that the smallest RSS corresponds to the closest approximation, not necessarily an exact solution.

Key things to remember about Residual Sum of Squares

• Residual Sum of Squares is the total of the squared differences between observed values and predicted values.

• In least squares problems, RSS is the number you minimize to find the best approximating vector or coefficient vector.

• RSS fits the geometry of Linear Algebra because it measures the squared distance from the data vector to the model subspace.

• A smaller RSS means the model tracks the data more closely, but it does not automatically mean the model is perfect.

• Do not confuse RSS with a single residual, because RSS combines all of the residuals into one total.

Frequently asked questions about Residual Sum of Squares

What is Residual Sum of Squares in Linear Algebra and Differential Equations?

Residual Sum of Squares, or RSS, is the sum of the squared residuals between observed values and the values predicted by a least squares model. In this course, it measures how close a matrix-based approximation is to the target data. The smaller the RSS, the better the fit.

How do you find Residual Sum of Squares?

First find each residual by subtracting the predicted value from the observed value. Then square every residual and add the results. If you are working from a least squares matrix problem, you can also get RSS from the residual vector after finding the best approximating vector.

Is Residual Sum of Squares the same as a residual?

No. A residual is one error term for one data point, while RSS is the total of all squared residuals. A residual can be positive or negative, but RSS is always nonnegative because the residuals are squared before you add them.

Why do we square the residuals instead of just adding them?

Squaring keeps positive and negative errors from canceling each other out. It also makes larger mistakes count more than smaller ones, which gives the least squares method a clear way to choose the best approximation. That is why RSS is the standard error measure in this topic.