Fundamental Linear Algebra Concepts

Why This Matters

Linear algebra is the language that makes linear modeling work. Every regression model you build, every system you solve, and every transformation you analyze relies on the concepts covered here. You need to understand why matrices represent transformations, how vector spaces constrain solutions, and what decompositions reveal about system behavior. These fundamentals show up everywhere: from solving $Ax = b$ to understanding why your least squares solution is optimal.

Think of this as your toolkit. Vectors and matrices are the basic instruments, but the real power comes from understanding concepts like linear independence, span, orthogonality, and eigenstructure. Don't just memorize definitions. Know what each concept tells you about the structure of your data and the behavior of your models. When a problem asks about solution existence or model stability, you need to connect these foundational ideas to practical outcomes.

---

Building Blocks: Vectors and Matrices

These are the fundamental objects you'll manipulate throughout linear modeling. Every linear model ultimately reduces to operations on vectors and matrices.

Vectors and Vector Operations

• Vectors represent quantities with both magnitude and direction. In modeling contexts, think of them as data points, coefficient lists, or directions in parameter space.
• Key operations include addition, scalar multiplication, and the dot product $\mathbf{u} \cdot \mathbf{v} = \sum u_i v_i$, which measures alignment between vectors. A dot product of zero means the vectors are perpendicular; a large positive value means they point in similar directions.
• Dimensionality determines the space in which your model operates. A vector in $\mathbb{R}^n$ has $n$ components and lives in $n$-dimensional space.

Matrices and Matrix Operations

• A matrix is a rectangular array that can represent linear transformations, systems of equations, or data organized in rows and columns. An $m \times n$ matrix has $m$ rows and $n$ columns.
• Matrix multiplication $AB$ composes transformations. The order matters because $AB \neq BA$ in general. For the product to be defined, the number of columns in $A$ must equal the number of rows in $B$.
• The transpose $A^T$ flips rows and columns. It shows up constantly in linear modeling, especially in the normal equations $A^T A \hat{x} = A^T b$.
• The inverse $A^{-1}$ allows you to solve $Ax = b$ directly as $x = A^{-1}b$, but only when $A$ is square and has full rank. If the determinant is zero, no inverse exists.

---

Structure of Vector Spaces

Understanding how vectors combine and what spaces they generate is essential for analyzing solution sets and model constraints. These concepts determine whether solutions exist and how many you'll find.

Linear Combinations and Linear Independence

• A linear combination $c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \cdots + c_n\mathbf{v}_n$ creates new vectors from existing ones using scalar weights.
• Linear independence means no redundancy. Vectors are independent if the only solution to $c_1\mathbf{v}_1 + \cdots + c_n\mathbf{v}_n = \mathbf{0}$ is all $c_i = 0$. In other words, no vector in the set can be written as a linear combination of the others.
• Dependent vectors indicate redundant information in your model. This reduces the rank and directly affects solution uniqueness. For example, if two predictor columns in a design matrix are proportional, they're linearly dependent, and you can't uniquely estimate both coefficients.

Span and Basis

• The span of a set of vectors is all possible linear combinations of those vectors. It defines the subspace they can "reach."
• A basis is a minimal spanning set: linearly independent vectors that span the entire space. Any vector in the space can be written as a unique linear combination of basis vectors.
• Dimension equals the number of basis vectors, telling you the degrees of freedom in your space. For $\mathbb{R}^3$, any basis has exactly 3 vectors.

Vector Spaces and Subspaces

• A vector space satisfies closure under addition and scalar multiplication. You can add any two vectors and multiply by any scalar without leaving the space. It must also contain the zero vector.
• Subspaces are vector spaces contained within larger spaces, such as the column space or null space of a matrix.
• The four fundamental subspaces of a matrix $A$ ($m \times n$) completely characterize its behavior:
  • Column space $\text{Col}(A)$: all possible outputs $A\mathbf{x}$; lives in $\mathbb{R}^m$
  • Null space $\text{Null}(A)$: all $\mathbf{x}$ satisfying $A\mathbf{x} = \mathbf{0}$; lives in $\mathbb{R}^n$
  • Row space $\text{Row}(A)$: the column space of $A^T$; lives in $\mathbb{R}^n$
  • Left null space $\text{Null}(A^T)$: lives in $\mathbb{R}^m$

The rank-nullity theorem ties these together: $\text{rank}(A) + \text{nullity}(A) = n$, where $n$ is the number of columns. The rank equals the dimension of the column space (and the row space), and the nullity equals the dimension of the null space.

---

Transformations and Mappings

Linear transformations are functions that preserve the structure of vector spaces. Matrices are the computational representation of these transformations.

Linear Transformations

• Linearity means $T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})$ and $T(c\mathbf{v}) = cT(\mathbf{v})$. The transformation respects addition and scaling.
• Every linear transformation from $\mathbb{R}^n$ to $\mathbb{R}^m$ has an $m \times n$ matrix representation, so analyzing transformations reduces to analyzing matrices.
• The kernel (null space) and image (column space) of a transformation reveal what gets "lost" and what can be "reached." If the kernel is just $\{\mathbf{0}\}$, the transformation is one-to-one (injective). If the image equals the entire codomain, it's onto (surjective).

Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors identify the "natural" directions of a transformation where behavior is simplest.

• Eigenvectors are directions that only get scaled (not rotated) by a transformation: $A\mathbf{v} = \lambda\mathbf{v}$. You find them by solving $\det(A - \lambda I) = 0$ for the eigenvalues $\lambda$, then solving $(A - \lambda I)\mathbf{v} = \mathbf{0}$ for each eigenvector.
• Eigenvalues $\lambda$ indicate the scaling factor. Positive means same direction, negative means reversal, and zero means collapse to the origin along that direction.
• Applications include stability analysis (eigenvalues determine whether a system grows, decays, or oscillates) and PCA (eigenvectors of the covariance matrix identify the directions of greatest variance in your data).

---

Solving Systems: Methods and Structure

The core application of linear algebra in modeling is solving $Ax = b$. Different methods and decompositions reveal different aspects of the solution.

Systems of Linear Equations

A system $Ax = b$ can have one solution, infinitely many, or none. Here's how to determine which:

1. Compute $\text{rank}(A)$ and $\text{rank}([A|b])$ (the augmented matrix).
2. If $\text{rank}(A) < \text{rank}([A|b])$, the system is inconsistent (no solution).
3. If $\text{rank}(A) = \text{rank}([A|b]) = n$ (number of unknowns), there's exactly one solution.
4. If $\text{rank}(A) = \text{rank}([A|b]) < n$, there are infinitely many solutions, parameterized by $n - \text{rank}(A)$ free variables.

• Row reduction (Gaussian elimination) transforms the system to echelon form, making solutions readable by back substitution.
• Homogeneous systems $Ax = \mathbf{0}$ always have at least the trivial solution $\mathbf{x} = \mathbf{0}$. Nontrivial solutions exist when the columns of $A$ are linearly dependent (i.e., when $\text{rank}(A) < n$).

Matrix Decomposition (LU, QR)

Decompositions trade one hard problem for multiple easy ones. Triangular systems and orthogonal matrices are computationally friendly.

• LU decomposition writes $A = LU$ where $L$ is lower triangular and $U$ is upper triangular. To solve $Ax = b$, you first solve $Ly = b$ (forward substitution), then $Ux = y$ (back substitution). This is efficient when you need to solve the same system with multiple right-hand sides.
• QR decomposition writes $A = QR$ where $Q$ is orthogonal ($Q^TQ = I$) and $R$ is upper triangular. This is essential for least squares problems because the normal equations simplify: since $Q^TQ = I$, you get $R\hat{x} = Q^Tb$, which is a simple triangular solve. QR is also more numerically stable than forming $A^TA$ directly.

---

Geometry and Optimization

Orthogonality provides geometric insight that's crucial for optimization, particularly in least squares problems. Perpendicularity means independence, and projections minimize error.

Orthogonality and Projections

• Orthogonal vectors satisfy $\mathbf{u} \cdot \mathbf{v} = 0$, meaning they're perpendicular and carry completely independent information.
• The projection of $\mathbf{b}$ onto the column space of $A$ finds the closest point in that subspace to $\mathbf{b}$. The projection matrix is $P = A(A^TA)^{-1}A^T$, so the projection is $\hat{\mathbf{b}} = P\mathbf{b}$. This formula requires $A^TA$ to be invertible, which happens when $A$ has full column rank.
• Least squares solutions minimize $\|A\mathbf{x} - \mathbf{b}\|^2$ by projecting $\mathbf{b}$ onto the column space of $A$. The residual $\mathbf{b} - A\hat{\mathbf{x}}$ is orthogonal to every column of $A$, which is exactly the condition $A^T(\mathbf{b} - A\hat{\mathbf{x}}) = \mathbf{0}$. Rearranging gives the normal equations: $A^TA\hat{\mathbf{x}} = A^T\mathbf{b}$.

This geometric picture is worth internalizing: the least squares solution isn't some arbitrary "best fit." It's the unique point where the error vector is perpendicular to the space of all possible predictions.

---

Quick Reference Table

---

Self-Check Questions

1. What do linear independence and orthogonality have in common, and how do they differ? Which property is stronger, and why?

2. Given a system $Ax = b$ where $A$ is $m \times n$ with $m > n$, which decomposition would you use to find the least squares solution, and why is it preferred over forming $A^TA$ directly?

3. If a matrix has an eigenvalue of zero, what does this tell you about its invertibility, its determinant, and its null space?

4. Compare the column space and null space of a matrix. How do their dimensions relate through the rank-nullity theorem, and what does each tell you about solutions to $Ax = b$?

5. Explain why the least squares residual $\mathbf{b} - A\hat{\mathbf{x}}$ is orthogonal to every column of $A$. Which concepts from this guide connect in your answer?