Fundamental Linear Algebra Concepts

Why This Matters

Linear algebra isn't just abstract mathematics—it's the language that makes linear modeling work. Every regression model you build, every system you solve, and every transformation you analyze relies on the concepts covered here. You're being tested on your ability to understand why matrices represent transformations, how vector spaces constrain solutions, and what decompositions reveal about system behavior. These fundamentals show up everywhere: from solving $Ax = b$ to understanding why your least squares solution is optimal.

Think of this section as your toolkit. Vectors and matrices are your basic instruments, but the real power comes from understanding concepts like linear independence, span, orthogonality, and eigenstructure. Don't just memorize definitions—know what each concept tells you about the structure of your data and the behavior of your models. When an exam asks about solution existence or model stability, you need to connect these foundational ideas to practical outcomes.

---

Building Blocks: Vectors and Matrices

These are the fundamental objects you'll manipulate throughout linear modeling. Every linear model ultimately reduces to operations on vectors and matrices.

Vectors and Vector Operations

• Vectors represent quantities with both magnitude and direction—in modeling contexts, think of them as data points, coefficient lists, or directions in parameter space
• Key operations include addition, scalar multiplication, and the dot product $\mathbf{u} \cdot \mathbf{v} = \sum u_i v_i$, which measures alignment between vectors
• Dimensionality determines the space in which your model operates—a vector in $\mathbb{R}^n$ has $n$ components and lives in $n$-dimensional space

Matrices and Matrix Operations

• A matrix is a rectangular array that can represent linear transformations, systems of equations, or data organized in rows and columns
• Matrix multiplication $AB$ composes transformations—the order matters since $AB \neq BA$ in general
• The inverse $A^{-1}$ allows you to solve $Ax = b$ directly as $x = A^{-1}b$, but only when the inverse exists

---

Structure of Vector Spaces

Understanding how vectors combine and what spaces they generate is essential for analyzing solution sets and model constraints. These concepts determine whether solutions exist and how many you'll find.

Linear Combinations and Linear Independence

• A linear combination $c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \cdots + c_n\mathbf{v}_n$ creates new vectors from existing ones using scalar weights
• Linear independence means no redundancy—vectors are independent if the only solution to $c_1\mathbf{v}_1 + \cdots + c_n\mathbf{v}_n = \mathbf{0}$ is all $c_i = 0$
• Dependent vectors indicate redundant information in your model, which affects rank and solution uniqueness

Span and Basis

• The span of vectors is all possible linear combinations—it defines the subspace those vectors can "reach"
• A basis is a minimal spanning set—linearly independent vectors that span the entire space, giving the most efficient representation
• Dimension equals the number of basis vectors, telling you the degrees of freedom in your space

Vector Spaces and Subspaces

• A vector space satisfies closure axioms—you can add any two vectors and multiply by any scalar without leaving the space
• Subspaces are vector spaces contained within larger spaces, such as the column space or null space of a matrix
• The four fundamental subspaces (column space, null space, row space, left null space) completely characterize a matrix's behavior

---

Transformations and Mappings

Linear transformations are functions that preserve the structure of vector spaces. Matrices are simply the computational representation of these transformations.

Linear Transformations

• Linearity means $T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})$ and $T(c\mathbf{v}) = cT(\mathbf{v})$—the transformation respects addition and scaling
• Every linear transformation has a matrix representation, so analyzing transformations reduces to analyzing matrices
• The kernel (null space) and image (column space) of a transformation reveal what gets "lost" and what can be "reached"

Eigenvalues and Eigenvectors

• Eigenvectors are special directions that only get scaled (not rotated) by a transformation: $A\mathbf{v} = \lambda\mathbf{v}$
• Eigenvalues $\lambda$ indicate the scaling factor—positive means same direction, negative means reversal, zero means collapse
• Applications include stability analysis (eigenvalues determine system behavior) and PCA (eigenvectors of covariance matrices identify principal directions)

---

Solving Systems: Methods and Structure

The core application of linear algebra in modeling is solving $Ax = b$. Different methods and decompositions reveal different aspects of the solution.

Systems of Linear Equations

• A system $Ax = b$ can have one solution, infinitely many, or none—determined by comparing rank($A$) to rank($[A|b]$) and the number of variables
• Row reduction (Gaussian elimination) transforms the system to echelon form, making solutions readable
• Homogeneous systems $Ax = \mathbf{0}$ always have at least the trivial solution—nontrivial solutions exist when columns are linearly dependent

Matrix Decomposition (LU, QR)

• LU decomposition writes $A = LU$ with lower and upper triangular factors, enabling efficient solving via forward and back substitution
• QR decomposition writes $A = QR$ with orthogonal $Q$ and upper triangular $R$, essential for least squares problems
• Decompositions trade one hard problem for multiple easy ones—triangular systems and orthogonal matrices are computationally friendly

---

Geometry and Optimization

Orthogonality provides geometric insight that's crucial for optimization, particularly in least squares problems. Perpendicularity means independence, and projections minimize error.

Orthogonality and Projections

• Orthogonal vectors satisfy $\mathbf{u} \cdot \mathbf{v} = 0$, meaning they're perpendicular and carry independent information
• The projection of $\mathbf{b}$ onto a subspace finds the closest point in that subspace—computed as $\text{proj}_A(\mathbf{b}) = A(A^TA)^{-1}A^T\mathbf{b}$
• Least squares solutions minimize $\|Ax - b\|^2$ by projecting $b$ onto the column space of $A$, making the residual orthogonal to all columns

---

Quick Reference Table

---

Self-Check Questions

1. What do linear independence and orthogonality have in common, and how do they differ? Which property is stronger?

2. Given a system $Ax = b$ where $A$ is $m \times n$ with $m > n$, which decomposition would you use to find the least squares solution, and why?

3. If a matrix has an eigenvalue of zero, what does this tell you about its invertibility and its null space?

4. Compare the column space and null space of a matrix—how do their dimensions relate, and what does each tell you about solutions to $Ax = b$?

5. An FRQ asks you to explain why the least squares residual $b - A\hat{x}$ is orthogonal to every column of $A$. Which concepts from this guide would you connect in your answer?