Key Concepts of Orthogonality

Why This Matters

Orthogonality is one of those foundational ideas that shows up everywhere in linear algebra—and if you're taking this course, you're being tested on your ability to recognize when and why vectors, matrices, and even functions are "perpendicular" in a generalized sense. The concepts here connect directly to solving systems of equations, simplifying matrix computations, data compression, and signal processing. When you understand orthogonality, you unlock powerful tools for breaking complex problems into simpler, independent pieces.

Here's the key insight: orthogonality isn't just about right angles in 2D or 3D space. It's about independence and efficiency—orthogonal components don't interfere with each other, which makes calculations cleaner and solutions more stable. Don't just memorize definitions—know what principle each concept demonstrates and how it connects to practical applications like least squares regression, QR decomposition, and SVD.

---

Foundational Definitions and the Dot Product

Before diving into applications, you need rock-solid understanding of what orthogonality actually means. The dot product is your primary tool for detecting orthogonality—when it equals zero, you've found perpendicular vectors.

Definition of Orthogonal Vectors

• Two vectors are orthogonal if their dot product equals zero—this is your go-to test for orthogonality in any dimension
• Orthogonal vectors are linearly independent, meaning neither can be written as a scalar multiple of the other
• Geometric interpretation: orthogonal vectors meet at right angles in Euclidean space, forming the basis for coordinate systems

Dot Product and Its Relation to Orthogonality

• The dot product formula $\mathbf{a} \cdot \mathbf{b} = ||\mathbf{a}|| \, ||\mathbf{b}|| \cos(\theta)$—when $\theta = 90°$, cosine is zero
• Computational form: $\mathbf{a} \cdot \mathbf{b} = a_1b_1 + a_2b_2 + \cdots + a_nb_n$ gives you a quick calculation method
• Sign interpretation: positive means acute angle, negative means obtuse, zero means orthogonal

Orthonormal Basis

• Orthonormal vectors are orthogonal AND unit length—they satisfy both $\mathbf{u}_i \cdot \mathbf{u}_j = 0$ for $i \neq j$ and $||\mathbf{u}_i|| = 1$
• Coordinate extraction becomes trivial: any vector's coefficient is simply its dot product with the basis vector
• Standard basis $\{\mathbf{e}_1, \mathbf{e}_2, \ldots\}$ is the classic example—this is why Cartesian coordinates work so cleanly

---

Building Orthogonal Sets: Gram-Schmidt and Projections

These techniques let you construct orthogonal vectors from arbitrary starting points. The core mechanism is subtraction of projections—you remove the component that lies along previously established directions.

Orthogonal Projections

• Projection formula: $\text{proj}_{\mathbf{b}} \mathbf{a} = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{b} \cdot \mathbf{b}} \mathbf{b}$—this extracts the component of $\mathbf{a}$ in the direction of $\mathbf{b}$
• Minimization property: the projection minimizes the distance between $\mathbf{a}$ and any scalar multiple of $\mathbf{b}$
• Residual is orthogonal: $\mathbf{a} - \text{proj}_{\mathbf{b}} \mathbf{a}$ is perpendicular to $\mathbf{b}$—this fact drives Gram-Schmidt

Gram-Schmidt Orthogonalization Process

• Input: any set of linearly independent vectors; Output: orthogonal vectors spanning the same subspace
• Algorithm: subtract projections onto all previously orthogonalized vectors—$\mathbf{v}_k = \mathbf{u}_k - \sum_{j=1}^{k-1} \text{proj}_{\mathbf{v}_j} \mathbf{u}_k$
• Normalize at the end to get an orthonormal set—divide each vector by its magnitude

Orthogonal Complements

• Definition: $V^\perp$ contains all vectors orthogonal to every vector in subspace $V$
• Dimension relationship: $\dim(V) + \dim(V^\perp) = n$ for subspaces of $\mathbb{R}^n$
• Direct sum property: $V \oplus V^\perp = \mathbb{R}^n$—any vector decomposes uniquely into components from each

---

Orthogonal Matrices and Transformations

When orthogonality is built into a matrix's structure, you get transformations that preserve geometry. These matrices satisfy $A^T A = I$, meaning the transpose equals the inverse.

Orthogonal Matrices

• Defining property: $A^T = A^{-1}$, equivalently $A^T A = AA^T = I$
• Column (and row) structure: columns form an orthonormal set—this is how you verify orthogonality
• Determinant constraint: $\det(A) = \pm 1$ always; +1 for rotations, -1 for reflections

Orthogonal Transformations

• Length preservation: $||A\mathbf{x}|| = ||\mathbf{x}||$ for all vectors—distances don't change
• Angle preservation: the angle between any two vectors remains unchanged after transformation
• Geometric examples: rotations and reflections in $\mathbb{R}^n$ are the classic orthogonal transformations

Orthogonal Diagonalization

• Applies to symmetric matrices: if $A = A^T$, then $A = QDQ^T$ where $Q$ is orthogonal and $D$ is diagonal
• Spectral theorem: the columns of $Q$ are orthonormal eigenvectors of $A$
• Computational advantage: powers and functions of $A$ become trivial—$A^n = QD^nQ^T$

---

Matrix Decompositions Using Orthogonality

These decompositions leverage orthogonality to solve practical problems efficiently. The key insight is that orthogonal factors are numerically stable and easy to invert.

QR Decomposition

• Factorization: $A = QR$ where $Q$ is orthogonal and $R$ is upper triangular
• Construction method: apply Gram-Schmidt to columns of $A$; the orthonormal results form $Q$
• Applications: solving $A\mathbf{x} = \mathbf{b}$ becomes $R\mathbf{x} = Q^T\mathbf{b}$, which is easy back-substitution

Singular Value Decomposition (SVD)

• Factorization: $A = U\Sigma V^T$ where $U$ and $V$ are orthogonal, $\Sigma$ is diagonal with singular values
• Works for ANY matrix—not just square or symmetric; this is its major advantage
• Reveals matrix structure: rank equals number of nonzero singular values; best rank-$k$ approximation uses top $k$ singular values

Orthogonality in Least Squares Problems

• Key insight: the residual $\mathbf{b} - A\hat{\mathbf{x}}$ is orthogonal to the column space of $A$
• Normal equations: $A^T A \hat{\mathbf{x}} = A^T \mathbf{b}$ arise directly from this orthogonality condition
• Best fit interpretation: orthogonality ensures minimum squared error—no other solution gets closer

---

Orthogonality in Function Spaces

The same principles extend beyond vectors to functions. The inner product becomes an integral, and "orthogonal" means the integral of the product equals zero.

Orthogonality in Function Spaces

• Inner product definition: $\langle f, g \rangle = \int_a^b f(x)g(x) \, dx$—functions are orthogonal when this equals zero
• Weight functions: sometimes $\langle f, g \rangle = \int_a^b w(x)f(x)g(x) \, dx$ with weight $w(x) > 0$
• Infinite-dimensional analogy: function spaces work like $\mathbb{R}^n$ but with infinitely many "directions"

Orthogonal Polynomials

• Definition: polynomials $\{P_0, P_1, P_2, \ldots\}$ satisfying $\langle P_m, P_n \rangle = 0$ for $m \neq n$
• Famous families: Legendre polynomials (weight 1 on $[-1,1]$), Chebyshev polynomials (weight $1/\sqrt{1-x^2}$)
• Applications: numerical integration, polynomial approximation, solving differential equations

Fourier Series and Orthogonal Functions

• Basis functions: $\{1, \cos(x), \sin(x), \cos(2x), \sin(2x), \ldots\}$ are orthogonal on $[0, 2\pi]$
• Coefficient extraction: $a_n = \frac{\langle f, \cos(nx) \rangle}{\langle \cos(nx), \cos(nx) \rangle}$—orthogonality makes this work
• Signal processing foundation: decomposing signals into frequency components relies entirely on this orthogonality

---

Quick Reference Table

---

Self-Check Questions

1. If you're given a set of three vectors and asked to produce an orthonormal basis for their span, which process do you use, and what's the key operation at each step?

2. Compare and contrast QR decomposition and SVD: what types of matrices can each handle, and when would you choose one over the other for solving $A\mathbf{x} = \mathbf{b}$?

3. A symmetric matrix $A$ can be orthogonally diagonalized as $A = QDQ^T$. What special property do the columns of $Q$ have, and why does this make computing $A^{100}$ straightforward?

4. In a least squares problem, the residual vector is orthogonal to what? Explain why this orthogonality condition guarantees the minimum error solution.

5. How does the concept of orthogonality in $\mathbb{R}^n$ (dot product = 0) generalize to function spaces, and why does this make Fourier coefficient calculation possible?