8.3 Orthogonal Projections and Complementary Subspaces

Orthogonal projections and complementary subspaces are key concepts in inner product spaces. They allow us to break down vectors into components that are perpendicular to each other, making it easier to analyze and manipulate them.

These ideas are crucial for understanding how vectors interact in higher dimensions. By projecting vectors onto subspaces and finding their orthogonal complements, we can solve complex problems in linear algebra and its applications.

Orthogonal Projections

Definition and Properties

• An orthogonal projection is a linear transformation $P$ from a vector space $V$ to itself such that $P^2 = P$ and the range of $P$ is orthogonal to the null space of $P$
• The matrix representation of an orthogonal projection is symmetric ($P^T = P$) and idempotent ($P^2 = P$)
• Orthogonal projections are characterized by the property that they minimize the distance between a vector and its projection onto a subspace
  • For any vector $v \in V$ and subspace $W \subseteq V$, the orthogonal projection $P_W(v)$ is the closest point in $W$ to $v$
  • Mathematically, $\|v - P_W(v)\| \leq \|v - w\|$ for all $w \in W$
• The eigenvalues of an orthogonal projection matrix are either 0 or 1
  • Eigenvectors corresponding to eigenvalue 1 span the range of the projection
  • Eigenvectors corresponding to eigenvalue 0 span the null space of the projection
• Orthogonal projections preserve the length of vectors in the subspace they project onto
  • For any vector $v \in W$, $\|P_W(v)\| = \|v\|$
  • This follows from the fact that $P_W$ is an identity transformation on $W$

Computing Orthogonal Projections

• Given a basis for a subspace $W$ of a vector space $V$, the orthogonal projection of a vector $v$ onto $W$ can be computed using the Gram-Schmidt process to find an orthonormal basis for $W$
  • The Gram-Schmidt process takes a basis $\{w_1, \ldots, w_k\}$ for $W$ and produces an orthonormal basis $\{u_1, \ldots, u_k\}$
  • The orthogonal projection of $v$ onto $W$ is then given by $P_W(v) = \sum_{i=1}^k \langle v, u_i \rangle u_i$
• The orthogonal projection matrix onto a subspace $W$ is given by $P = A(A^T A)^{-1} A^T$, where $A$ is a matrix whose columns form a basis for $W$
  • This formula can be derived using the properties of inner products and linear algebra
  • If the columns of $A$ are orthonormal, then $A^T A = I$ and the formula simplifies to $P = AA^T$
• The orthogonal projection of a vector $v$ onto a subspace $W$ can be computed as the sum of the inner products of $v$ with each basis vector of $W$, multiplied by the corresponding basis vector
  • If $\{w_1, \ldots, w_k\}$ is a basis for $W$, then $P_W(v) = \sum_{i=1}^k \langle v, w_i \rangle w_i$
  • This formula follows from the linearity of inner products and the properties of bases
• In $\mathbb{R}^n$, the orthogonal projection of a vector $v$ onto a subspace $W$ can be found by solving the linear system $Ax = v$ for $x$, where $A$ is a matrix whose columns form an orthonormal basis for $W$
  • The solution $x$ gives the coordinates of $P_W(v)$ with respect to the orthonormal basis
  • This approach is computationally efficient and numerically stable

Orthogonal Complements

Definition and Properties

• The orthogonal complement of a subspace $W$ in a vector space $V$ is the set of all vectors in $V$ that are orthogonal to every vector in $W$
  • Mathematically, $W^{\perp} = \{v \in V : \langle v, w \rangle = 0 \text{ for all } w \in W\}$
  • Orthogonality is defined in terms of the inner product on $V$
• The orthogonal complement of $W$ is denoted by $W^{\perp}$ and is itself a subspace of $V$
  • This follows from the linearity of inner products and the subspace properties
  • If $v_1, v_2 \in W^{\perp}$ and $c \in \mathbb{R}$, then $\langle cv_1 + v_2, w \rangle = c\langle v_1, w \rangle + \langle v_2, w \rangle = 0$ for all $w \in W$
• For any subspace $W$ of a finite-dimensional vector space $V$, the dimension of $W^{\perp}$ is equal to the codimension of $W$, i.e., $\dim(W^{\perp}) = \dim(V) - \dim(W)$
  • This result follows from the rank-nullity theorem applied to the orthogonal projection map onto $W$
  • The range of the projection has dimension $\dim(W)$, while the null space has dimension $\dim(V) - \dim(W)$
• The orthogonal complement of the orthogonal complement of a subspace $W$ is $W$ itself, i.e., $(W^{\perp})^{\perp} = W$
  • This "double complement" property follows from the definition of orthogonal complements and linear algebra
  • It shows that the orthogonal complement operation is an involution on subspaces
• The existence and uniqueness of orthogonal complements can be proved using the rank-nullity theorem and the properties of inner products
  • For any subspace $W \subseteq V$, the orthogonal projection map onto $W$ is a surjective linear transformation
  • By the rank-nullity theorem, the null space of this map (which is $W^{\perp}$) has dimension $\dim(V) - \dim(W)$
  • The uniqueness of $W^{\perp}$ follows from the fact that orthogonality is a well-defined relation on vectors

Relationship to Orthogonal Projections

• The orthogonal complement of a subspace $W$ is closely related to the orthogonal projection onto $W$
  • The null space of the orthogonal projection map $P_W$ is precisely $W^{\perp}$
  • This means that a vector $v \in V$ is in $W^{\perp}$ if and only if $P_W(v) = 0$
• The orthogonal projection onto $W^{\perp}$ is complementary to the orthogonal projection onto $W$, i.e., $P_{W^{\perp}} = I - P_W$
  • This follows from the properties of orthogonal projections and the definition of orthogonal complements
  • For any vector $v \in V$, we have $v = P_W(v) + P_{W^{\perp}}(v)$, where $P_W(v) \in W$ and $P_{W^{\perp}}(v) \in W^{\perp}$
• The matrix representation of the orthogonal projection onto $W^{\perp}$ can be computed from the matrix representation of the orthogonal projection onto $W$
  • If $P$ is the matrix representing $P_W$, then $I - P$ is the matrix representing $P_{W^{\perp}}$
  • This follows from the complementary property of orthogonal projections and matrix algebra
• Orthogonal complements and orthogonal projections are fundamental tools in the study of inner product spaces and their applications
  • They allow us to decompose vectors and subspaces into orthogonal components
  • They provide a way to find the best approximation of a vector in a given subspace
  • They are used in various fields such as quantum mechanics, signal processing, and data analysis

Orthogonal Subspace Decompositions

Decomposing Vectors

• Any vector $v$ in a vector space $V$ can be uniquely decomposed into a sum of two orthogonal vectors: one in a subspace $W$ and the other in its orthogonal complement $W^{\perp}$
  • Mathematically, $v = P_W(v) + P_{W^{\perp}}(v)$, where $P_W(v) \in W$ and $P_{W^{\perp}}(v) \in W^{\perp}$
  • This decomposition follows from the properties of orthogonal projections and complements
• The orthogonal decomposition of a vector $v$ with respect to a subspace $W$ is given by $v = P_W(v) + (I - P_W)(v)$, where $P_W$ is the orthogonal projection matrix onto $W$ and $I$ is the identity matrix
  • The first term $P_W(v)$ is the orthogonal projection of $v$ onto $W$, representing the component of $v$ in $W$
  • The second term $(I - P_W)(v)$ is the orthogonal projection of $v$ onto $W^{\perp}$, representing the component of $v$ orthogonal to $W$
• The orthogonal decomposition of a vector is unique and independent of the choice of basis for $W$ and $W^{\perp}$
  • This follows from the uniqueness of orthogonal projections and complements
  • Different bases may lead to different expressions for the components, but the resulting decomposition is the same
• Orthogonal decompositions are useful in solving various problems in linear algebra and its applications
  • They allow us to separate a vector into its relevant and irrelevant components with respect to a given subspace
  • They provide a way to find the closest vector in a subspace to a given vector, which is important in least-squares approximations and regression analysis
  • They are used in quantum mechanics to decompose state vectors into orthogonal subspaces corresponding to different observables or symmetries

Decomposing Vector Spaces

• The orthogonal decomposition theorem states that a finite-dimensional vector space $V$ is the direct sum of a subspace $W$ and its orthogonal complement $W^{\perp}$, i.e., $V = W \oplus W^{\perp}$
  • This means that every vector $v \in V$ can be uniquely written as $v = w + u$, where $w \in W$ and $u \in W^{\perp}$
  • The direct sum notation $\oplus$ emphasizes that the decomposition is unique and that $W \cap W^{\perp} = \{0\}$
• The orthogonal decomposition theorem is a consequence of the properties of orthogonal projections and complements
  • The orthogonal projection onto $W$ maps $V$ onto $W$, while the orthogonal projection onto $W^{\perp}$ maps $V$ onto $W^{\perp}$
  • The sum of these two projections is the identity map on $V$, which implies that $V = W + W^{\perp}$
  • The uniqueness of the decomposition follows from the fact that $W \cap W^{\perp} = \{0\}$, which is a consequence of the definition of orthogonal complements
• The orthogonal decomposition theorem can be generalized to more than two subspaces
  • If $W_1, \ldots, W_k$ are mutually orthogonal subspaces of $V$ (i.e., $W_i \perp W_j$ for all $i \neq j$), then $V = W_1 \oplus \cdots \oplus W_k \oplus (W_1 + \cdots + W_k)^{\perp}$
  • This decomposition is called an orthogonal direct sum and is unique
  • It allows us to decompose a vector space into orthogonal components corresponding to different properties or behaviors
• Orthogonal decompositions of vector spaces have numerous applications in mathematics, physics, and engineering
  • In quantum mechanics, the state space of a system is decomposed into orthogonal subspaces corresponding to different eigenvalues of an observable
  • In signal processing, a signal is decomposed into orthogonal components corresponding to different frequencies or time scales (e.g., Fourier or wavelet decompositions)
  • In data analysis, a dataset is decomposed into orthogonal components corresponding to different sources of variation or latent factors (e.g., principal component analysis)

Matrix Decompositions

• The orthogonal decomposition of a matrix $A$ can be used to find its best low-rank approximation, which has applications in data compression, signal processing, and machine learning
  • The best rank-$k$ approximation of $A$ is given by $A_k = U_k \Sigma_k V_k^T$, where $U_k$ and $V_k$ contain the first $k$ columns of $U$ and $V$, respectively, and $\Sigma_k$ contains the first $k$ singular values of $A$
  • This approximation is optimal in the sense that it minimizes the Frobenius norm of the difference between $A$ and any rank-$k$ matrix
  • The matrices $U_k$ and $V_k$ can be interpreted as the principal components of the row and column spaces of $A$, respectively, while the singular values in $\Sigma_k$ represent the importance of each component
• The orthogonal decomposition of a symmetric matrix $A$ is given by its eigendecomposition $A = Q \Lambda Q^T$, where $Q$ is an orthogonal matrix and $\Lambda$ is a diagonal matrix containing the eigenvalues of $A$
  • The columns of $Q$ are eigenvectors of $A$ and form an orthonormal basis for the underlying vector space
  • The eigenvalues in $\Lambda$ represent the variances of the data along each eigenvector direction
  • This decomposition is used in principal component analysis, spectral clustering, and other dimensionality reduction techniques
• The orthogonal decomposition of a matrix can also be used to solve linear systems and least-squares problems
  • If $A = QR$ is the QR decomposition of $A$, where $Q$ is orthogonal and $R$ is upper triangular, then the linear system $Ax = b$ can be solved by first solving $Ry = Q^T b$ for $y$ and then setting $x = Qy$
  • If $A = U \Sigma V^T$ is the singular value decomposition of $A$, then the least-squares solution to $Ax = b$ is given by $x = V \Sigma^+ U^T b$, where $\Sigma^+$ is the pseudoinverse of $\Sigma$
  • These methods are numerically stable and efficient, especially when $A$ is ill-conditioned or has a high condition number
• Matrix decompositions based on orthogonal subspaces are a fundamental tool in numerical linear algebra and have numerous applications in science and engineering
  • They provide a way to reveal the underlying structure and properties of a matrix, such as its rank, range, null space, and singular values
  • They allow us to compress, denoise, or regularize large datasets by focusing on the most important or informative components
  • They enable us to solve various optimization and approximation problems by reducing them to simpler subproblems in orthogonal subspaces