5.3 Orthogonality, projections, and Gram-Schmidt process

Orthogonality in vector spaces is all about perpendicular relationships between vectors. It's a key concept in inner product spaces and Hilbert spaces, where we can measure angles and distances between vectors using inner products.

Orthogonal projections are super useful for finding the closest point in a subspace to a given vector. We can compute these projections using inner products and orthonormal bases, which helps solve problems in many fields like signal processing and data analysis.

Orthogonality and Inner Product Spaces

Orthogonality in vector spaces

• Orthogonality is a geometric concept describing the perpendicular relationship between vectors in inner product spaces and Hilbert spaces
• Two vectors $x$ and $y$ are orthogonal if their inner product equals zero: $\langle x, y \rangle = 0$
• Inner product spaces are vector spaces with an inner product function assigning a scalar value to each pair of vectors
  • Inner product satisfies properties for all vectors $x, y, z$ and scalars $a$:
    • Conjugate symmetry: $\langle x, y \rangle = \overline{\langle y, x \rangle}$
    • Linearity in the second argument: $\langle x, ay + z \rangle = a\langle x, y \rangle + \langle x, z \rangle$
    • Positive definiteness: $\langle x, x \rangle \geq 0$ and $\langle x, x \rangle = 0$ if and only if $x = 0$
• Hilbert spaces are complete inner product spaces where every Cauchy sequence converges to an element within the space
  • Completeness allows application of powerful tools and theorems in functional analysis ($L^2$ spaces, Fourier series)

Computation of orthogonal projections

• Orthogonal projections find the closest point in a subspace to a given vector
• The orthogonal projection of vector $x$ onto closed subspace $M$ is the unique vector $P_M(x)$ in $M$ minimizing distance between $x$ and any vector in $M$
  • Mathematically: $\|x - P_M(x)\| = \min_{y \in M} \|x - y\|$
• To compute the orthogonal projection of vector $x$ onto subspace $M$ spanned by orthonormal basis $\{e_1, e_2, \ldots, e_n\}$:
  • $P_M(x) = \sum_{i=1}^n \langle x, e_i \rangle e_i$
• Vector $x - P_M(x)$ is orthogonal to every vector in $M$, representing the component of $x$ perpendicular to the subspace (residual, error term)

Gram-Schmidt Process and Orthonormal Bases

Gram-Schmidt process for orthonormal bases

• The Gram-Schmidt process constructs an orthonormal basis from a linearly independent set of vectors
• Given linearly independent set $\{v_1, v_2, \ldots, v_n\}$:
  1. Define $e_1 = \frac{v_1}{\|v_1\|}$
  2. For $i = 2, 3, \ldots, n$:
     • Compute $u_i = v_i - \sum_{j=1}^{i-1} \langle v_i, e_j \rangle e_j$
     • Define $e_i = \frac{u_i}{\|u_i\|}$
• The resulting set $\{e_1, e_2, \ldots, e_n\}$ is an orthonormal basis for the space spanned by the original linearly independent set
• Orthonormal bases simplify calculations and provide a convenient coordinate system (QR factorization, least squares)

Properties of orthogonal projections

• In Hilbert spaces, orthogonal projections onto closed subspaces always exist and are unique
• Existence proof:
  • Let $M$ be a closed subspace of Hilbert space $H$ and $x \in H$
  • Show set $\{x - y : y \in M\}$ contains a unique element of minimal norm using parallelogram law and completeness of $M$
  • This element is the orthogonal projection of $x$ onto $M$
• Uniqueness proof:
  • Suppose two orthogonal projections $P_M(x)$ and $Q_M(x)$ of $x$ onto $M$ exist
  • Show $\|P_M(x) - Q_M(x)\|^2 = 0$ using properties of orthogonal projections and Pythagorean theorem
  • Conclude $P_M(x) = Q_M(x)$, proving uniqueness
• Existence and uniqueness of orthogonal projections underlie many applications (least squares approximation, signal processing)