Abstract Linear Algebra II Unit 4 Review: Inner Product Spaces

Key Concepts and Definitions

• Inner product spaces combine vector spaces with an inner product, a generalization of the dot product
• Inner products are denoted as $\langle u, v \rangle$ for vectors $u$ and $v$
  • Must satisfy properties of conjugate symmetry, linearity in the second argument, and positive definiteness
• Norm of a vector $v$ is defined as $\|v\| = \sqrt{\langle v, v \rangle}$, generalizing the concept of length
• Cauchy-Schwarz inequality states that $|\langle u, v \rangle| \leq \|u\| \|v\|$ for any vectors $u$ and $v$
  • Equality holds if and only if $u$ and $v$ are linearly dependent
• Angle between two vectors $u$ and $v$ can be defined using the inner product as $\cos \theta = \frac{\langle u, v \rangle}{\|u\| \|v\|}$
• Orthogonality: two vectors $u$ and $v$ are orthogonal if $\langle u, v \rangle = 0$
• Orthonormal basis is a basis consisting of orthogonal unit vectors (vectors with norm 1)

Properties of Inner Product Spaces

• Inner product spaces are normed vector spaces, with the norm induced by the inner product
• Parallelogram law holds in inner product spaces: $\|u + v\|^2 + \|u - v\|^2 = 2(\|u\|^2 + \|v\|^2)$ for any vectors $u$ and $v$
• Polarization identity allows the inner product to be recovered from the norm: $\langle u, v \rangle = \frac{1}{4}(\|u + v\|^2 - \|u - v\|^2)$
• Orthogonal complements: for any subspace $U$ of an inner product space $V$, there exists an orthogonal complement $U^\perp$ such that $V = U \oplus U^\perp$
  • $U^\perp = \{v \in V : \langle u, v \rangle = 0 \text{ for all } u \in U\}$
• Pythagorean theorem generalizes to inner product spaces: if $u_1, \ldots, u_n$ are pairwise orthogonal, then $\|\sum_{i=1}^n u_i\|^2 = \sum_{i=1}^n \|u_i\|^2$
• Bessel's inequality states that for any orthonormal set $\{e_1, \ldots, e_n\}$ and vector $v$, $\sum_{i=1}^n |\langle v, e_i \rangle|^2 \leq \|v\|^2$
  • Equality holds if and only if $\{e_1, \ldots, e_n\}$ is an orthonormal basis

Examples and Non-Examples

• Euclidean spaces $\mathbb{R}^n$ with the standard dot product are inner product spaces
  • For $u = (u_1, \ldots, u_n)$ and $v = (v_1, \ldots, v_n)$, $\langle u, v \rangle = \sum_{i=1}^n u_i v_i$
• Complex vector spaces $\mathbb{C}^n$ with the Hermitian inner product are inner product spaces
  • For $u = (u_1, \ldots, u_n)$ and $v = (v_1, \ldots, v_n)$, $\langle u, v \rangle = \sum_{i=1}^n \overline{u_i} v_i$
• Space of continuous functions $C[a, b]$ with the inner product $\langle f, g \rangle = \int_a^b f(x) g(x) dx$ is an inner product space
• Space of square-integrable functions $L^2[a, b]$ with the inner product $\langle f, g \rangle = \int_a^b f(x) \overline{g(x)} dx$ is an inner product space
• Not all normed vector spaces are inner product spaces
  • $\ell^p$ spaces with $p \neq 2$ are normed vector spaces but not inner product spaces
• Not all bilinear forms on vector spaces are inner products
  • Bilinear form $B(u, v) = u_1 v_1 - u_2 v_2$ on $\mathbb{R}^2$ is not an inner product (fails positive definiteness)

Orthogonality and Orthonormal Bases

• Orthogonality is a generalization of perpendicularity in inner product spaces
  • Two vectors $u$ and $v$ are orthogonal if $\langle u, v \rangle = 0$
• Orthogonal sets are linearly independent
  • If $\{u_1, \ldots, u_n\}$ is an orthogonal set, then it is linearly independent
• Orthonormal sets are orthogonal sets consisting of unit vectors (vectors with norm 1)
• Orthonormal bases are bases consisting of orthonormal vectors
  • Provide a convenient way to represent vectors and compute inner products
• Parseval's identity: if $\{e_1, \ldots, e_n\}$ is an orthonormal basis and $v = \sum_{i=1}^n \langle v, e_i \rangle e_i$, then $\|v\|^2 = \sum_{i=1}^n |\langle v, e_i \rangle|^2$
• Orthogonal projection of a vector $v$ onto a subspace $U$ is the closest point in $U$ to $v$
  • Can be computed using an orthonormal basis $\{e_1, \ldots, e_n\}$ of $U$ as $\text{proj}_U(v) = \sum_{i=1}^n \langle v, e_i \rangle e_i$

Gram-Schmidt Process

• Gram-Schmidt process is an algorithm for constructing an orthonormal basis from a linearly independent set
• Given a linearly independent set $\{v_1, \ldots, v_n\}$, the Gram-Schmidt process produces an orthonormal set $\{e_1, \ldots, e_n\}$ spanning the same subspace
• Algorithm:
  1. Set $e_1 = \frac{v_1}{\|v_1\|}$
  2. For $i = 2, \ldots, n$:
     • Set $u_i = v_i - \sum_{j=1}^{i-1} \langle v_i, e_j \rangle e_j$
     • Set $e_i = \frac{u_i}{\|u_i\|}$
• Resulting set $\{e_1, \ldots, e_n\}$ is an orthonormal basis for the span of $\{v_1, \ldots, v_n\}$
• Gram-Schmidt process is numerically unstable due to rounding errors
  • Modified Gram-Schmidt process and Householder transformations are more stable alternatives

Projections and Best Approximations

• Orthogonal projection of a vector $v$ onto a subspace $U$ is the closest point in $U$ to $v$
  • Denoted as $\text{proj}_U(v)$ or $P_U(v)$
• Projection theorem states that for any vector $v$ and subspace $U$, $v = \text{proj}_U(v) + \text{proj}_{U^\perp}(v)$
  • Decomposition is unique and orthogonal
• Best approximation problem: given a subspace $U$ and a vector $v$, find the vector $u \in U$ that minimizes $\|v - u\|$
  • Solution is the orthogonal projection $\text{proj}_U(v)$
• Least squares approximation: given a set of data points $(x_i, y_i)$ and a subspace $U$ of functions, find the function $f \in U$ that minimizes $\sum_{i=1}^n (y_i - f(x_i))^2$
  • Solution is the orthogonal projection of the data vector $y = (y_1, \ldots, y_n)$ onto the subspace $\{(f(x_1), \ldots, f(x_n)) : f \in U\}$
• Projections onto orthogonal complements: for any subspace $U$ and vector $v$, $\text{proj}_{U^\perp}(v) = v - \text{proj}_U(v)$

Applications in Linear Algebra

• Orthogonal diagonalization: symmetric matrices can be diagonalized by an orthonormal basis of eigenvectors
  • Eigenvalues are real and eigenvectors corresponding to distinct eigenvalues are orthogonal
• Singular value decomposition (SVD): any matrix $A$ can be written as $A = U \Sigma V^*$, where $U$ and $V$ are orthogonal matrices and $\Sigma$ is a diagonal matrix with non-negative entries
  • Columns of $U$ and $V$ are orthonormal bases for the left and right singular subspaces of $A$
• Principal component analysis (PCA): technique for dimensionality reduction and data compression using orthogonal projections
  • Finds orthogonal directions (principal components) that maximize the variance of the projected data
• Quantum mechanics: state vectors are elements of a complex inner product space (Hilbert space)
  • Observables are represented by Hermitian operators, and their eigenvectors form an orthonormal basis
• Fourier analysis: functions can be represented as linear combinations of orthogonal basis functions (e.g., trigonometric functions, wavelets)
  • Coefficients are computed using inner products (Fourier coefficients)

Common Pitfalls and Misconceptions

• Not all bilinear forms are inner products
  • Inner products must satisfy additional properties (conjugate symmetry, positive definiteness)
• Orthogonality does not imply orthonormality
  • Orthonormal vectors are orthogonal and have unit norm
• Orthogonal projection onto a subspace is not the same as the vector projection (scalar projection)
  • Vector projection computes the component of a vector along another vector
• Orthonormal bases are not unique
  • Any orthonormal basis can be transformed into another by an orthogonal transformation (rotation or reflection)
• Gram-Schmidt process is not the only way to construct orthonormal bases
  • Other methods include Householder transformations and Givens rotations
• Orthogonal matrices are not always symmetric
  • Orthogonal matrices satisfy $A^T A = A A^T = I$, but may not be symmetric ($A^T \neq A$)
• Orthogonal projections do not commute in general
  • $\text{proj}_U(\text{proj}_V(v)) \neq \text{proj}_V(\text{proj}_U(v))$ unless $U$ and $V$ are orthogonal subspaces