abstract linear algebra ii unit 4 study guides

Key Concepts and Definitions

• Inner product spaces combine vector spaces with an inner product, a generalization of the dot product
• Inner products are denoted as $\langle u, v \rangle$ for vectors $u$ and $v$
  • Must satisfy properties of conjugate symmetry, linearity in the second argument, and positive definiteness
• Norm of a vector $v$ is defined as $|v| = \sqrt{\langle v, v \rangle}$, generalizing the concept of length
• Cauchy-Schwarz inequality states that $|\langle u, v \rangle| \leq |u| |v|$ for any vectors $u$ and $v$
  • Equality holds if and only if $u$ and $v$ are linearly dependent
• Angle between two vectors $u$ and $v$ can be defined using the inner product as $\cos \theta = \frac{\langle u, v \rangle}{|u| |v|}$
• Orthogonality: two vectors $u$ and $v$ are orthogonal if $\langle u, v \rangle = 0$
• Orthonormal basis is a basis consisting of orthogonal unit vectors (vectors with norm 1)

Properties of Inner Product Spaces

• Inner product spaces are normed vector spaces, with the norm induced by the inner product
• Parallelogram law holds in inner product spaces: $|u + v|^2 + |u - v|^2 = 2(|u|^2 + |v|^2)$ for any vectors $u$ and $v$
• Polarization identity allows the inner product to be recovered from the norm: $\langle u, v \rangle = \frac{1}{4}(|u + v|^2 - |u - v|^2)$
• Orthogonal complements: for any subspace $U$ of an inner product space $V$, there exists an orthogonal complement $U^\perp$ such that $V = U \oplus U^\perp$
  • $U^\perp = {v \in V : \langle u, v \rangle = 0 \text{ for all } u \in U}$
• Pythagorean theorem generalizes to inner product spaces: if $u_1, \ldots, u_n$ are pairwise orthogonal, then $|\sum_{i=1}^n u_i|^2 = \sum_{i=1}^n |u_i|^2$
• Bessel's inequality states that for any orthonormal set ${e_1, \ldots, e_n}$ and vector $v$, $\sum_{i=1}^n |\langle v, e_i \rangle|^2 \leq |v|^2$
  • Equality holds if and only if ${e_1, \ldots, e_n}$ is an orthonormal basis

Examples and Non-Examples

• Euclidean spaces $\mathbb{R}^n$ with the standard dot product are inner product spaces
  • For $u = (u_1, \ldots, u_n)$ and $v = (v_1, \ldots, v_n)$, $\langle u, v \rangle = \sum_{i=1}^n u_i v_i$
• Complex vector spaces $\mathbb{C}^n$ with the Hermitian inner product are inner product spaces
  • For $u = (u_1, \ldots, u_n)$ and $v = (v_1, \ldots, v_n)$, $\langle u, v \rangle = \sum_{i=1}^n \overline{u_i} v_i$
• Space of continuous functions $C[a, b]$ with the inner product $\langle f, g \rangle = \int_a^b f(x) g(x) dx$ is an inner product space
• Space of square-integrable functions $L^2[a, b]$ with the inner product $\langle f, g \rangle = \int_a^b f(x) \overline{g(x)} dx$ is an inner product space
• Not all normed vector spaces are inner product spaces
  • $\ell^p$ spaces with $p \neq 2$ are normed vector spaces but not inner product spaces
• Not all bilinear forms on vector spaces are inner products
  • Bilinear form $B(u, v) = u_1 v_1 - u_2 v_2$ on $\mathbb{R}^2$ is not an inner product (fails positive definiteness)

Orthogonality and Orthonormal Bases

• Orthogonality is a generalization of perpendicularity in inner product spaces
  • Two vectors $u$ and $v$ are orthogonal if $\langle u, v \rangle = 0$
• Orthogonal sets are linearly independent
  • If ${u_1, \ldots, u_n}$ is an orthogonal set, then it is linearly independent
• Orthonormal sets are orthogonal sets consisting of unit vectors (vectors with norm 1)
• Orthonormal bases are bases consisting of orthonormal vectors
  • Provide a convenient way to represent vectors and compute inner products
• Parseval's identity: if ${e_1, \ldots, e_n}$ is an orthonormal basis and $v = \sum_{i=1}^n \langle v, e_i \rangle e_i$, then $|v|^2 = \sum_{i=1}^n |\langle v, e_i \rangle|^2$
• Orthogonal projection of a vector $v$ onto a subspace $U$ is the closest point in $U$ to $v$
  • Can be computed using an orthonormal basis ${e_1, \ldots, e_n}$ of $U$ as $\text{proj}U(v) = \sum{i=1}^n \langle v, e_i \rangle e_i$

Gram-Schmidt Process

• Gram-Schmidt process is an algorithm for constructing an orthonormal basis from a linearly independent set
• Given a linearly independent set ${v_1, \ldots, v_n}$, the Gram-Schmidt process produces an orthonormal set ${e_1, \ldots, e_n}$ spanning the same subspace
• Algorithm:
  1. Set $e_1 = \frac{v_1}{|v_1|}$
  2. For $i = 2, \ldots, n$:
     • Set $u_i = v_i - \sum_{j=1}^{i-1} \langle v_i, e_j \rangle e_j$
     • Set $e_i = \frac{u_i}{|u_i|}$
• Resulting set ${e_1, \ldots, e_n}$ is an orthonormal basis for the span of ${v_1, \ldots, v_n}$
• Gram-Schmidt process is numerically unstable due to rounding errors
  • Modified Gram-Schmidt process and Householder transformations are more stable alternatives

Projections and Best Approximations

• Orthogonal projection of a vector $v$ onto a subspace $U$ is the closest point in $U$ to $v$
  • Denoted as $\text{proj}_U(v)$ or $P_U(v)$
• Projection theorem states that for any vector $v$ and subspace $U$, $v = \text{proj}U(v) + \text{proj}{U^\perp}(v)$
  • Decomposition is unique and orthogonal
• Best approximation problem: given a subspace $U$ and a vector $v$, find the vector $u \in U$ that minimizes $|v - u|$
  • Solution is the orthogonal projection $\text{proj}_U(v)$
• Least squares approximation: given a set of data points $(x_i, y_i)$ and a subspace $U$ of functions, find the function $f \in U$ that minimizes $\sum_{i=1}^n (y_i - f(x_i))^2$
  • Solution is the orthogonal projection of the data vector $y = (y_1, \ldots, y_n)$ onto the subspace ${(f(x_1), \ldots, f(x_n)) : f \in U}$
• Projections onto orthogonal complements: for any subspace $U$ and vector $v$, $\text{proj}_{U^\perp}(v) = v - \text{proj}_U(v)$

Applications in Linear Algebra

• Orthogonal diagonalization: symmetric matrices can be diagonalized by an orthonormal basis of eigenvectors
  • Eigenvalues are real and eigenvectors corresponding to distinct eigenvalues are orthogonal
• Singular value decomposition (SVD): any matrix $A$ can be written as $A = U \Sigma V^*$, where $U$ and $V$ are orthogonal matrices and $\Sigma$ is a diagonal matrix with non-negative entries
  • Columns of $U$ and $V$ are orthonormal bases for the left and right singular subspaces of $A$
• Principal component analysis (PCA): technique for dimensionality reduction and data compression using orthogonal projections
  • Finds orthogonal directions (principal components) that maximize the variance of the projected data
• Quantum mechanics: state vectors are elements of a complex inner product space (Hilbert space)
  • Observables are represented by Hermitian operators, and their eigenvectors form an orthonormal basis
• Fourier analysis: functions can be represented as linear combinations of orthogonal basis functions (e.g., trigonometric functions, wavelets)
  • Coefficients are computed using inner products (Fourier coefficients)

Common Pitfalls and Misconceptions

• Not all bilinear forms are inner products
  • Inner products must satisfy additional properties (conjugate symmetry, positive definiteness)
• Orthogonality does not imply orthonormality
  • Orthonormal vectors are orthogonal and have unit norm
• Orthogonal projection onto a subspace is not the same as the vector projection (scalar projection)
  • Vector projection computes the component of a vector along another vector
• Orthonormal bases are not unique
  • Any orthonormal basis can be transformed into another by an orthogonal transformation (rotation or reflection)
• Gram-Schmidt process is not the only way to construct orthonormal bases
  • Other methods include Householder transformations and Givens rotations
• Orthogonal matrices are not always symmetric
  • Orthogonal matrices satisfy $A^T A = A A^T = I$, but may not be symmetric ($A^T \neq A$)
• Orthogonal projections do not commute in general
  • $\text{proj}_U(\text{proj}_V(v)) \neq \text{proj}_V(\text{proj}_U(v))$ unless $U$ and $V$ are orthogonal subspaces