11.2 Positive Definite Operators and Matrices

Positive definite operators and matrices are key players in linear algebra, with far-reaching applications. They're special because they always produce positive results when applied to non-zero vectors, making them crucial in many mathematical and real-world scenarios.

These operators have unique properties that set them apart. They're always invertible, have positive eigenvalues, and possess a unique square root. Understanding these characteristics is essential for grasping their role in the broader context of self-adjoint operators and their applications.

Positive Definite Operators and Matrices

Definition and Key Characteristics

• A positive definite operator is a linear operator that maps a vector space to itself and satisfies the condition $<Ax, x> > 0$ for all non-zero vectors $x$ in the vector space, where $<.,.>$ denotes the inner product
• A positive definite matrix is a symmetric matrix $A$ such that $x^T A x > 0$ for all non-zero vectors $x$, where $x^T$ denotes the transpose of $x$
• Positive definite operators and matrices are self-adjoint (Hermitian in the complex case)
  • A self-adjoint operator $A$ satisfies $<Ax, y> = <x, Ay>$ for all vectors $x$ and $y$ in the vector space
  • A Hermitian matrix $A$ satisfies $A^* = A$, where $A^*$ denotes the conjugate transpose of $A$
• The eigenvalues of a positive definite operator or matrix are all strictly positive
  • This property follows from the definition of positive definiteness and the spectral theorem for self-adjoint operators
  • If $\lambda$ is an eigenvalue of a positive definite operator $A$ with eigenvector $v$, then $<Av, v> = \lambda <v, v> > 0$, implying $\lambda > 0$
• The determinant of a positive definite matrix is always positive
  • The determinant of a matrix is the product of its eigenvalues
  • Since all eigenvalues of a positive definite matrix are positive, their product (the determinant) is also positive

Geometric Interpretation

• Positive definite operators and matrices can be interpreted geometrically as transformations that preserve or increase the length of vectors
  • For a positive definite operator $A$ and a non-zero vector $x$, $<Ax, Ax> > <x, x>$, meaning the transformed vector $Ax$ has a greater length than the original vector $x$
  • In the case of matrices, a positive definite matrix $A$ satisfies $||Ax||^2 > ||x||^2$ for all non-zero vectors $x$, where $||.||$ denotes the Euclidean norm
• Positive definite matrices define ellipsoids in the vector space
  • The set of points $x$ satisfying $x^T A x = 1$ for a positive definite matrix $A$ forms an ellipsoid
  • The principal axes of the ellipsoid are determined by the eigenvectors of $A$, and the lengths of the axes are inversely proportional to the square roots of the corresponding eigenvalues

Properties of Positive Definite Operators

Invertibility and Preservation of Positive Definiteness

• Positive definite operators and matrices are invertible, and their inverses are also positive definite
  • If $A$ is a positive definite operator, then for any non-zero vector $x$, $<Ax, x> > 0$, implying that $Ax \neq 0$ for all non-zero $x$, which is the definition of invertibility
  • To show that the inverse of a positive definite operator $A$ is also positive definite, consider $<A^{-1}x, x> = <A^{-1}x, AA^{-1}x> = <x, A^{-1}x> > 0$ for all non-zero $x$
• The sum of two positive definite operators or matrices is also positive definite
  • If $A$ and $B$ are positive definite operators, then for any non-zero vector $x$, $<(A+B)x, x> = <Ax, x> + <Bx, x> > 0$, proving that $A+B$ is positive definite
  • The same property holds for positive definite matrices
• The product of two positive definite operators or matrices is positive definite if and only if they commute
  • If $A$ and $B$ are commuting positive definite operators, then $<ABx, x> = <BAx, x> = <Ax, Bx> > 0$ for all non-zero $x$, proving that $AB$ is positive definite
  • However, if $A$ and $B$ do not commute, their product may not be positive definite
  • For matrices, $AB$ is positive definite if and only if $A$ and $B$ commute and are both positive definite

Powers and Schur Complement

• If $A$ is a positive definite matrix, then $A^n$ is also positive definite for any positive integer $n$
  • This property follows from the fact that the eigenvalues of $A^n$ are the $n$-th powers of the eigenvalues of $A$, which are all positive
  • For a positive definite operator $A$, the same property holds for the operator power $A^n$
• The Schur complement of a positive definite matrix is also positive definite
  • Consider a partitioned positive definite matrix $A = \begin{pmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{pmatrix}$, where $A_{11}$ is invertible
  • The Schur complement of $A_{11}$ in $A$ is defined as $S = A_{22} - A_{21}A_{11}^{-1}A_{12}$
  • The Schur complement $S$ is positive definite if $A$ is positive definite
  • This property is useful in various applications, such as in the analysis of block matrices and in optimization problems

Cholesky Decomposition

• Positive definite matrices have a unique Cholesky decomposition, $A = LL^T$, where $L$ is a lower triangular matrix with positive diagonal entries
  • The Cholesky decomposition is a special case of the LU decomposition for positive definite matrices
  • The existence and uniqueness of the Cholesky decomposition follow from the positive definiteness of $A$
  • The Cholesky decomposition is numerically stable and efficient for solving linear systems and computing matrix inverses
  • Applications of the Cholesky decomposition include least squares problems, Kalman filtering, and Gaussian process regression

Unique Positive Definite Square Root

Existence and Definition

• The positive definite square root of a positive definite operator $A$ is an operator $B$ such that $B^2 = A$ and $B$ is also positive definite
  • The existence of the positive definite square root can be proved using the spectral theorem for self-adjoint operators
  • The spectral theorem states that a self-adjoint operator $A$ can be diagonalized by an orthonormal basis of eigenvectors, $A = UDU^*$, where $U$ is unitary and $D$ is a diagonal matrix of eigenvalues
  • Define the positive definite square root as $B = UD^{1/2}U^*$, where $D^{1/2}$ is the diagonal matrix of square roots of the eigenvalues
  • Since the eigenvalues of $A$ are all positive, their square roots are real and positive, ensuring that $B$ is positive definite

Uniqueness Proof

• The positive definite square root is unique, which can be shown by assuming the existence of two positive definite square roots and proving they must be equal
  • Suppose $B_1$ and $B_2$ are two positive definite square roots of $A$, i.e., $B_1^2 = A$ and $B_2^2 = A$
  • Consider the operator $C = B_1B_2$, which satisfies $C^2 = B_1B_2B_1B_2 = B_1AB_2 = A^2$
  • Since $A$ is positive definite, $A^2$ is also positive definite, implying that $C$ is self-adjoint
  • A self-adjoint operator satisfying $C^2 = A^2$ must be equal to $A$ (by the spectral theorem and the uniqueness of the square root of positive numbers)
  • Therefore, $B_1B_2 = A$, and multiplying both sides by $B_1^{-1}$ yields $B_2 = B_1$, proving the uniqueness of the positive definite square root

Properties and Applications

• The positive definite square root inherits many properties from the original positive definite operator
  • If $A$ is invertible, then its positive definite square root $B$ is also invertible, and $B^{-1} = (A^{-1})^{1/2}$
  • If $A$ and $B$ are commuting positive definite operators, then $(AB)^{1/2} = A^{1/2}B^{1/2}$
  • The positive definite square root is continuous with respect to the operator norm topology
• The positive definite square root has applications in various fields
  • In quantum mechanics, the positive definite square root of the density matrix is used to define the purification of a mixed state
  • In statistics, the positive definite square root of the covariance matrix is used to whiten random vectors and to define the Mahalanobis distance
  • In machine learning, the positive definite square root is used in kernel methods, such as Gaussian process regression and support vector machines, to transform the feature space

Applications of Positive Definite Operators

Linear Algebra Problems

• Apply the properties of positive definite operators and matrices to solve linear algebra problems
  • Use the invertibility of positive definite matrices to solve linear systems of the form $Ax = b$, where $A$ is positive definite
  • Exploit the Cholesky decomposition to efficiently solve linear systems and compute matrix inverses
  • Utilize the spectral theorem and the positive definite square root to analyze and manipulate positive definite operators
  • Example: Given a positive definite matrix $A$ and a vector $b$, solve the linear system $Ax = b$ using the Cholesky decomposition $A = LL^T$ by first solving $Ly = b$ for $y$ and then solving $L^Tx = y$ for $x$

Convex Optimization

• Recognize and exploit the connections between positive definite matrices and convex optimization problems
  • Many convex optimization problems, such as quadratic programming and semidefinite programming, involve positive definite matrices in their objective functions or constraints
  • The positive definiteness of the Hessian matrix ensures the convexity of the objective function in unconstrained optimization problems
  • Interior point methods for convex optimization often rely on the properties of positive definite matrices to ensure the convergence and efficiency of the algorithms
  • Example: Consider the quadratic programming problem $\min_x \frac{1}{2}x^TAx - b^Tx$ subject to $Cx \leq d$, where $A$ is positive definite. The positive definiteness of $A$ ensures the convexity of the problem, allowing the use of efficient optimization algorithms

Real-World Applications

• Apply positive definite operators and matrices in various fields to model and solve real-world problems
  • In quantum mechanics, positive definite operators represent observables, and their eigenvalues and eigenvectors correspond to the possible measurement outcomes and the associated quantum states
  • In statistics, positive definite matrices appear as covariance matrices of random vectors, and their properties are crucial for parameter estimation, hypothesis testing, and model selection
  • In machine learning, positive definite kernels are used to measure the similarity between data points and to implicitly map the data into high-dimensional feature spaces, enabling the application of linear methods to non-linear problems
  • Example: In Gaussian process regression, the covariance matrix of the Gaussian process is a positive definite matrix that encodes the similarity between input points and determines the smoothness and the uncertainty of the regression function

Numerical Methods

• Develop and analyze numerical methods for computing with positive definite operators and matrices
  • Exploit the structure and properties of positive definite matrices to design efficient algorithms for matrix factorization, eigenvalue computation, and matrix function evaluation
  • Use iterative methods, such as the conjugate gradient method and the Lanczos algorithm, to solve large-scale linear systems and eigenvalue problems involving positive definite matrices
  • Investigate the stability and convergence properties of numerical methods for positive definite operators and matrices, and develop error analysis and preconditioning techniques to improve their performance
  • Example: Implement the conjugate gradient method to solve a large-scale linear system $Ax = b$, where $A$ is a sparse positive definite matrix, and compare its performance with direct methods, such as the Cholesky factorization, in terms of computational complexity and memory requirements