∞Calculus IV Unit 6 – Directional Derivatives and Gradients

Key Concepts and Definitions

• Directional derivatives measure the rate of change of a function in a specific direction
• Gradients are vectors that point in the direction of the greatest rate of increase of a function
• Partial derivatives are derivatives of a function with respect to one variable while holding other variables constant
• Level curves are curves in the domain of a function where the function value remains constant
• Tangent planes are planes that touch a surface at a single point and are parallel to the surface at that point
• Stationary points are points where the gradient vector is zero (includes local maxima, local minima, and saddle points)
  • Local maxima are points where the function value is greater than or equal to nearby points
  • Local minima are points where the function value is less than or equal to nearby points
  • Saddle points are points where the function increases in some directions and decreases in others

Vector Functions and Partial Derivatives

• Vector functions map input vectors to output vectors and can be used to represent curves and surfaces in higher dimensions
• Partial derivatives of vector functions can be computed by differentiating each component function separately
• The Jacobian matrix contains all the partial derivatives of a vector function and is used in various applications (optimization, coordinate transformations)
• Partial derivatives can be used to find tangent planes and normal vectors to surfaces defined by vector functions
• The chain rule for partial derivatives allows for computing derivatives of compositions of functions
  • If $z = f(x, y)$ and $x = g(t)$, $y = h(t)$, then $\frac{\partial z}{\partial t} = \frac{\partial f}{\partial x}\frac{dx}{dt} + \frac{\partial f}{\partial y}\frac{dy}{dt}$
• Higher-order partial derivatives can be computed by repeatedly differentiating with respect to different variables
• Mixed partial derivatives (derivatives taken with respect to different variables in succession) are equal if the function is continuously differentiable

Understanding Directional Derivatives

• Directional derivatives measure the rate of change of a function in a specific direction specified by a unit vector
• The directional derivative of a function $f$ at a point $\mathbf{p}$ in the direction of a unit vector $\mathbf{u}$ is denoted as $D_{\mathbf{u}}f(\mathbf{p})$
• Directional derivatives can be computed using the gradient vector: $D_{\mathbf{u}}f(\mathbf{p}) = \nabla f(\mathbf{p}) \cdot \mathbf{u}$
• The direction of the gradient vector at a point is the direction of the greatest rate of increase of the function at that point
• The magnitude of the directional derivative is maximum when the direction is parallel to the gradient vector and zero when perpendicular to the gradient vector
• Directional derivatives can be used to find the rate of change of a function along curves or paths in higher dimensions
  • If $\mathbf{r}(t)$ is a parametric curve, then the rate of change of $f$ along the curve at $t$ is $D_{\mathbf{r}'(t)}f(\mathbf{r}(t))$
• Directional derivatives are linear: $D_{\mathbf{u}}(af + bg) = aD_{\mathbf{u}}f + bD_{\mathbf{u}}g$ for constants $a$ and $b$

The Gradient Vector

• The gradient of a function $f(x, y)$ is a vector $\nabla f = \left(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}\right)$ that points in the direction of the greatest rate of increase of $f$
• The gradient vector is perpendicular to level curves of the function at each point
• The magnitude of the gradient vector $\|\nabla f\|$ represents the maximum rate of change of the function at a point
• The gradient vector can be used to find directional derivatives: $D_{\mathbf{u}}f(\mathbf{p}) = \nabla f(\mathbf{p}) \cdot \mathbf{u}$
• The gradient vector is zero at stationary points (local maxima, local minima, and saddle points)
• The gradient vector can be generalized to functions of more than two variables: $\nabla f = \left(\frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}, \ldots, \frac{\partial f}{\partial x_n}\right)$
• The gradient vector is used in optimization problems to find the direction of steepest ascent or descent

Relationship Between Gradients and Directional Derivatives

• The directional derivative can be computed using the gradient vector: $D_{\mathbf{u}}f(\mathbf{p}) = \nabla f(\mathbf{p}) \cdot \mathbf{u}$
• The gradient vector points in the direction of the maximum directional derivative
• The magnitude of the gradient vector is equal to the maximum value of the directional derivative
• The directional derivative is zero in directions perpendicular to the gradient vector
• Level curves (or surfaces) are perpendicular to the gradient vector at each point
• The chain rule relates the gradient of a composition of functions: $\nabla(f \circ \mathbf{g}) = J_{\mathbf{g}}^T(\nabla f \circ \mathbf{g})$, where $J_{\mathbf{g}}$ is the Jacobian matrix of $\mathbf{g}$
• The gradient vector and directional derivatives are essential tools in optimization problems, as they help identify the direction and magnitude of the greatest change in a function

Applications in Optimization

• Optimization problems involve finding the maximum or minimum values of a function subject to constraints
• The gradient vector and directional derivatives are used to determine the direction of steepest ascent (for maximization) or descent (for minimization)
• Gradient descent is an iterative optimization algorithm that moves in the direction of the negative gradient to find local minima
  • The update rule for gradient descent is $\mathbf{x}_{n+1} = \mathbf{x}_n - \gamma \nabla f(\mathbf{x}_n)$, where $\gamma$ is the learning rate
• Gradient ascent is similar to gradient descent but moves in the direction of the positive gradient to find local maxima
• Constrained optimization problems can be solved using Lagrange multipliers, which incorporate the constraints into the objective function
• The method of steepest descent (or ascent) follows the direction of the negative (or positive) gradient with a step size determined by a line search
• Newton's method uses second-order derivatives (the Hessian matrix) to find stationary points more efficiently than gradient descent
• Stochastic gradient descent is a variant of gradient descent that uses random subsets of data to compute the gradient, making it more efficient for large datasets

Computational Methods and Tools

• Numerical differentiation techniques (finite differences) can be used to approximate partial derivatives and gradients
• Automatic differentiation is a family of techniques that compute derivatives of functions defined by computer programs
  • Forward mode automatic differentiation computes directional derivatives by applying the chain rule to elementary operations
  • Reverse mode automatic differentiation (backpropagation) computes the gradient by traversing the computation graph backwards
• Machine learning frameworks (TensorFlow, PyTorch) provide automatic differentiation capabilities for training neural networks
• Optimization libraries (SciPy, CVXPY) implement various optimization algorithms and can handle constraints
• Finite element methods are used to solve partial differential equations by discretizing the domain and approximating the solution with basis functions
• Gradient-based optimization is a key component of many machine learning algorithms (neural networks, logistic regression, support vector machines)
• Visualization tools (matplotlib, plotly) can be used to plot functions, level curves, and gradient vectors for better understanding and interpretation

Practice Problems and Examples

• Find the directional derivative of $f(x, y) = x^2 + xy + y^2$ at $(1, 2)$ in the direction of $\mathbf{u} = \frac{1}{\sqrt{2}}(1, 1)$
• Compute the gradient vector of $g(x, y, z) = x^2yz + \sin(xy) + e^{z}$ at $(0, \pi, 1)$
• Find the direction of steepest ascent of $h(x, y) = x^3 + y^3 - 3xy$ at $(1, 1)$
• Use gradient descent to minimize $f(x, y) = (x - 2)^2 + (y - 3)^2$, starting at $(0, 0)$ with a learning rate of $0.1$ for 10 iterations
• Find the maximum value of $f(x, y) = x + y$ subject to the constraint $x^2 + y^2 = 1$ using Lagrange multipliers
• Compute the directional derivative of $\mathbf{f}(x, y) = (xy, x^2 - y^2)$ at $(1, 1)$ in the direction of $\mathbf{v} = (1, -1)$
• Use the method of steepest descent to minimize $g(x, y) = x^4 + y^4 - 4xy$, starting at $(1, 1)$ with a step size of $0.1$ for 5 iterations
• Implement gradient descent in Python to minimize the Rosenbrock function $f(x, y) = (1 - x)^2 + 100(y - x^2)^2$