6.1 Directional derivatives and their properties

Directional derivatives generalize partial derivatives by measuring how a function changes along any direction, not just along the coordinate axes. They tie together gradient vectors and unit vectors into a single computation, and they're the foundation for identifying directions of steepest ascent and descent in optimization.

Directional Derivatives and Unit Vectors

Calculating Directional Derivatives

The directional derivative of $f$ at a point $\vec{x}$ in the direction of a unit vector $\vec{u}$ is the rate of change of $f$ as you move away from $\vec{x}$ along $\vec{u}$. It's denoted $D_{\vec{u}}f(\vec{x})$.

The formula is a dot product:

$D_{\vec{u}}f(\vec{x}) = \nabla f(\vec{x}) \cdot \vec{u}$

To compute it step by step:

1. Find all partial derivatives of $f$ and assemble the gradient $\nabla f(\vec{x})$.
2. Make sure your direction vector is a unit vector (magnitude 1). If it isn't, normalize it first.
3. Take the dot product of the gradient with the unit vector.

Example. Let $f(x,y) = x^2 + 3xy$ and suppose you want the directional derivative at $(1, 2)$ in the direction of $\vec{v} = \langle 3, 4 \rangle$.

1. $\nabla f = \langle 2x + 3y,\; 3x \rangle$, so $\nabla f(1,2) = \langle 8, 3 \rangle$.
2. Normalize: $|\vec{v}| = 5$, so $\vec{u} = \langle 3/5,\; 4/5 \rangle$.
3. Dot product: $D_{\vec{u}}f(1,2) = 8(3/5) + 3(4/5) = 24/5 + 12/5 = 36/5$.

The function is increasing at a rate of $36/5$ per unit length in that direction.

Unit Vectors and Their Properties

A unit vector has magnitude 1 and specifies a pure direction. The standard basis unit vectors are $\hat{i}$, $\hat{j}$, and $\hat{k}$, pointing along the $x$-, $y$-, and $z$-axes respectively.

To convert any nonzero vector $\vec{v}$ into a unit vector, divide by its magnitude:

$\vec{u} = \frac{\vec{v}}{|\vec{v}|}$

The directional derivative formula requires a unit vector. If you plug in a non-unit vector, you'll scale the answer incorrectly. This is a common mistake on exams.

Interpreting Directional Derivatives

The sign of $D_{\vec{u}}f$ tells you what the function is doing in that direction:

• Positive: $f$ is increasing along $\vec{u}$
• Negative: $f$ is decreasing along $\vec{u}$
• Zero: $f$ is momentarily constant along $\vec{u}$, meaning $\vec{u}$ is tangent to a level curve (or level surface)

Geometrically, the directional derivative is the scalar projection of $\nabla f$ onto $\vec{u}$. Since $\nabla f \cdot \vec{u} = |\nabla f|\cos\theta$, where $\theta$ is the angle between the gradient and $\vec{u}$, the directional derivative depends entirely on that angle.

Partial Derivatives and Linearity

Partial Derivatives and Gradient Vectors

Partial derivatives are the special case of directional derivatives along coordinate axes. For $f(x, y)$:

• $\frac{\partial f}{\partial x}$ is the rate of change holding $y$ constant (equivalent to $D_{\hat{i}}f$)
• $\frac{\partial f}{\partial y}$ is the rate of change holding $x$ constant (equivalent to $D_{\hat{j}}f$)

The gradient vector packages all partial derivatives together:

$\nabla f(x, y) = \left\langle \frac{\partial f}{\partial x},\; \frac{\partial f}{\partial y} \right\rangle$

Two key properties of the gradient:

• It points in the direction of greatest rate of increase of $f$.
• It is perpendicular to the level curves of $f$ at every point. This perpendicularity is what makes level curves "flat" relative to the gradient: moving along a level curve gives a directional derivative of zero.

Linearity of Directional Derivatives

Directional derivatives are linear operators. For functions $f$ and $g$ and scalars $a$ and $b$:

$D_{\vec{u}}(af + bg) = a\,D_{\vec{u}}f + b\,D_{\vec{u}}g$

This follows directly from the linearity of the gradient (since the gradient of a linear combination is the linear combination of the gradients) and the linearity of the dot product. In practice, this means you can compute directional derivatives of complicated expressions term by term.

Chain Rule for Directional Derivatives

When you have a composition of differentiable functions, the chain rule extends to directional derivatives. If $h(\vec{x}) = f(g(\vec{x}))$ where $g: \mathbb{R}^n \to \mathbb{R}^m$ and $f: \mathbb{R}^m \to \mathbb{R}$, then:

$D_{\vec{u}}h(\vec{x}) = \nabla f\big(g(\vec{x})\big) \cdot Dg(\vec{x})\,\vec{u}$

where $Dg(\vec{x})$ is the Jacobian matrix of $g$. For the simpler case where $g: \mathbb{R}^n \to \mathbb{R}$ and $f: \mathbb{R} \to \mathbb{R}$, this reduces to:

$D_{\vec{u}}(f \circ g)(\vec{x}) = f'(g(\vec{x}))\;D_{\vec{u}}g(\vec{x})$

This is the same "outer derivative times inner derivative" pattern from single-variable calculus, just applied in a directional context.

Steepest Ascent and Descent

Finding the Direction of Steepest Ascent

Since $D_{\vec{u}}f = |\nabla f|\cos\theta$, the directional derivative is maximized when $\theta = 0$, i.e., when $\vec{u}$ points in the same direction as $\nabla f$.

• The direction of steepest ascent is $\vec{u} = \frac{\nabla f}{|\nabla f|}$
• The maximum directional derivative equals $|\nabla f|$, the magnitude of the gradient

So the gradient does double duty: its direction tells you where the function increases fastest, and its magnitude tells you how fast.

Finding the Direction of Steepest Descent

By the same cosine argument, the directional derivative is minimized when $\theta = \pi$, i.e., when $\vec{u}$ points opposite to $\nabla f$.

• The direction of steepest descent is $\vec{u} = -\frac{\nabla f}{|\nabla f|}$
• The minimum directional derivative equals $-|\nabla f|$

Notice the symmetry: the steepest ascent and steepest descent have the same magnitude but opposite signs.

Applications of Steepest Ascent and Descent

Gradient descent in machine learning works by repeatedly stepping in the direction of $-\nabla f$ to minimize a cost function. Each iteration updates parameters as $\vec{x}_{n+1} = \vec{x}_n - \alpha\,\nabla f(\vec{x}_n)$, where $\alpha$ is the learning rate.

At a critical point, $\nabla f = \vec{0}$, which means the directional derivative is zero in every direction. These are the candidates for local maxima, minima, and saddle points.

Beyond machine learning, steepest ascent/descent analysis shows up in physics (heat flow follows the negative gradient of temperature), engineering (stress analysis on surfaces), and economics (maximizing utility or profit over multiple variables). The core idea is always the same: the gradient tells you the best direction to move.