f <- function(x) x^2
h <- 10^-(1:4)
(f(3 + h) - f(3)) / h[1] 6.1000 6.0100 6.0010 6.000111 Derivatives
A derivative answers one question: if I nudge the input, how much does the output move?
That is the question behind every optimization algorithm. Training a model means adjusting parameters to reduce a loss, and to adjust them intelligently you need to know which direction each one should move and by how much. The derivative is that information.
11.1 Rate of change
Start with something familiar. If you travel 120 km in 2 hours, your average speed is 60 km/h. That is a rate of change: output change divided by input change.
\[ \text{average rate} = \frac{f(b) - f(a)}{b - a} \]
Geometrically this is the slope of the line through \((a, f(a))\) and \((b, f(b))\) — the secant line.
But average speed over two hours tells you little about the moment you passed a speed camera. For that you want the rate at an instant, and an instant has zero duration, so the formula gives \(0/0\).
This is exactly the difficulty limits were built for.
11.2 The derivative as a limit
Shrink the interval. Fix \(a\), let the second point be \(a + h\), and take \(h \to 0\):
\[ f'(a) = \lim_{h \to 0}\frac{f(a+h) - f(a)}{h} \tag{11.1}\]
The quotient is the slope of a secant; the limit is the slope of the tangent. When the limit exists, \(f\) is differentiable at \(a\).
Note that we never divide by zero. The limit asks what the quotient approaches as \(h\) shrinks, never evaluating at \(h = 0\) — the loophole from Section 10.3, doing real work.
Worked example. For \(f(x) = x^2\) at \(a = 3\):
\[ \frac{(3+h)^2 - 9}{h} = \frac{9 + 6h + h^2 - 9}{h} = \frac{6h + h^2}{h} = 6 + h \]
The cancellation is legitimate because \(h \neq 0\) throughout. Now let \(h \to 0\): \(f'(3) = 6\).
Heading to 6, as promised.
xg <- seq(-0.5, 2.6, length.out = 200)
sec <- function(h) {
s <- ((1 + h)^2 - 1) / h
cbind(xg, 1 + s * (xg - 1))
}
draw_plane(
curves = list(
`x^2` = cbind(xg, xg^2),
sec(1.5), sec(0.75), sec(0.25),
tangent = cbind(xg, 1 + 2 * (xg - 1))
),
curve_color = c(
amds_colors[1], amds_gray, amds_gray,
amds_gray, amds_colors[2]
),
curve_line_type = c(
"solid", "dashed", "dashed", "dashed", "solid"
),
points = cbind(1, 1),
xlab = "x", ylab = "f(x)"
)11.3 Notation
Four notations for the same thing, and you will meet all of them.
| Notation | Named for | Reads as |
|---|---|---|
| \(f'(x)\) | Lagrange | “f prime of x” |
| \(\dfrac{df}{dx}\) | Leibniz | “d f d x” |
| \(\dfrac{d}{dx}f(x)\) | Leibniz | “d by d x of f” |
| \(\dot{x}\) | Newton | derivative with respect to time |
Leibniz notation is the most useful because it names the variable you are differentiating with respect to — essential once there is more than one (Section 12.3) — and because it makes the chain rule look like fractions cancelling.
To evaluate at a point: \(f'(3)\), or \(\left.\frac{df}{dx}\right|_{x=3}\).
11.4 Derivatives of common functions
| \(f(x)\) | \(f'(x)\) | Note |
|---|---|---|
| \(c\) | \(0\) | constants do not change |
| \(x\) | \(1\) | |
| \(x^n\) | \(nx^{n-1}\) | the power rule; any real \(n\) |
| \(e^x\) | \(e^x\) | its own derivative |
| \(e^{ax}\) | \(ae^{ax}\) | |
| \(\log x\) | \(1/x\) | for \(x > 0\) |
| \(\sin x\) | \(\cos x\) | |
| \(\cos x\) | \(-\sin x\) | |
| \(\sqrt{x}\) | \(\frac{1}{2\sqrt{x}}\) | power rule with \(n = 1/2\) |
The power rule covers more than it looks: \(1/x = x^{-1}\) differentiates to \(-x^{-2}\), and \(\sqrt{x} = x^{1/2}\) to \(\frac{1}{2}x^{-1/2}\).
\(e^x\) being its own derivative is the reason \(e\) is the natural base, and the reason it appears throughout probability and optimization.
11.5 Rules of differentiation
\[ \begin{aligned} (cf)' &= cf' &&\text{constant multiple} \\ (f + g)' &= f' + g' &&\text{sum} \\ (fg)' &= f'g + fg' &&\text{product} \\ \left(\frac{f}{g}\right)' &= \frac{f'g - fg'}{g^2} &&\text{quotient} \end{aligned} \]
The first two say differentiation is linear — it distributes over sums and pulls out constants. That is why the derivative of a loss \(\frac{1}{n}\sum_i \ell_i\) is \(\frac{1}{n}\sum_i \ell_i'\), and why you can reason about one observation at a time.
The product rule is the one people mis-remember. It is not \(f'g'\):
\[ (x^2 \cdot x^3)' = (x^5)' = 5x^4 \]
and via the rule, \(2x \cdot x^3 + x^2 \cdot 3x^2 = 2x^4 + 3x^4 = 5x^4\). Whereas \(f'g' = 2x \cdot 3x^2 = 6x^3\), which is wrong.
11.6 The chain rule
The most important rule in applied mathematics, because composition is how models are built (Section 3.7).
\[ \frac{d}{dx}f(g(x)) = f'(g(x)) \cdot g'(x) \tag{11.2}\]
In Leibniz notation it reads like cancellation, which is the way to remember it:
\[ \frac{dy}{dx} = \frac{dy}{du}\cdot\frac{du}{dx} \]
In words: differentiate the outside, keep the inside, multiply by the derivative of the inside.
Worked example. \(\frac{d}{dx}\sin(x^2)\). Outside is \(\sin\), inside is \(x^2\):
\[ \frac{d}{dx}\sin(x^2) = \cos(x^2)\cdot 2x \]
Another, which is the backbone of logistic regression. With \(\sigma(x) = 1/(1+e^{-x})\):
\[ \sigma'(x) = \sigma(x)\bigl(1 - \sigma(x)\bigr) \tag{11.3}\]
A derivative expressible in terms of the function itself — which is why logistic regression’s gradient is so tidy.
numeric formula
0.2217129 0.2217129
NoteIn machine learning
Backpropagation is Equation 11.2 applied repeatedly down a stack of layers. A network is \(f_L \circ \cdots \circ f_1\), and the gradient with respect to an early parameter is a product of derivatives from every later layer. This is also why gradients vanish or explode: a product of \(L\) numbers each below 1 collapses toward zero, and each above 1 runs away.
11.7 Higher-order derivatives
Differentiate again and you get the second derivative \(f''(x)\) or \(\frac{d^2f}{dx^2}\), the rate of change of the rate of change.
For \(f(x) = x^3\): \(f'(x) = 3x^2\), \(f''(x) = 6x\), \(f'''(x) = 6\).
The first derivative is the slope; the second is the curvature:
| \(f''\) | Shape | Called |
|---|---|---|
| \(> 0\) | curving upward, like a bowl | convex |
| \(< 0\) | curving downward, like a dome | concave |
| \(= 0\) | possibly an inflection point | — |
Curvature is what tells a second-order optimizer how big a step to take: high curvature means the slope is changing fast, so step cautiously. That intuition becomes Newton’s method in Section 16.10, and the multivariable version is the Hessian (Section 12.8).
11.8 Critical points and extrema
At a maximum or minimum of a smooth function, the tangent is horizontal:
\[ f'(x) = 0 \]
Such an \(x\) is a critical point (or stationary point). This is the foundation of optimization: to minimize, look where the derivative vanishes.
The second derivative test classifies them:
| At a critical point | Then it is a |
|---|---|
| \(f'' > 0\) | local minimum |
| \(f'' < 0\) | local maximum |
| \(f'' = 0\) | inconclusive |
Worked example. \(f(x) = x^3 - 3x\). Then \(f'(x) = 3x^2 - 3 = 0\) gives \(x = \pm 1\), and \(f''(x) = 6x\).
At \(x = 1\): \(f'' = 6 > 0\), a local minimum, value \(f(1) = -2\). At \(x = -1\): \(f'' = -6 < 0\), a local maximum, value \(f(-1) = 2\).
fc <- function(x) x^3 - 3 * x
c(f_at_1 = fc(1), f_at_minus1 = fc(-1)) f_at_1 f_at_minus1
-2 2
WarningWatch out
\(f'(x) = 0\) is necessary but not sufficient. \(f(x) = x^3\) has \(f'(0) = 0\) but 0 is neither a maximum nor a minimum — it is a saddle, and the second derivative test is silent because \(f''(0) = 0\) too.
Also: critical points are local. A function can have many, and finding one tells you nothing about whether a better one exists elsewhere. This is precisely the difficulty non-convex optimization faces (Section 15.8).
11.9 Linear approximation
Near a point, a differentiable function looks like its tangent line:
\[ f(x) \approx f(a) + f'(a)(x - a) \tag{11.4}\]
This is the single most-used idea in applied mathematics. Gradient descent, Newton’s method, the delta method, and error propagation are all this approximation applied somewhere.
Worked example. Estimate \(\sqrt{4.1}\) without a calculator. Take \(f(x) = \sqrt{x}\), \(a = 4\), so \(f(a) = 2\) and \(f'(a) = \frac{1}{2\sqrt{4}} = 0.25\):
\[ \sqrt{4.1} \approx 2 + 0.25(0.1) = 2.025 \]
Correct to three decimal places, from arithmetic you can do in your head.
11.10 Taylor series
Linear approximation uses one derivative. Using more gives a better fit:
\[ f(x) \approx f(a) + f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2 + \cdots + \frac{f^{(n)}(a)}{n!}(x-a)^n \tag{11.5}\]
This is the Taylor polynomial of degree \(n\) about \(a\). Each term corrects the previous one using one more derivative’s worth of information.
For \(e^x\) about \(a = 0\), every derivative is 1, so
\[ e^x = 1 + x + \frac{x^2}{2} + \frac{x^3}{6} + \frac{x^4}{24} + \cdots \]
The quadratic case is worth naming separately, because it is what second-order optimizers actually use:
\[ f(x) \approx f(a) + f'(a)(x-a) + \tfrac{1}{2}f''(a)(x-a)^2 \]
Newton’s method minimizes this quadratic exactly and jumps to its minimum (Section 16.10).
11.11 Numerical differentiation
When you have no formula, approximate Equation 11.1 directly. Two schemes:
\[ \text{forward: } \frac{f(x+h)-f(x)}{h} \qquad \text{central: } \frac{f(x+h)-f(x-h)}{2h} \]
Taylor expansion shows the forward difference has error \(O(h)\) and the central difference \(O(h^2)\) — so halving \(h\) halves one error and quarters the other. Central is strictly better for the same number of evaluations plus one.
But Section 10.8 warned that smaller is not always better.
err <- function(h, method) {
truth <- exp(1)
if (method == "forward") {
abs((exp(1 + h) - exp(1)) / h - truth)
} else {
abs((exp(1 + h) - exp(1 - h)) / (2 * h) - truth)
}
}
hs <- 10^-(1:16)
tab <- data.frame(
h = hs,
forward = sapply(hs, err, "forward"),
central = sapply(hs, err, "central")
)
tab[c(1, 4, 5, 8, 11, 16), ] h forward central
1 1e-01 1.405601e-01 4.532735e-03
4 1e-04 1.359186e-04 4.530566e-09
5 1e-05 1.359150e-05 5.858691e-11
8 1e-08 6.602751e-09 6.602751e-09
11 1e-11 3.263395e-05 1.042949e-05
16 1e-16 2.718282e+00 2.718282e+00Both curves are V-shaped. Going left to right as \(h\) shrinks: error falls while truncation error dominates, bottoms out, then climbs as rounding error takes over — because \(f(x+h) - f(x)\) subtracts two nearly equal numbers, and the leading digits cancel.
The optimum is around \(h \approx \sqrt{\epsilon} \approx 10^{-8}\) for forward differences and \(h \approx \epsilon^{1/3} \approx 10^{-5}\) for central ones, where \(\epsilon\) is machine epsilon.
NoteIn machine learning
This is why deep learning does not use finite differences. Automatic differentiation computes derivatives exactly — to machine precision, with no step size to tune — by applying the chain rule to the operations as they execute. It costs about the same as evaluating the function, where finite differences would cost one evaluation per parameter. For a model with millions of parameters that difference is decisive (Section 13.9).
11.12 Summary
| Idea | Statement |
|---|---|
| Definition | \(f'(a) = \lim_{h\to0}\frac{f(a+h)-f(a)}{h}\) |
| Power rule | \((x^n)' = nx^{n-1}\) |
| Product | \((fg)' = f'g + fg'\) |
| Chain | \((f\circ g)' = f'(g(x))\,g'(x)\) |
| Critical point | \(f'(x) = 0\) |
| Second derivative test | \(f''>0\) min, \(f''<0\) max |
| Linear approximation | \(f(x) \approx f(a) + f'(a)(x-a)\) |
| Taylor | add \(\frac{f^{(k)}(a)}{k!}(x-a)^k\) terms |
11.13 Exercises
1. Differentiate \(f(x) = 3x^4 - 2x^2 + 7\) and evaluate \(f'(2)\).
TipSolution
Term by term: \(f'(x) = 12x^3 - 4x\). At \(x = 2\): \(12(8) - 8 = 88\).
fq <- function(x) 3 * x^4 - 2 * x^2 + 7
hq <- 1e-6
c(
formula = 12 * 2^3 - 4 * 2,
numeric = (fq(2 + hq) - fq(2 - hq)) / (2 * hq)
)formula numeric
88 88 The constant 7 contributes nothing — shifting a function vertically does not change any slope.
2. Use the chain rule on \(f(x) = e^{-x^2/2}\).
TipSolution
Outside \(e^u\), inside \(u = -x^2/2\) with \(u' = -x\). So
\[ f'(x) = e^{-x^2/2}\cdot(-x) = -x\,e^{-x^2/2} \]
formula numeric
-0.5584246 -0.5584246 This is the (unnormalized) standard normal density. Its derivative is zero only at \(x=0\), which is why the normal distribution peaks at its mean.
3. Find and classify the critical points of \(f(x) = x^4 - 4x^2\).
TipSolution
\(f'(x) = 4x^3 - 8x = 4x(x^2 - 2) = 0\) at \(x = 0\) and \(x = \pm\sqrt{2}\).
\(f''(x) = 12x^2 - 8\). At \(x=0\): \(f'' = -8 < 0\), a local maximum. At \(x = \pm\sqrt{2}\): \(f'' = 24 - 8 = 16 > 0\), local minima.
[,1] [,2] [,3]
x -1.4142 0 1.4142
f -4.0000 0 -4.0000
f2 16.0000 -8 16.0000A W shape: two equal minima at \(-4\) either side of a local maximum at 0. Two distinct global minima — a reminder that “the” minimum need not be unique.
4. Approximate \(\sqrt{9.2}\) using Equation 11.4 about \(a = 9\), and find the error.
TipSolution
\(f(9) = 3\), \(f'(9) = \frac{1}{2\cdot3} = \frac{1}{6}\), so \(\sqrt{9.2} \approx 3 + 0.2/6 = 3.0\overline{3}\).
approx actual error
3.0333333333 3.0331501776 0.0001831557 The error is about \(1.8\times10^{-4}\) — small, and it scales with \((x-a)^2\), which is exactly the first term Equation 11.4 throws away. Halving the distance from \(a\) would quarter it.
5. Verify Equation 11.3 numerically at several points.
TipSolution
[,1] [,2] [,3] [,4] [,5]
numeric 0.04517666 0.1966119 0.25 0.1966119 0.04517666
formula 0.04517666 0.1966119 0.25 0.1966119 0.04517666They agree. The derivative peaks at \(x=0\) with value \(0.25\) and decays symmetrically — so the logistic is least sensitive to its input exactly where it is most confident, which is why saturated units learn slowly.
6. Find the \(h\) minimizing the central-difference error for \(e^x\) at \(x=1\), and compare it to \(\epsilon^{1/3}\).
TipSolution
best_h eps_cube_root best_error
1.000000e-05 6.055454e-06 5.858691e-11 The empirical optimum sits close to \(\epsilon^{1/3} \approx 6\times10^{-6}\), as the theory predicts. The achievable accuracy is about \(10^{-11}\) — far short of machine precision, and the reason automatic differentiation is preferred whenever it is available.