certain fair_coin one_in_1000
0.000000 1.000000 9.965784 23 Information Theory
Twice already this book has told you that binary cross-entropy is the negative log-likelihood of a Bernoulli (Section 20.2, Section 21.5). That is true, and it is only half the story — it explains what cross-entropy equals without explaining what it measures.
This chapter supplies the missing half. Information theory gives a way to quantify uncertainty, to measure the distance between two distributions, and to say precisely why cross-entropy is the natural loss for a classifier rather than an arbitrary choice that happens to work.
23.1 What information is
Start with a single event. How much do you learn when something of probability \(p\) occurs?
Three requirements pin the answer down almost completely:
- A certain event teaches you nothing, so the information must be \(0\) when \(p = 1\).
- A rarer event teaches you more, so information must decrease in \(p\).
- Learning two independent things should give the sum of what each gives — but their probabilities multiply (Equation 18.4).
The last requirement is the binding one. A function turning products into sums is a logarithm (Section 3.8), so:
\[ I(x) = -\log p(x) \tag{23.1}\]
This is the surprisal of an outcome. The minus sign makes it positive, since probabilities are at most 1 and their logarithms are at most 0.
The base of the logarithm is a choice of unit:
| Base | Unit | Used by |
|---|---|---|
| 2 | bits | information theory, this chapter |
| \(e\) | nats | statistics, machine learning |
Base 2 is easier to interpret — one bit is the information in one fair coin flip — so the exposition here uses bits. Software almost always uses nats, because log() in R and every ML framework is natural. The two differ only by the constant \(\log 2\).
A certain event carries zero bits, a coin flip one, and a one-in-a-thousand event about ten.
23.2 Entropy
Surprisal describes one outcome. Entropy is the surprisal you should expect — the average over the whole distribution:
\[ H(p) = -\sum_x p(x)\log p(x) = \mathbb{E}\bigl[-\log p(X)\bigr] \tag{23.2}\]
It is an expectation (Section 19.5) of a function of \(X\), and nothing more exotic. Read it as: how uncertain am I, on average, before observing this variable?
fair_coin biased_coin near_certain fair_die
1.00000000 0.46899559 0.08079314 2.58496250 A fair coin has exactly 1 bit of entropy — that is the definition of the unit. A coin that lands heads 99% of the time has only 0.081 bits, because you can nearly always guess correctly. A fair die has \(\log_2 6 = 2.585\) bits.
The convention \(0\log 0 = 0\) is not a fudge: \(p\log p \to 0\) as \(p \to 0\), which is a limit of exactly the kind Chapter 10 handles.
That shape should look familiar: the Bernoulli variance \(p(1-p)\) also peaks at \(0.5\) and vanishes at the ends (Section 20.2). Both are measures of unpredictability, and they agree about where it is greatest.
They are not the same function, though, and the difference matters.
[,1] [,2] [,3] [,4]
p 0.50 0.2500 0.100 0.0100
entropy 1.00 0.8113 0.469 0.0808
variance 0.25 0.1875 0.090 0.0099Going from \(p = 0.1\) to \(p = 0.01\) cuts the variance by a factor of about 11 but the entropy by only about 6. Entropy penalizes near-certainty more gently than variance does, because it counts how hard the outcome is to describe, not how far it spreads.
NoteIn machine learning
Entropy is the impurity criterion in decision trees. A node whose labels are all one class has entropy 0 and needs no further splitting; a 50/50 node has 1 bit and is maximally impure. Information gain — the entropy reduction a split achieves — is exactly the mutual information of Section 23.6.
23.3 Cross-entropy
Now suppose the truth is \(p\) but you believe \(q\). Your expected surprisal uses your beliefs to measure surprise, and reality to weight it:
\[ H(p, q) = -\sum_x p(x)\log q(x) \tag{23.3}\]
This is the cross-entropy of \(q\) relative to \(p\). Note the asymmetry built in: \(p\) decides how often each outcome happens, \(q\) decides how surprised you are when it does.
Worked example. The truth is a fair coin, \(p = (0.5, 0.5)\), but you believe it is heavily biased, \(q = (0.9, 0.1)\).
entropy_of_truth cross_entropy
1.000000 1.736966 Being wrong costs you: 1.737 bits of expected surprise where 1 bit was unavoidable. Half the time the coin comes up tails, an outcome you assigned probability 0.1 and are therefore \(\log_2 10 = 3.32\) bits surprised by.
Cross-entropy is never smaller than entropy, and equals it only when \(q = p\):
\[ H(p, q) \geq H(p) \tag{23.4}\]
which is Gibbs’ inequality. The gap between them is the subject of the next section.
23.4 Kullback–Leibler divergence
Subtract the unavoidable part and what remains is the cost of being wrong:
\[ D_{\text{KL}}(p \,\|\, q) = H(p,q) - H(p) = \sum_x p(x)\log\frac{p(x)}{q(x)} \tag{23.5}\]
The Kullback–Leibler divergence measures how far belief \(q\) is from truth \(p\), in units of wasted bits. Two properties do all the work:
It is non-negative, and zero only when \(p = q\). This follows from Jensen’s inequality (Section 19.5): \(-\log\) is convex (Section 15.3), so
\[ D_{\text{KL}}(p\|q) = \mathbb{E}_p\left[-\log\frac{q}{p}\right] \geq -\log \mathbb{E}_p\left[\frac{q}{p}\right] = -\log 1 = 0 \]
A one-line proof, entirely from convexity.
It is not symmetric. \(D_{\text{KL}}(p\|q) \neq D_{\text{KL}}(q\|p)\), so it is not a distance in the geometric sense of Section 4.9 — there is no triangle inequality either.
kl_p_q kl_q_p check
0.7369656 0.5310044 0.7369656 Both directions are positive but they differ — 0.737 against 0.531. The asymmetry is not a defect, and which direction you minimize changes what you get:
| Minimize | Behavior | Called |
|---|---|---|
| \(D_{\text{KL}}(p\|q)\) over \(q\) | \(q\) must cover everywhere \(p\) has mass | mean-seeking, forward |
| \(D_{\text{KL}}(q\|p)\) over \(q\) | \(q\) may concentrate on one mode of \(p\) | mode-seeking, reverse |
The reason is visible in Equation 23.5. In the forward direction, any \(x\) where \(p(x) > 0\) but \(q(x) \approx 0\) makes \(\log(p/q)\) explode — so \(q\) is forced to spread out. In the reverse direction those terms are weighted by \(q(x) \approx 0\) and cost nothing, so \(q\) is free to ignore parts of \(p\) entirely.
[,1] [,2] [,3] [,4]
q_heads 0.5 0.6000 0.8000 0.9900
kl_bits 0.0 0.0294 0.3219 2.3292Zero when the belief is correct, growing rapidly as it becomes confidently wrong.
NoteIn machine learning
The forward/reverse distinction is not academic. Variational inference minimizes the reverse KL, which is why a variational approximation to a multimodal posterior often collapses onto a single mode and understates uncertainty.
Maximum likelihood minimizes the forward KL, which is why fitted models tend to over-disperse rather than miss regions of the data.
23.5 Why cross-entropy is the classification loss
Here is the payoff.
Training a classifier means choosing a predicted distribution \(q_\theta\) to be close to the true label distribution \(p\). The natural objective is to minimize \(D_{\text{KL}}(p\|q_\theta)\) — but by Equation 23.5,
\[ D_{\text{KL}}(p \,\|\, q_\theta) = \underbrace{H(p,q_\theta)}_{\text{depends on }\theta} - \underbrace{H(p)}_{\text{fixed}} \]
The entropy of the truth does not depend on your model. So minimizing KL divergence and minimizing cross-entropy are the same optimization, differing by a constant that changes nothing about where the minimum sits.
For a single observation with true label \(y \in \{0,1\}\) and predicted probability \(\hat{p}\), the true distribution puts all its mass on \(y\), and Equation 23.3 collapses to
\[ -\bigl[y\log\hat{p} + (1-y)\log(1-\hat{p})\bigr] \tag{23.6}\]
which is binary cross-entropy — and also, term for term, the negative log-likelihood of a Bernoulli (Section 21.5). Three descriptions, one quantity:
\[ \text{minimize cross-entropy} \;\Longleftrightarrow\; \text{minimize } D_{\text{KL}} \;\Longleftrightarrow\; \text{maximize likelihood} \]
cross_entropy_nats neg_loglik_per_obs
0.2990012 0.2990012 Identical, because they are the same formula written twice.
The shape of the penalty explains the loss’s character:
draw_fun(
function(q) -log2(q),
from = 0.01, to = 1,
xlab = "probability assigned to the true outcome",
ylab = "penalty (bits)"
) [,1] [,2] [,3] [,4] [,5]
predicted 0.900 0.5 0.100 0.010 0.001
penalty_bits 0.152 1.0 3.322 6.644 9.966Predicting 0.9 for something that happens costs 0.15 bits; predicting 0.001 costs 9.97. Cross-entropy is not merely counting errors — it is scoring calibration, and a confident mistake costs far more than a hedged one.
WarningWatch out
The unbounded penalty is why \(\log(0)\) crashes training runs. A model that assigns probability exactly 0 to something that occurs incurs infinite loss, and the gradient follows.
Implementations clamp predictions away from 0 and 1, or work in log-space throughout — the same numerical-survival argument as Section 21.5.
23.6 Mutual information
How much does knowing \(Y\) tell you about \(X\)? Compare the joint distribution against what it would be if they were independent:
\[ I(X;Y) = \sum_{x,y} p(x,y)\log\frac{p(x,y)}{p(x)p(y)} = D_{\text{KL}}\bigl(p(x,y) \,\|\, p(x)p(y)\bigr) \tag{23.7}\]
Mutual information is the KL divergence between the joint and the product of the marginals — literally a measure of how far from independent (Equation 19.10) the variables are.
Equivalently, it is the uncertainty in \(X\) that observing \(Y\) removes:
\[ I(X;Y) = H(X) - H(X \mid Y) = H(X) + H(Y) - H(X,Y) \]
mutual_info <- function(joint) {
px <- rowSums(joint)
py <- colSums(joint)
ind <- outer(px, py)
keep <- joint > 0
sum(joint[keep] * log2(joint[keep] / ind[keep]))
}
weak <- matrix(c(0.1, 0.2, 0.3, 0.4), nrow = 2, byrow = TRUE)
strong <- matrix(
c(0.45, 0.05, 0.05, 0.45),
nrow = 2, byrow = TRUE
)
independent <- outer(rowSums(weak), colSums(weak))
c(
weak = mutual_info(weak),
strong = mutual_info(strong),
independent = mutual_info(independent)
) weak strong independent
0.005802149 0.531004406 0.000000000 Exactly zero for independent variables — and that is an if and only if, since KL is zero only when its two arguments agree.
The strong case removes 0.531 of the 1 bit in \(X\), so observing \(Y\) answers about 53% of the question.
H_X I H_X_given_Y
1.0000000 0.5310044 0.4689956
NoteIn machine learning
Mutual information is correlation’s more general cousin. Correlation detects only linear association (Section 19.10) — recall the \(Y = X^2\) example with correlation exactly zero and total dependence. Mutual information catches that, because it compares whole distributions rather than second moments.
It is the basis of information-gain feature selection, of the InfoNCE objective in contrastive learning, and of the information bottleneck view of representation learning. Its drawback is practical: estimating it from continuous data is hard, whereas correlation is trivial.
23.7 Maximum entropy
If entropy measures uncertainty, the distribution with the most entropy is the one assuming the least. That gives a principle for choosing a distribution: among all distributions consistent with what you know, pick the one with maximum entropy.
Anything else smuggles in structure you have no evidence for.
uniform slightly_loaded heavily_loaded nearly_certain
2.5850 2.4710 2.1610 0.7012 The uniform distribution attains \(\log_2 6 = 2.585\) bits and every alternative falls short. Add constraints and the answer changes in a recognizable way:
| Given | Maximum-entropy distribution |
|---|---|
| a finite set of outcomes | uniform |
| support \([0,\infty)\) and a fixed mean | exponential |
| support \(\mathbb{R}\) and a fixed variance | normal |
| a fixed mean on \(\{0,1,2,\dots\}\) | geometric |
The third is the striking one. The normal distribution is not just the limit of sums (Section 22.4) — it is also the least presumptuous distribution with a given variance. Two completely different arguments arrive at the same family, which is a large part of why it is so hard to avoid.
23.8 Summary
| Quantity | Formula | Measures |
|---|---|---|
| Surprisal | \(-\log p(x)\) | information in one outcome |
| Entropy | \(-\sum p\log p\) | expected surprisal; uncertainty |
| Cross-entropy | \(-\sum p\log q\) | expected surprisal under belief \(q\) |
| KL divergence | \(\sum p\log(p/q)\) | wasted bits; \(\geq 0\), asymmetric |
| Mutual information | \(D_{\text{KL}}(p_{xy}\|p_xp_y)\) | dependence, linear or not |
| Key identity | \(D_{\text{KL}} = H(p,q) - H(p)\) | why cross-entropy is the loss |
23.9 Exercises
1. Compute the entropy of a fair 8-sided die and of a coin that lands heads 75% of the time.
TipSolution
eight_sided log2_of_8 biased_coin
3.0000000 3.0000000 0.8112781 Exactly 3 bits for the die — you could encode the outcome in three binary digits, which is what a bit is. The biased coin gives 0.811 bits, less than a fair coin’s 1, because guessing heads is right three times in four.
2. Show numerically that cross-entropy is minimized when the belief equals the truth.
TipSolution
[,1] [,2] [,3] [,4] [,5]
belief_heads 0.5 0.6000 0.7000 0.8000 0.900
cross_entropy 1.0 0.9125 0.8813 0.9219 1.103c(entropy_of_truth = entropy(truth))entropy_of_truth
0.8812909 The minimum is at \(q = 0.7\), where cross-entropy equals the entropy of the truth — 0.8813 bits. That floor is unavoidable: even a perfect model faces the randomness in the data, which is Section 21.10’s irreducible error, measured in bits.
3. Verify that \(D_{\text{KL}}\) is asymmetric using \(p = (0.7, 0.3)\) and \(q = (0.4, 0.6)\).
TipSolution
kl_p_q kl_q_p difference
0.26514845 0.27705803 -0.01190959 Both positive, and unequal. Because KL is not symmetric and has no triangle inequality, “KL distance” is a misnomer — divergence is the correct word, and the direction always has to be stated.
4. A model predicts probability 0.99 for the correct class on 99 examples and 0.01 on one. Compare its mean cross-entropy with a model that predicts 0.7 every time.
TipSolution
confident_mean_bits hedged_mean_bits accuracy_confident accuracy_hedged
0.08079314 0.51457317 0.99000000 1.00000000 The confident model is 99% accurate and the hedged model 100%, yet their cross-entropies are close — the single disastrous prediction costs 6.64 bits on its own, wiping out the gains from 99 excellent ones.
Accuracy and cross-entropy rank models differently. Cross-entropy cares about calibration, and one confidently wrong prediction is expensive.
5. Compute the mutual information between two variables that are perfectly dependent.
TipSolution
I H_X H_X_given_Y
1 1 0 \(I(X;Y) = H(X) = 1\) bit, so \(H(X \mid Y) = 0\): knowing \(Y\) removes all uncertainty about \(X\). Mutual information is bounded above by \(\min(H(X), H(Y))\) — you cannot learn more about \(X\) than there was to know.
6. Among all distributions on six outcomes with \(P(\text{outcome } 1) = 0.5\), which has the highest entropy?
TipSolution
The one spreading the remaining \(0.5\) uniformly over the other five, giving \(0.1\) each.
uniform_rest skewed_rest concentrated
2.1610 1.8913 1.5295 The maximum-entropy principle at work: honor the constraint you were given, and assume nothing whatsoever about the rest. Any other allocation asserts structure the constraint does not justify.