Entropy, cross entropy and KL divergence on the same distribution

Entropy measures uncertainty under a distribution. Cross entropy measures the expected negative log probability assigned by another distribution. KL divergence is the excess cross entropy relative to the original distribution's entropy. Calculate all three on the same outcomes and logarithm base before comparing their numerical values.

Our invented source produces A with probability 0.75 and B with probability 0.25. A proposed model assigns probabilities 0.5 and 0.5. We can calculate the mismatch exactly, without training a model or treating one sampled sequence as the population.

Write the definitions with their direction

For true distribution p and model distribution q, entropy is H(p) = -sum p_i log(p_i). Cross entropy is H(p,q) = -sum p_i log(q_i). KL divergence is D_KL(p || q) = sum p_i log(p_i/q_i).

Subtracting the first expression from the second gives H(p,q) = H(p) + D_KL(p || q). The equality requires compatible support and consistent log units. Natural logarithms produce nats; logarithms base 2 produce bits.

Here the uniform model always assigns a negative log probability of log(2), so its cross entropy is about 0.693 nats. The source itself is less uncertain than a fair binary outcome, so its entropy is smaller. The difference is the mismatch penalty.

Verify the arithmetic and support check

import numpy as np
p = np.array([.75, .25])
q = np.array([.5, .5])
def cross_entropy(source, model):
    positive = source > 0
    if np.any(model[positive] <= 0):
        return float('inf')
    return float(-np.sum(source[positive] * np.log(model[positive])))
H = cross_entropy(p, p)
CE = cross_entropy(p, q)
KL = CE - H
reverse = cross_entropy(q, p) - cross_entropy(q, q)
print(f'H(p)={H:.6f} H(p,q)={CE:.6f} KL(p||q)={KL:.6f}')
print(f'KL(q||p)={reverse:.6f}')
print(f'cross entropy in bits={CE / np.log(2):.6f}')
print('zero model probability:', cross_entropy(p, np.array([1., 0.])))
assert np.isclose(KL, np.sum(p * np.log(p / q)))
assert KL >= 0

Checked output:

H(p)=0.562335 H(p,q)=0.693147 KL(p||q)=0.130812
KL(q||p)=0.143841
cross entropy in bits=1.000000
zero model probability: inf

Explain why direction matters

The reverse KL value is different because the expectation now weights outcomes using q. KL is not generally symmetric and is not a distance metric in the usual geometric sense. Naming only “the KL between two distributions” leaves out information needed to reproduce the calculation.

A zero source probability contributes zero by the usual limiting convention, provided it is handled without evaluating an invalid 0 × log(0) expression. A zero model probability for an outcome with positive source probability is a more serious issue: its expected negative log probability is infinite.

Clipping model probabilities avoids a numerical infinity but changes the quantity being measured. If you clip, document the threshold and, when treating the result as a probability distribution, account for normalization. The helper here chooses to report infinity rather than silently change the distribution.

Connect cross entropy to a training example

For a single observed class label, the empirical target is one-hot. Cross entropy then becomes the negative log probability assigned to that observed class. Averaging those losses estimates an expectation over the sample, which need not equal the population expectation in our exact two-outcome calculation.

A classifier's logits are not probabilities. A stable cross-entropy implementation combines normalization and logarithms without explicitly computing very small probabilities. Log-sum-exp numerical stability explains the arithmetic behind that implementation. Passing probabilities into a function that expects logits can produce the wrong loss without an obvious shape error.

Extend the distribution

Move q gradually from [0.5, 0.5] toward p and confirm that the mismatch penalty decreases. Then reverse p and q at each step and compare the curves. Finally, add a third outcome with zero source probability and state how you handle its contribution.

The Statistics course connects distributions to uncertainty. Use logistic regression from scratch to inspect a binary cross-entropy objective inside an actual fitted classifier, and keep the distinction between sample loss and exact distributional quantities visible.

References

SciPy entropy API — Entropy, relative entropy, log bases and distribution normalization.