Shannon Entropy, explained.
Shannon entropy measures the average uncertainty of a probability distribution, expressed in bits when logarithms use base two.
Why it happens
An outcome that was unlikely carries more surprise when it occurs. Entropy averages that surprise across all outcomes, weighting each by its probability. A predictable binary source has low entropy; an evenly balanced one has the maximum of one bit per outcome.
Entropy measures average surprise under a specified probability model.
Read the result
Move the binary probability toward certainty and compare the average uncertainty. Do not confuse the surprise of one rare outcome with the average entropy of the entire source.
A worked example
Predicting a coin-like source
One source produces heads half the time. Another always produces heads.
The balanced source has one bit of entropy per independent output; the certain source has zero.
The difference measures uncertainty about the output, not whether either output is meaningful to a person.
OPTIONAL DEEPER DETAILGo deeper: inside the model
Inside this model
For a binary event with probability p, H(p)=−p log₂ p−(1−p) log₂(1−p) bits. Entropy peaks at one bit for p=0.5 and approaches zero at the extremes.
Where this idea is useful
A practical use
Estimate how much information a yes/no answer may carry when one answer is rare.
A common misconception
“High entropy means valuable or intelligent information.”
Entropy describes a probability pattern. It does not measure truth, usefulness, or meaning.
What this explanation leaves out
- Entropy describes uncertainty in a model, not meaning or usefulness of the message.
Can a rare event be surprising in a low-entropy system?
Yes. A nearly certain source has low average uncertainty, but its rare alternative can carry a large amount of surprise when it occurs.
Are you measuring unpredictability, or the value of understanding the message?
Associated thinkers
Further reading
Explore the original research or the teaching reference behind this experiment.