Entropy

Entropy is a concept which originally arose from the study of the physics of heat engines. It can be described as a measure of the amount of disorder in a system. An organized structure, such as a crystal or a living organism, is very highly ordered and consequently has low entropy. When the crystal is heated sufficiently, it melts and becomes liquid, a much less ordered state. When the organism dies, it decays and becomes completely disrupted. In either system, its entropy increases.

Another way of expressing entropy is to consider the spread of states which a system can adopt. A low entropy system occupies a small number of such states, while a high entropy system occupies a large number of states.

In the case of an image, these states correspond to the gray levels which the individual pixels can adopt. For example, in an 8-bit pixel there are 256 such states. If all such states are equally occupied, as they are in the case of an image which has been perfectly histogram equalized, the spread of states is a maximum, as is the entropy of the image. On the other hand, if the image has been thresholded, so that only two states are occupied, the entropy is low. If all of the pixels have the same value, the entropy of the image is zero.

Note that, in this progression, as the entropy of the image is decreased, so is its information content. We moved from a full gray scale image, with high entropy, to a thresholded binary image, with low entropy, to a single-valued image, with zero entropy.

Information, in this context, refers to the announcement of the unexpected. If the pixels of an image were inspected, and found to be the same. This information could have been communicated in a very short message. The information content is said to be low simply because it can be communicated in a short message. If the pixels are changing in unexpected ways, however, longer messages are required to communicate this fact and the information is said to increase. This assumes, of course, that all changes in the image are meaningful. Changes due to noise are still considered to be information in that they describe the image as it actually is, rather than as it should be.

Noise and heat play similar roles in increasing the entropy of systems. The entropy H of an image is defined as

where M is the number of gray levels and p_k is the probability associated with gray level k.

Maximum entropy is achieved in the case of a uniform probability distribution. If , then p_k is constant and given by

The maximum entropy is found from

Minimum entropy is achieved when the image itself is constant, that is, all of the pixels have the same gray level k. For that gray level, p_k = 1, and H = -log(1) = 0.

Entropy sets a lower bound on the average number of bits per pixel required to encode an image without distortion. This is only true, however, for uncorrelated images. Consider the images below,

The image on the left (uniform random noise) has an entropy of 8 bits and is uncompressible (look at the GIF file size). The image on the right has the same distribution of gray levels but is highly spatially correlated.

References

Rhys Lewis, Practical Image Processing, Ellis Horwood Limited, 1990. ISBN 0-13-6383525-0. p 90-91.
William Pratt, Digital Image Processing, Second Edition, John Wiley & Sons, Inc., 1991. ISBN 0-471-85766-1. p. 561-563.
Azriel Rosenfeld and Avinash Kak, Digital Picture Processing, Second Edition, Volume 1, Academic Press, 1982. ISBN 0-12-597301-2. p. 194-198.

Maintained by John Loomis, last updated 10 June 1998