This is just a preliminary definition.
(or more precisely , so that is defined). Notice that, despite the minus sign in this formula, is a nonnegative function (since for ). It is also important that is concave.
Both and are , but for different reasons; because, upon verifying a statement with probability , one gains no information; while because one expects never to verify a statement with probability . In general, is the information gained by verifying a statement of probability , but this will happen only with probability , hence .
We have not specified the base of the logarithm, which amounts to a constant factor (proportional to the logarithm of the base), which we think of as specifying the unit of measurement of entropy. Common choices for the base are (whose unit is the bit, originally a unit of memory in computer science), (byte: bits), (trit), (nat or neper), (bel, originally a unit of power intensity in telegraphy, or ban, dit, or hartley), and (decibel: of a bel). In applications to statistical physics, common bases are approximately (joule per kelvin), (calorie per mole-kelvin), etc.
This is a general mathematical definition of entropy.
In words, the entropy is the supremum, over all ways of expressing as an internal disjoint union of finitely many elements of the -algebra , of the sum, over these measurable sets, of the expected information of verification of these sets. This supremum can also be expressed as a limit as we take to be finer and finer, since is concave and the partitions are directed.
(Without loss of generality, we do not need the elements of to be disjoint, as long as their intersections are null sets. Similarly, we do not need their union to be all of , as long as their union is a full set. In constructive mathematics, it seems that we must weaken the latter condition in this way.)
This definition is very general, and it is instructive to look at special cases.
Given a probability space , the entropy of this probability space is the entropy, with respect to , of the -algebra of all measurable subsets of .
Recall that a partition of a set is a family of such that is the union of and any two distinct elements of are disjoint. (That is, the supremum in (1) is taken over finite partitions of into elements of .)
Every partition of a measure space into measurable sets (indeed, any family of measurable subsets of ) generates a -algebra of measurable sets. The entropy of a measurable partition of a probability measure space is the entropy, with respect to , of the -algebra generated by . The formula (1) may then be written
since an infinite sum (of positive terms) may also be defined as a supremum. (Actually, the supremum in the infinite sum does not quite match the supremum in (1), so there is a bit of a theorem to prove here.)
In most of the following special cases, we will consider only partitions, although it would be possible also to consider more general -algebras.
Recall that a discrete probability space is a set equipped with a function such that ; since is possible for only countably many , we may assume that is countable. We make into a measure space (with every subset measurable) by defining . Since every set is measurable, any partition of is a partition into measurable sets.
Given a discrete probability space and a partition of , the entropy of with respect to is defined to be the entropy of with respect to the probability measure induced by . Simplifying (2), we find
More specially, the entropy of the discrete probability space is the entropy of the partition of into singletons; we find
This is actually a special case of the entropy of a probability space, since the -algebra generated by the singletons is the power set of (at least when is countable, and the formulas agree in any case).
Yet more specially, the entropy of a finite set is the entropy of equipped with the uniform discrete probability measure; we find
which is probably the earliest mathematical formula for entropy, due to Boltzmann. (Its physical interpretation appears below.)
Recall that a Borel measure? on an interval in the real line is absolutely continuous if whenever is a null set (with respect to Lebesgue measure). In this case, we can take the Radon-Nikodym derivative of with respect to Lebesgue measure, to get an integrable function , called the probability distribution function; we recover by
where the integral is taken with respect to Lebesgue measure.
If is a partition of an interval into Borel sets, then the entropy of with respect to an integrable function is the entropy of with respect to the measure induced by using the integral formula (4); we find
On the other hand, the entropy of the probability distribution space is the entropy of the entire -algebra of all Borel sets with respect to ; we find
by a fairly complicated argument.
I haven't actually managed to check this argument yet, although my memory tags it as a true fact. —Toby
Note that this -algebra is not generated by a partition.
So just as the entropy of a probability distribution is given by , so the entropy of a density operator is
using the functional calculus.
There is a way to fit this into the framework given by (1), but I don't remember it (and never really understood it).
For two finite probability distributions and , their relative entropy is
Or alternatively, for two density matrices, their relative entropy is
For more on this see relative entropy.
As hinted above, any probability distribution on a phase space in classical physics has an entropy, and any density matrix on a Hilbert space in quantum physics has an entropy. However, these are microscopic entropy, which is not the usual entropy in thermodynamics and most other branches of physics. (In particular, microscopic entropy is conserved, rather than increasing with time.)
Instead, physicists use coarse-grained entropy, which corresponds mathematically to taking the entropy of a -algebra much smaller than the -algebra of all measurable sets. Given a classical system with microscopic degrees of freedom, we identify macroscopic degrees of freedom that we can reasonably expect to measure, giving a map from to (or more generally, a map from an -dimensional microscopic phase space to an -dimensional macroscopic phase space). Then the -algebra of all measurable sets in pulls back to a -algebra on , and the macroscopic entropy of a statistical state is the entropy of this -algebra. (Typically, is on the order of Avogadro's number, while is rarely more than half a dozen, and often as small as .)
Generally, we specify a state by a point in , a macroscopic pure state, and assume a uniform probability distribution on its fibre in . If this fibre were a finite set, then we would recover Boltzmann's formula (3). This is never exactly true in classical statistical physics, but it is often nevertheless a very good approximation. (Boltzmann's formula actually makes better physical sense in quantum statistical physics, even though Boltzmann himself did not live to see this.)
The concept of entropy was introduced, by Rudolf Clausius in 1865, in the context of physics, and then adapted to information theory by Claude Shannon in 1948, to ergodic theory by Andrey Kolmogorov and Sinai in 1958, and to topological dynamics by Adler, Konheim and McAndrew in 1965.
A survey of entropy in operator algebras is in
A large collection of references on quantum entropy is in
After the concept of entropy proved enormously useful in practice, many people have tried to find a more abstract foundation for the concept (and its variants) by characterizing it as the unique measure satisfying some list of plausible-sounding axioms.