nLab
relative entropy

Idea

The notion of relative entropy of states is a generalization of the notion of entropy to a situation where the entropy of one state is measured “relative to” another state.

is also called

• Kullback-Leibler divergence

• information divergence

• information gain .

Definition

For states on finite probability spaces

For two finite probability distributions $(p_i)$ and $(q_i)$, their relative entropy is

$$S(p/q) := \sum_{k = 1}^n p_k(log p_k - log q_k) \,.$$

Alternatively, for $\rho, \phi$ two density matrices, their relative entropy is

$$S(\rho/\phi) := tr \rho(log \rho - log \phi) \,.$$

For states on classical probability spaces

Definition

For $X$ a measurable space and $P$ and $Q$ two probability measures on $X$, such that $Q$ is absolutely continuous with respect to $P$, their relative entropy is the integral

$$S(Q|P) = \int_X log \frac{d Q}{d P} d P \,,$$

where $d Q / d P$ is the Radon-Nikodym derivative of $Q$ with respect to $P$.

For states on quantum probability spaces (von Neumann algebras)

Let $A$ be a von Neumann algebra and let $\phi$, $\psi : A \to \mathbb{C}$ be two states on it (faithful, positive linear functionals).

Definition

The relative entropy $S(\phi/\psi)$ of $\psi$ relative to $\phi$ is

$$S(\phi/\psi) := - (\Psi, (log \Delta_{\Phi,\Psi}) \Psi) \,,$$

where $\Delta_{\Phi,\Psi}$ is the relative modular operator? of any cyclic and separating vector representatives $\Phi$ and $\Psi$ of $\phi$ and $\psi$.

This is due to (Araki).

Proposition

• This definition is independent of the choice of these representatives.

• In the case that $A$ is finite dimensional and $\rho_\phi$ and $\rho_\psi$ are density matrices of $\phi$ and $\psi$, respectively, this reduces to the above definition.

References

Relative entropy of states on von Neumann algebras was introduced in

• Huzihiro Araki, Relative Entropy of States of von Neumann Algebras (pdf)

A characterization of relative entropy on finite-dimensional C-star algebras is given in

• D. Petz, Characterization of the relative entropy of states of matrix algebras (pdf)

A survey of entropy in operator algebras is in

• Erling Størmer, Entropy in operator algebras (pdf)