In information theory, Rényi entropy refers to a class of measures, of entropy that are essentially logarithms of diversity indices.
For special values of its parameter, the notion of Rényy entropy reproduces all of: Shannon entropy, Hartley entropy/max-entropy and min-entropy.
Let $p$ be a probability distribution over $n \in \mathbb{N}$ elements, and let $\alpha$ be a non-negative real number not equal to 1:
The Rényi entropy of $p$ at order $\alpha$ is:
For various (limiting) values of $\alpha$ the Rényi entropy reduces to notions of entropy that are known by their own names:
for $\alpha = 0$, Rényi entropy equals the Hartley entropy/max-entropy,
in the limit $\alpha \to 1$, Rényi entropy equals the Shannon entropy,
for $\alpha = 2$, Rényi entropy equals the collision entropy,
in the limit $\alpha \to \infty$, Rényi entropy equals the min-entropy.
order | $\phantom{\to} 0$ | $\to 1$ | $\phantom{\to}2$ | $\to \infty$ | ||||
---|---|---|---|---|---|---|---|---|
Rényi entropy | Hartley entropy | $\geq$ | Shannon entropy | $\geq$ | collision entropy | $\geq$ | min-entropy |
The Rényi entropy is an anti-monotone function in the order-parameter $\alpha$:
(e.g. Ram & Sason 16, Fact 1)
In particular, in terms of the above special cases, this means that
Due to:
Textbook account:
See also
Wikipedia, Rényi entropy
Eshed Ram, Igal Sason, On Renyi Entropy Power Inequalities (arXiv:1601.06555)
On holographic Renyi entropy in relation to holographic entanglement entropy and quantum error correcting codes:
Xi Dong, The Gravity Dual of Renyi Entropy, Nature Communications 7, 12472 (2016) (arXiv:1601.06788, doi:10.1038/ncomms12472)
Chris Akers, Pratik Rath, Holographic Renyi entropy from quantum error correction, J. High Energ. Phys. 2019, 52 (2019) (arXiv:1811.05171)
Last revised on May 28, 2021 at 12:14:23. See the history of this page for a list of all contributions to it.