# nLab Rényi entropy

Contents

### Context

#### Measure and probability theory

measure theory

probability theory

# Contents

## Idea

In information theory, Rényi entropy refers to a class of measures, of entropy that are essentially logarithms of diversity indices.

For special values of its parameter, the notion of Rényy entropy reproduces all of: Shannon entropy, Hartley entropy/max-entropy and min-entropy.

## Definition

Let $p$ be a probability distribution over $n \in \mathbb{N}$ elements, and let $\alpha$ be a non-negative real number not equal to 1:

$\alpha \;\in\; \mathbb{R}_{\geq 0} \setminus \{1\} \,.$

The Rényi entropy of $p$ at order $\alpha$ is:

$H_\alpha(p) \;\coloneqq\; \frac{1}{1-\alpha} \log \left( \sum_{i=1}^n (p_i)^\alpha \right) \,.$

## Properties

### Relation to other notions of entropy

For various (limiting) values of $\alpha$ the Rényi entropy reduces to notions of entropy that are known by their own names:

• for $\alpha = 0$, Rényi entropy equals the Hartley entropy/max-entropy,

• in the limit $\alpha \to 1$, Rényi entropy equals the Shannon entropy,

• for $\alpha = 2$, Rényi entropy equals the collision entropy,

• in the limit $\alpha \to \infty$, Rényi entropy equals the min-entropy.

order$\phantom{\to} 0$$\to 1$$\phantom{\to}2$$\to \infty$
Rényi entropyHartley entropy$\geq$Shannon entropy$\geq$collision entropy$\geq$min-entropy

### Monotonicity

The Rényi entropy is an anti-monotone function in the order-parameter $\alpha$:

$\alpha_1 \;\leq\; \alpha_2 \;\;\;\;\;\;\; \Rightarrow \;\;\;\;\;\;\; H_{\alpha_1}(p) \;\geq\; H_{\alpha_2}(p) \,.$

(e.g. Ram & Sason 16, Fact 1)

In particular, in terms of the above special cases, this means that

$HartlyEntropy \;\geq\; ShannonEntropy \;\geq\; CollisionEntropy \;\geq\; MinEntropy \,.$

Due to:

• Alfréd Rényi, On Measures of Entropy and Information, Berkeley Symposium on Mathematical Statistics and Probability, 1961: 547-561 (1961) (euclid)

Textbook account:

• J. Aczél, Z. Daróczy, Chapter 5 of: On Measures of Information and their Characterizations, Mathematics in Science and Engineering 115, Academic Press 1975 (ISBN:978-0-12-043760-3)