为什么当概率分布均匀时熵最大？

32

我知道熵是过程/变量随机性的量度，可以定义如下。对于集合的随机变量：。在MacKay撰写的《熵和信息论》一书中，他在第二章中提供了这一陈述 $X \in$ $A$ $H(X)= \sum_{x_i \in A} -p(x_i) \log (p(x_i))$

如果p是均匀的，则熵最大。

直观地说，我能够理解，如果像在集合中的所有数据点都以相同的概率拾取（为组的基数），则随机性或熵的增加。但是，如果我们知道集合中的某些点比其他点更有可能发生（例如，在正态分布的情况下，数据点的最大集中度在均值附近，并且标准偏差区域较小，则随机性或熵应减少。 $A$ $1/m$ $m$ $A$ $A$

但是，对此有任何数学证明吗？像的方程式一样，我针对对其进行微分，并将其设置为0或类似的值。 $H(X)$ $p(x)$

附带说明一下，信息理论中出现的熵和化学（热力学）中的熵计算之间是否有联系？

uniform entropy maximum-entropy

— 用户名
source

2

可以通过stats.stackexchange.com/a/49174/919回答（通过）此问题。

— ub

我对克里斯托弗·毕晓普斯（Christopher Bishops）的书中的另一句话感到非常困惑，该书指出“对于单个实变量，使熵最大化的分布就是高斯”。它还指出“对于给定的协方差，具有最大熵的多元分布是高斯”。此声明如何有效？均匀分布的熵不是总是最大吗？

— user76170 2014年

6

始终在可能解决方案的约束条件下执行最大化。当约束条件是所有概率都必须超过预定义的限制时，最大熵解是统一的。相反，当约束条件是期望值和方差必须等于预定义值时，则ME解决方案是高斯式的。您引用的陈述必须在声明或至少隐含理解了这些约束的特定上下文中进行。

— ub

2

我可能还应该提到，“熵”一词在高斯环境中的含义与此处原始问题中的含义有所不同，因为那时我们正在讨论连续分布的熵。这种“微分熵”是与离散分布的熵不同的动物。主要区别在于，在变量变化的情况下，微分熵不是不变的。

— ub

那么，这意味着最大化总是与约束有关的？如果没有约束怎么办？我的意思是，不能有这样的问题吗？哪个概率分布具有最大熵？

— user76170 '02

25

启发式地，上的概率密度函数具有最大熵的证明是与最小知识量相对应的，即均匀分布。 $\{x_1, x_2,..,.x_n\}$ $\{x_1, x_2,..,.x_n\}$

现在，要获得更正式的证明，请考虑以下内容：

上的概率密度函数是一组非负实数加起来为1。熵是一个连续函数元组，并且这些点位于的紧凑子组，所以有一个 $\{x_1, x_2,..,.x_n\}$ $p_1,...,p_n$ $n$ $(p_1,...,p_n)$ $\mathbb{R}^n$ $n$ -熵最大的元组。我们要展示这发生在和其他地方。 $(1/n,...,1/n)$

假设并非全部相等，例如。（显然。）我们将发现具有更高熵的新概率密度。然后，由于熵在某个元组处最大化，因此该熵在元组处唯一地最大化，所有。 $p_j$ $p_1 < p_2$ $n\neq 1$ $n$ $n$ $p_i = 1/n$ $i$

由于，对于小的正我们有。的熵减去的熵 $p_1 < p_2$ $\varepsilon$ $p_1 + \varepsilon < p_2 -\varepsilon$ $\{p_1 + \varepsilon, p_2 -\varepsilon,p_3,...,p_n\}$ 等于 $\{p_1,p_2,p_3,...,p_n\}$

为了完成证明，我们要证明这是积极的足够小。改写上述公式为

- p_{1} \log (\frac{p_{1} + ε}{p_{1}}) - ε \log (p_{1} + ε) - p_{2} \log (\frac{p_{2} - ε}{p_{2}}) + ε \log (p_{2} - ε)

$-p_1\log\left(\frac{p_1+\varepsilon}{p_1}\right)-\varepsilon\log(p_1+\varepsilon)-p_2\log\left(\frac{p_2-\varepsilon}{p_2}\right)+\varepsilon\log(p_2-\varepsilon)$

ε

$\varepsilon$

- p_{1} \log (1 + \frac{ε}{p_{1}}) - ε (\log p_{1} + \log (1 + \frac{ε}{p_{1}})) - p_{2} \log (1 - \frac{ε}{p_{2}}) + ε (\log p_{2} + \log (1 - \frac{ε}{p_{2}}))

$-p_1\log\left(1+\frac{\varepsilon}{p_1}\right)-\varepsilon\left(\log p_1+\log\left(1+\frac{\varepsilon}{p_1}\right)\right)-p_2\log\left(1-\frac{\varepsilon}{p_2}\right)+\varepsilon\left(\log p_2+\log\left(1-\frac{\varepsilon}{p_2}\right)\right)$

回顾为小，上述公式是，其为正时 $\log(1 + x) = x + O(x^2)$ $x$

- ε - ε \log p_{1} + ε + ε \log p_{2} + O (ε^{2}) = ε \log (p_{2} / p_{1}) + O (ε^{2})

$-\varepsilon-\varepsilon\log p_1 + \varepsilon + \varepsilon \log p_2 + O(\varepsilon^2) = \varepsilon\log(p_2/p_1) + O(\varepsilon^2)$

足够小，因为

。

ε

$\varepsilon$

p_{1} < p_{2}

$p_1 < p_2$

不太严格的证据如下：

首先考虑以下引理：

让和是对的间隔连续概率密度函数在实数，用和上。我们有，如果存在两个积分。此外，当且仅当 $p(x)$ $q(x)$ $I$ $p\geq 0$ $q > 0$ $I$

- \int_{I} p \log p d x \leq - \int_{I} p \log q d x

$-\int_I p\log p dx\leq -\int_I p\log q dx$

对于所有

。

p (x) = q (x)

$p(x) = q(x)$

x

$x$

现在，令为上的任何概率密度函数，其中。令所有， $p$ $\{x_1,...,x_n\}$ $p_i = p(x_i)$ $q_i = 1/n$ $i$ 是的熵。因此，我们的引理说，用平等的当且仅当是均匀的。

- \sum_{i = 1}^{n} p_{i} \log q_{i} = \sum_{i = 1}^{n} p_{i} \log n = \log n

$-\sum_{i=1}^n p_i\log q_i = \sum_{i=1}^n p_i \log n=\log n$

q

$q$

h (p) \leq h (q)

$h(p)\leq h(q)$

p

$p$

此外，维基百科对此也进行了简短的讨论：维基

— 米奇斯
source

11

我很欣赏呈现基本（无微积分）证明的努力。通过加权，可以通过加权AM-GM不等式进行严格的单线演示。

\exp (H)

$\exp(H)$

\prod {(\frac{1}{p_{i}})}^{p_{i}} \leq \sum p_{i} \frac{1}{p_{i}} = n

$\prod\left(\frac{1}{p_i}\right)^{p_i}\le\sum p_i\frac{1}{p_i}=n$

1 / p_{i}

$1/p_i$

I don't understand how

\sum \log n

$\sum{\log{n}}$ can be equal to

\log n

$\log{n}$ .

— user1603472

4

@user1603472 do you mean

\sum_{i = 1}^{n} p_{i} \log n = \log n

$\sum\limits_{i=1}^n p_i \log n = \log n$ ? Its because

\sum_{i = 1}^{n} p_{i} \log n = \log n \sum_{i = 1}^{n} p_{i} = \log n \times 1

$\sum\limits_{i=1}^n p_i \log n = \log n \sum\limits_{i=1}^n p_i = \log n \times 1$

— HBeel

@Roland I pulled the

\log n

$\log n$ outside of the sum since it does not depend on

i

$i$ . Then the sum is equal to

1

$1$ because

p_{1}, \dots, p_{n}

$p_1,\ldots,p_n$ are the densities of a probability mass function.

— HBeel

Same explanation with more details can be found here: math.uconn.edu/~kconrad/blurbs/analysis/entropypost.pdf

— Roland

14

Entropy in physics and information theory are not unrelated. They're more different than the name suggests, yet there's clearly a link between. The purpose of entropy metric is to measure the amount of information. See my answer with graphs here to show how entropy changes from uniform distribution to a humped one.

The reason why entropy is maximized for a uniform distribution is because it was designed so! Yes, we're constructing a measure for the lack of information so we want to assign its highest value to the least informative distribution.

Example. I asked you "Dude, where's my car?" Your answer is "it's somewhere in USA between Atlantic and Pacific Oceans." This is an example of the uniform distribution. My car could be anywhere in USA. I didn't get much information from this answer.

However, if you told me "I saw your car one hour ago on Route 66 heading from Washington, DC" - this is not a uniform distribution anymore. The car's more likely to be in 60 miles distance from DC, than anywhere near Los Angeles. There's clearly more information here.

Hence, our measure must have high entropy for the first answer and lower one for the second. The uniform must be least informative distribution, it's basically "I've no idea" answer.

— Aksakal
source

7

The mathematical argument is based on Jensen inequality for concave functions. That is, if $f(x)$ is a concave function on $[a,b]$ and $y_1, \ldots y_n$ are points in $[a,b]$ , then: $n \cdot f(\frac{y_1 + \ldots y_n}{n}) \geq f(y_1) + \ldots + f(y_n)$

Apply this for the concave function $f(x) = -x \log(x)$ and Jensen inequality for $y_i = p(x_i)$ and you have the proof. Note that $p(x_i)$ define a discrete probability distribution, so their sum is 1. What you get is $log(n) \geq \sum_{i=1}^n - p(x_i) log(p(x_i))$ , with equality for the uniform distribution.

— Octavian Ganea
source

1

I actually find the Jensen's inequality proof to be a much deeper proof conceptually than the AM-GM one.

— Casebash

4

On a side note, is there any connnection between the entropy that occurs information theory and the entropy calculations in chemistry (thermodynamics) ?

Yes, there is! You can see the work of Jaynes and many others following his work (such as here and here, for instance).

But the main idea is that statistical mechanics (and other fields in science, also) can be viewed as the inference we do about the world.

As a further reading I'd recommend Ariel Caticha's book on this topic.

— kaslusimoes
source

1

An intuitive explanation:

If we put more probability mass into one event of a random variable, we will have to take away some from other events. The one will have less information content and more weight, the others more information content and less weight. Therefore the entropy being the expected information content will go down since the event with lower information content will be weighted more.

As an extreme case imagine one event getting probability of almost one, therefore the other events will have a combined probability of almost zero and the entropy will be very low.

— Roland
source

0

Main idea: take partial derivative of each $p_i$ , set them all to zero, solve the system of linear equations.

Take a finite number of $p_i$ where $i=1,...,n$ for an example. Denote $q = 1-\sum_{i=0}^{n-1} p_i$ .

\begin{aligned} H & = - \sum_{i = 0}^{n - 1} p_{i} \log p_{i} - (1 - q) \log q \\ H * \ln 2 & = - \sum_{i = 0}^{n - 1} p_{i} \ln p_{i} - (1 - q) \ln q \end{aligned}

$\begin{align} H &= -\sum_{i=0}^{n-1} p_i \log p_i - (1-q)\log q\\ H*\ln 2 &= -\sum_{i=0}^{n-1} p_i \ln p_i - (1-q)\ln q \end{align}$

\begin{aligned} \frac{\partial H}{\partial p_{i}} & = \ln \frac{q}{p_{i}} = 0 \end{aligned}

$\begin{align} \frac{\partial H}{\partial p_i} &= \ln \frac{q}{p_i} = 0 \end{align}$ Then

q = p_{i}

$q = p_i$ for every

i

$i$ , i.e.,

p_{1} = p_{2} = . . . = p_{n}

$p_1=p_2=...=p_n$ .

— Jan Fan
source

I am glad you pointed out this is the "main idea," because it's only a part of the analysis. The other part--which might not be intuitive and actually is a little trickier--is to verify this is a global minimum by studying the behavior of the entropy as one or more of the

p_{i}

$p_i$ shrinks to zero.

— whuber