熵告诉我们什么？

我正在阅读有关熵的信息，并且在概念上很难理解连续情况下的含义。Wiki页面指出以下内容：

事件的概率分布与每个事件的信息量一起形成一个随机变量，其期望值为该分布生成的平均信息量或熵。

因此，如果我计算出与连续概率分布相关的熵，那到底能告诉我什么？他们给出了一个有关抛硬币的例子，所以是离散情况，但是如果有一种直观的方式来解释一个连续的例子，那就太好了！

如果有帮助，则连续随机变量的熵定义如下：$X$

$$H(X)=-\int P(x)\log_b P(x)dx$$ where $P(x)$ is a probability distribution function.

---

To try and make this more concrete, consider the case of $X\sim \text{Gamma}(\alpha,\beta)$, then, according to Wikipedia, the entropy is

$$\begin{align} H(X)&=\mathbb{E}[-\ln(P(X))]\\ &=\mathbb{E}[-\alpha\ln(\beta)+\ln(\Gamma(\alpha))+\ln(\Gamma(\alpha))-(\alpha-1)\ln(X)+\beta X]\\ &=\alpha-\ln(\beta)+\ln(\Gamma(\alpha))+(1-\alpha)\left(\frac{d}{d\alpha}\ln(\Gamma(\alpha))\right) \end{align}$$

And so now we have calculated the entropy for a continuous distribution (the Gamma distribution) and so if I now evaluate that expression, $H(X)$, given $\alpha$ and $\beta$, what does that quantity actually tell me?

The entropy tells you how much uncertainty is in the system. Let's say you're looking for a cat, and you know that it's somewhere between your house and the neighbors, which is 1 mile away. Your kids tell you that the probability of a cat being on the distance $x$ from your house is described best by beta distribution $f(x;2,2)$. So a cat could be anywhere between 0 and 1, but more likely to be in the middle, i.e. $x_{max}=1/2$.

Let's plug the beta distribution into your equation, then you get $H=-0.125$.

Next, you ask your wife and she tells you that the best distribution to describe her knowledge of your cat is the uniform distribution. If you plug it to your entropy equation, you get $H=0$.

Both uniform and beta distributions let the cat be anywhere between 0 and 1 miles from your house, but there's more uncertainty in the uniform, because your wife has really no clue where the cat is hiding, while kids have some idea, they think it's more likely to be somewhere in the middle. That's why Beta's entropy is lower than Uniform's.

You might try other distributions, maybe your neighbor tells you the cat likes to be near either of the houses, so his beta distribution is with $\alpha=\beta=1/2$. Its $H$ must be lower than that of uniform again, because you get some idea about where to look for a cat. Guess whether your neighbor's information entropy is higher or lower than your kids'? I'd bet on kids any day on these matters.

UPDATE:

How does this work? One way to think of this is to start with a uniform distribution. If you agree that it's the one with the most uncertainty, then think of disturbing it. Let's look at the discrete case for simplicity. Take $\Delta p$ from one point and add it to another like follows:

$$p_i'=p-\Delta p$$ $$p_j'=p+\Delta p$$

Now, let's see how the entropy changes:

$$H-H'=p_i\ln p_i-p_i\ln (p_i-\Delta p)+p_j\ln p_j-p_j\ln (p_j+\Delta p)$$ $$=p\ln p-p\ln [p(1-\Delta p/p)]+p\ln p-p\ln [p(1+\Delta p/p)]$$ $$=-\ln (1-\Delta p/p)-\ln (1+\Delta p/p)>0$$ This means that any disturbance from the uniform distribution reduces the entropy (uncertainty). To show the same in continuous case, I'd have to use calculus of variations or something along this line, but you'll get the same kind of result, in principle.

UPDATE 2: The mean of $n$ uniform random variables is a random variable itself, and it's from Bates distribution. From CLT we know that this new random variable's variance shrinks as $n\to\infty$. So, uncertainty of its location must reduce with increase in $n$: we're more and more certain that a cat's in the middle. My next plot and MATLAB code shows how the entropy decreases from 0 for $n=1$ (uniform distribution) to $n=13$. I'm using distributions31 library here.

x = 0:0.01:1;
for k=1:5
    i = 1 + (k-1)*3;
    idx(k) = i;
    f = @(x)bates_pdf(x,i);
    funb=@(x)f(x).*log(f(x));
    fun = @(x)arrayfun(funb,x);
    h(k) = -integral(fun,0,1);
    subplot(1,5+1,k)

    plot(x,arrayfun(f,x))
    title(['Bates(x,' num2str(i) ')'])
    ylim([0 6])
end

subplot(1,5+1,5+1)
plot(idx,h)
title 'Entropy'

I'd like to add a straightforward answer to this question:

what does that quantity actually tell me?

It's intuitive to illustrate that in a discrete scenario. Suppose that you toss a heavily biased coin, saying the probability of seeing a head on each flip is 0.99. Every actual flip tells you very little information because you almost already know that it will be head. But when it comes to a fairer coin, it't harder for you to have any idear what to expect, then every flip tells you more information than any more biased coin. The quantity of information obtained by observing a single toss is equated with $\log \frac{1}{p(x)}$.

What the quantity of the entropy tells us is the information every actual flipping on (weighted) average can convey: $E \log \frac{1}{p(x)} = \sum p(x) \log \frac{1}{p(x)}$. The fairer the coin the larger the entropy, and a completely fair coin will be maximally informative.