信息获取，相互信息及相关措施

Andrew More 将信息获取定义为：

$IG(Y|X) = H(Y) - H(Y|X)$

其中$H(Y|X)$是条件熵。但是，维基百科称上述数量互为信息。

另一方面，维基百科将信息增益定义为两个随机变量之间的Kullback-Leibler散度（又名信息散度或相对熵）：

$D_{KL}(P||Q) = H(P,Q) - H(P)$

其中被定义为交叉熵。$H(P,Q)$

这两个定义似乎彼此不一致。

我还看到其他作者在谈论另外两个相关概念，即微分熵和相对信息增益。

这些数量之间的确切定义或关系是什么？有没有一本涵盖所有内容的好教科书？

• 信息获取
• 相互信息
• 交叉熵
• 条件熵
• 微分熵
• 相对信息获取

我认为将Kullback-Leibler差异称为“信息增益”是非标准的。

第一个定义是标准的。

编辑：但是，$H(Y)−H(Y|X)$也可以称为互信息。

请注意，我认为您不会找到真正具有标准化，精确且一致的命名方案的科学学科。因此，您始终必须查看公式，因为它们通常会为您提供更好的主意。

教科书：请参阅“很好地介绍各种熵”。

另外：Cosma Shalizi：《复杂系统科学的方法和技术：概述》，Thomas S. Deisboeck和J. Yasha Kresh（ed。）的第1章（pp。33--114），生物医学中的复杂系统科学， http：// arxiv.org/abs/nlin.AO/0307015

罗伯特·格雷（Robert M. Gray）：熵和信息论 http://ee.stanford.edu/~gray/it.html

David MacKay：信息理论，推理和学习算法 http://www.inference.phy.cam.ac.uk/mackay/itila/book.html

还有，“什么是“熵和信息增益”？”

$p(X,Y)$$P(X)P(Y)$

$$\begin{align} I(X; Y) &= H(Y) - H(Y \mid X)\\ &= - \sum_y p(y) \log p(y) + \sum_{x,y} p(x) p(y\mid x) \log p(y\mid x)\\ &= \sum_{x,y} p(x, y) \log{p(y\mid x)} - \sum_{y} \left(\sum_{x}p(x,y)\right) \log p(y)\\ &= \sum_{x,y} p(x, y) \log{p(y\mid x)} - \sum_{x,y}p(x, y) \log p(y)\\ &= \sum_{x,y} p(x, y) \log \frac{p(y\mid x)}{p(y)}\\ &= \sum_{x,y} p(x, y) \log \frac{p(y\mid x)p(x)}{p(y)p(x)}\\ &= \sum_{x,y} p(x, y) \log \frac{p(x, y)}{p(y)p(x)}\\ &= \mathcal D_{KL} (P(X,Y)\mid\mid P(X)P(Y)) \end{align}$$

Note: $p(y) = \sum_x p(x,y)$

Mutual information can be defined using Kullback-Liebler as

$$\begin{align*} I(X;Y) = D_{KL}(p(x,y)||p(x)p(y)). \end{align*}$$

Extracting mutual information from textual datasets as a feature to train machine learning model: ( the task was to predict age, gender and personality of bloggers)

Both definitions are correct, and consistent. I'm not sure what you find unclear as you point out multiple points that might need clarification.

Firstly: $MI_{Mutual Information}\equiv$ $IG_{InformationGain}\equiv I_{Information}$ are all different names for the same thing. In different contexts one of these names may be preferable, i will call it hereon Information.

The second point is the relation between the Kullback–Leibler divergence-$D_{KL}$, and Information. The Kullback–Leibler divergence is simply a measure of dissimilarity between two distributions. The Information can be defined in these terms of distributions' dissimilarity (see Yters' response). So information is a special case of $K_{LD}$, where $K_{LD}$ is applied to measure the difference between the actual joint distribution of two variables (which captures their dependence) and the hypothetical joint distribution of the same variables, were they to be independent. We call that quantity Information.

The third point to clarify is the inconsistent, though standard notation being used, namely that $\operatorname{H} (X,Y)$ is both the notation for Joint entropy and for Cross-entropy as well.

So, for example, in the definition of Information:

$$\begin{aligned}\operatorname {I} (X;Y)&{}\equiv \mathrm {H} (X)-\mathrm {H} (X|Y)\\&{}\equiv \mathrm {H} (Y)-\mathrm {H} (Y|X)\\&{}\equiv \mathrm {H} (X)+\mathrm {H} (Y)-\mathrm {H} (X,Y)\\&{}\equiv \mathrm {H} (X,Y)-\mathrm {H} (X|Y)-\mathrm {H} (Y|X)\end{aligned}$$ in both last lines, $\operatorname{H}(X,Y)$ is the joint entropy. This may seem inconsistent with the definition in the Information gain page however: $DKL(P||Q)=H(P,Q)−H(P)$ but you did not fail to quote the important clarification - $\operatorname{H}(P,Q)$ is being used there as the cross-entropy (as is the case too in the cross entropy page).

Joint-entropy and Cross-entropy are NOT the same.

Check out this and this where this ambiguous notation is addressed and a unique notation for cross-entropy is offered - $H_q(p)$

I would hope to see this notation accepted and the wiki-pages updated.