证明1:
首先要注意为所有一> 0。lna≤a−1a>0
现在,我们将证明,这意味着d ķ 大号(p | | q )≥ 0−DKL(p||q)≤0DKL(p||q)≥0
−D(p||q)=−∑xp(x)lnp(x)q(x)=∑xp(x)lnq(x)p(x)≤(a)∑xp(x)(q(x)p(x)−1)=∑xq(x)−∑xp(x)=1−1=0
ln
−∑xp(x)log2p(x)≤−∑xp(x)log2q(x)
∑xp(x)log2p(x)−∑xp(x)log2q(x)≥0∑xp(x)log2p(x)q(x)≥0
我之所以不将其作为单独的证明是因为,如果您要让我证明吉布斯不等式,我将不得不从KL散度的非负性开始,并从顶部进行同样的证明。
∑i=1nailog2aibi≥(∑i=1nai)log2∑ni=1ai∑ni=1bi
Then we can show that DKL(p||q)≥0:
D(p||q)=∑xp(x)log2p(x)q(x)≥(b)(∑xp(x))log2∑xp(x)∑xq(x)=1⋅log211=0
where we have used the Log sum inequality at (b).
Proof 3:
(Taken from the book "Elements of Information Theory" by Thomas M. Cover and Joy A. Thomas)
−D(p||q)=−∑xp(x)log2p(x)q(x)=∑xp(x)log2q(x)p(x)≤(c)log2∑xp(x)q(x)p(x)=log21=0
where at (c) we have used Jensen's inequality and the fact that log is a concave function.