两个伽马分布之间的Kullback–Leibler散度


15

选择通过pdf g x ; b c = 1参数化伽马分布Γ(b,c)g(x;b,c)=1Γ(c)xc1bcex/b 之间的相对熵Γ(bq,cq)Γ(bp,cp)是由为[1]中给出

KLGa(bq,cq;bp,cp)=(cq1)Ψ(cq)logbqcqlogΓ(cq)+logΓ(cp)+cplogbp(cp1)(Ψ(cq)+logbq)+bqcqbp

我猜Ψ(x):=Γ(x)/Γ(x)digamma函数

这是没有派生的。我找不到任何可以得出这一点的参考。有什么帮助吗?一个好的参考就足够了。困难的部分是将与gamma pdf 集成。logx

[1] WD Penny,法线,伽马,狄利克雷和Wishart密度的KL散度,请访问:www.fil.ion.ucl.ac.uk/~wpenny/publications/densities.ps


2
将pdf的导数相对于引入您要寻找的因子l o g x :这就是为什么digamma出现的原因。clog(x)
ub

如果您碰巧遇到Pierre Baldi和Laurent Itti(2010),“惊叹不已:应用引起关注的贝叶斯惊奇理论”,《神经网络》 23:649-666,您会发现等式73给出了两个伽玛pdf之间的KL差异。但是请注意,该公式看起来打印错误。
Clarinet先生2012年

我要寻找一个解决同样的问题,觉得这一个是有用的。
易阳

Answers:


15

KL散度是形式积分的差

$$ \ eqalign {I(a,b,c,d)&= \ int_0 ^ {\ infty} \ log \ left(\ frac {e ^ {-x / a} x ^ {b-1}} {a ^ b \ Gamma(b)} \ right)\ frac {e ^ {-x / c} x ^ {d-1}} {c ^ d \ Gamma(d)} dx \

&=-\ frac {1} {a} \ int_0 ^ \ infty \ frac {x ^ de ^ {-x / c}} {c ^ d \ Gamma(d)} \,dx-\ log(a ^ b \ Gamma(b))\ int_0 ^ \ infty \ frac {e ^ {-x / c} x ^ {d-1}} {c ^ d \ Gamma(d)} \,dx \&\ quad +(b- 1)\ int_0 ^ \ infty \ log(x)\ frac {e ^ {-x / c} x ^ {d-1}} {c ^ d \ Gamma(d)} \,dx \

&=-\ frac {cd} {a}-\ log(a ^ b \ Gamma(b))+(b-1)\ int_0 ^ \ infty \ log(x)\ frac {e ^ {-x / c } x ^ {d-1}} {c ^ d \ Gamma(d)} \,dx} $$

我们只需要处理通过观察得到的右手积分

dΓ(d)=d0ex/cxd1cddx=d0ex/c(x/c)d1cdx=0ex/cxd1cdlogxcdx=0log(x)ex/cxd1cddxlog(c)Γ(d).

何处

b1Γ(d)0log(x)ex/c(x/c)d1dx=(b1)Γ(d)Γ(d)+(b1)log(c).

插入先前的收益

I(a,b,c,d)=cdalog(abΓ(b))+(b1)Γ(d)Γ(d)+(b1)log(c).

Γ(c,d)Γ(a,b)I(c,d,c,d)I(a,b,c,d)


实施细节

Gamma函数快速增长,因此为避免溢出,请不要计算Gamma并取其对数:而是使用在任何统计计算平台(包括Excel)中都可以找到的log-Gamma函数。

Γ(d)/Γ(d)Γ,ψ,

RIψ

#
# `b` and `d` are Gamma shape parameters and
# `a` and `c` are scale parameters.
# (All, therefore, must be positive.)
#
KL.gamma <- function(a,b,c,d) {
  i <- function(a,b,c,d)
    - c * d / a - b * log(a) - lgamma(b) + (b-1)*(psigamma(d) + log(c))
  i(c,d,c,d) - i(a,b,c,d)
}
print(KL.gamma(1/114186.3, 202, 1/119237.3, 195), digits=12)

2
好答案。谢谢!我相信在第四等式中有一个符号错误。另外,您的gamma pdf在分母中应有一个额外的因子“ c”。您要我编辑吗?
伊恩·兰摩尔

@Ian你是对的;我通常将度量写为dX/X 通过不这样做,我省略了额外的因素 C. Good catch on the sign mistake. If you would like to make the edits, feel free!
whuber

2
I made the corrections.
Ian Langmore

10

The Gamma distribution is in the exponential family because its density can be expressed as:

f(xθ)=exp(η(θ)T(x)g(θ)+h(x))

Looking at the Gamma density function, its log-normalizer is

g(θ)=log(Γ(c))+clog(b)
with natural parameters
θ=[c11b]

All distributions in the exponential family have KL divergence:

KL(q;p)=g(θp)g(θq)(θpθq)g(θq).

There's a really nice proof of that in:

Frank Nielsen, École Polytechnique, and Richard Nock, Entropies and cross-entropies of exponential families.


Didn't know this. Just a quick question - the g(.) function, does it have to be the same for θp as for θq? So for example, would the above formula be valid for KL divergence of normal pdf from gamma pdf?
probabilityislogic

1
Yes, this formula is for two distributions in the same exponential family.
Neil G
By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.