KL divergence between two univariate Gaussians

Question

I need to determine the KL-divergence between two Gaussians. I am comparing my results to these, but I can't reproduce their result. My result is obviously wrong, because the KL is not 0 for KL(p, p).

I wonder where I am doing a mistake and ask if anyone can spot it.

Let p(x)=N(μ1,σ1) and q(x)=N(μ2,σ2) . From Bishop's PRML I know that

K L (p, q) = - \int p (x) log q (x) d x + \int p (x) log p (x) d x

where integration is done over all real line, and that

\int p (x) log p (x) d x = 1 2 (1 + log 2 π σ 21),

so I restrict myself to ∫p(x)logq(x)dx , which I can write out as

- \int p (x) log 1 ( 2 π σ 2 2 ) ( 1 / 2 ) e - ( x - μ 2 ) 2 2 σ 2 2 d x,

which I can separate into

1 2 log (2 π σ 22) - \int p (x) log e - ( x - μ 2 ) 2 2 σ 2 2 d x .

Taking the log I get

1 2 log (2 π σ 22) - \int p (x) - ( x - μ 2 ) 2 2 σ 2 2 d x,

where I separate the sums and get σ22 out of the integral.

1 2 log (2 π σ 22) + \int p ( x ) x 2 d x - \int p ( x ) 2 x μ d x + \int p ( x ) μ 2 d x 2 σ 2 2

Letting ⟨⟩ denote the expectation operator under p , I can rewrite this as

1 2 log (2 π σ 22) + ⟨ x 2 ⟩ - 2 ⟨ x ⟩ μ 2 + μ 2 2 2 σ 2 2 .

We know that var(x)=⟨x2⟩−⟨x⟩2 . Thus

⟨ x 2 ⟩ = σ 21 + μ 21

and therefore

1 2 log (2 π σ 2) + σ 2 1 + μ 2 1 - 2 μ 1 μ 2 + μ 2 2 2 σ 2 2,

which I can put as

1 2 log (2 π σ 22) + σ 2 1 + ( μ 1 - μ 2 ) 2 2 σ 2 2 .

Putting everything together, I get to

K L (p, q) = - \int p (x) log q (x) d x + \int p (x) log p (x) d x = 1 2 log (2 π σ 22) + σ 2 1 + ( μ 1 - μ 2 ) 2 2 σ 2 2 - 1 2 (1 + log 2 π σ 21) = log σ 2 σ 1 + σ 2 1 + ( μ 1 - μ 2 ) 2 2 σ 2 2 .

Which is wrong since it equals

1 for two identical Gaussians.

Can anyone spot my error?

Update

Thanks to mpiktas for clearing things up. The correct answer is:

KL(p,q)=logσ2σ1+σ21+(μ1−μ2)22σ22−12

sorry for posting the incorrect answer in the first place. I just looked at x−μ1 and immediately thought that the integral is zero. The point that it was squared completely missed my mind :) — Feb 21 '11 at 12:02
I have just seen in a research paper that kld should be $KL(p, q) = ½ * ((μ₁-μ₂)² + σ₁²+σ₂²) * ( (1/σ₁²) + (1/σ₂²) ) - 2 — Aug 1 '13 at 14:26

score 6 · Accepted Answer · Feb 21 '11 at 17:58

up vote 6 down vote accepted

OK, my bad. The error is in the last equation:

K L (p, q) = - \int p (x) log q (x) d x + \int p (x) log p (x) d x = 1 2 log (2 π σ 22) + σ 2 1 + ( μ 1 - μ 2 ) 2 2 σ 2 2 - 1 2 (1 + log 2 π σ 21) = log σ 2 σ 1 + σ 2 1 + ( μ 1 - μ 2 ) 2 2 σ 2 2

−12 is missing in the last line. With it the last line becomes zero when μ1=μ2 and σ1=σ2 .

edited Feb 21 '11 at 17:58

answered Feb 21 '11 at 11:55

mpiktas
17.7k 3 28 75

The first line is trivially 0 because the two terms are equal with opposite sign. The "x" at the bottom of the integral signs has no meaning, either. Copy-and-paste errors? – whuber ♦ Feb 21 '11 at 17:39

@whuber, yes copy-paste. The same error is in the OP, will fix it. – mpiktas Feb 21 '11 at 17:58

It's much clearer now, thanks. – whuber ♦ Feb 21 '11 at 18:05

add comment

Community ♦ 1 · Answer 2 · Nov 19 '12 at 18:50

I did not have a look at your calculation but here is mine with a lot of details. Suppose p is the density of a normal random variable with mean μ1 and variance σ21 , and that q is the density of a normal random variable with mean μ2 and variance σ22 . The Kullback-Leibler distance from q to p is:

∫[log(p(x))−log(q(x))]p(x)dx

=∫[−12log(2π)−log(σ1)−12(x−μ1σ1)2+12log(2π)+log(σ2)+12(x−μ2σ2)2] ×12π√σ1exp[−12(x−μ1σ1)2]dx

=∫{log(σ2σ1)+12[(x−μ2σ2)2+(x−μ1σ1)2]} ×12π√σ1exp[−12(x−μ1σ1)2]dx

=E1{log(σ2σ1)+12[(x−μ2σ2)2+(x−μ1σ1)2]}

=log(σ2σ1)+12σ22E1{(X−μ2)2}−12σ21E1{(X−μ1)2}

=−12log(σ21σ22)+12σ22E1{(X−μ2)2}−12

(Now note that (X−μ2)2=(X−μ1+μ1−μ2)2=(X−μ1)2+2(X−μ1)(μ1−μ2)+(μ1−μ2)2 )

=−12log(σ21σ22)+12σ22[E1{(X−μ1)2}+2(μ1−μ2)E1{X−μ1}+(μ1−μ2)2]−12

=−12log(σ21σ22)+12σ21σ22+12σ22(μ1−μ2)2−12

=(μ1−μ2)22σ22+12(σ21σ22−1−logσ21σ22)

KL divergence between two univariate Gaussians

KL divergence between two univariate Gaussians

2 Answers

你可能感兴趣的:(KL divergence between two univariate Gaussians)