EM (Expectation–Maximization) Algorithm 思路分析及推导

往期文章链接目录

文章目录

    • 往期文章链接目录
    • Jensen’s inequality
    • Overview of Expectation–Maximization (EM) algorithm
    • Intuition: Why we need EM algorithm
      • Complete Case v.s. Incomplete Case
    • EM Algorithm Derivation (Using *MLE*)
      • summary
      • Coordinate Ascent/ descent - view EM from a different prospect
    • Convergence of EM algorithm
    • What's next?
    • 往期文章链接目录

Jensen’s inequality

EM (Expectation–Maximization) Algorithm 思路分析及推导_第1张图片
  • Theorem: Let f f f be a convex function, and let X X X be a random variable. Then:

E [ f ( X ) ] ≥ f ( E [ X ] ) E[f(X)] \geq f(E[X]) E[f(X)]f(E[X])

\quad Moreover, if f f f is strictly convex, then E [ f ( X ) ] = f ( E [ X ] ) E[f(X)] = f(E[X]) E[f(X)]=f(E[X]) holds true if and
only if X X X is a constant.

  • Later in the post we are going to use the following fact from the Jensen’s inequality:
    Suppose λ j ≥ 0 \lambda_j \geq 0 λj0 for all j j j and ∑ j λ j = 1 \sum_j \lambda_j = 1 jλj=1, then

log ⁡ ∑ j λ j y j ≥ ∑ j λ j   l o g   y j \log \sum_j \lambda_j y_j \geq \sum_j \lambda_j \, log \, y_j logjλjyjjλjlogyj

\quad where the log ⁡ \log log function is concave.

Overview of Expectation–Maximization (EM) algorithm

In this post, let Y Y Y be a set of observed data, Z Z Z a set of unobserved latent data, and θ \theta θ the unknown parameters.

(After this post, you will be comfortable with the following description about the EM algorithm.)

Expectation–Maximization (EM) algorithm is an iterative method to find (local) maximum likelihood estimation (MLE) of L ( θ ) = p ( Y ∣ θ ) L(\theta) = p(Y|\theta) L(θ)=p(Yθ), where the model depends on unobserved latent variables Z Z Z.

Algorithm:

  1. Initialize peremeters θ 0 \theta_0 θ0.

Iterate between steps 2 and 3 until convergence:

  1. an expectation (E) step, which creates a function Q ( θ , θ i ) Q(\theta, \theta_i) Q(θ,θi) for the expectation of the log-likelihood log ⁡ p ( Y , Z ∣ θ ) \log p(Y,Z|\theta) logp(Y,Zθ) evaluated using the current conditional distribution of
    Z Z Z given Y Y Y and the current estimate of the parameters θ i \theta_i θi, where

Q ( θ , θ i ) = ∑ Z P ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Y , Z ∣ θ ) = E Z ∼ P ( Z ∣ Y , θ i ) [ log ⁡ p ( Y , Z ∣ θ ) ] \begin{aligned} Q(\theta, \theta_i) &= \sum_Z P(Z|Y,\theta_i) \cdot \log p(Y,Z|\theta) \\ &= E_{Z \sim P(Z|Y,\theta_i)}[\log p(Y,Z|\theta)] \end{aligned} Q(θ,θi)=ZP(ZY,θi)logp(Y,Zθ)=EZP(ZY,θi)[logp(Y,Zθ)]

  1. A maximization (M) step, which computes parameters maximizing the expected log-likelihood Q ( θ , θ i ) Q(\theta, \theta_i) Q(θ,θi) found on the E E E step and then update parameters to θ i + 1 \theta_{i+1} θi+1.

These parameter-estimates are then used to determine the distribution of the latent variables in the next E E E step. We say it converges if the increase in successive iterations is smaller than some tolerance parameter.

In general, multiple maxima may occur, with no guarantee that the global maximum will be found.

Intuition: Why we need EM algorithm

Sometimes maximizing the likelihood ℓ ( θ ) \ell(\theta) (θ) explicitly might be difficult since there are some unknown latent variables. In such a setting, the EM algorithm gives an efficient method for maximum likelihood estimation.

Complete Case v.s. Incomplete Case

Complete case

( Y , Z ) (Y, Z) (Y,Z) is observable, and the log likelihood can be written as

ℓ ( θ ) = log ⁡ p ( Y , Z ∣ θ ) = log ⁡ p ( Z ∣ θ ) ⋅ p ( Y ∣ Z , θ ) = log ⁡ p ( Z ∣ θ ) + log ⁡ p ( Y ∣ Z , θ ) \begin{aligned} \ell(\theta) &= \log p(Y, Z | \theta) \\ &= \log p(Z|\theta) \cdot p(Y|Z, \theta) \\ &= \log p(Z|\theta) + \log p(Y|Z, \theta) \\ \end{aligned} (θ)=logp(Y,Zθ)=logp(Zθ)p(YZ,θ)=logp(Zθ)+logp(YZ,θ)

We subdivide our task of maximizing ℓ ( θ ) \ell(\theta) (θ) into two sub-tasks. Note that in both log ⁡ p ( Z ∣ θ ) \log p(Z|\theta) logp(Zθ) and log ⁡ p ( Y ∣ Z , θ ) \log p(Y|Z, \theta) logp(YZ,θ), the only unknown parameter is θ \theta θ. They are just two standard MLE problems which could be easily solved by methods such as gradient descent.

Incomplete case

( Y ) (Y) (Y) is observable, but ( Z ) (Z) (Z) is unknown. We need to introduce the marginal distribution of variable Z Z Z:

ℓ ( θ ) = log ⁡ p ( Y ∣ θ ) = log ⁡ ∑ Z p ( Y , Z ∣ θ ) = log ⁡ ∑ Z p ( Z ∣ θ ) ⋅ p ( Y ∣ Z , θ ) \begin{aligned} \ell(\theta) &= \log p(Y | \theta) \\ &= \log \sum_Z p(Y, Z | \theta) \\ &= \log \sum_Z p(Z|\theta) \cdot p(Y|Z, \theta) \\ \end{aligned} (θ)=logp(Yθ)=logZp(Y,Zθ)=logZp(Zθ)p(YZ,θ)

Here we have a summation inside the log, so it’s hard to use optimization methods or take derivatives. This is the case where EM algorithm comes into aid.

EM Algorithm Derivation (Using MLE)

Given the observed data Y Y Y, we want to maximize the likelihood ℓ ( θ ) = p ( Y ∣ θ ) \ell(\theta) = p(Y|\theta) (θ)=p(Yθ), and it’s the same as maximizing the log-likelihood log ⁡ p ( Y ∣ θ ) \log p(Y|\theta) logp(Yθ). Therefore, from now on we will try to maximize the likelihood

ℓ ( θ ) = log ⁡ p ( Y ∣ θ ) \ell(\theta) = \log p(Y|\theta) (θ)=logp(Yθ)

by taking the unknown variable Z Z Z into account, we rewrite the objective function as

ℓ ( θ ) = log ⁡ p ( Y ∣ θ ) = log ⁡ ∑ Z p ( Y , Z ∣ θ ) = log ⁡ ∑ Z p ( Y ∣ Z , θ ) ⋅ p ( Z ∣ θ ) \begin{aligned} \ell(\theta) &= \log p(Y|\theta) \\ &= \log \sum_Z p(Y, Z | \theta) \\ &= \log \sum_Z p(Y|Z,\theta) \cdot p(Z|\theta) \\ \end{aligned} (θ)=logp(Yθ)=logZp(Y,Zθ)=logZp(YZ,θ)p(Zθ)

Note that in the last step, the log ⁡ \log log is outside of the ∑ \sum , which is hard to compute and optimize. Check out my previous post to know why we prefer to have log ⁡ \log log inside of ∑ \sum , instead of outside. So later we would find a way to approximate it (Jensen’s inequality).

Suppose we follow the iteration step 2 (E) and 3 (M) repeatedly, and have updated parameters to θ i \theta_i θi, then the difference between ℓ ( θ ) \ell(\theta) (θ) and our estimate ℓ ( θ i ) \ell(\theta_i) (θi) is ℓ ( θ ) − ℓ ( θ i ) \ell(\theta) - \ell(\theta_i) (θ)(θi). You can think of this difference as the improvement that later estimate of θ \theta θ tries to achieve. So our next step is to find θ i + 1 \theta_{i+1} θi+1 such that it improves the difference the most. That is, to make the difference ℓ ( θ ) − ℓ ( θ i ) \ell(\theta) - \ell(\theta_i) (θ)(θi) as large as possible. So we want our next estimate θ i + 1 \theta_{i+1} θi+1 to be

θ i + 1 = a r g   m a x θ    ℓ ( θ ) − ℓ ( θ i ) \theta_{i+1} = \mathop{\rm arg\,max}\limits_{\theta} \,\, \ell(\theta) - \ell(\theta_i) θi+1=θargmax(θ)(θi)

Note that we know the value of θ i \theta_i θi, so as ℓ ( θ i ) \ell(\theta_i) (θi). And

ℓ ( θ ) − ℓ ( θ i ) = log ⁡ p ( Y ∣ θ ) = log ⁡ ∑ Z p ( Y ∣ Z , θ ) ⋅ p ( Z ∣ θ ) − log ⁡ p ( Y ∣ θ i ) = log ⁡ ∑ Z P ( Z ∣ Y , θ i ) p ( Y ∣ Z , θ ) ⋅ p ( Z ∣ θ ) P ( Z ∣ Y , θ i ) − log ⁡ p ( Y ∣ θ i ) ( 1 ) \begin{aligned} \ell(\theta) - \ell(\theta_i)&= \log p(Y|\theta) \\ &= \log \sum_Z p(Y|Z,\theta) \cdot p(Z|\theta) - \log p(Y|\theta_i) \\ &= \log \sum_Z P(Z|Y,\theta_i) \frac{p(Y|Z,\theta) \cdot p(Z|\theta)}{P(Z|Y,\theta_i)} - \log p(Y|\theta_i) & & &(1)\\ \end{aligned} (θ)(θi)=logp(Yθ)=logZp(YZ,θ)p(Zθ)logp(Yθi)=logZP(ZY,θi)P(ZY,θi)p(YZ,θ)p(Zθ)logp(Yθi)(1)

Since P ( Z ∣ Y , θ i ) ≥ 0 P(Z|Y,\theta_i) \geq 0 P(ZY,θi)0 for all z ∈ Z z\in Z zZ and ∑ Z P ( Z ∣ Y , θ i ) = 1 \sum_Z P(Z|Y,\theta_i) = 1 ZP(ZY,θi)=1, we can use the Jensen’s inequality and then re-write ( 1 ) (1) (1) as

ℓ ( θ ) − ℓ ( θ i ) ≥ ∑ Z P ( Z ∣ Y , θ i ) log ⁡ p ( Y ∣ Z , θ ) ⋅ p ( Z ∣ θ ) P ( Z ∣ Y , θ i ) − ∑ Z P ( Z ∣ Y , θ i ) ) ⋅ log ⁡ p ( Y ∣ θ i ) = ∑ Z P ( Z ∣ Y , θ i ) ( log ⁡ p ( Y ∣ Z , θ ) ⋅ p ( Z ∣ θ ) P ( Z ∣ Y , θ i ) − log ⁡ p ( Y ∣ θ i ) ) ℓ ( θ ) ≥ ℓ ( θ i ) + ∑ Z P ( Z ∣ Y , θ i ) log ⁡ p ( Y ∣ Z , θ ) ⋅ p ( Z ∣ θ ) P ( Z ∣ Y , θ i ) ⋅ p ( Y ∣ θ i ) \begin{aligned} \ell(\theta) - \ell(\theta_i) &\geq \sum_Z P(Z|Y,\theta_i) \log \frac{p(Y|Z,\theta) \cdot p(Z|\theta)}{P(Z|Y,\theta_i)} - \sum_Z P(Z|Y,\theta_i)) \cdot \log p(Y|\theta_i)\\ &= \sum_Z P(Z|Y,\theta_i) \left( \log \frac{p(Y|Z,\theta) \cdot p(Z|\theta)}{P(Z|Y,\theta_i)} - \log p(Y|\theta_i) \right) \\ \ell(\theta) &\geq \ell(\theta_i) + \sum_Z P(Z|Y,\theta_i) \log \frac{p(Y|Z,\theta) \cdot p(Z|\theta)}{P(Z|Y,\theta_i) \cdot p(Y|\theta_i)} \end{aligned} (θ)(θi)(θ)ZP(ZY,θi)logP(ZY,θi)p(YZ,θ)p(Zθ)ZP(ZY,θi))logp(Yθi)=ZP(ZY,θi)(logP(ZY,θi)p(YZ,θ)p(Zθ)logp(Yθi))(θi)+ZP(ZY,θi)logP(ZY,θi)p(Yθi)p(YZ,θ)p(Zθ)

Now we define

B ( θ , θ i ) ≜ ℓ ( θ i ) + ∑ Z P ( Z ∣ Y , θ i ) log ⁡ p ( Y ∣ Z , θ ) ⋅ p ( Z ∣ θ ) P ( Z ∣ Y , θ i ) ⋅ p ( Y ∣ θ i ) B(\theta, \theta_i) \triangleq \ell(\theta_i) + \sum_Z P(Z|Y,\theta_i) \log \frac{p(Y|Z,\theta) \cdot p(Z|\theta)}{P(Z|Y,\theta_i) \cdot p(Y|\theta_i)} B(θ,θi)(θi)+ZP(ZY,θi)logP(ZY,θi)p(Yθi)p(YZ,θ)p(Zθ)

So we see that

ℓ ( θ ) ≥ B ( θ , θ i ) \ell(\theta) \geq B(\theta, \theta_i) (θ)B(θ,θi)

which implies that B ( θ , θ i ) B(\theta, \theta_i) B(θ,θi) is a lower bound of ℓ ( θ ) \ell(\theta) (θ) for all i i i. Therefore our next step is to maximize the lower bound B ( θ , θ i ) B(\theta, \theta_i) B(θ,θi) and make it as tight as possible. In the M M M step, we define

θ i + 1 = a r g   m a x θ B ( θ , θ i ) = a r g   m a x θ   ( ℓ ( θ i ) + ∑ Z P ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Y ∣ Z , θ ) ⋅ p ( Z ∣ θ ) P ( Z ∣ Y , θ i ) ⋅ p ( Y ∣ θ i ) ) ( 2 ) = a r g   m a x θ   ( ∑ Z P ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Y ∣ Z , θ ) ⋅ p ( Z ∣ θ ) P ( Z ∣ Y , θ i ) ⋅ p ( Y ∣ θ i ) ) ( 3 ) = a r g   m a x θ   ( ∑ Z P ( Z ∣ Y , θ i ) ⋅ [ log ⁡ p ( Y ∣ Z , θ ) ⋅ p ( Z ∣ θ ) − log ⁡ P ( Z ∣ Y , θ i ) ⋅ p ( Y ∣ θ i ) ] ) ( 4 ) = a r g   m a x θ   ( ∑ Z P ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Y ∣ Z , θ ) ⋅ p ( Z ∣ θ ) ) ( 5 ) = a r g   m a x θ   ( ∑ Z P ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Y , Z ∣ θ ) ) ( 6 ) = a r g   m a x θ   Q ( θ , θ i ) ( 7 ) \begin{aligned} \theta_{i+1} &= \mathop{\rm arg\,max}\limits_{\theta} B(\theta, \theta_i)\\ &= \mathop{\rm arg\,max}\limits_{\theta} \, \left(\ell(\theta_i) + \sum_Z P(Z|Y,\theta_i) \cdot \log \frac{p(Y|Z,\theta) \cdot p(Z|\theta)}{P(Z|Y,\theta_i) \cdot p(Y|\theta_i)} \right) && (2)\\ &= \mathop{\rm arg\,max}\limits_{\theta} \, \left(\sum_Z P(Z|Y,\theta_i) \cdot \log \frac{p(Y|Z,\theta) \cdot p(Z|\theta)}{P(Z|Y,\theta_i) \cdot p(Y|\theta_i)} \right) && (3)\\ &= \mathop{\rm arg\,max}\limits_{\theta} \, \left(\sum_Z P(Z|Y,\theta_i) \cdot [\log p(Y|Z,\theta) \cdot p(Z|\theta) - \log P(Z|Y,\theta_i) \cdot p(Y|\theta_i)] \right) && (4)\\ &= \mathop{\rm arg\,max}\limits_{\theta} \, \left(\sum_Z P(Z|Y,\theta_i) \cdot \log p(Y|Z,\theta) \cdot p(Z|\theta) \right) && (5)\\ &= \mathop{\rm arg\,max}\limits_{\theta} \, \left(\sum_Z P(Z|Y,\theta_i) \cdot \log p(Y,Z|\theta) \right) &&(6)\\ &= \mathop{\rm arg\,max}\limits_{\theta} \, Q(\theta, \theta_i) &&(7)\\ \end{aligned} θi+1=θargmaxB(θ,θi)=θargmax((θi)+ZP(ZY,θi)logP(ZY,θi)p(Yθi)p(YZ,θ)p(Zθ))=θargmax(ZP(ZY,θi)logP(ZY,θi)p(Yθi)p(YZ,θ)p(Zθ))=θargmax(ZP(ZY,θi)[logp(YZ,θ)p(Zθ)logP(ZY,θi)p(Yθi)])=θargmax(ZP(ZY,θi)logp(YZ,θ)p(Zθ))=θargmax(ZP(ZY,θi)logp(Y,Zθ))=θargmaxQ(θ,θi)(2)(3)(4)(5)(6)(7)

Remark:

  • ( 2 ) → ( 3 ) (2) \to (3) (2)(3) since ℓ ( θ i ) \ell(\theta_i) (θi) does not contain θ \theta θ.

  • ( 3 ) → ( 4 ) (3) \to (4) (3)(4) we used log ⁡ A B = log ⁡ A − log ⁡ B \log \frac{A}{B} = \log A - \log B logBA=logAlogB.

  • ( 4 ) → ( 5 ) (4) \to (5) (4)(5) since log ⁡ P ( Z ∣ Y , θ i ) ⋅ p ( Y ∣ θ i ) \log P(Z|Y,\theta_i) \cdot p(Y|\theta_i) logP(ZY,θi)p(Yθi) does not contain θ \theta θ.

  • ( 6 ) → ( 7 ) (6) \to (7) (6)(7) we define Q ( θ , θ i ) = ∑ Z P ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Y , Z ∣ θ ) Q(\theta, \theta_i) = \sum_Z P(Z|Y,\theta_i) \cdot \log p(Y,Z|\theta) Q(θ,θi)=ZP(ZY,θi)logp(Y,Zθ).

  • since both Y Y Y and θ i \theta_i θi are known, we have the distribution of Z ∼ p ( Z ∣ Y , θ i ) Z \sim p(Z|Y,\theta_i) Zp(ZY,θi). Therefore, the only unknown parameter is θ \theta θ, which means this is now a complete case I mentioned early in the post. So this is now a MLE problem.

summary

  1. θ i + 1 = a r g   m a x θ ℓ ( θ ) − ℓ ( θ i ) = a r g   m a x θ   Q ( θ , θ i ) \theta_{i+1} = \mathop{\rm arg\,max}\limits_{\theta} \ell(\theta) - \ell(\theta_i) = \mathop{\rm arg\,max}\limits_{\theta} \, Q(\theta, \theta_i) θi+1=θargmax(θ)(θi)=θargmaxQ(θ,θi). This implies that maximizing ℓ ( θ ) − ℓ ( θ i ) \ell(\theta) - \ell(\theta_i) (θ)(θi) is the same as maximizing Q ( θ , θ i ) Q(\theta, \theta_i) Q(θ,θi).

  2. Note that Q ( θ , θ i ) = ∑ Z P ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Y , Z ∣ θ ) Q(\theta, \theta_i) = \sum_Z P(Z|Y,\theta_i) \cdot \log p(Y,Z|\theta) Q(θ,θi)=ZP(ZY,θi)logp(Y,Zθ) is just the expectation of log ⁡ p ( Y , Z ∣ θ ) \log p(Y,Z|\theta) logp(Y,Zθ), where Z Z Z is drawn from the current conditional distribution P ( Z ∣ Y , θ i ) P(Z|Y,\theta_i) P(ZY,θi). Therefore we have
    Q ( θ , θ i ) = E Z ∼ p ( Z ∣ Y , θ i ) [ log ⁡ p ( Y , Z ∣ θ ) ] Q(\theta, \theta_i) = E_{Z \sim p(Z|Y,\theta_i)}[\log p(Y,Z|\theta)] Q(θ,θi)=EZp(ZY,θi)[logp(Y,Zθ)]
    That’s why it’s called the Expectation step. In E step, we are actually trying to calculate the expectation of the term log ⁡ p ( Y , Z ∣ θ ) \log p(Y,Z|\theta) logp(Y,Zθ), where unknown variable Z Z Z follows the current conditional distribution given by Y Y Y and θ i \theta_i θi. Then in the Maximazation step, we are trying to maximize this expectation. That’s why it’s called the M step.

Coordinate Ascent/ descent - view EM from a different prospect

EM (Expectation–Maximization) Algorithm 思路分析及推导_第2张图片

In the Expectation Step, we actually fixed θ i \theta_i θi, and tried to optimize Q ( θ , θ i ) Q(\theta, \theta_i) Q(θ,θi).

In the Maximization step, we actually fixed Q ( θ , θ i ) Q(\theta, \theta_i) Q(θ,θi), and tried to optimize θ \theta θ to get θ i + 1 \theta_{i+1} θi+1.

Every time we only optimize one variable and fix the rest. Therefore, from the graph above we see that in every iteration the gradient changes either vertically or horizontally.

To see how to perform the coordinate descent, check out my previous post.

Convergence of EM algorithm

By following the algorithm, we keep updating parameter θ i \theta_i θi and calculating approximated log-likelihood ℓ ( θ i ) \ell (\theta_i) (θi). But do we actually keep ℓ ( θ i ) \ell (\theta_i) (θi) getting closer to l ( θ ) l(\theta) l(θ) as we do more iterations? Keep in mind that in MLE our final goal is to maximize ℓ ( θ ) \ell (\theta) (θ).

Suppose θ i \theta_i θi and θ i + 1 \theta_{i+1} θi+1 are the parameters from two successive iterations of EM. We will now prove that ℓ ( θ i ) ≤ ℓ ( θ i + 1 ) \ell(\theta_i) \leq \ell(\theta_{i+1}) (θi)(θi+1), which shows EM always monotonically improves the log-likelihood.

ℓ ( θ ) = log ⁡ p ( Y ∣ θ ) = log ⁡ p ( Y , Z ∣ θ ) p ( Z ∣ Y , θ ) = log ⁡ p ( Y , Z ∣ θ ) − log ⁡ p ( Z ∣ Y , θ ) = ∑ Z p ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Y , Z ∣ θ ) − ∑ Z p ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Z ∣ Y , θ ) = Q ( θ , θ i ) + H ( θ , θ i ) ( 8 ) \begin{aligned} \ell(\theta) &= \log p(Y|\theta) \\ &= \log \frac {p(Y,Z|\theta)}{p(Z|Y, \theta)} \\ &= \log p(Y,Z|\theta) - \log p(Z|Y, \theta) \\ &= \sum_Z p(Z|Y, \theta_i)\cdot \log p(Y,Z|\theta) - \sum_Z p(Z|Y, \theta_i)\cdot \log p(Z|Y, \theta) \\ &= Q(\theta, \theta_i) + H(\theta, \theta_i) && (8)\\ \end{aligned} (θ)=logp(Yθ)=logp(ZY,θ)p(Y,Zθ)=logp(Y,Zθ)logp(ZY,θ)=Zp(ZY,θi)logp(Y,Zθ)Zp(ZY,θi)logp(ZY,θ)=Q(θ,θi)+H(θ,θi)(8)

where H ( θ , θ i ) = − ∑ Z p ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Z ∣ Y , θ ) H(\theta, \theta_i) = - \sum_Z p(Z|Y, \theta_i)\cdot \log p(Z|Y, \theta) H(θ,θi)=Zp(ZY,θi)logp(ZY,θ). This last equation ( 8 ) (8) (8) holds for every value of θ \theta θ, including θ = θ i \theta=\theta_i θ=θi, which means

ℓ ( θ i ) = Q ( θ i , θ i ) + H ( θ i , θ i ) \ell(\theta_i) = Q(\theta_i, \theta_i) + H(\theta_i, \theta_i) (θi)=Q(θi,θi)+H(θi,θi)

Therefore subtracting ℓ ( θ i ) \ell(\theta_i) (θi) from ℓ ( θ i + 1 ) \ell(\theta_{i+1}) (θi+1) gives

ℓ ( θ i + 1 ) − ℓ ( θ i ) = [ Q ( θ i + 1 , θ i ) + H ( θ i + 1 , θ i ) ] − [ Q ( θ i , θ i ) + H ( θ i , θ i ) ] = [ Q ( θ i + 1 , θ i ) − Q ( θ i , θ i ) ] + [ H ( θ i + 1 , θ i ) − H ( θ i , θ i ) ] \begin{aligned} \ell(\theta_{i+1}) - \ell(\theta_i) &= [Q(\theta_{i+1}, \theta_{i}) + H(\theta_{i+1}, \theta_{i})] - [Q(\theta_i, \theta_i) + H(\theta_i, \theta_i)]\\ &= [Q(\theta_{i+1}, \theta_{i}) - Q(\theta_i, \theta_{i})] + [H(\theta_{i+1}, \theta_{i}) - H(\theta_i, \theta_{i})] \\ \end{aligned} (θi+1)(θi)=[Q(θi+1,θi)+H(θi+1,θi)][Q(θi,θi)+H(θi,θi)]=[Q(θi+1,θi)Q(θi,θi)]+[H(θi+1,θi)H(θi,θi)]

Since θ i + 1 = a r g   m a x θ   Q ( θ , θ i ) \theta_{i+1} = \mathop{\rm arg\,max}\limits_{\theta} \, Q(\theta, \theta_i) θi+1=θargmaxQ(θ,θi), we have Q ( θ i + 1 , θ i ) ≥ Q ( θ i , θ i ) Q(\theta_{i+1}, \theta_i) \geq Q(\theta_i, \theta_i) Q(θi+1,θi)Q(θi,θi). The second parenthesis gives

H ( θ i + 1 , θ i ) − H ( θ i , θ i ) = − ∑ Z p ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Z ∣ Y , θ i + 1 ) + ∑ Z p ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Z ∣ Y , θ i ) = − ( ∑ Z p ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Z ∣ Y , θ i + 1 ) − ∑ Z p ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Z ∣ Y , θ i ) ) = − ( ∑ Z p ( Z ∣ Y , θ i ) ⋅ log ⁡ p ( Z ∣ Y , θ i + 1 ) p ( Z ∣ Y , θ i ) )     ( 9 ) ≥ − log ⁡ ( ∑ Z p ( Z ∣ Y , θ i ) ⋅ p ( Z ∣ Y , θ i + 1 ) p ( Z ∣ Y , θ i ) ) ( 10 ) = − log ⁡ ∑ Z p ( Z ∣ Y , θ i + 1 ) = − log ⁡ 1 = 0 \begin{aligned} H(\theta_{i+1}, \theta_{i}) - H(\theta_i, \theta_{i}) &= - \sum_Z p(Z|Y, \theta_i)\cdot \log p(Z|Y, \theta_{i+1}) + \sum_Z p(Z|Y, \theta_i)\cdot \log p(Z|Y, \theta_i) \\ &= - \left( \sum_Z p(Z|Y, \theta_i)\cdot \log p(Z|Y, \theta_{i+1}) - \sum_Z p(Z|Y, \theta_i)\cdot \log p(Z|Y, \theta_i)\right) \\ &= - \left( \sum_Z p(Z|Y, \theta_i) \cdot \log \frac{p(Z|Y, \theta_{i+1})}{p(Z|Y, \theta_{i})} \right) && \,\,\,(9)\\ &\geq - \log \left( \sum_Z p(Z|Y, \theta_i) \cdot \frac{p(Z|Y, \theta_{i+1})}{p(Z|Y, \theta_{i})} \right) && (10)\\ &= - \log \sum_Z p(Z|Y, \theta_{i+1}) = - \log 1 = 0 \end{aligned} H(θi+1,θi)H(θi,θi)=Zp(ZY,θi)logp(ZY,θi+1)+Zp(ZY,θi)logp(ZY,θi)=(Zp(ZY,θi)logp(ZY,θi+1)Zp(ZY,θi)logp(ZY,θi))=(Zp(ZY,θi)logp(ZY,θi)p(ZY,θi+1))log(Zp(ZY,θi)p(ZY,θi)p(ZY,θi+1))=logZp(ZY,θi+1)=log1=0(9)(10)

From ( 9 ) (9) (9) to ( 10 ) (10) (10) we use the Jensen’s inequality (note that there is a negative sign in the front so we reverse the inequality).

Since Q ( θ i + 1 , θ i ) ≥ Q ( θ i , θ i ) Q(\theta_{i+1}, \theta_{i}) \geq Q(\theta_i, \theta_{i}) Q(θi+1,θi)Q(θi,θi) and H ( θ i + 1 , θ i ) ≥ H ( θ i , θ i ) H(\theta_{i+1}, \theta_{i}) \geq H(\theta_i, \theta_{i}) H(θi+1,θi)H(θi,θi), we have

ℓ ( θ i + 1 ) ≥ ℓ ( θ i ) for all   i \ell(\theta_{i+1}) \geq \ell(\theta_i) \qquad\qquad \text{for all} \, i (θi+1)(θi)for alli

Since it’s monotonically increasing and bounded above, we say ℓ ( θ i ) \ell (\theta_i) (θi) converges.

What’s next?

I’m going to write a post to discuss how EM algorithm is applied in K-means and GMM in the future. Stay tuned!


Reference:

  • https://en.wikipedia.org/wiki/Expectation–maximization_algorithm
  • Part IX: The EM algorithm from CS229 Lecture notes by Andrew Ng http://cs229.stanford.edu/notes/cs229-notes8.pdf
  • http://guillefix.me/cosmos/static/Jensen%27s%2520inequality.html
  • https://en.wikipedia.org/wiki/Jensen%27s_inequality
  • http://www.adeveloperdiary.com/data-science/machine-learning/introduction-to-coordinate-descent-using-least-squares-regression/

往期文章链接目录

你可能感兴趣的:(机器学习,核心推导)