From EM to VAE

1. Expectaion Maximization

• Observed data: and unobserved data
• Goal
  
• For single observation
  Note: since are i r.v., we can consider the following first

• Evidence Lower Bound
  
  The last step is by Jensus Inequality( if is convex and equality holds when is constant.), and the equality above is achieved when
  
  Basically, the equality is achieved when the prior is equal to the posterior.

  We define Evidence Lower Bound as the following
  
  with

• For multiple observation
  With the help the notation, we have
  `