EM算法概念

EM算法是一种非常经典的alternative optimizing算法。alternative optimizing的思想就是对于一个最优化问题，可以计算分为两步或者参数分为两个，就可以随机任意的选择一个起始值或位置，固定一个参数A，以另一个参数B进行优化，然后固定参数B，以参数A进行优化，直到收敛未知。前面博文中所讲述的K-means也就这样的一个过程，或者meanshift均值漂移也是这样的一个思想。今天学习的一个算法也是这样一个概念。这里依然做一个入门级的概念理解指导，不做原理性的深入，后续等用到时在进行深入学习。参考维基百科。

In statistics, an expectation–maximization (EM) algorithm is an iterative method for finding maximum likelihood or maximum a posteriori (MAP) estimates of parameters in statistical models, where the model depends on unobserved latent variables. The EM iteration alternates between performing an expectation (E) step, which creates a function for the expectation of the log-likelihood evaluated using the current estimate for the parameters, and a maximization (M) step, which computes parameters maximizing the expected log-likelihood found on the E step. These parameter-estimates are then used to determine the distribution of the latent variables in the next E step.

上面讲述了EM算法的轮廓，EM算法是一个用来寻找model参数的最大似然估计或最大后验估计的迭代方法，而这个model依赖于没有观测的潜在变量。所以，EM算法采用alternative optimizing的思想，首先执行expectation 步，利用当前的参数构建最大似然函数（log），然后执行maximization 步，生成新的参数，就这样不断地交替计算优化迭代下去，直到收敛为止。

The EM algorithm is used to find (locally) maximum likelihood parameters of a statistical model in cases where the equations cannot be solved directly. Typically these models involve latent variables in addition to unknown parameters and known data observations. That is, either there are missing values among the data, or the model can be formulated more simply by assuming the existence of additional unobserved data points. For example, a mixture model can be described more simply by assuming that each observed data point has a corresponding unobserved data point, or latent variable, specifying the mixture component that each data point belongs to.

Finding a maximum likelihood solution typically requires taking the derivatives of the likelihood function with respect to all the unknown values — viz. the parameters and the latent variables — and simultaneously solving the resulting equations. In statistical models with latent variables, this usually is not possible. Instead, the result is typically a set of interlocking equations in which the solution to the parameters requires the values of the latent variables and vice versa, but substituting one set of equations into the other produces an unsolvable equation.

The EM algorithm proceeds from the observation that the following is a way to solve these two sets of equations numerically. One can simply pick arbitrary values for one of the two sets of unknowns, use them to estimate the second set, then use these new values to find a better estimate of the first set, and then keep alternating between the two until the resulting values both converge to fixed points. It’s not obvious that this will work at all, but in fact it can be proven that in this particular context it does, and that the derivative of the likelihood is (arbitrarily close to) zero at that point, which in turn means that the point is either a maximum or a saddle point.[12] In general there may be multiple maxima, and there is no guarantee that the global maximum will be found. Some likelihoods also have singularities in them, i.e. nonsensical maxima. For example, one of the “solutions” that may be found by EM in a mixture model involves setting one of the components to have zero variance and the mean parameter for the same component to be equal to one of the data points.

EM算法通常用来找到无法直接解决的（局部的）最大似然参数。在最优化问题求解时，一个参数的解需要另一个潜在的变量值，而这个潜在变量的值则需要那一个参数的解，就这样交织在一起。这个时候最好的办法就是alternative
optimizing。这里对应的是EM算法。

EM算法可以任意的选择两个set中的一个，并给定任意的值，利用它们去估计第二个，然后用这个新的值去估计找到一个更好的第一个set的值，就这样交替的执行，直到结果收敛到某些固定的点为止。这个流程并不一定能很好地工作，但是在一些特别的上下文中可以证明它是可行的。只是很多时候可能是局部最优或者是到了马鞍点。

2015-8-28 艺少