My Note of Maximum Entropy

Note for Maximum Entropy

Notations

• $P$: model distr.
• $\tilde{P}$: empirical/sample distr. (Dirac distr.)
• $\{x_i\}$: sample
• $f_j$: features
• $E_{P}f$: expectation under the distr. $P$

Maximum Entropy

Def. Max Entropy(ME)
$$\begin{gathered} \underset{P\in \mathcal{P}}{\max }H(X) \\ {E}_P(f)=E_{\tilde{P}}(f)(\star ) \end{gathered}$$
where $f(x)$ are features.

Fact. $P_w(x)\sim e^{{\sum }_j{w}_j{f}_j(x)}$ is the solution to ${\inf }_{P}L(P,w)$, where $L$ is the Laplacian of ($\star$), $w$ is the Lagrange multiplier.

Laplacian function: the likelihood of the sample,
$$\begin{gathered} \Psi (w):=L(P_w,w) \\ =-\ln Z(w)+E_{\tilde{P}}(f)w \\ =\sum _i\ln P_w(x_i) \end{gathered}$$

Dual problem: Max. likelihood estimation(MLE)
$$\underset{w}{\max }\Psi (w)$$
where $\Psi (w):={\sum }_i\ln p(x_i)$.

Fact. the dual of ME($\star$) is MLE($\star \star$).

ME for Machine learning/conditional likelihood

Assume that $P(Y|X)$ is a determinative model.

Def. Max Entropy for $P(Y|X)$
$$\begin{gathered} \underset{P\in \mathcal{P}}{\max }H(Y|X) \\ {E}_P(f)=E_{\tilde{P}}(f) \end{gathered}$$
where $f(x,y)$ are features, and $P(y|x)=\tilde{P}(x)P(y|x)$

Fact. $P_w(y|x)\sim e^{{\sum }_j{w}_j{f}_j(x,y)}$ is the solution to max $L(P,w)$.

Laplacian function: $\Psi (w):=L(P_w,w)={\sum }_i\ln P_w(y_i|x_i)$, the conditional likelihood.

Dual problem (conditional MLE):
$$\underset{w}{\max }\Psi (w)$$
where $\Psi (w):={\sum }_i\ln p(y_i|x_i)$.

---

Exercise
plz consider ME for the generative model $P(X,Y)$