机器学习-白板推导系列笔记（二十四）-直面配分函数

此文章主要是结合哔站shuhuai008大佬的白板推导视频：直面配分函数_134min

全部笔记的汇总贴：机器学习-白板推导系列笔记

直面配分函数-对应花书第十八章

动机：Learning问题、evaluation问题

一、对数似然梯度

$x\in {\mathbb{R}}^p,\{0,1{\}}^p$

许多概率图模型（通常无向图模型），由一个为归一化的$\hat{p}(x;\theta )$定义，我们必须除以一个配分函数$Z(\theta )$来归一化， $$p(x;\theta )=\frac{1}{Z(\theta )}\hat{p}(x;\theta )$$

配分函数$Z(\theta )$是对未归一化概率所有状态的积分或者求和：
$$Z(\theta )=\int \hat{p}(x)dxorZ(\theta )=\sum _x\hat{p}(x)$$

ML learning:Given $X=\{x_i{\}}_{i=1}^N,estimate:\theta$

$$\theta =\underset{\theta }{\mathrm{argmax}}p(x;\theta )=\underset{\theta }{\mathrm{argmax}}\prod _{i=1}^{N}p(x_i;\theta )$$
引入一个$\log$
$$\begin{gathered} \theta =\underset{\theta }{\mathrm{argmax}}\log \prod _{i=1}^{N}p(x_i;\theta ) \\ =\underset{\theta }{\mathrm{argmax}}\sum _{i=1}^N\log p(x_i;\theta ) \\ =\underset{\theta }{\mathrm{argmax}}\sum _{i=1}^N(\log \hat{p}(x_i;\theta )-\log Z(\theta )) \\ =\underset{\theta }{\mathrm{argmax}}\sum _{i=1}^N\log \hat{p}(x_i;\theta )-N\cdot \log Z(\theta ) \\ =\underset{\theta }{\mathrm{argmax}}\frac{1}{N}\sum _{i=1}^N\log \hat{p}(x_i;\theta )-\log Z(\theta ) \\ （提一个N对计算无影响） \end{gathered}$$
$$l(\theta )=\frac{1}{N}\sum _{i=1}^N\log \hat{p}(x_i;\theta )-\log Z(\theta )$$

求梯度，

$${\nabla }_{\theta }l(\theta )=\frac{1}{N}\sum _{i=1}^N{\nabla }_{\theta }\log \hat{p}(x_i;\theta )-{\nabla }_{\theta }\log Z(\theta )$$

$$\begin{gathered} {\nabla }_{\theta }\log Z(\theta )=\frac{1}{Z(\theta )}{\nabla }_{\theta }Z(\theta ) \\ =\frac{p(x;\theta )}{\hat{p}(x;\theta )}{\nabla }_{\theta }\int \hat{p}(x)dx \\ =\frac{p(x;\theta )}{\hat{p}(x;\theta )}\int {\nabla }_{\theta }\hat{p}(x)dx \\ =\int \frac{p(x;\theta )}{\hat{p}(x;\theta )}{\nabla }_{\theta }\hat{p}(x)dx \\ =\int p(x;\theta ){\nabla }_{\theta }\log \hat{p}(x)dx \\ =E_{p(x;\theta )}[{\nabla }_{\theta }\log \hat{p}(x)] \end{gathered}$$

二、随机最大似然

$$\begin{gathered} {\nabla }_{\theta }l(\theta )=\frac{1}{N}\sum _{i=1}^N{\nabla }_{\theta }\log \hat{p}(x_i;\theta )-E_{p(x;\theta )}[{\nabla }_{\theta }\log \hat{p}(x;\theta )] \\ =\underset{postivephase}{E_{P_{data}}[{\nabla }_{\theta }\log \hat{p}(x;\theta )]}-\underset{negativephase}{E_{P_{model}}[{\nabla }_{\theta }\log \hat{p}(x;\theta )]} \end{gathered}$$

$$\{\begin{array}{c}datadistrobution:P_{data}\\ modeldistribution:P_{model}\stackrel{\Delta }{=}P(x;\theta )\end{array}$$

Gradient Ascend:

$${\theta }^{(t+1)}={\theta }^{(t)}+\eta {\nabla }_{\theta }(l_{\theta }^{(t)})$$

Gibbs Sampling from $P_{model}=P(x;{\theta }^{(t)})，\{{\hat{x}}_i{\}}_{i=1}^m$
$$\begin{array}{c}{\hat{x}}_1\sim P(x;{\theta }^{(t)})\\ {\hat{x}}_2\sim P(x;{\theta }^{(t)})\\ ⋮\\ {\hat{x}}_m\sim P(x;{\theta }^{(t)})\end{array}\}\to fantacyparticles$$

所以，Gradient Ascend based on MCMC:

$${\theta }^{(t+1)}={\theta }^{(t)}+\eta (\sum _{i=1}^m{\nabla }_{\theta }\log \hat{p}(x_i;{\theta }^{(t)})-\sum _{i=1}^m{\nabla }_{\theta }\log \hat{p}({\hat{x}}_i;{\theta }^{(t)}))$$

三、对比散度

Gibbs采样时采用空间换时间，每一个需要$k-step$，这个k可能是极大的，所以我们需要采用对比散度的方法，使初始化马尔科夫链为采样自数据分布中的样本，可以大大降低混合时间（mixing time）。

CD-k（k=$1,2,\cdots$）：

$${\hat{x}}_i=x_i(x_i为trainingdata，是从P_{data}中采样得到的)$$
$\hat{\theta }=\underset{\theta }{\mathrm{argmin}}\left(\underset{ContrastiveDivergence}{KL(P^{(0)}|P^{(\infty )})-KL(P^{(k)}|P^{(\infty )})}\right)$

极大似然估计：
$$\begin{gathered} \hat{\theta }=\underset{\theta }{\mathrm{argmax}}\frac{1}{N}\sum _{i=1}^N\log p(x_i;\theta ) \\ =\underset{\theta }{\mathrm{argmax}}{E}_{P_{data}}[\log P_{model}(x;\theta )] \\ =\underset{\theta }{\mathrm{argmax}}\int P_{data}\log P_{model}(x;\theta )dx \\ \stackrel{P_{data}与\theta 无关}{=}\underset{\theta }{\mathrm{argmax}}\int P_{data}\log \frac{P_{model}(x;\theta )}{P_{data}}dx \\ =\underset{\theta }{\mathrm{argmax}}-KL(P_{data}|P_{model}(x;\theta )) \\ =\underset{\theta }{\mathrm{argmin}}KL(P_{data}|P_{model}(x;\theta )) \\ =\underset{\theta }{\mathrm{argmin}}KL(P^{(0)}|P^{(\infty )}) \end{gathered}$$

四、RBM的Learning问题

$$x=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_p\end{array}\right)=\left(\begin{array}{c}h\\ v\end{array}\right)h=\left(\begin{array}{c}{h}_1\\ h_2\\ ⋮\\ h_m\end{array}\right)v=\left(\begin{array}{c}{v}_1\\ v_2\\ ⋮\\ v_n\end{array}\right)p=m+n$$
$$W=[w_{ij}{]}_{m\ast n}\alpha =\left(\begin{array}{c}{\alpha }_1\\ {\alpha }_2\\ ⋮\\ {\alpha }_n\end{array}\right)\beta =\left(\begin{array}{c}{\beta }_1\\ {\beta }_2\\ ⋮\\ {\beta }_m\end{array}\right)$$
$$\{\begin{array}{c}P(x)=\frac{1}{Z}\exp \{-E(h,v)\}\\ \\ E(h,v)=-(h^{T}Wv+{\alpha }^{T}v+{\beta }^{T}h)\end{array}$$

log-likelihood:

training set $V\in S,|S|=N$
$$\frac{1}{N}\sum _{V\in S}\log P(V)$$

log-likelihood gradient:

$$\frac{\partial }{\partial \theta }\frac{1}{N}\sum _{V\in S}\log P(V)$$

所以，

$$\begin{gathered} \log P(V)=\log \sum _{h}P(h,v) \\ =\log \sum _h\frac{1}{Z}\exp \{-E(h,v)\} \\ =\underset{①}{\log \sum _h\exp \{-E(h,v)\}}-\underset{②}{\log \sum _{h,v}\exp \{-E(h,v)\}} \end{gathered}$$
$$\frac{\partial }{\partial \theta }\log P(V)=\frac{\partial }{\partial \theta }①-\frac{\partial }{\partial \theta }②$$
$$\begin{gathered} \frac{\partial }{\partial \theta }①=\frac{\partial }{\partial \theta }\log \sum _h\exp \{-E(h,v)\} \\ =-\frac{1}{{\sum }_h\exp \{-E(h,v)\}}\sum _h\exp \{-E(h,v)\}\cdot \frac{\partial E(h,v)}{\partial \theta } \\ =-\sum _h\frac{\exp \{-E(h,v)\}}{{\sum }_h\exp \{-E(h,v)\}}\cdot \frac{\partial E(h,v)}{\partial \theta } \\ =-\sum _h\frac{\frac{1}{Z}\exp \{-E(h,v)\}}{\frac{1}{Z}{\sum }_h\exp \{-E(h,v)\}}\cdot \frac{\partial E(h,v)}{\partial \theta } \\ =-\sum _{h}P(h|v)\cdot \frac{\partial E(h,v)}{\partial \theta } \end{gathered}$$
$$\begin{gathered} \frac{\partial }{\partial \theta }②=\frac{\partial }{\partial \theta }\log \sum _{h,v}\exp \{-E(h,v)\} \\ =-\frac{1}{{\sum }_{h,v}\exp \{-E(h,v)\}}\sum _{h,v}\exp \{-E(h,v)\}\cdot \frac{\partial E(h,v)}{\partial \theta } \\ =-\sum _{h,v}\frac{\exp \{-E(h,v)\}}{{\sum }_{h,v}\exp \{-E(h,v)\}}\cdot \frac{\partial E(h,v)}{\partial \theta } \\ =-\sum _{h,v}\frac{\frac{1}{Z}\exp \{-E(h,v)\}}{\frac{1}{Z}{\sum }_{h,v}\exp \{-E(h,v)\}}\cdot \frac{\partial E(h,v)}{\partial \theta } \\ =-\sum _{h,v}P(h,v)\cdot \frac{\partial E(h,v)}{\partial \theta } \end{gathered}$$

所以，

$$\begin{gathered} \frac{\partial }{\partial \theta }\log P(V)=\frac{\partial }{\partial \theta }①-\frac{\partial }{\partial \theta }② \\ =\sum _{h,v}P(h,v)\cdot \frac{\partial E(h,v)}{\partial \theta }-\sum _{h}P(h|v)\cdot \frac{\partial E(h,v)}{\partial \theta } \\ ⟺\frac{\partial }{\partial w_{ij}}\log P(V)=-\sum _{h}P(h|v)\cdot \frac{\partial E(h,v)}{\partial w_{ij}}+\sum _{h,v}P(h,v)\cdot \frac{\partial E(h,v)}{\partial w_{ij}} \end{gathered}$$

因为：

$$E(h,v)=-(h^{T}Wv+\Delta )=-(\sum _{i=1}^m\sum _{j=1}^n{h}_i{w}_{ij}{v}_j+\Delta )$$

所以：

$$\begin{gathered} \frac{\partial }{\partial w_{ij}}\log P(V)=-\sum _{h}P(h|v)\cdot (-h_i{v}_j)+\sum _{h,v}P(h,v)\cdot (-h_i{v}_j) \\ =\underset{①}{\sum _{h}P(h|v)h_i{v}_j}-\underset{②}{\sum _{h,v}P(h,v)h_i{v}_j} \end{gathered}$$
$$\begin{gathered} ①=\sum _{h_1}\sum _{h_2}\cdots \sum _{h_i}\cdots \sum _{h_m}P(h_1,h_2,\cdots ,h_i,\cdots ,h_m|v)\cdot h_i{v}_j \\ =\sum _{h_i}P(h_i|v)\cdot h_i{v}_j \\ =P(h_i=1|v)\cdot v_j \end{gathered}$$
$$\begin{gathered} ②=\sum _h\sum _{v}P(v)\cdot P(h|v)\cdot h_i{v}_j \\ =\sum _{v}P(v)\sum _{h}P(h|v)\cdot h_i{v}_j \\ =\sum _{v}P(v)\cdot P(h_i=1|v)\cdot v_j \end{gathered}$$

即，

$$\frac{\partial }{\partial w_{ij}}\log P(V)=P(h_i=1|v)\cdot v_j-\sum _{v}P(v)\cdot P(h_i=1|v)\cdot v_j$$
$$\frac{\partial }{\partial w_{ij}}\frac{1}{N}\sum _{V\in S}\log P(V)=\frac{1}{N}\sum _{V\in S}\frac{\partial }{\partial w_{ij}}\log P(V)$$

CD-k for RBM:

For each $v\in S:$
     \;\; $v^{(0)}\leftarrow v$
     \;\; For $l=0,1,2,\cdots ,k-1:$$(kstepsblockGibbsSampling)$
     \;\;      \;\;      \;\; For $i=1,2,\cdots ,m:$
     \;\;      \;\;      \;\;      \;\;      \;\; sample $h_i^{(l)}\sim P(h_i|v^{(l)})$
     \;\;      \;\;      \;\; For $j=1,2,\cdots ,n:$
     \;\;      \;\;      \;\;      \;\;      \;\; sample $v_j^{(l+1)}\sim P(v_j|h^{(l)})$
For $i=1,2,\cdots ,m,j=1,2,\cdots ,n:$
     \;\; $\Delta w_{ij}\leftarrow \Delta w_{ij}+\frac{\partial }{\partial w_{ij}}\log P(V)$
     \;\; $\frac{\partial }{\partial w_{ij}}\log P(V)\approx P(h_i=1|v^{(0)})\cdot v_j^{(0)}-P(h_i=1|v^{(k)})\cdot v_J^{(k)}$

所以，

$$\frac{\partial }{\partial w_{ij}}\frac{1}{N}\sum _{V\in S}\log P(V)=\frac{1}{N}\sum _{V\in S}\frac{\partial }{\partial w_{ij}}\log P(V)\approx \frac{1}{N}\Delta w_{ij}$$

下一章传送门：白板推导系列笔记（二十五）-近似推断