【系列二】数学基础—概率—高斯分布3

1. 边缘概率与条件概率

$$x_i\backsim N(\mu ,\Sigma )=\frac{1}{(2\pi {)}^{\frac{p}{2}}{\Sigma }^{\frac{1}{2}}}\exp (-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))$$$$x=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_p\end{array}\right),\mu =\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ⋮\\ {\mu }_p\end{array}\right),\Sigma =\left(\begin{array}{c}{\sigma }_{11},{\sigma }_{12}\dots ,{\sigma }_{1p}\\ {\sigma }_{21},{\sigma }_{22}\dots ,{\sigma }_{2p}\\ ⋮\\ {\sigma }_{p1},{\sigma }_{p2}\dots ,{\sigma }_{pp}\end{array}\right)$$
将$x,\mu ,\Sigma$分组，$x=\left(\begin{array}{c}{x}_a\\ x_b\end{array}\right),\mu =\left(\begin{array}{c}{\mu }_a\\ {\mu }_b\end{array}\right),\Sigma =\left(\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right),\{\begin{array}{l}a=1,2,\dots ,m\\ b=1,2,\dots ,n\\ m+n=p\end{array}$
已知一个高维高斯分布，求它的边缘概率分布$P(x_a),P(x_b)$和条件概率分布$P(x_a|x_b),P(x_b|x_a)$？

PRML这本书中介绍了“配方法”来求解。下面介绍一种稍微简单一点的方法。
首先引入一个定理(不证)：$$\begin{gathered} x\backsim N(\mu ,\Sigma ) \\ y=Ax+B \\ y\backsim N(A\mu +B,A\Sigma A^T) \end{gathered}$$
求边缘概率分布：
$x_a=\left(\begin{array}{cc}{I}_m& 0\end{array}\right)\left(\begin{array}{c}{x}_a\\ x_b\end{array}\right)=Ax$，有上面的定理$x_a\backsim N(A\mu ,A\Sigma A^T)=N({\mu }_a,{\Sigma }_{aa})$
求条件概率分布：
构造一个新的随机变量：
$$\begin{gathered} x_{b\cdot a}=x_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a=\left(\begin{array}{cc}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}& I_m\end{array}\right)\left(\begin{array}{c}{x}_a\\ x_b\end{array}\right)=Ax \\ E[x_{b\cdot a}]={\mu }_{b\cdot a}={\mu }_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\mu }_a \\ var[x_{b\cdot a}]=x_{bb\cdot a}={\Sigma }_{bb}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\Sigma }_{ab} \end{gathered}$$
$$x_{b\cdot a}\backsim N({\mu }_{b\cdot a},{\Sigma }_{bb\cdot a})$$
$x_b={\mu }_{b\cdot a}+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a$$$x_b|x_a\backsim N({\mu }_{b\cdot a}+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a,{\Sigma }_{bb\cdot a})$$
$x_{b.a}与x_a$的独立性证明具体步骤如下：（独立说明$P(x_b|x_a)=P(x_b)$)

Note：

1. 一般情况下两个随机变量之间独立一定不相关，不相关不一定独立（也就是独立的概念更“苛刻”一点，不相关稍微“弱”一点）
2. 如果两个随机变量均服从高斯分布，那么“不相关”等价于“独立”

2. 联合概率分布

已知：$p(x)=N(x|\mu ,{\Lambda }^{-1}),p(y|x)=N(y|Ax+b,L^{-1})$
求：$p(y),p(x|y)$
①：$$\begin{gathered} y=Ax+b+ϵ,ϵ\backsim N(0,L^{-1}),ϵ\perp x \\ E[y]=E(Ax+b+ϵ)=A\mu +b \\ var[y]=var[Ax+b]+var[ϵ]=A{\Lambda }^{-1}{A}^T+L^{-1} \\ y\backsim N(A\mu +b,A{\Lambda }^{-1}{A}^T+L^{-1}) \end{gathered}$$
②：$$\begin{gathered} z=\left(\begin{array}{c}x\\ y\end{array}\right)\backsim N\left(\begin{array}{cc}\left[\begin{array}{c}\mu \\ A\mu +b\end{array}\right]& \left[\begin{array}{cc}{\Lambda }^{-1}& \Delta \\ \Delta & A{\Lambda }^{-1}{A}^T+L^{-1}\end{array}\right]\end{array}\right) \\ \Delta =Cov(x,y)=E[(x-E[x])(y-E[y]{)}^T]= \\ E[(x-\mu )(Ax-A\mu +ϵ{)}^T]=E[(x-\mu )(Ax-A\mu {)}^T+(x-\mu ){ϵ}^T]= \\ E[(x-\mu )(Ax-A\mu {)}^T]+E[(x-\mu ){ϵ}^T]=E[(x-\mu )(x-\mu {)}^T{A}^T]= \\ var[x]A^T={\Lambda }^{-1}{A}^T \end{gathered}$$
一些基本概念

原视频链接

【机器学习】【白板推导系列】【合集 1～23】