关于单变量高斯分布的一些数学推导

对 Christopher M. Bishop 大神的 《Pattern Recognition and Machine Learning》书中第一章相关单变量高斯分布的一些公式进行了详细推导。涉及到书中的公式，其编号与原书一致。

单变量高斯分布定义为：

$$\mathcal{N}(x|\mu ,{\sigma }^2)=\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}\exp \left\{-\frac{1}{2{\sigma }^2}(x-\mu {)}^2\right\}\text{\hspace{1em}(1.46)}$$

1. 满足概率密度函数的标准化条件的推导,即推导式(1.48) :

$${\int }_{-\infty }^{\infty }\mathcal{N}(x|\mu ,{\sigma }^2)\mathrm{d}x=1\text{\hspace{1em}(1.48)}$$

$$\begin{array}{rl}{\int }_{-\infty }^{\infty }\mathcal{N}(x|\mu ,{\sigma }^2)\mathrm{d}x& ={\int }_{-\infty }^{\infty }\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}\exp \left\{-\frac{1}{2{\sigma }^2}(x-\mu {)}^2\right\}\mathrm{d}x\\ & =\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}{\int }_{-\infty }^{\infty }\exp \left\{-\frac{1}{2{\sigma }^2}(x-\mu {)}^2\right\}\mathrm{d}x\\ & =\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}{\int }_{-\infty }^{\infty }\exp \left\{-\frac{1}{2}(\frac{x-\mu }{\sigma }{)}^2\right\}\mathrm{d}x\\ & =\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}\sigma {\int }_{-\infty }^{\infty }\exp \left\{-\frac{1}{2}(\frac{x-\mu }{\sigma }{)}^2\right\}\mathrm{d}\frac{x-\mu }{\sigma }\\ & =\frac{1}{(2\pi {)}^{1/2}}{\int }_{-\infty }^{\infty }\exp \left(-\frac{1}{2}{t}^2\right)\mathrm{d}t\text{这里使用t=σx−μ进行代换 }\\ & =\frac{1}{(2\pi {)}^{1/2}}\sqrt{2}{\int }_{-\infty }^{\infty }\exp \left[-{\left(\frac{t}{\sqrt{2}}\right)}^2\right]\mathrm{d}\frac{t}{\sqrt{2}}\\ & =\frac{1}{\sqrt{\pi }}{\int }_{-\infty }^{\infty }\exp (-z^2)\mathrm{d}z\\ & =1\end{array}$$

注：关于 ${\int }_{-\infty }^{\infty }{e}^{-x^2}\mathrm{d}x=\sqrt{\pi }$ 的证明

令

$$I={\int }_{-\infty }^{\infty }\exp (-x^2)\mathrm{d}x$$

则有

$$\begin{array}{rl}{I}^2& ={\int }_{-\infty }^{\infty }\exp (-x^2)\mathrm{d}x{\int }_{-\infty }^{\infty }\exp (-y^2)\mathrm{d}y\\ & ={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }\exp (-x^2-y^2)\mathrm{d}x\mathrm{d}y\end{array}$$

又令 $x=r\cos \theta$ ， $y=r\sin \theta$ ，则有

$$\begin{array}{rl}{I}^2& ={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }\exp (-x^2-y^2)\mathrm{d}x\mathrm{d}y\\ & ={\int }_0^{2\pi }{\int }_0^{\infty }\exp (-r^2)r\mathrm{d}r\mathrm{d}\theta \\ & ={\int }_0^{2\pi }\mathrm{d}\theta {\int }_0^{\infty }\exp (-r^2)r\mathrm{d}r\\ & =-\pi {\int }_0^{\infty }\exp (-r^2)\mathrm{d}(-r^2)\\ & =-\pi {\left[\exp (-r^2)\right]}_0^{\infty }\\ & =\pi \end{array}$$

故有

$$I={\int }_{-\infty }^{\infty }\exp (-x^2)\mathrm{d}x=\sqrt{\pi }$$

2. 均值的推导

$$\begin{array}{rl}\mathbb{E}[x]& ={\int }_{-\infty }^{\infty }\mathcal{N}(x|\mu ,{\sigma }^2)x\mathrm{d}x\\ & =\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}{\int }_{-\infty }^{\infty }x\exp \left\{-\frac{1}{2{\sigma }^2}(x-\mu {)}^2\right\}\mathrm{d}x\\ & =\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}{\int }_{-\infty }^{\infty }(x-\mu )\exp \left\{-\frac{1}{2{\sigma }^2}(x-\mu {)}^2\right\}+\mu \exp \left\{-\frac{1}{2{\sigma }^2}(x-\mu {)}^2\right\}\mathrm{d}(x-\mu )\\ & =\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}{\int }_{-\infty }^{\infty }z\exp \left\{-\frac{1}{2{\sigma }^2}{z}^2\right\}+\mu \exp \left\{-\frac{1}{2{\sigma }^2}{z}^2\right\}\mathrm{d}z\text{这里使用z=x−μ进行代换}\\ & =\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}{\int }_{-\infty }^{\infty }z\exp \left\{-\frac{1}{2{\sigma }^2}{z}^2\right\}\mathrm{d}z+\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}{\int }_{-\infty }^{\infty }\mu \exp \left\{-\frac{1}{2{\sigma }^2}{z}^2\right\}\mathrm{d}z\text{第一项为奇函数，在对称区间上的定积分为0}\\ & =\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}{\int }_{-\infty }^{\infty }\mu \exp \left\{-\frac{1}{2{\sigma }^2}{z}^2\right\}\mathrm{d}z\\ & =\mu {\int }_{-\infty }^{\infty }\frac{1}{(2\pi {\sigma }^2{)}^{1/2}}\exp \left\{-\frac{1}{2{\sigma }^2}{z}^2\right\}\mathrm{d}z\\ & =\mu {\int }_{-\infty }^{\infty }\mathcal{N}(z|0,{\sigma }^2)\mathrm{d}z\\ & =\mu \end{array}$$

3. 方差的推导

为书写方便，令 ${\sigma }^2=t$， $(1.48)$ 式左边对 $t$ 求微分

$$\begin{array}{rl}\frac{\partial }{\partial t}{\int }_{-\infty }^{\infty }\mathcal{N}(x|\mu ,t)\mathrm{d}x& ={\int }_{-\infty }^{\infty }\frac{\partial }{\partial t}\mathcal{N}(x|\mu ,t)\mathrm{d}x\\ & ={\int }_{-\infty }^{\infty }\frac{\partial }{\partial t}\left(\frac{1}{(2\pi t{)}^{1/2}}\exp \left\{-\frac{1}{2t}(x-\mu {)}^2\right\}\right)\mathrm{d}x\\ & =\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }\frac{\partial }{\partial t}\left(t^{-1/2}\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\right)\mathrm{d}x\end{array}$$

因为

$$\begin{array}{rl}\frac{\partial }{\partial t}\left(t^{-1/2}\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\right)& =\frac{\partial t^{-1/2}}{\partial t}\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}+t^{-1/2}\frac{\partial }{\partial t}\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\\ & =-\frac{1}{2}{t}^{-3/2}\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}+\frac{(x-\mu {)}^2}{2}{t}^{-5/2}\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\\ & =\frac{1}{2}{t}^{-3/2}\left[(x-\mu {)}^2{t}^{-1}-1\right]\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\end{array}$$

于是，有

$$\begin{array}{rl}\frac{\partial }{\partial t}{\int }_{-\infty }^{\infty }\mathcal{N}(x|\mu ,t)\mathrm{d}x& =\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }\frac{1}{2}{t}^{-3/2}\left[(x-\mu {)}^2{t}^{-1}-1\right]\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\mathrm{d}x\\ & =\frac{1}{\sqrt{2\pi }}\frac{1}{2}{t}^{-3/2}{\int }_{-\infty }^{\infty }\left[(x-\mu {)}^2{t}^{-1}-1\right]\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\mathrm{d}x\end{array}$$

因为 $(1.48)$ 式右边的微分等于零，即

$$\frac{\partial }{\partial t}{\int }_{-\infty }^{\infty }\mathcal{N}(x|\mu ,t)\mathrm{d}x=\frac{1}{\sqrt{2\pi }}\frac{1}{2}{t}^{-3/2}{\int }_{-\infty }^{\infty }\left[(x-\mu {)}^2{t}^{-1}-1\right]\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\mathrm{d}x=0$$

即

$$\begin{array}{rl}{\int }_{-\infty }^{\infty }\left[(x-\mu {)}^2{t}^{-1}-1\right]\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\mathrm{d}x& =0\\ t^{-1}{\int }_{-\infty }^{\infty }(x-\mu {)}^2\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\mathrm{d}x& ={\int }_{-\infty }^{\infty }\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\mathrm{d}x\\ t^{-1}{\int }_{-\infty }^{\infty }(x-\mu {)}^2\frac{1}{\sqrt{2\pi t}}\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\mathrm{d}x& ={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi t}}\exp \left\{-\frac{t^{-1}}{2}(x-\mu {)}^2\right\}\mathrm{d}x\end{array}$$

将 $t$ 换回 ${\sigma }^2$ :

$$\begin{array}{rl}{\sigma }^{-2}{\int }_{-\infty }^{\infty }(x-\mu {)}^2\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left\{-\frac{1}{2{\sigma }^2}(x-\mu {)}^2\right\}\mathrm{d}x& ={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left\{-\frac{1}{2{\sigma }^2}(x-\mu {)}^2\right\}\mathrm{d}x\\ {\sigma }^{-2}{\int }_{-\infty }^{\infty }(x-\mu {)}^2\mathcal{N}(x|\mu ,{\sigma }^2)\mathrm{d}x& ={\int }_{-\infty }^{\infty }\mathcal{N}(x|\mu ,{\sigma }^2)\mathrm{d}x\\ {\sigma }^{-2}\mathbb{E}[(x-\mu {)}^2]& =1\text{左边项用了函数期望的定义，右边项用了概率密度函数的标准化条件}\\ \mathbb{E}[(x-\mu {)}^2]& ={\sigma }^2\end{array}$$

又

$$\begin{array}{rl}{\sigma }^2=\mathbb{E}[(x-\mu {)}^2]=\mathrm{v}\mathrm{a}\mathrm{r}[x]& =\mathbb{E}[x^2-2x\mu +{\mu }^2]\\ & =\mathbb{E}[x^2]-2\mu \mathbb{E}[x]+{\mu }^2\\ & =\mathbb{E}[x^2]-2{\mu }^2+{\mu }^2\\ & =\mathbb{E}[x^2]-{\mu }^2\end{array}$$

即

$$\mathbb{E}[x^2]={\sigma }^2+{\mu }^2\text{\hspace{1em}(1.50)}$$

最终得

$$\begin{array}{rl}\mathrm{v}\mathrm{a}\mathrm{r}[x]& =\mathbb{E}[x^2]-\mathbb{E}[x{]}^2\\ & ={\sigma }^2+{\mu }^2-{\mu }^2\\ & ={\sigma }^2\end{array}$$

注：关于 $\mathrm{v}\mathrm{a}\mathrm{r}[x]=\mathbb{E}[x^2]-\mathbb{E}[x{]}^2$ 的证明

$$\begin{array}{rl}\mathrm{v}\mathrm{a}\mathrm{r}[x]& =\mathbb{E}\left[(x-\mathbb{E}[x]{)}^2\right]\\ & =\mathbb{E}\left[x^2-2x\mathbb{E}[x]+\mathbb{E}[x{]}^2\right]\\ & =\mathbb{E}[x^2]-2\mathbb{E}[x{]}^2+\mathbb{E}[x{]}^2\text{期望算子是线性算子}\\ & =\mathbb{E}[x^2]-\mathbb{E}[x{]}^2\end{array}\text{\hspace{1em}(1.40)}$$

4. 均值和方差的最大似然估计推导

设数据集 $\mathcal{X}=\{x_1,x_2,\cdots ,x_N\}$ 中的样本是独立同分布的，服从 $\mathcal{N}(x|\mu ,{\sigma }^2)$ 。因此，有

$$p(\mathcal{X}|\mu ,{\sigma }^2)=\prod _{n=1}^N\mathcal{N}(x_n|\mu ,{\sigma }^2)=\frac{1}{(2\pi {\sigma }^2{)}^{N/2}}\exp \left\{\sum _{n=1}^N-\frac{1}{2{\sigma }^2}(x_n-\mu {)}^2\right\}\text{\hspace{1em}(1.53)}$$

两边取对数，得

$$\ln p(\mathcal{X}|\mu ,{\sigma }^2)=-\frac{N}{2}\ln 2\pi -\frac{N}{2}\ln {\sigma }^2-\frac{1}{2{\sigma }^2}\sum _{n=1}^N(x_n-\mu {)}^2\text{\hspace{1em}(1.54)}$$

令

$$\frac{\partial \ln p(\mathcal{X}|\mu ,{\sigma }^2)}{\partial \mu }=-\frac{1}{2{\sigma }^2}\sum _{n=1}^N\frac{\partial (x_n-\mu {)}^2}{\partial \mu }=\frac{1}{{\sigma }^2}\sum _{n=1}^N(x_n-\mu )=0$$

即

$$\sum _{n=1}^N(x_n-{\mu }_{ML})=\sum _{n=1}^N{x}_n-N{\mu }_{ML}=0$$

$${\mu }_{ML}=\frac{1}{N}\sum _{n=1}^N{x}_n\text{\hspace{1em}(1.55)}$$

令

$$\frac{\partial \ln p(\mathcal{X}|{\mu }_{ML},{\sigma }^2)}{\partial {\sigma }^2}=-\frac{N}{2{\sigma }^2}+\frac{1}{2{\sigma }^4}\sum _{n=1}^N(x_n-{\mu }_{ML}{)}^2=0$$

得

$${\sigma }_{ML}^2=\frac{1}{N}\sum _{n=1}^N(x_n-{\mu }_{ML}{)}^2\text{\hspace{1em}(1.56)}$$

注：均值的最大似然估计是无偏的，方差的最大似然估计是渐进无偏的

均值最大似然估计的无偏性
$$\mathbb{E}[{\mu }_{ML}]=\mathbb{E}[\frac{1}{N}\sum _{n=1}^N{x}_n]=\frac{1}{N}\sum _{n=1}^N\mathbb{E}[x_n]=\mu \text{\hspace{1em}(1.57)}$$

方差最大似然估计的渐进无偏性

$$\begin{array}{rl}\mathbb{E}[{\sigma }_{ML}^2]& =\mathbb{E}[\frac{1}{N}\sum _{n=1}^N(x_n-{\mu }_{ML}{)}^2]\\ & =\mathbb{E}[\frac{1}{N}\sum _{i=1}^N(x_i-\frac{1}{N}\sum _{j=1}^N{x}_j{)}^2]\\ & =\frac{1}{N}\sum _{i=1}^N\mathbb{E}\left[(x_i-\frac{1}{N}\sum _{j=1}^N{x}_j{)}^2\right]\\ & =\frac{1}{N}\sum _{i=1}^N\mathbb{E}\left[x_i^2-\frac{2}{N}{x}_i\sum _{j=1}^N{x}_j+\frac{1}{N^2}{\left(\sum _{j=1}^N{x}_j\right)}^2\right]\\ & =\frac{1}{N}\sum _{i=1}^N\left\{\mathbb{E}[x_i^2]-\frac{2}{N}\mathbb{E}\left[x_i\sum _{j=1}^N{x}_j\right]+\frac{1}{N^2}\mathbb{E}\left[{\left(\sum _{j=1}^N{x}_j\right)}^2\right]\right\}\\ & =\frac{1}{N}\sum _{i=1}^N\left\{({\sigma }^2+{\mu }^2)-\frac{2}{N}(N{\mu }^2+{\sigma }^2)+\frac{1}{N^2}(N^2{\mu }^2+N{\sigma }^2)\right\}\\ & =\frac{1}{N}\sum _{i=1}^N\left\{\frac{N-1}{N}{\sigma }^2\right\}\\ & =\frac{N-1}{N}{\sigma }^2\end{array}\text{\hspace{1em}(1.58)}$$

故

$$\underset{N\to \infty }{\lim }\mathbb{E}[{\sigma }_{ML}^2]=\underset{N\to \infty }{\lim }\left[\frac{N-1}{N}{\sigma }^2\right]={\sigma }^2$$

上面的推导中用到了两个结论：一是式$(1.50)$，二是独立随机变量的乘积的期望等于它们各自期望的乘积。

关于单变量高斯分布的推导就写这些吧，应该是把原书中相关公式的来龙去脉推清楚了。

若有什么地方写错了或者没写清楚，欢迎留言指正！

下一篇准备推导一下《Pattern Recognition and Machine Learning》第一章中关于曲线拟合的那些东西。