【系列二】数学基础—概率—高斯分布1

1. 一维高斯分布参数$\mu ,\sigma$的极大似然估计

datas: $X=(x_1,x_2,\dots ,x_n{)}_{n\times p}^T=\left(\begin{array}{c}{x}_{11},x_{12}\dots ,x_{1p}\\ x_{21},x_{22}\dots ,x_{2p}\\ ⋮\\ x_{n1},x_{n2}\dots ,x_{np}\end{array}\right)$
$x_i\stackrel{iid}{\backsim }N(\mu ,\Sigma )$，
iid(Independent and identically distributed，独立同分布，之后没特殊说明。均为独立同分布)
$$\theta =(\mu ,\Sigma ),p(x)=\frac{1}{(2\pi {)}^{\frac{p}{2}}{\Sigma }^{\frac{1}{2}}}\exp (-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))$$
令p=1(p为$x_i$的维度)，则$$\theta =(\mu ,{\sigma }^2),p(x)=\frac{1}{\sqrt{2\pi }\sigma }\exp (-\frac{(x-\mu {)}^2}{2{\sigma }^2})$$即高维高斯分布退化为一维高斯分布。
极大似然估计：$$\begin{gathered} {\theta }_{MLE}=arg\underset{\theta }{\max }P(X|\theta ) \\ \log P(X|\theta )=\log \prod _{i=1}^{n}p(x_i|\theta )=\sum _{i=1}^n\log P(x_i|\theta ) \\ =\sum _{i=1}^n\log \frac{1}{\sqrt{2\pi }\sigma }\exp (-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}) \\ =\sum _{i=1}^n[\log \frac{1}{\sqrt{2\pi }}+\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}] \end{gathered}$$
由于是极大似然估计，所以常数项可以约去，同时对$\mu$的最大似然估计与$\sigma$无关（即视为常数项，其他同理），化简得$$\begin{gathered} {\mu }_{MLE}=arg\underset{\mu }{\max }P(X|\theta ) \\ =arg\underset{\mu }{\min }\sum _{i=1}^n(x_i-\mu {)}^2 \end{gathered}$$
令偏导数为0，即$$\begin{gathered} \frac{\partial }{\partial \mu }\sum _{i=1}^n(x_i-\mu {)}^2=\sum _{i=1}^{n}2(x_i-\mu )(-1)=0 \\ \sum _{i=1}^n(x_i-\mu )=0 \\ \sum _{i=1}^n{x}_i-\sum _{i=1}^n\mu =0 \\ {\mu }_{MLE}=\frac{1}{n}\sum _{i=1}^n{x}_i \end{gathered}$$
同理：$$\begin{gathered} {\sigma }_{MLE}^2=arg\underset{\sigma }{\max }P(X|\theta ) \\ =arg\underset{\sigma }{\max }(\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2})=arg\underset{\sigma }{\max }\ell (x) \end{gathered}$$
令：$$\begin{gathered} \frac{\partial \ell }{\partial \sigma }=\sum _{i=1}^n(-\frac{1}{\sigma }+(x_i-\mu {)}^2{\sigma }^{-3}=0 \\ \sum _{i=1}^n(x_i-\mu {)}^2=\sum _{i=1}^n{\sigma }^2 \\ {\sigma }_{MLE}^2=\frac{1}{n}\sum _{i=1}^n(x_i-\mu {)}^2 \end{gathered}$$

2. 无偏估计与有偏估计

上节解出来的${\mu }_{MLE}$是无偏估计量，而${\sigma }_{MLE}^2$是有偏估计量。
一个总体如果服从高斯分布，那么必客观存在参数$\mu ,\sigma$。通过对样本的观察，采样极大似然的方法，对总体的参数进行估计，如果估计量的期望等于被估计的参数，则称此估计量为无偏估计，否则为有偏估计。
$$\begin{gathered} E[{\mu }_{MLE}]=E[\frac{1}{n}\sum _{i=1}^n{x}_i]=\frac{1}{n}\sum _{i=1}^{n}E[x_i]=\frac{1}{n}\sum _{i=1}^n\mu =\mu \\ {\sigma }_{MLE}^2=\frac{1}{n}\sum _{i=1}^n(x_i-{\mu }_{MLE}{)}^2 \\ =\frac{1}{n}\sum _{i=1}^n(x_i^2-2x_i{\mu }_{MLE}+{\mu }_{MLE}^2)=\frac{1}{n}\sum _{i=1}^n{x}_i^2-2{\mu }_{MLE}^2+{\mu }_{MLE}^2 \end{gathered}$$$$\begin{gathered} E[{\sigma }_{MLE}^2]=E[(\frac{1}{n}\sum _{i=1}^n{x}_i^2-{\mu }^2)-({\mu }_{MLE}^2-{\mu }^2)] \\ =E[\frac{1}{n}\sum _{i=1}^n(x_i^2-{\mu }^2)]-E[{\mu }_{MLE}^2-{\mu }^2] \\ =\frac{1}{n}\sum _{i=1}^{n}E[(x_i^2-{\mu }^2)]-E[{\mu }_{MLE}^2-{\mu }^2] \\ =(E[{\mu }^2]-E^2[\mu ])-(E[{\mu }_{MLE}^2]-E^2[{\mu }_{MLE}]) \\ ={\sigma }^2-var[{\mu }_{MLE}] \\ ={\sigma }^2-\frac{1}{n}{\sigma }^2=\frac{n-1}{n}{\sigma }^2 \end{gathered}$$其中$$\begin{gathered} var[{\mu }_{MLE}]=var[\frac{1}{n}\sum _{i=1}^n{x}_i] \\ =\frac{1}{n^2}\sum _{i=1}^{n}var[x_i]=\frac{{\sigma }^2}{n} \end{gathered}$$

原视频链接

【机器学习】【白板推导系列】【合集 1～23】