机器学习 之 作业2

作业 1: 线性回归模型的极大似然估计

1. $E\{{\text{w}}_{MLE}\}$

$$E\{\text{w}\}=\int \text{w}p(y|\text{X},w,{\delta }^2)dt$$

其中$\text{w}=({\text{X}}^T\text{X}{)}^{-1}{\text{X}}^T\text{y}$，带入
$$\begin{gathered} E\{\text{w}\}=({\text{X}}^T\text{X}{)}^{-1}{\text{X}}^T\int \text{y}p(y|\text{X},w,{\delta }^2)dt \\ =({\text{X}}^T\text{X}{)}^{-1}{\text{X}}^{T}E\{\text{y}\} \\ =({\text{X}}^T\text{X}{)}^{-1}{\text{X}}^T\text{X}\text{w} \\ =\text{w} \end{gathered}$$

2. $Cov\{{\text{w}}_{MLE}\}$

$$\begin{gathered} cov\{\text{w}\}=E\{\text{w}{\text{w}}^T\}-E\{\text{w}\}E\{\text{w}{\}}^T \\ =E\{\text{w}{\text{w}}^T\}-\text{w}{\text{w}}^T \end{gathered}$$

其中
$$\begin{gathered} E\{\text{w}{\text{w}}^T\}=E\{(({\text{X}}^T\text{X}{)}^{-1}{\text{X}}^T\text{y})(({\text{X}}^T\text{X}{)}^{-1}{\text{X}}^T\text{y}{)}^T\} \\ =({\text{X}}^T\text{X}{)}^{-1}{\text{X}}^{T}E\{\text{y}{\text{y}}^T\}\text{X}({\text{X}}^T\text{X}{)}^{-1} \end{gathered}$$

其中
$$\begin{gathered} cov\{\text{y}\}={\delta }^{2}I=E\{\text{y}{\text{y}}^T\}-E\{\text{y}\}E\{\text{y}{\}}^T \\ E\{\text{y}{\text{y}}^T\}=E\{\text{y}\}E\{\text{y}{\}}^T+{\delta }^{2}I \\ =\text{X}\text{w}(\text{X}\text{w}{)}^T+{\delta }^{2}I \\ =\text{X}\text{w}{\text{w}}^T{\text{X}}^T+{\delta }^{2}I \end{gathered}$$

因此
$$\begin{gathered} E\{\text{w}{\text{w}}^T\}=({\text{X}}^T\text{X}{)}^{-1}{\text{X}}^T\text{X}\text{w}{\text{w}}^T{\text{X}}^T\text{X}({\text{X}}^T\text{X}{)}^{-1} \\ +{\delta }^2({\text{X}}^T\text{X}{)}^{-1}{\text{X}}^T\text{X}({\text{X}}^T\text{X}{)}^{-1} \\ =\text{w}{\text{w}}^T+{\delta }^2({\text{X}}^T\text{X}{)}^{-1} \end{gathered}$$

$$\begin{gathered} cov\{\text{w}\}=\text{w}{\text{w}}^T+{\delta }^2({\text{X}}^T\text{X}{)}^{-1}-\text{w}{\text{w}}^T \\ ={\delta }^2({\text{X}}^T\text{X}{)}^{-1} \end{gathered}$$

作业 2: 关于岭回归与正则化

1. $求P(w|D)$的均值和方差

$$\begin{gathered} p(w|D)\simeq p(D|w)p(w) \\ =(\frac{1}{\sqrt{2\pi }\delta }{)}^n{e}^{-\frac{(\text{y}-\text{X}\text{w}{)}^T(\text{y}-\text{X}\text{w})}{2{\delta }^2\text{I}}}\times (\frac{1}{\sqrt{2\pi {\Sigma }_0}}{)}^n{e}^{-\frac{(\text{w}-{\mu }_0{)}^T(\text{w}-{\mu }_0)}{2{\Sigma }_0}} \\ \simeq e^{-\frac{1}{2}(\frac{(\text{y}-\text{X}\text{w}{)}^T(\text{y}-\text{X}\text{w})}{{\delta }^2\text{I}}+\frac{(\text{w}-{\mu }_0{)}^T(\text{w}-{\mu }_0)}{{\Sigma }_0})} \\ \simeq e^{-\frac{1}{2}(\frac{-2{\text{y}}^T\text{X}\text{w}+{\text{w}}^T{\text{X}}^T\text{X}\text{w}}{{\delta }^2\text{I}}+\frac{{\text{w}}^T\text{w}-2{\mu }_0^T\text{w}}{{\Sigma }_0})} \end{gathered}$$

后验概率应该为
$$\begin{gathered} p(\text{w}|D)=N({\mu }_w,{\Sigma }_w) \\ \simeq e^{-\frac{(\text{w}-{\mu }_w{)}^T(\text{w}-{\mu }_w)}{2{\Sigma }_w}} \\ \simeq e^{-\frac{{\text{w}}^T\text{w}-2{\mu }_w^T\text{w}}{2{\Sigma }_w}} \end{gathered}$$

这里面有一个线性二次项，对应相等，方差可解
$$\begin{gathered} \frac{{\text{w}}^T\text{w}}{{\Sigma }_w}=\frac{{\text{w}}^T{\text{X}}^T\text{X}\text{w}}{{\delta }^2\text{I}}+\frac{{\text{w}}^T\text{w}}{{\Sigma }_0} \\ ={\text{w}}^T(\frac{{\text{X}}^T\text{X}}{{\delta }^2\text{I}}+\frac{1}{{\Sigma }_0})\text{w} \\ \frac{1}{{\Sigma }_w}=\frac{{\text{X}}^T\text{X}}{{\delta }^2\text{I}}+\frac{1}{{\Sigma }_0} \\ {\Sigma }_w=(\frac{1}{{\delta }^2}{\text{X}}^T\text{X}+{\Sigma }_0^{-1}{)}^{-1} \end{gathered}$$

同样的，将线性一次项相等，会获得均值${\mu }_w$，这样$w$的方差，均值都获得了
$$\begin{gathered} \frac{-2{\mu }_w^T\text{w}}{{\Sigma }_w}=\frac{-2{\text{y}}^T\text{X}\text{w}}{{\delta }^2}+\frac{-2{\mu }_0^T\text{w}}{{\Sigma }_0} \\ \frac{{\mu }_w^T}{{\Sigma }_w}=\frac{{\text{y}}^T\text{X}}{{\delta }^2}+\frac{{\mu }_0^T}{{\Sigma }_0} \\ {\mu }_w^T=(\frac{{\text{y}}^T\text{X}}{{\delta }^2}+\frac{{\mu }_0^T}{{\Sigma }_0}){\Sigma }_w \\ {\mu }_w={\Sigma }_w(\frac{1}{{\delta }^2}{\text{X}}^T\text{y}+{\Sigma }_0^{-1}{\mu }_0) \\ =(\frac{1}{{\delta }^2}{\text{X}}^T\text{X}+{\Sigma }_0^{-1}{)}^{-1}(\frac{1}{{\delta }^2}{\text{X}}^T\text{y}+{\Sigma }_0^{-1}{\mu }_0) \end{gathered}$$

2. 为什么$w_{MAP}=(X^T+\lambda I{)}^{-1}{X}^{T}y?$

因为后验概率是高斯分布，$w$最有可能的地方就是在均值处，如果${\mu }_0=[0,0,...,0{]}^T$，那么就变成了$$w_{MAP}={\mu }_w=(\frac{1}{{\delta }^2}{\text{X}}^T\text{X}+{\Sigma }_0^{-1}{)}^{-1}\frac{1}{{\delta }^2}{\text{X}}^T\text{y}$$

基本相同。

3. 为什么$X^{T}X+\lambda I$是可逆的？

$({\text{X}}^T\text{X}{)}^T={\text{X}}^T\text{X}$，因此${\text{X}}^T\text{X}$是对称阵，明显的特点是对称轴都大于等于0，加上$\lambda I$后对称轴上都大于0，说明它为正定矩阵，正定矩阵可逆。