机器学习常见算法推导过程

机器学习

线性回归

预测值与误差： $$y^{(i)}={\theta }^T{x}^{(i)}+{\epsilon }^{(i)}（1）$$
由于误差服从高斯分布：$$\rho ({\epsilon }^{(i)})=\frac{1}{\sqrt{2\pi }\sigma }exp(-\frac{({\epsilon }^{(i)}{)}^2}{2{\sigma }^2})（2）$$

将（1）带入（2）,可得：

$$\rho ({\epsilon }^{(i)})=\frac{1}{\sqrt{2\pi }\sigma }exp(-\frac{(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2}{2{\sigma }^2})$$

似然函数： $$L(\theta )=\prod _{i=1}^{m}p(y^{(i)}|x^{(i)};\theta )=\prod _{i=1}^m\frac{1}{\sqrt{2\pi }\sigma }exp(-\frac{(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2}{2{\sigma }^2})$$

解释：什么样的参数和数据组合恰好就是真实值 以此来解释似然函数

对数似然：将两边Log一下，转化为加法：
$$\log L(\theta )=\prod _{i=1}^{m}p(y^{(i)}|x^{(i)};\theta )=\log \prod _{i=1}^m\frac{1}{\sqrt{2\pi }\sigma }exp(-\frac{(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2}{2{\sigma }^2})$$

展开式:$$\log \prod _{i=1}^m\frac{1}{\sqrt{2\pi }\sigma }exp(-\frac{(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2}{2{\sigma }^2})=m\log \frac{1}{\sqrt[]{2\pi }\sigma }-\frac{1}{{\sigma }^2}.\frac{1}{2}\sum _{i=1}^m(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2$$

故:极大似然：保证似然函数越大越好，在上述推导中，除去常数项方程为：
$$-\frac{1}{2}\sum _{i=1}^m(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2$$

所以保证：$$J(\theta )=\frac{1}{2}\sum _{i=1}^m(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2$$最小即可

$$J(\theta )=\frac{1}{2}\sum _{i=1}^m(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2=\frac{1}{2}(X\theta -y{)}^T(X\theta -y)$$

$${\nabla }_{\theta }J(\theta )={\nabla }_{\theta }(\frac{1}{2}(X\theta -y{)}^T(X\theta -y))={\nabla }_{\theta }(\frac{1}{2}({\theta }^T{X}^T-y^T)(X\theta -y))$$

$$={\nabla }_{\theta }(\frac{1}{2}({\theta }^T{X}^{T}X\theta -{\theta }^T{X}^{T}y-y^{T}X\theta +y^{T}y))$$

$$=\frac{1}{2}(2X^{T}X\theta -X^{T}y-(y^{T}X{)}^T)=X^{T}X\theta -X^{T}y$$

当偏导等于0：$\theta =(X^{T}X{)}^{-1}{X}^{T}y$