线性回归模型求解

1.线性回归损失函数推导

1.1 相关函数

• 线性回归函数
  $$\begin{array}{rl}{y}_{(i)}& ={\theta }^{T}X\end{array}\text{\hspace{1em}(1)}$$
• 高斯分布
  $$\begin{array}{rl}f(x)& =\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{(x-u{)}^2}{2{\sigma }^2}}\end{array}\text{\hspace{1em}(2)}$$
• 联合概率密度
  • 如果两随机变量互相独立，则联合密度函数等于边缘分布函数的乘积
    $$\begin{array}{rl}P(AB)& =P(A)\ast P(B)\end{array}$$

1.2 损失函数推导

• 已知：线性回归误差$ϵ$服从均值为0，方差为${\sigma }^2$的高斯分布
• 最大可能性估计，我们要求误差最大似然估计，求最大值
  $$\begin{array}{rl}{L}_{\theta }\{{ϵ}_1,{ϵ}_2,{ϵ}_3...{ϵ}_M\}& =f({ϵ}_1,{ϵ}_2,{ϵ}_3...{ϵ}_m|u,{\sigma }^2)\\ & =f({ϵ}_1|u,{\sigma }^2)\ast f({ϵ}_2|u,{\sigma }^2)\dots f({ϵ}_m|u,{\sigma }^2)\\ & =\prod _{i=1}^{m}f({ϵ}_i|u,{\sigma }^2)\\ & =\prod _{i=1}^m\frac{1}{\sqrt{2\pi }\sigma }\ast e^{-\frac{(y^i-{\theta }^T{X}^i{)}^2}{2{\sigma }^2}}\end{array}\text{\hspace{1em}(带入1,2)}$$
• 对公式取对数
  $$\begin{array}{rl}lo{g}_e^{L_{\theta }\{{ϵ}_1,{ϵ}_2,{ϵ}_3...{ϵ}_M\}}& ={\log }_e^{{\prod }_{i=1}^m\frac{1}{\sqrt{2\pi }\sigma }\ast e^{-\frac{(y^i-{\theta }^T{X}^i{)}^2}{2{\sigma }^2}}}\\ & =\sum _{i=1}^m{\log }_e^{\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{(y^i-{\theta }^T{X}^i{)}^2}{2{\sigma }^2}}}\\ & =\sum _{i=1}^m{\log }_e^{\frac{1}{\sqrt{2\pi }\sigma }}+\sum _{i=1}^m{\log }_e^{e^{-\frac{(y^i-{\theta }^T{X}^i{)}^2}{2{\sigma }^2}}}\\ & =m{\log }_e^{\frac{1}{\sqrt{2\pi }\sigma }}-m\frac{(y^i-{\theta }^T{X}^i{)}^2}{2{\sigma }^2}\end{array}$$
• 已知以下值是确定的
  $$\begin{array}{r}m{\log }_e^{\frac{1}{\sqrt{2\pi }\sigma }},-\frac{m}{2{\sigma }^2}\end{array}$$
• 即求$L_{\theta }\{{ϵ}_1,{ϵ}_2,{ϵ}_3...{ϵ}_M\}$，最大值，就是求$\frac{(y^i-{\theta }^T{X}^i-b{)}^2}{2{\sigma }^2}$的最小值，就是求$(y^i-{\theta }^T{X}^i-b{)}^2$的最小值

1.3 总结

• 线性回归损失函数为【基于均方误差MSE】
  $$\begin{array}{rl}Los{s}_{\theta }& =\frac{1}{2m}\sum _{i=1}^m({\theta }^T{X}^i+b-y^i{)}^2\\ & =\frac{1}{2m}\sum _{i=1}^m({\theta }^T{X}^i-y^i)({\theta }^T{X}^i-y^i)\\ & =\frac{1}{2}({\theta }^{T}X-y{)}^T({\theta }^{T}X-y)\\ & =\frac{1}{2}(X\theta -y{)}^T(X\theta -y)\\ & =\frac{1}{2}({\theta }^T{X}^T-y^T)(X\theta -y)\\ & =\frac{1}{2}({\theta }^T{X}^{T}X\theta -{\theta }^T{X}^{T}y-y^{T}X\theta +y^{T}y)\end{array}$$

2.损失函数求解推导

2.1 矩阵运算法则

$$\begin{gathered} (KA{)}^T=K{A}^T \\ (A+B{)}^T=A^T+B^T \\ (AB{)}^T=B^T{A}^T \\ (A^T{)}^T=A \\ \frac{\partial {\theta }^{T}A\theta }{\partial \theta }=2A\theta \\ \frac{\partial {\theta }^{T}A}{\partial \theta }=A \\ \frac{\partial A\theta }{\partial \theta }=A^T \end{gathered}$$

2.2 损失函数求解【求导】

$$\begin{array}{rl}\frac{Los{s}_{\theta }}{\partial \theta }& =\frac{\frac{1}{2}(X^T{\theta }^T\theta X-X^T{\theta }^{T}y-y^T\theta X+y^{T}y)}{\partial \theta }\\ & =\frac{\frac{1}{2}(X^T{\theta }^T{X}^T\theta -X^T{\theta }^{T}y-y^T\theta X+y^{T}y)}{\partial \theta }\\ & =\frac{1}{2}(X^{T}X\theta -X^T{\theta }^{T}y-y^T\theta X+y^{T}y)\end{array}$$