【ML】- 001 线性回归01-梯度下降法基础理论

• 数据集
  给定数据集$\{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),\cdots ,(x^{(m)},y^{(m)})\}$，其中$x^{(i)}=\{x_1^{(i)},x_2^{(i)},\cdots ,x_n^{(i)}\}$，$y^{(i)}\in R$。

• 待拟合的函数(线性函数)

$$\begin{array}{rl}{h}_{\theta }(x^{(i)})& ={\theta }_0+{\theta }_1{x}_1^{(i)}+{\theta }_2{x}_2^{(i)}+\cdots +{\theta }_n{x}_n^{(i)}\\ & =\sum _{j=0}^n{\theta }_j{x}_j^{(i)}\\ & ={\theta }^T{x}^{(i)}\end{array}$$

• 损失函数(目标函数)

$$\begin{gathered} J(\theta )=\frac{1}{2}\sum _{i=1}^m{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2 \\ \underset{\theta }{\min }J(\theta ) \end{gathered}$$

• 梯度下降法求解

• Batch Gradient Descent (m为样本数)
  $$\begin{array}{rl}{\theta }_j& :={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J(\theta )\\ & ={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}\frac{1}{2}\sum _{i=1}^m{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2\\ & ={\theta }_j-\alpha \sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\frac{\partial }{\partial {\theta }_j}\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\\ & ={\theta }_j-\alpha \sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)x_j^{(i)}\end{array}$$

• Stochastic Gradient Descent (随机取一个样本)
  $$\begin{array}{rl}{\theta }_j& :={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J(\theta )\\ & ={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}\frac{1}{2}{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2\\ & ={\theta }_j-\alpha \left(h_{\theta }(x^{(i)})-y^{(i)}\right)\frac{\partial }{\partial {\theta }_j}\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\\ & ={\theta }_j-\alpha \left(h_{\theta }(x^{(i)})-y^{(i)}\right)x_j^{(i)}\end{array}$$

• mini-batch Stochastic Gradient Descent (b为batch size)
  $$\begin{array}{rl}{\theta }_j& :={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J(\theta )\\ & ={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}\frac{1}{2}\sum _{i=1}^b{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2\\ & ={\theta }_j-\alpha \sum _{i=1}^b\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\frac{\partial }{\partial {\theta }_j}\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\\ & ={\theta }_j-\alpha \sum _{i=1}^b\left(h_{\theta }(x^{(i)})-y^{(i)}\right)x_j^{(i)}\end{array}$$

• 三种梯度下降算法对比：

1. BGD：全局最优解，能保证每一次更新权值，都能降低损失函数；样本数量大的时候训练慢。
2. SGD：训练速度快；局部最优解，不能保证每次迭代都向着整体最优化方向，需要迭代的次数多。
3. mbSGD：以上二者折衷。