分类与回归梯度下降公式推导

1.相关函数公式及求导

• 链式推导法则
  $$\begin{array}{r}\frac{\partial J_{(\theta )}}{\partial \theta }=\frac{\partial J_{(h)}}{\partial h}\ast \frac{\partial h_{(z)}}{\partial z}\ast \frac{\partial z_{(\theta )}}{\partial \theta }\end{array}$$

1.1.线性回归公式

• 原函数
  $$z_{(\theta )}={\theta }^{T}x$$
• 求导过程
  $$\begin{array}{rl}\frac{\partial z_{(\theta )}}{\partial {\theta }_{}}& =\frac{\partial {\theta }^{T}x}{\partial \theta }\\ & =x\end{array}$$

1.2.sigmoid函数

• 原函数
  $$\begin{array}{rl}{h}_{(z)}& =\frac{1}{1+e^{-z}}\end{array}$$
• 求导过程

$$\begin{array}{rl}\frac{\partial h_{(z)}}{\partial z}& =\frac{\partial (\frac{1}{1+e^{-z}})}{\partial z}\\ & =\frac{0-\partial (1+e^{-z})}{(1+e^{-z}{)}^2}\\ & =\frac{e^{-z}}{(1+e^{-z}{)}^2}\\ & =\frac{1+e^{-z}-1}{(1+e^{-z})(1+e^{-z})}\\ & =\frac{1+e^{-z}-1}{(1+e^{-z})}.\frac{1}{(1+e^{-z})}\\ & =[1-\frac{1}{(1+e^{-z})}].\frac{1}{(1+e^{-z})}\\ & =(1-h_{(z)})\ast h_{(z)}\end{array}$$

2. 逻辑回归

2.1损失函数公式【交叉熵公式】

• 原函数
  $$\begin{array}{rl}Los{s}_{(w)}& =-\frac{1}{m}[\sum _{i=1}^m(y^i\ast lo{g}^{h(z)}+(1-y^i)\ast lo{g}^{(1-h(z)})]\end{array}$$
• 求偏导过程
  $$\begin{array}{rl}\frac{\partial Los{s}_{(h)}}{\partial h}& =-\frac{\partial \frac{1}{m}[{\sum }_{i=1}^m(y^i\ast lo{g}^{h(z)}+(1-y^i)\ast lo{g}^{(1-h(z)})]}{\partial h}\\ & =-\frac{1}{m}[\sum _{i=1}^m(y^i\ast \frac{1}{h(z)}+(1-y^i)\ast \frac{1}{1-h(z)}\ast (-1))]\\ & =-\frac{1}{m}[\sum _{i=1}^m(\frac{y^i}{h(z)}+\frac{(1-y^i)}{1-h(z)}\ast (-1))]\\ & =-\frac{1}{m}[\sum _{i=1}^m(\frac{y^i\ast (1-h_{(z)})+(y^i-1)\ast h_{(z)}}{h_{(z)}\ast (1-h_{(z)})})]\\ & =-\frac{1}{m}[\sum _{i=1}^m(\frac{y^i-h_{(z)}}{h_{(z)}\ast (1-h_{(z)})})]\\ & =\frac{1}{m}[\sum _{i=1}^m(\frac{h_{(z)}-y^i}{h_{(z)}\ast (1-h_{(z)})})]\end{array}$$

2.2 求导

$$\begin{array}{rl}\frac{\partial Los{s}_{(\theta )}}{\partial {\theta }_j}& =\frac{\partial Los{s}_{(h)}}{\partial h}\ast \frac{\partial h_{(z)}}{\partial z}\ast \frac{\partial z_{(\theta )}}{\partial {\theta }_j}\\ & =\frac{1}{m}[\sum _{i=1}^m(\frac{h_{(z)}-y^i}{h_{(z)}\ast (1-h_{(z)})})]\ast (1-h_{(z)})\ast h_{(z)}\ast x_j^i\\ & =\frac{1}{m}\sum _{i=1}^m(h_{(z)}-y^i)\ast x_j^i\end{array}$$

2.2 逻辑回归梯度下降公式

• 对$\theta$求偏导
  $$\begin{array}{rl}{\theta }_j& :={\theta }_j-\alpha \ast \frac{\partial Los{s}_{(\theta )}}{\partial {\theta }_j}\\ & :={\theta }_j-\alpha \ast \frac{1}{m}\sum _{i=1}^m(h_{(z)}-y^i)\ast x_j^i\end{array}$$

3. 线性回归

3.1损失函数公式【MSE公式】

• 原函数
  $$\begin{array}{rl}Los{s}_{(\theta )}& =\frac{1}{2}(z_{(\theta )}-y{)}^2\end{array}$$
• 求导过程
  $$\begin{array}{rl}\frac{\partial Los{s}_{(\theta )}}{\partial {\theta }_j}& =\frac{\partial \frac{1}{2}(z_{(\theta )}-y^i{)}^2}{\partial {\theta }_j}\\ & =\frac{1}{2}\ast 2\ast (z_{({\theta }_j)}-y^i)\ast \partial z_{({\theta }_j)}\\ & =(z_{({\theta }_j)}-y^i)\ast x_j^i\end{array}$$

3.2 线性回归梯度下降公式

$$\begin{array}{rl}{\theta }_j& :={\theta }_j-\alpha \ast \frac{\partial Los{s}_{(\theta )}}{\partial {\theta }_j}\\ & :={\theta }_j-\alpha \ast (z_{({\theta }_j)}-y^i)\ast x_j^i\end{array}$$