深度学习基础1：神经网络反向传播(BackPropagation, BP)算法

链式求导法则

方程1. $f(x,y)=0$
方程2. $g(x,y)=0$
方程3. $z(f,g)=0$
计算$z$对$x,y$的偏导：

$\frac{\partial z}{\partial x}=\frac{\partial z}{\partial f}\ast \frac{\partial f}{\partial x}+\frac{\partial z}{\partial g}\ast \frac{\partial g}{\partial x}$
$\frac{\partial z}{\partial y}=\frac{\partial z}{\partial f}\ast \frac{\partial f}{\partial y}+\frac{\partial z}{\partial g}\ast \frac{\partial g}{\partial y}$

神经网络结构

下面对符号做一下解释：

符号	意义
X	训练数据
Y	训练标签(目标值)
1…M；1…N	隐藏层 j, k 的神经元序号
1…T	输出层神经元序号
$Ne{t}_m^j$	网络层j的第$m$个神经元的输入（节点输入）
$H_m^j$	网络层$j$的第$m$个神经元的输出 （节点输出）
$f(.)$	激活函数
$W_{n,m}^j$	网络层$j$第$m$个神经元输入到网络层$j+1$第$n$个神经元的权重
${\delta }_m^j$	网络层$j$第$m$个神经元的输出偏差(残差)
$\eta$	梯度下降的步长

神经网络前向传播

1. 输出层第t个神经元的输出$H_t^4$:
   $H_t^4=f(Ne{t}_t^4)$---------------------------------------------- 第四层输出-----------(1)
   $Ne{t}_t^4={\sum }_{n=1}^N(W_{t,n}^3\ast H_n^3)$---------------------------- 第四层输入------------(2)
   $H_n^3=f(Ne{t}_n^3)$, for $n=1,...,N$--------------------- 第三层输出------------(3)
   $Ne{t}_n^3={\sum }_{m=1}^M{W}_{n,m}^2\ast H_m^2$, for $n=1,...,N$---- 第三层输入------------(4)
   $H_m^2=f(Ne{t}_m^2)$, for $m=1,...,M$------------------ 第二层输出------------(5)
   $Ne{t}_m^2=W_{m,1}^1\ast X1+W_{m,2}^1\ast X2$------------------- 第二层输入-----------(6)
   第一层输入输出均为: $[X1,X2{]}^T$------------------------ 第一层：输入层------(7)
   以上即为输出层第一个神经元$H_1^4$的传播过程

反向传播的网络结构

神经网络反向传播推导

1. 首先我们看一下网络的代价函数，其实就是：
   %% 计算（目标值减去最后一层网络的输出值）的模的平方
   $$\begin{array}{rl}Loss& =\frac{1}{2}\ast ||Y-H^4|{|}_2^2\\ & =\frac{1}{2}\ast \sum _{t=1}^T(y_t-H_t^4{)}^2\end{array}\text{\hspace{1em}(8)}$$
2. 将公式(1)(2)代入(8)
   $$\begin{array}{rl}Loss& =\frac{1}{2}\ast \sum _{t=1}^T(y_t-f(Ne{t}_t^4){)}^2\\ & =\frac{1}{2}\ast \sum _{t=1}^T(y_t-f(\sum _{n=1}^N{W}_{t,n}^3\ast H_n^3){)}^2\\ & =\frac{1}{2}\ast \sum _{t=1}^T(y_t-f(\sum _{n=1}^N{W}_{t,n}^3\ast f(Ne{t}_n^3)){)}^2\\ & =......\end{array}\text{\hspace{1em}(9)}$$
3. 计算导数$Loss$对第三层网络权重的倒数：
   $$\begin{array}{r}\frac{\partial Loss}{\partial W_{t,n}^3}=\frac{\partial Loss}{\partial Ne{t}_t^4}\ast \frac{\partial Ne{t}_t^4}{\partial W_{t,n}^3}\end{array}\text{\hspace{1em}(10)}$$
   $$\begin{array}{r}\frac{\partial Loss}{\partial W_{n,m}^2}=\frac{\partial Loss}{\partial Ne{t}_n^3}\ast \frac{\partial Ne{t}_n^3}{\partial W_{n,m}^2}\end{array}\text{\hspace{1em}(11)}$$
   $$\begin{array}{r}\frac{\partial Loss}{\partial W_{m,i}^1}=\frac{\partial Loss}{\partial Ne{t}_n^2}\ast \frac{\partial Ne{t}_n^2}{\partial W_{m,i}^1}\end{array}\text{\hspace{1em}(12)}$$
   令${\delta }_t^4=-\frac{\partial Loss}{\partial Ne{t}_t^4}$, ${\delta }_n^3=-\frac{\partial Loss}{\partial Ne{t}_n^3}$, ${\delta }_m^2=-\frac{\partial Loss}{\partial Ne{t}_m^2}$
   3.1. 下面计算权重增量：
   $$\begin{array}{r}\Delta W_{t,n}^3=-\eta \ast \frac{\partial Loss}{\partial W_{t,n}^3}=-\eta \ast \frac{\partial Loss}{\partial Ne{t}_t^4}\ast \frac{\partial Ne{t}_t^4}{\partial W_{t,n}^3}=\eta \ast {\delta }_t^4\ast \frac{\partial Ne{t}_t^4}{\partial W_{t,n}^3}\end{array}\text{\hspace{1em}(13)}$$
   $$\begin{array}{r}\Delta W_{n,m}^2=-\eta \ast \frac{\partial Loss}{\partial W_{n,m}^2}=-\eta \ast \frac{\partial Loss}{\partial Ne{t}_n^3}\ast \frac{\partial Ne{t}_n^3}{\partial W_{n,m}^2}=\eta \ast {\delta }_n^3\ast \frac{\partial Ne{t}_n^3}{\partial W_{n,m}^2}\end{array}\text{\hspace{1em}(14)}$$
   $$\begin{array}{r}\Delta W_{m,i}^1=-\eta \ast \frac{\partial Loss}{\partial W_{m,i}^1}=-\eta \ast \frac{\partial Loss}{\partial Ne{t}_m^2}\ast \frac{\partial Ne{t}_m^2}{\partial W_{m,i}^1}=\eta \ast {\delta }_m^2\ast \frac{\partial Ne{t}_m^2}{\partial W_{m,i}^1}\end{array}\text{\hspace{1em}(15)}$$
   3.2. 下面分别计算${\delta }_t^4,{\delta }_n^3,{\delta }_m^2$
   $$\begin{array}{r}{\delta }_t^4=-\frac{\partial Loss}{\partial Ne{t}_t^4}=(y_t-H_t^4)\ast f^{\prime }(Ne{t}_t^4)\end{array}\text{\hspace{1em}(16)}$$
   $$\begin{array}{rl}{\delta }_n^3=-\frac{\partial Loss}{\partial Ne{t}_n^3}& \\ & =\sum _{t=1}^T(y_t-H_t^4)\ast f^{\prime }(Ne{t}_t^4)\ast W_{t,n}^3\ast f^{\prime }(Ne{t}_n^3)\\ & =\sum _{t=1}^T{\delta }_t^4\ast W_{t,n}^3\ast f^{\prime }(Ne{t}_n^3)\end{array}\text{\hspace{1em}(17)}$$
   $$\begin{array}{rl}{\delta }_m^2=-\frac{\partial Loss}{\partial Ne{t}_m^2}& \\ & =\sum _{t=1}^T(y_t-H_t^4)\ast f^{\prime }(Ne{t}_t^4)\ast \sum _{n=1}^N{W}_{t,n}^3\ast f^{\prime }(Ne{t}_n^3)\ast W_{n,m}^2\ast f^{\prime }(Ne{t}_m^2)\\ & =\sum _{t=1}^T{\delta }_t^4\ast \sum _{n=1}^N{W}_{t,n}^3\ast f^{\prime }(Ne{t}_n^3)\ast W_{n,m}^2\ast f^{\prime }(Ne{t}_m^2)\\ & =\sum _{n=1}^N(\sum _{t=1}^T{\delta }_t^4\ast W_{t,n}^3\ast f^{\prime }(Ne{t}_n^3)\ast W_{n,m}^2\ast f^{\prime }(Ne{t}_m^2))\\ & =\sum _{n=1}^N{\delta }_n^3\ast W_{n,m}^2\ast f^{\prime }(Ne{t}_m^2)\end{array}\text{\hspace{1em}(18)}$$
   3.3. 计算$\frac{\partial Ne{t}_t^4}{\partial W_{t,n}^3}$, $\frac{\partial Ne{t}_n^3}{\partial W_{n,m}^2}$, $\frac{\partial Ne{t}_m^2}{\partial W_{m,i}^1}$:
   $$\begin{array}{r}\frac{\partial Ne{t}_t^4}{\partial W_{t,n}^3}=\frac{\partial {\sum }_{n=1}^N(W_{t,n}^3\ast H_n^3)}{\partial W_{t,n}^3}=H_n^3\end{array}\text{\hspace{1em}(19)}$$
   $$\begin{array}{r}\frac{\partial Ne{t}_n^3}{\partial W_{n,m}^2}=\frac{\partial {\sum }_{m=1}^M(W_{n,m}^2\ast H_m^2)}{\partial W_{n,m}^2}=H_m^2\end{array}\text{\hspace{1em}(20)}$$
   $$\begin{array}{r}\frac{\partial Ne{t}_m^2}{\partial W_{m,i}^1}=\frac{\partial {\sum }_{i=1}^I(W_{m,i}^1\ast H_i^1)}{\partial W_{m,i}^1}=H_i^1\end{array}\text{\hspace{1em}(21)}$$
   3.4. 总结
   将公式（19-21）带入（13-15），得到
   $$\begin{array}{r}\Delta W_{t,n}^3=\eta \ast {\delta }_t^4\ast \frac{\partial Ne{t}_t^4}{\partial W_{t,n}^3}=\eta \ast {\delta }_t^4\ast H_n^3\end{array}\text{\hspace{1em}(22)}$$
   $$\begin{array}{r}\Delta W_{n,m}^2=\eta \ast {\delta }_n^3\ast \frac{\partial Ne{t}_n^3}{\partial W_{n,m}^2}=\eta \ast {\delta }_n^3\ast H_m^2\end{array}\text{\hspace{1em}(23)}$$
   $$\begin{array}{r}\Delta W_{m,i}^1=\eta \ast {\delta }_m^2\ast \frac{\partial Ne{t}_m^2}{\partial W_{m,i}^1}=\eta \ast {\delta }_m^2\ast H_i^1\end{array}\text{\hspace{1em}(24)}$$
   3.5. 权值更新：
   $$\begin{array}{r}{W}_{t,n}^3=W_{t,n}^3+\Delta W_{t,n}^3\end{array}\text{\hspace{1em}(25)}$$
   $$\begin{array}{r}{W}_{n,m}^2=W_{n,m}^2+\Delta W_{n,m}^2\end{array}\text{\hspace{1em}(26)}$$
   $$\begin{array}{r}{W}_{m,i}^1=W_{m,i}^1+\Delta W_{m,i}^1\end{array}\text{\hspace{1em}(27)}$$

偏置

对于偏置项，推倒方法与上述一样。