全连接与卷积神经网络反向传播公式推导

全连接与卷积神经网络反向传播公式推导

全连接网络反向传播公式

BP四项基本原则：

$$\begin{array}{rl}{\delta }_i^{(L)}& ={▽}_{y_i}Cost\cdot {\sigma }^{\prime }(logi{t}_i^{(L)})\\ {\delta }_i^{(l)}& =\sum _j{\delta }_j^{(l+1)}{w}_{ji}^{(l+1)}{\sigma }^{\prime }(logi{t}_i^{(l)})\\ \frac{\partial Cost}{\partial bia{s}_i^{(l)}}& ={\delta }_i^{(l)}\\ \frac{\partial Cost}{\partial w_{ij}^{(l)}}& ={\delta }_i^{(l)}{h}_j^{(l-1)}\end{array}$$

其中，$(l)$表示第$l$层，一共有L层，$i,j$表示当前层神经元的序号。

反向传播公式的目的主要是得到：$\frac{\partial Cost}{\partial bia{s}_i^{(l)}}$和$\frac{\partial Cost}{\partial w_{ij}^{(l)}}$。

在推导的过程中

$$\begin{array}{rl}\frac{\partial Cost}{\partial bia{s}_i^{(l)}}& =\frac{\partial Cost}{\partial logi{t}_i^{(l)}}\cdot \frac{\partial logi{t}_i^{(l)}}{\partial bia{s}_i^{(l)}}\\ \frac{\partial Cost}{\partial w_{ij}^{(l)}}& =\frac{\partial Cost}{\partial logi{t}_i^{(l)}}\cdot \frac{\partial logi{t}_i^{(l)}}{\partial w_{ij}^{(l)}}\end{array}$$

会发现都要用到$\frac{\partial Cost}{\partial logi{t}_i^{(l)}}$。

而

$$logi{t}_i^{(l)}=w_{ij}^{(l)}{h}_j^{(l)}+\sum _{k\ne j}{w}_{ik}^{(l)}{h}_k^{(l)}+bia{s}_i^{(l)}$$

所以

$$\begin{array}{rl}\frac{\partial logi{t}_i^{(l)}}{\partial bia{s}_i^{(l)}}& =1\\ \frac{\partial logi{t}_i^{(l)}}{\partial w_{ij}^{(l)}}& =h_j^{(l)}\end{array}$$

那接下来的问题就只有求$\frac{\partial Cost}{\partial logi{t}_i^{(l)}}$了，求它可以用递推法：

为公式看起来简洁，我们把$\frac{\partial Cost}{\partial logi{t}_i^{(l)}}$记为${\delta }_i^{(l)}$，那么

$${\delta }_i^{(l)}=\frac{\partial Cost}{\partial logi{t}_i^{(l)}}=\sum _j\frac{\partial Cost}{\partial logi{t}_j^{(l+1)}}\cdot \frac{\partial logi{t}_j^{(l+1)}}{\partial logi{t}_i^{(l)}}=\sum _j{\delta }_j^{(l+1)}\cdot \frac{\partial logi{t}_j^{(l+1)}}{\partial logi{t}_i^{(l)}}$$

易知：

$$logi{t}_j^{(l+1)}=w_{ji}^{(l+1)}\sigma (logi{t}_i^{(l)})+\sum _{k\ne i}{w}_{jk}^{(l+1)}\sigma (logi{t}_k^{(l)})+bia{s}_j^{(l+1)}$$

所以

$$\frac{\partial logi{t}_j^{(l+1)}}{\partial logi{t}_i^{(l)}}=\frac{\partial [w_{ji}^{(l+1)}\sigma (logi{t}_i^{(l)})+{\sum }_{k\ne i}{w}_{jk}^{(l+1)}\sigma (logi{t}_k^{(l)})+bia{s}_j^{(l+1)}]}{\partial logi{t}_i^{(l)}}=w_{ji}^{(l+1)}{\sigma }^{\prime }(logi{t}_i^{(l)})$$

因此

$${\delta }_i^{(l)}=\sum _j{\delta }_j^{(l+1)}{w}_{ji}^{(l+1)}{\sigma }^{\prime }(logi{t}_i^{(l)})$$

以上就是${\delta }_i^{(l)}$的递推公式，当然递推公式需要一个终止条件，那就是${\delta }_i^{(L)}$：

$${\delta }_i^{(L)}=\frac{\partial Cost}{\partial logi{t}_i^{(L)}}=\frac{\partial Cost}{\partial y_i}\cdot \frac{\partial y_i}{\partial logi{t}_i^{(L)}}=\frac{\partial Cost}{\partial y_i}\cdot \frac{\partial \sigma (logi{t}_i^{(L)})}{\partial logi{t}_i^{(L)}}={▽}_{y_i}Cost\cdot {\sigma }^{\prime }(logi{t}_i^{(L)})$$

这就是BP四项基本原则的思考过程。

下面看矩阵形式：

$$\begin{array}{rl}{\delta }^{(L)}& =\left[\begin{array}{c}{\delta }_1^{(L)}\\ {\delta }_2^{(L)}\\ ⋮\\ {\delta }_M^{(L)}\end{array}\right]=\left[\begin{array}{c}{▽}_{y_1}Cost\cdot {\sigma }^{\prime }(logi{t}_1^{(L)})\\ {▽}_{y_2}Cost\cdot {\sigma }^{\prime }(logi{t}_2^{(L)})\\ ⋮\\ {▽}_{y_M}Cost\cdot {\sigma }^{\prime }(logi{t}_M^{(L)})\end{array}\right]=\left[\begin{array}{c}{▽}_{y_1}Cost\\ {▽}_{y_2}Cost\\ ⋮\\ {▽}_{y_M}Cost\end{array}\right]\odot \left[\begin{array}{c}{\sigma }^{\prime }(logi{t}_1^{(L)})\\ {\sigma }^{\prime }(logi{t}_2^{(L)})\\ ⋮\\ {\sigma }^{\prime }(logi{t}_M^{(L)})\end{array}\right]\\ \Rightarrow {\delta }^{(L)}& ={▽}_{y}Cost\odot {\sigma }^{\prime }(logi{t}^{(L)})\end{array}$$

这里，$\odot$表示Hadamard乘积，即对应元素相乘，M表示第$L$层有M个神经元。

$$\begin{array}{rl}{\delta }^{(l)}& =\left[\begin{array}{c}{\delta }_1^{(l)}\\ {\delta }_2^{(l)}\\ ⋮\\ {\delta }_M^{(l)}\end{array}\right]=\left[\begin{array}{c}{\sum }_j{\delta }_j^{(l+1)}{w}_{j1}^{(l+1)}{\sigma }^{\prime }(logi{t}_1^{(l)})\\ {\sum }_j{\delta }_j^{(l+1)}{w}_{j2}^{(l+1)}{\sigma }^{\prime }(logi{t}_2^{(l)})\\ ⋮\\ {\sum }_j{\delta }_j^{(l+1)}{w}_{jM}^{(l+1)}{\sigma }^{\prime }(logi{t}_M^{(l)})\end{array}\right]\\ & =\left[\begin{array}{c}{\sum }_j{\delta }_j^{(l+1)}{w}_{j1}^{(l+1)}\\ {\sum }_j{\delta }_j^{(l+1)}{w}_{j2}^{(l+1)}\\ ⋮\\ {\sum }_j{\delta }_j^{(l+1)}{w}_{jM}^{(l+1)}\end{array}\right]\odot \left[\begin{array}{c}{\sigma }^{\prime }(logi{t}_1^{(l)})\\ {\sigma }^{\prime }(logi{t}_2^{(l)})\\ ⋮\\ {\sigma }^{\prime }(logi{t}_M^{(l)})\end{array}\right]\\ & =\left[\begin{array}{cccc}{w}_{11}^{(l+1)}& w_{21}^{(l+1)}& \dots & w_{N1}^{(l+1)}\\ w_{12}^{(l+1)}& w_{22}^{(l+1)}& \dots & w_{N2}^{(l+1)}\\ ⋮& ⋮& \ddots & ⋮\\ w_{1M}^{(l+1)}& w_{2M}^{(l+1)}& \dots & w_{NM}^{(l+1)}\end{array}\right]\left[\begin{array}{c}{\delta }_1^{(l+1)}\\ {\delta }_2^{(l+1)}\\ ⋮\\ {\delta }_N^{(l+1)}\end{array}\right]\odot \left[\begin{array}{c}{\sigma }^{\prime }(logi{t}_1^{(l)})\\ {\sigma }^{\prime }(logi{t}_2^{(l)})\\ ⋮\\ {\sigma }^{\prime }(logi{t}_M^{(l)})\end{array}\right]\\ & ={\left[\begin{array}{cccc}{w}_{11}^{(l+1)}& w_{12}^{(l+1)}& \dots & w_{1M}^{(l+1)}\\ w_{21}^{(l+1)}& w_{22}^{(l+1)}& \dots & w_{2M}^{(l+1)}\\ ⋮& ⋮& \ddots & ⋮\\ w_{N1}^{(l+1)}& w_{N2}^{(l+1)}& \dots & w_{NM}^{(l+1)}\end{array}\right]}^T\left[\begin{array}{c}{\delta }_1^{(l+1)}\\ {\delta }_2^{(l+1)}\\ ⋮\\ {\delta }_N^{(l+1)}\end{array}\right]\odot \left[\begin{array}{c}{\sigma }^{\prime }(logi{t}_1^{(l)})\\ {\sigma }^{\prime }(logi{t}_2^{(l)})\\ ⋮\\ {\sigma }^{\prime }(logi{t}_M^{(l)})\end{array}\right]\\ \Rightarrow {\delta }^{(l)}& =[(W^{(L+1)}{)}^T{\delta }^{(l+1)}]\odot {\sigma }^{\prime }(logi{t}^{(l)})\end{array}$$

因为$\frac{\partial Cost}{\partial bia{s}_i^{(l)}}={\delta }_i^{(l)}$，所以$\frac{\partial Cost}{\partial bia{s}^{(l)}}={\delta }^{(l)}$。那就剩下

$$\begin{array}{rl}\frac{\partial Cost}{\partial w^{(l)}}& =\left[\begin{array}{cccc}\frac{\partial Cost}{\partial w_{11}^{(l)}}& \frac{\partial Cost}{\partial w_{12}^{(l)}}& \dots & \frac{\partial Cost}{\partial w_{1N}^{(l)}}\\ \frac{\partial Cost}{\partial w_{21}^{(l)}}& \frac{\partial Cost}{\partial w_{22}^{(l)}}& \dots & \frac{\partial Cost}{\partial w_{2N}^{(l)}}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial Cost}{\partial w_{M1}^{(l)}}& \frac{\partial Cost}{\partial w_{M2}^{(l)}}& \dots & \frac{\partial Cost}{\partial w_{MN}^{(l)}}\end{array}\right]=\left[\begin{array}{cccc}{\delta }_1^{(l)}{h}_1^{(l-1)}& {\delta }_1^{(l)}{h}_2^{(l-1)}& \dots & {\delta }_1^{(l)}{h}_N^{(l-1)}\\ {\delta }_2^{(l)}{h}_1^{(l-1)}& {\delta }_2^{(l)}{h}_2^{(l-1)}& \dots & {\delta }_2^{(l)}{h}_N^{(l-1)}\\ ⋮& ⋮& \ddots & ⋮\\ {\delta }_M^{(l)}{h}_1^{(l-1)}& {\delta }_M^{(l)}{h}_2^{(l-1)}& \dots & {\delta }_M^{(l)}{h}_N^{(l-1)}\end{array}\right]\\ & =\left[\begin{array}{c}{\delta }_1^{(l)}\\ {\delta }_2^{(l)}\\ ⋮\\ {\delta }_M^{(l)}\end{array}\right]\left[\begin{array}{cccc}{h}_1^{(l-1)}& h_2^{(l-1)}& \dots & h_N^{(l-1)}\end{array}\right]\\ \Rightarrow \frac{\partial Cost}{\partial w^{(l)}}& ={\delta }^{(l)}\cdot (h^{(l-1)}{)}^T\end{array}$$

综上，可以得到BP四项基本原则的矩阵形式：

$$\begin{array}{rl}{\delta }^{(L)}& ={▽}_{y}Cost\odot {\sigma }^{\prime }(logi{t}^{(L)})\\ {\delta }^{(l)}& =[(W^{(L+1)}{)}^T{\delta }^{(l+1)}]\odot {\sigma }^{\prime }(logi{t}^{(l)})\\ \frac{\partial Cost}{\partial bia{s}^{(l)}}& ={\delta }^{(l)}\\ \frac{\partial Cost}{\partial w^{(l)}}& ={\delta }^{(l)}\cdot (h^{(l-1)}{)}^T\end{array}$$

卷积网络反向传播公式

卷积神经网络反向传播理论推导：http://www.uml.org.cn/ai/201809102.asp?artid=21154

卷积神经网络(CNN)反向传播算法：https://www.cnblogs.com/pinard/p/6494810.html