反向传播算法推导过程（非常详细）

1. 前向传播

假设$X$为$N\times m$的矩阵（其中，$N$为样本个数（batch size），$m$为特征维数）

$h_1$与$Z_1$的维数为$m_1\to W_1$为$m\times m_1$的矩阵，$b_1\in {\mathbb{R}}^{m_1},$
$h_2$与$Z_2$的维数为$m_2\to W_2$为$m_1\times m_2$的矩阵，$b_2\in {\mathbb{R}}^{m_2},$
$⋮$
$h_L$与$Z_L$的维数为$m_L\to W_L$为$m_{L-1}\times m_L$的矩阵，$b_L\in {\mathbb{R}}^{m_L}$

前向算法：

$$\begin{array}{l}{h}_1=x{W}_1+{\tilde{b}}_1,Z_1=f_1\left(h_1\right),{\tilde{b}}_1为b_1^T沿着行方向扩展成N行\\ h_2=Z_1{W}_2+{\tilde{b}}_2,Z_2=f_2\left(h_2\right)\\ ⋮\\ h_L=Z_{L-1}{W}_L+{\tilde{b}}_L,Z_L=f_L\left(h_L\right)\\ \text{ out }=Z_L{W}_{L+1}+{\tilde{b}}_{L+1}\end{array}$$
假设输出为$n$维，则$out$为大小为$N\times n$的矩阵，根据MSE或CE准则可以求得$\frac{\partial J}{\partial out}$，对于回归问题与分类问题，$\frac{\partial J}{\partial out}$的求解方法如下：

回归问题

分类问题

• 对于回归问题，对out直接计算损失，损失函数为MSE。 损失：$J=\frac{1}{2N}{\sum }_{i=1}^N||y_i-\widetilde{y_i}|{|}^2$
  $$\begin{array}{rl}\frac{\partial J}{\partial y_i}& =\frac{1}{2N}\sum _{i=1}^N(y_i-\widetilde{y_i})\times 2\\ & =\frac{1}{N}\sum _{i=1}^N(y_i-\widetilde{y_i})\end{array}$$
• 对于分类问题，out后接softmax进行分类，然后使用CE(cross entropy)计算loss.$S_k=\frac{e^{y_k}}{{\sum }_{i=1}^n{e}^{y_i}}$一个样本对应的网络的输出$S(s_1,s_2,...,s_n)$是一个概率分布，而这个样本的标注$\tilde{S}$一般为$(0,0,...,1,0,0,...,0)$，也可以看做一个概率分布（硬分布）。cross entropy可以看成是$S$与$\tilde{S}$之间的KL距离：
  $$D(\tilde{S}||S)=\Sigma \tilde{S}\log \frac{\tilde{S}}{S}$$
  • 假设$\tilde{S}=(0,0,...,1,0,0,...,0)$，其中1为第$k$个元素(索引从0开始)，令$S=(s_0,s_1,...,s_k,...,s_{n-1})$.
    损失：
    $$\begin{array}{rl}J=D(\tilde{S}||S)& =1\times \log \frac{1}{s_k}\\ & =-\log s_k(CE损失函数,可看做目标类别概率最大)\\ & =-\log \frac{e^{y_k}}{{\sum }_{i=0}^{n-1}{e}^{y_i}}\end{array}$$
    $$\begin{array}{rl}& \frac{\partial J}{\partial y_m}=\frac{\partial J}{\partial y_m}(\log \sum _{i=0}^{n-1}{e}^{y_i}-y_k)=\frac{e^{y_m}}{{\sum }_{i=0}^{n-1}{e}^{y_i}}-\delta (m=k)=s_m-\delta (m=k)\\ & 写成向量形式为:\frac{\partial J}{\partial y}=S-\tilde{S}\end{array}$$

KL距离（相对熵）：是Kullback-Leibler Divergence的简称，也叫相对熵(Relative Entropy).它衡量的是相同事件空间里的两个概率分布的差异情况。其物理意义是：在相同事件空间里，概率分布$P(x)$对应的每个事件，若用概率分布$Q(x)$编码时，平均每个基本事件(符号)编码长度增加了多少比特。我们用$D(P||Q)$表示KL距离，计算公式如下：
$$D(P||Q)=\sum _{x\in X}P(x)\log \frac{P(x)}{Q(x)}$$
当两个概率分布完全相同时，即$P(X)=Q(X)$,其相对熵为0.

2.反向传播

$\text{ out }=Z_L{W}_{L+1}+{\tilde{b}}_{L+1}$，为了便于详细说明反向传播算法，假设$Z_L$为$2\times 3$的向量，$W_{L+1}$为$3\times 2$的向量：
$$\begin{array}{l}{Z}_L={\left(\begin{array}{ccc}{z}_{11}& z_{12}& z_{13}\\ z_{21}& z_{22}& z_{23}\end{array}\right)}_{2\times 3},W_{L+1}={\left(\begin{array}{cc}{w}_{11}& w_{12}\\ w_{21}& w_{22}\\ w_{31}& w_{32}\end{array}\right)}_{3\times 2}{\tilde{b}}_{L+1}={\left(\begin{array}{cc}{b}_1& b_2\\ b_1& b_2\end{array}\right)}_{2\times 2},\text{ out }=\left(\begin{array}{cc}{o}_{11}& o_{12}\\ o_{21}& o_{22}\end{array}\right)\\ \Rightarrow Z_L{W}_{L+1}+{\tilde{b}}_{L+1}=\left(\begin{array}{cc}{z}_{11}{w}_{11}+z_{12}{w}_{21}+z_{13}{w}_{31}+b_1& z_{11}{w}_{12}+z_{12}{w}_{22}+z_{13}{w}_{32}+b_2\\ z_{21}{w}_{11}+z_{22}{w}_{21}+z_{23}{w}_{31}+b_1& z_{21}{w}_{12}+z_{22}{w}_{22}+z_{23}{w}_{32}+b_2\end{array}\right)=\text{out}.\end{array}$$
所以，
$$\begin{array}{l}{o}_{11}=z_{11}{w}_{11}+z_{12}{w}_{21}+z_{13}{w}_{31}+b_1\\ o_{12}=z_{11}{w}_{12}+z_{12}{w}_{22}+z_{13}{w}_{32}+b_2\\ o_{21}=z_{21}{w}_{11}+z_{22}{w}_{21}+z_{23}{w}_{31}+b_1\\ o_{22}=z_{21}{w}_{12}+z_{22}{w}_{22}+z_{23}{w}_{32}+b_2\end{array}$$

1) 损失$J$对$W$的导数：

$$\begin{array}{rl}\frac{\partial J}{\partial w_{11}}& =\frac{\partial J}{\partial o_{11}}{z}_{11}+\frac{\partial J}{\partial o_{21}}{z}_{21},\frac{\partial J}{\partial w_{12}}=\frac{\partial J}{\partial o_{12}}{z}_{11}+\frac{\partial J}{\partial o_{22}}{z}_{21}\\ \frac{\partial J}{\partial w_{21}}& =\frac{\partial J}{\partial o_{11}}{z}_{12}+\frac{\partial J}{\partial o_{21}}{z}_{22},\frac{\partial J}{\partial w_{22}}=\frac{\partial J}{\partial o_{12}}{z}_{12}+\frac{\partial J}{\partial o_{22}}{z}_{22}\\ \frac{\partial J}{\partial w_{31}}& =\frac{\partial J}{\partial o_{11}}{z}_{13}+\frac{\partial J}{\partial o_{21}}{z}_{23},\frac{\partial J}{\partial w_{32}}=\frac{\partial J}{\partial o_{12}}{z}_{13}+\frac{\partial J}{\partial o_{22}}{z}_{23}\end{array}$$
$$\Rightarrow \left(\begin{array}{cc}\frac{\partial J}{\partial w_{11}}& \frac{\partial J}{\partial w_{12}}\\ \frac{\partial J}{\partial w_{21}}& \frac{\partial J}{\partial w_{22}}\\ \frac{\partial J}{\partial w_{31}}& \frac{\partial J}{\partial w_{32}}\end{array}\right)=\left(\begin{array}{cc}{z}_{11}& z_{21}\\ z_{12}& z_{22}\\ z_{13}& z_{23}\end{array}\right)\left(\begin{array}{cc}\frac{\partial J}{\partial o_{11}}& \frac{\partial J}{\partial o_{12}}\\ \frac{\partial J}{\partial o_{21}}& \frac{\partial J}{\partial o_{22}}\end{array}\right)$$
即，$\frac{\partial J}{\partial W_{L+1}}=Z_L^T\frac{\partial J}{\partial out}$

2) 损失对偏置b的导数等于将$\frac{\partial J}{\partial out}$的每一列加起来：

$$\{\begin{array}{l}\frac{\partial J}{\partial b_1}=\frac{\partial J}{\partial o_{11}}+\frac{\partial J}{\partial o_{21}}\\ \frac{\partial J}{\partial b_2}=\frac{\partial J}{\partial o_{12}}+\frac{\partial J}{\partial o_{22}}\end{array}\Rightarrow {\left(\frac{\partial J}{\partial b}\right)}^T=\left(\frac{\partial J}{\partial b_1}\frac{\partial J}{\partial b_2}\right)=\left(\frac{\partial J}{\partial o_{11}}+\frac{\partial J}{\partial o_{21}}\frac{\partial J}{\partial o_{12}}+\frac{\partial J}{\partial o_{22}}\right)$$

3) 损失$J$对$Z$的导数：

$$\begin{array}{rl}\frac{\partial J}{\partial z_{11}}& =\frac{\partial J}{\partial o_{11}}{w}_{11}+\frac{\partial J}{\partial o_{12}}{w}_{12};\frac{\partial J}{\partial z_{12}}=\frac{\partial J}{\partial o_{11}}{w}_{21}+\frac{\partial J}{\partial o_{12}}{w}_{22};\frac{\partial J}{\partial z_{13}}=\frac{\partial J}{\partial o_{11}}{w}_{31}+\frac{\partial J}{\partial o_{12}}{w}_{32}\\ \frac{\partial J}{\partial z_{21}}& =\frac{\partial J}{\partial o_{21}}{w}_{11}+\frac{\partial J}{\partial o_{22}}{w}_{12};\frac{\partial J}{\partial z_{22}}=\frac{\partial J}{\partial o_{21}}{w}_{21}+\frac{\partial J}{\partial o_{12}}{w}_{22};\frac{\partial J}{\partial z_{23}}=\frac{\partial J}{\partial o_{21}}{w}_{31}+\frac{\partial J}{\partial o_{22}}{w}_{32}\end{array}$$
即，
$$\left(\begin{array}{ccc}\frac{\partial J}{\partial z_{11}}& \frac{\partial J}{\partial z_{12}}& \frac{\partial J}{\partial z_{13}}\\ \frac{\partial J}{\partial z_{21}}& \frac{\partial J}{\partial z_{22}}& \frac{\partial J}{\partial z_{23}}\end{array}\right)=\left(\begin{array}{cc}\frac{\partial J}{\partial o_{11}}& \frac{\partial J}{\partial o_{12}}\\ \frac{\partial J}{\partial {\theta }_{21}}& \frac{\partial J}{\partial o_{22}}\end{array}\right)\left(\begin{array}{ccc}{w}_{11}& w_{21}& w_{31}\\ w_{12}& w_{22}& w_{32}\end{array}\right)$$
$\Rightarrow \frac{\partial J}{\partial Z_L}=\frac{\partial J}{\partial out}{W}_{L+1}^T$

4) 损失$J$对$h$的导数：

$$Z_L=f_L(h_L)$$

• $f_L$为sigmoid时，$Z_L=\frac{1}{1+e^{-h_L}}$.
  $$\begin{array}{l}\frac{\partial J}{\partial h_L}=\frac{\partial J}{\partial Z_L}\frac{d{z}_L}{d{h}_L}=\frac{\partial J}{\partial Z_L}\frac{e^{-hL}}{{\left(1+e^{-h_L}\right)}^2}=\frac{\partial J}{\partial Z_L}\frac{1}{1+e^{-h_L}}\frac{e^{-h_L}}{1+e^{-h_L}}\\ =\frac{\partial J}{\partial Z_L}{Z}_L\left(1-Z_L\right)\end{array}$$
• $f_L$为tanh时，$Z_L=\frac{e^{h_L}-e^{-h_L}}{e^{h_L}+e^{-h_L}}$.
  $$\begin{array}{l}\frac{\partial J}{\partial h_L}=\frac{\partial J}{\partial Z_L}\frac{d{Z}_L}{d{h}_L}=\frac{\partial J}{\partial Z_L}\frac{4}{{\left(e^{h_L}+e^{-h_L}\right)}^2}=\frac{\partial J}{\partial Z_L}\left[1-{\left(\frac{e^{h_L}-e^{-h_L}}{e^{h_L}+e^{-h_L}}\right)}^2\right]\\ =\frac{\partial J}{\partial z_L}\left[1-z_L^2\right]\end{array}$$
• $f_L$为relu时，$Z_L=relu(h_L)=\{\begin{array}{cc}0,& h_L\le 0\\ h_L,& h_L>0\end{array}$.
  $$\begin{array}{l}\frac{\partial J}{\partial h_L}=\frac{\partial J}{\partial Z_L}\frac{\partial Z_L}{\partial h_L}=\{\begin{array}{cc}0,& h_L\le 0\\ \frac{\partial J}{\partial Z_L},& h_L>0\end{array}\end{array}$$

3. 梯度更新

对于不同算法 ，梯度更新方式如下：
$$\frac{\partial J}{\partial out}\Rightarrow \{\begin{array}{c}\frac{\partial J}{\partial W_{L+1}}=Z_L^T\frac{\partial J}{\partial out}\\ \frac{\partial J}{\partial Z_L}=\frac{\partial J}{\partial out}{W}_{L+1}^T\\ {\left(\frac{\partial J}{\partial b}\right)}^T=SumCol(\frac{\partial J}{\partial out})\\ W_{L+1}^{t+1}=W_{L+1}^t-\eta \frac{\partial J}{\partial W_{L+1}}\\ b_{L+1}^{t+1}=b_{L+1}^t-\eta \frac{\partial J}{\partial b_{L+1}}\end{array}\Rightarrow \frac{\partial J}{\partial h_L}=\frac{\partial J}{\partial Z_L}\frac{\partial Z_L}{\partial h_L}\Rightarrow \{\begin{array}{c}\frac{\partial J}{\partial W_L}=Z_{L-1}^T\frac{\partial J}{\partial h_L}\\ \frac{\partial J}{\partial Z_{L-1}}=\frac{\partial J}{\partial h_L}{W}_L^T\\ ⋮\\ ⋮\end{array}\Rightarrow \cdots$$
参考：深度之眼公众号