关于softmax，cross entropy，三层全连接的导数计算以及反向传播

• 在本文中，我们主要介绍softmax，softmax+crossentropy，三层全连接的导数计算和反向传播

softmax

• 定义：$S(a_i)=\frac{e^{a_i}}{{\sum }_{j=1}^N{e}^{a_j}}$
• 倒数计算过程（令$S_i$表示$S(a_i)$）：
  $$\begin{gathered} ifi==k： \\ \frac{\partial S_i}{\partial a_k} \\ =\frac{e^{a_i}}{{\sum }_{j=1}^N{e}^{a_j}}+\frac{e^{a_i}\ast -1\ast e^{a_i}}{({\sum }_{j=1}^N{e}^{a_j}{)}^2} \\ =\frac{e^{a_i}}{{\sum }_{j=1}^N{e}^{a_j}}\ast (1-\frac{e^{a_i}}{{\sum }_{j=1}^N{e}^{a_j}}) \\ =S_i(1-S_i) \\ ifi!=k： \\ \frac{\partial S_i}{\partial a_k} \\ =\frac{e^{a_i}\ast -1\ast e^{a_k}}{({\sum }_{j=1}^N{e}^{a_j}{)}^2} \\ =-S_i\ast S_k \end{gathered}$$

Softmax+CrossEntropy

• 定义：$L_{ce}={\sum }_{i=1}^N-y_i\ast log(S_i)$
• 导数计算过程：
  $$\begin{gathered} \frac{\partial L_{ce}}{\partial a_k}=\sum _{i=1}^N-y_i\ast \frac{\partial log(S_i)}{\partial a_k} \\ =-y_i\ast \frac{1}{S_i}\ast S_i\ast (1-S_i)-\sum _{i!=k}{y}_i\ast \frac{1}{S_i}\ast (-S_i\ast S_k) \\ =-y_i\ast (1-S_i)+\sum _{i!=k}{y}_i\ast S_k \\ =-y_i+\sum _{i=1}^N{y}_i\ast S_k \\ =S_k-y_i \end{gathered}$$
• 物理意义：softmax 和 cross entropy组合，对softmax前的第i维度的导数就是对应softmax后第i维度和ground truth 的差值

三层全联接求导

• 网络结构如下所示
  
• 我们首先定义如下符合
  • $L=\frac{1}{2}\ast |y-z{|}^2$
  • $z_k=f({\sum }_{j=1}^h{w}_{kj}{y}_j)$, f is the non-linear function, such as softmax
  • $ne{t}_k={\sum }_{j=1}^h{w}_{kj}{y}_j$
  • $y_j=f({\sum }_{i=1}^d{w}_{ji}{x}_i)$
  • $ne{t}_j={\sum }_{i=1}^d{w}_{ji}{x}_i$
  • $x_i$ represents the input value on the i-th dimension.
• 接下来我们分别计算对$w_{kj}and{w}_{ji}$的导数
  • derivation for $w_{kj}$
    $$\begin{gathered} \frac{\partial L}{\partial w_{kj}}=(y_k-z_k)\ast -1\ast \frac{\partial z_k}{\partial w_{kj}} \\ =-1\ast (y_k-z_k)\ast \frac{\partial z_k}{\partial ne{t}_k}\ast \frac{\partial ne{t}_k}{\partial w_{kj}} \\ =-1\ast (y_k-z_k)\ast f^{\prime }(ne{t}_k)\ast y_j \\ \therefore w_{kj}=w_{kj}-\alpha \Delta w_{kj} \\ =w_{kj}+(y_k-z_k)\ast f^{\prime }(ne{t}_k)\ast y_j \end{gathered}$$
  • derivation for $w_{j}i$
    $$\begin{gathered} \frac{\partial L}{\partial w_{ji}}=\frac{\partial L}{\partial y_j}\ast \frac{\partial y_j}{\partial ne{t}_j}\ast \frac{\partial ne{t}_j}{\partial w_{ji}} \\ \frac{\partial L}{\partial y_j}=\sum _{k=1}^h\frac{\partial L}{\partial ne{t}_k}\ast \frac{\partial ne{t}_k}{\partial y_j} \\ =\sum _{k=1}^h-(y_k-z_k)\ast f^{\prime }(ne{t}_k)\ast w_{kj} \\ \therefore \\ \frac{\partial L}{\partial w_{ji}}=\sum _{k=1}^h-(y_k-z_k)\ast f^{\prime }(ne{t}_k)\ast w_{kj}\ast f^{\prime }(ne{t}_j)\ast x_i \end{gathered}$$
  • 至此， 我们已经计算了这两层里面所有的参数的梯度，然后我们就可以通过反向传播去计算梯度啦。
  • 至于$f^{\prime }(ne{t}_k)$的计算参照softmax梯度的计算方法。$f^{\prime }(ne{t}_k)=ne{t}_k\ast (1-ne{t}_k)$