Softmax回归交叉熵损失函数求导

softmax函数的表达式：$a_i=\frac{e^{z_i}}{{\sum }_k{e}^{z_k}}$

交叉熵 损失函数：$C=-{\sum }_i{y}_i\ln a_i$

根据复合函数求导法则：$\frac{\partial C}{\partial z_i}={\sum }_j\left(\frac{\partial C_j}{\partial a_j}\frac{\partial a_j}{\partial z_i}\right)$

计算前面一项：$\frac{\partial C_j}{\partial a_j}=\frac{\partial \left(-y_j\ln a_j\right)}{\partial a_j}=-y_j\frac{1}{a_j}$

计算后面一项：

• 如果$i=j$：
  $$\frac{\partial a_i}{\partial z_i}=\frac{\partial \left(\frac{e^{z_i}}{{\sum }_k{e}^{z_k}}\right)}{\partial z_i}=\frac{{\sum }_k{e}^{z_k}{e}^{z_i}-{\left(e^{z_i}\right)}^2}{{\left({\sum }_k{e}^{z_k}\right)}^2}=\left(\frac{e^{z_i}}{{\sum }_k{e}^{z_k}}\right)\left(1-\frac{e^{z_i}}{{\sum }_k{e}^{z_k}}\right)=a_i\left(1-a_i\right)$$

• 如果$i\ne j$：
  $$\frac{\partial a_j}{\partial z_i}=\frac{\partial \left(\frac{e^{ij}}{\sum t^j{e}^k}\right)}{\partial z_i}=-e^{z_j}{\left(\frac{1}{{\sum }_k{e}^{z_k}}\right)}^2{e}^{z_i}=-a_i{a}_j$$

组合起来得到：
$$\begin{array}{l}\frac{\partial C}{\partial z_i}={\sum }_j\left(\frac{\partial C_j}{\partial a_j}\frac{\partial a_j}{\partial z_i}\right)={\sum }_{j\ne i}\left(\frac{\partial C_j}{\partial a_j}\frac{\partial a_j}{\partial z_i}\right)+{\sum }_{i=j}\left(\frac{\partial C_j}{\partial a_j}\frac{\partial a_j}{\partial z_i}\right)\\ ={\sum }_{j\ne i}-y_j\frac{1}{a_j}\left(-a_i{a}_j\right)+\left(-y_i\frac{1}{a_i}\right)\left(a_i\left(1-a_i\right)\right)\\ ={\sum }_{j\ne i}{a}_i{y}_j+\left(-y_i\left(1-a_i\right)\right)\\ ={\sum }_{j\ne i}{a}_i{y}_j+a_i{y}_i-y_i\\ =a_i{\sum }_j{y}_j-y_i\end{array}$$
对于分类问题，这个梯度等于：
$$\frac{\partial C}{\partial z_i}=a_i-y_i$$