Softmax Cross Entropy 梯度推导

Softmax的梯度

Softmax定义：
$$p_i=\frac{e^{a_i}}{{\sum }_{k=1}^N{e}^{a_k}},i\in N$$
数值稳定的Softmax：
$$\frac{e^{a_i}}{{\sum }_{k=1}^N{e}^{a_k}}=\frac{C{e}^{a_i}}{C{\sum }_{k=1}^N{e}^{a_k}}=\frac{e^{a_i+\log C}}{{\sum }_{k=1}^N{e}^{a_k+\log C}},i\in N$$
其中$\log C=-\max (\mathbf{a})$

对于向量函数Softmax，第i个输出相对于第j个输入的偏导数可以定义如下：
$$\frac{\partial p_i}{\partial a_j}=\frac{\partial \frac{e^{a_i}}{{\sum }_{k=1}^N{e}^{a_k}}}{\partial a_j}$$
那么根据商的求导法则：
$$f(x)=\frac{g(x)}{h(x)},f^{\prime }(x)=\frac{g^{\prime }(x)h(x)-h^{\prime }(x)g(x)}{(h(x){)}^2}$$
在这里，$g_i=e^{a_i},h_i={\sum }_{k=1}^N{e}^{a_k}$
因此：
$$\frac{\partial g_i}{\partial a_j}=\frac{\partial e^{a_i}}{\partial a_j}=\{\begin{array}{l}0,i̸=j\\ e^{a_i},i=j\end{array}$$
$$\frac{\partial h_i}{\partial a_j}=\frac{\partial {\sum }_{k=1}^N{e}^{a_k}}{\partial a_j}=e^{a_j}$$
所以，当$i̸=j$：
$$\begin{array}{rl}\frac{\partial p_i}{\partial a_j}& =\frac{0{\sum }_{k=1}^N{e}^{a_k}-e^{a_j}{e}^{a_i}}{({\sum }_{k=1}^N{e}^{a_k}{)}^2}\\ & =-\frac{e^{a_j}}{{\sum }_{k=1}^N{e}^{a_k}}\times \frac{e^{a_i}}{{\sum }_{k=1}^N{e}^{a_k}}\\ & =-p_j{p}_i\end{array}$$
而当$i=j$：
$$\begin{array}{rl}\frac{\partial p_i}{\partial a_j}& =\frac{e^{a_i}{\sum }_{k=1}^N{e}^{a_k}-e^{a_j}{e}^{a_i}}{({\sum }_{k=1}^N{e}^{a_k}{)}^2}\\ & =\frac{e^{a_i}}{{\sum }_{k=1}^N{e}^{a_k}}\times \frac{({\sum }_{k=1}^N{e}^{a_k}-e^{a_j})}{{\sum }_{k=1}^N{e}^{a_k}}\\ & =p_i(1-p_j)\end{array}$$
总结一下，就是：
$$\frac{\partial p_i}{\partial a_j}=\{\begin{array}{l}-p_j{p}_i,i̸=j\\ p_i(1-p_j),i=j\end{array}$$

Softmax交叉熵的梯度

交叉熵定义为：
$$H(p,q)=-\sum _{x}p(x)\log q(x)$$
在这里，可以改写为：
$$H(y,p)=-\sum _{i=1}^N{y}_i\log (p_i)=-log(p_k)$$
其中$k$是label；那么交叉熵针对第j个输入的偏导数是：
$$\begin{array}{rl}\frac{\partial L}{\partial o_j}& =-\frac{{\sum }_{i=1}^N{y}_i\log (p_i)}{\partial o_j}\\ & =-\sum _{i=1}^N{y}_i\frac{\partial \log p_i}{\partial p_i}\frac{\partial p_i}{\partial o_j}\\ & =-\sum _{i=1}^N{y}_i\frac{1}{p_i}\frac{\partial p_i}{\partial o_j}\end{array}$$
现在，把之前推导的Softmax的梯度代入$p_i$：
$$\begin{array}{rl}\frac{\partial L}{\partial o_j}& =-\sum _{i=1}^N{y}_i\frac{1}{p_i}\frac{\partial p_i}{\partial o_j}\\ & =-y_j\frac{1}{p_j}(p_j(1-p_j))-\sum _{i̸=j}^N{y}_i\frac{1}{p_i}(-p_j{p}_i))\\ & =-y_j(1-p_j)+\sum _{i̸=j}^N{y}_i{p}_j\\ & =y_j{p}_j+\sum _{i̸=j}^N{y}_i{p}_j-y_j\\ & =\sum _{i=1}^N{y}_i{p}_j-y_j\\ & =p_j-y_j(\text{since}\sum _{i=1}^N{y}_i=1)\end{array}$$
结束。

引用

[1]softmax-crossentropy
[2]the-softmax-function-and-its-derivative