关于MSELoss、BCELoss、CELoss损失函数求导的推导

1、MSELoss求导

• y:   ~  真实值
• p:   ~  预测值
  $$\begin{gathered} MS{E}_{loss}=\frac{1}{2}\cdot \sum _{i=1}^n(y_i-p_i{)}^2 \\ {p}_i=x_i\cdot w+b \end{gathered}$$
  $$\frac{\delta loss}{\delta w}=2\cdot \frac{1}{2}\cdot \sum _{i=1}^n(y_i-p_i)\cdot (-1)\cdot x_i=\sum _{i=1}^n(p_i-y_i)\cdot x_i$$
  $$\frac{\delta loss}{\delta b}=2\cdot \frac{1}{2}\cdot \sum _{i=1}^n(y_i-p_i)\cdot (-1)=\sum _{i=1}^n(p_i-y_i)$$
• 注：线性回归损失函数的其中一种由来是对误差进行正态分布概率建模推导而来

2、BCELoss求导

$$\begin{gathered} BE{C}_{loss}=-\sum _{i=1}^n[y_i\cdot log(p_i)+(1-y_i)\cdot log(1-p_i)] \\ {p}_i=sigmoid(x_i)=\frac{1}{1+e^{-x_i}} \\ \frac{\delta p_i}{\delta x_i}=p_i\cdot (1-p_i) \\ \frac{\delta loss}{\delta x_i}=\frac{\delta loss}{\delta p_i}\cdot \frac{\delta p_i}{\delta x_i}=-\sum _{i=1}^n(y_i\cdot \frac{1}{p_i}+(1-p_i)\cdot \frac{1}{p_i-1})\cdot p_i\cdot (1-p_i)=\sum _{i=1}^n(p_i-y_i) \\ 所以：\frac{\delta loss}{\delta x_i}=\sum _{i=1}^n(p_i-y_i) \end{gathered}$$

• 注：二元交叉熵损失函数是通过伯努利0、1分布概率建模推导而来

3、CELoss求导

$$\begin{gathered} C{E}_{loss}=-\sum _i^n{y}_i\cdot log{p}_i \\ {p}_i=\frac{e^{z_i}}{{\sum }_{j=1}^k{e}^{z_j}}=softmax(z_i) \end{gathered}$$
考虑以两个神经元输出为例：$loss=-(y_1\cdot log{p}_1+y_2\cdot log{p}_2)$
$$y=(y_1,y_2)=(0,1),p=(p_1,p_2)=(\frac{e^{z_1}}{e^{z_1}+e^{z_2}},\frac{e^{z_2}}{e^{z_1}+e^{z_2}})$$
$$\frac{\delta loss}{\delta z_1}=\frac{\delta loss}{\delta p_1}\cdot \frac{\delta p_1}{\delta z_1}+\frac{\delta loss}{\delta p_2}\cdot \frac{\delta p_2}{\delta z_1}$$
$$\begin{gathered} =-(\frac{y_1}{p_1}\cdot \frac{e^{z_1}\sum -(e^{z_1}{)}^2}{(\sum {)}^2}+\frac{y_2}{p_2}\cdot \frac{0-e^{z_1}\cdot e^{z_2}}{(\sum {)}^2}) \\ =-(\frac{y_1}{p_1}\cdot (p_1-p_1^2)+\frac{y_2}{p_2}\cdot (-p_1\cdot p_2)) \\ =-(y_1-p_1(y_1+y_2)) \\ =-(y_1-p_1) \end{gathered}$$
同理：$$\frac{\delta loss}{\delta z_2}=-(y_2-p_2)$$
$$\begin{gathered} 所以最终\frac{\delta loss}{\delta z}=(p-y) \\ (这是向量形式，并且很巧合，和sigmoidloss对x求导的结果是一样的) \end{gathered}$$

• 注：多元交叉熵损失函数是通过信息熵相关理论推导而来