神经网络常见激活函数求导

1.求导公式

2.sigmoid函数

• 原函数
  $$\begin{array}{rl}sigmoid(x)& =\frac{1}{1+e^{-x}}\end{array}$$
• 求导过程
  $$\begin{array}{rl}sigmoi{d}^{\prime }(x)& =(\frac{1}{1+e^{-x}}{)}^{\prime }\\ & =\frac{0-(1+e^{-x}{)}^{\prime }}{(1+e^{-x}{)}^2}\\ & =\frac{e^{-x}}{(1+e^{-x}{)}^2}\\ & =\frac{1+e^{-x}-1}{(1+e^{-x})(1+e^{-x})}\\ & =\frac{1+e^{-x}-1}{(1+e^{-x})}.\frac{1}{(1+e^{-x})}\\ & =1-\frac{1}{(1+e^{-x})}.\frac{1}{(1+e^{-x})}\\ & =1-sigmoi{d}^2(x)\end{array}$$

3.Tanh函数

• 原函数
  $$\begin{array}{rl}Tanh(x)& =\frac{e^x-e^{-x}}{e^x+e^{-x}}\end{array}$$
• 求导过程
  $$\begin{array}{rl}Tan{h}^{\prime }(x)& =\frac{e^x-e^{-x}}{e^x+e^{-x}}{)}^{\prime }\\ & =\frac{(e^x-e^{-x}{)}^{\prime }(e^x+e^{-x})-(e^x-e^{-x})(e^x+e^{-x}{)}^{\prime }}{(e^x+e^{-x}{)}^2}\\ & =\frac{(e^x+e^{-x})(e^x+e^{-x})-(e^x-e^{-x})(e^x-e^{-x})}{(e^x+e^{-x}{)}^2}\\ & =\frac{(e^x+e^{-x}{)}^2-(e^x-e^{-x}{)}^2}{(e^x+e^{-x}{)}^2}\\ & =1-\frac{(e^x-e^{-x}{)}^2}{(e^x+e^{-x}{)}^2}\\ & =1-(\frac{e^x-e^{-x}}{e^x+e^{-x}}{)}^2\\ & =1-Tan{h}^2(x)\end{array}$$

4.ReLU函数

• 原函数
  $$\begin{array}{r}Relu(x）=\{\begin{array}{ll}x& x\ge 0\\ 0& x<0\end{array}\end{array}$$
• 求导
  $$Rel{u}^{\prime }(x)=\{\begin{array}{ll}1& x\ge 0\\ 0& x<0\end{array}$$