LSTM缓解梯度消失的原因

LSTM缓解梯度消失的原因_第1张图片

\(c_{t}=c_{t-1} \otimes \sigma\left(W_{f} \cdot\left[H_{t-1}, X_{t}\right]\right) \oplus \tanh \left(W_{c} \cdot\left[H_{t-1}, X_{t}\right]\right) \otimes \sigma\left(W_{i} \cdot\left[H_{t-1}, X_{t}\right]\right)\)

反向传播公式:

\(\begin{aligned} \frac{\partial E_{k}}{\partial W}=& \frac{\partial E_{k}}{\partial H_{k}} \frac{\partial H_{k}}{\partial C_{k}} \frac{\partial C_{k}}{\partial C_{k-1}} \ldots \frac{\partial C_{2}}{\partial C_{1}} \frac{\partial C_{1}}{\partial W}=\\ & \frac{\partial E_{k}}{\partial H_{k}} \frac{\partial H_{k}}{\partial C_{k}}\left(\prod_{t=2}^{k} \frac{\partial C_{t}}{\partial C_{t-1}}\right) \frac{\partial C_{1}}{\partial W} \end{aligned}\)

括号中的部分是累乘项:

$\frac{\partial c_{t}}{\partial c_{t-1}}=\sigma\left(W_{f} \cdot\left[H_{t-1}, X_{t}\right]\right) + $
\(\frac{d}{d \mathcal{C}_{t-1}}\left(\tanh \left(W_{c} \cdot\left[H_{t-1}, X_{t}\right]\right) \otimes \sigma\left(W_{i} \cdot\left[H_{t-1}, X_{t}\right]\right)\right)\)

也就是说,这里的累乘单元是两项和形式,其中前部分是遗忘门的值。遗忘门决定了上一个细胞状态的保留比例,其取值可以接近于1,也就是说可以把遗忘门看成:\(\sigma\left(W_{f} \cdot\left[H_{t-1}, X_{t}\right]\right) \approx \overrightarrow{1}\),所以LSTM中:

\(\frac{\partial E_{k}}{\partial W} \approx \frac{\partial E_{k}}{\partial H_{k}} \frac{\partial H_{k}}{\partial c_{k}}\left(\Pi_{t=2}^{k} \sigma\left(W_{f} \cdot\left[H_{t-1}, X_{t}\right]\right)\right) \frac{\partial C_{1}}{\partial w} \nrightarrow 0\)

所以,LSTM能缓解梯度消失。

转载于:https://www.cnblogs.com/Elaine-DWL/p/11240213.html

你可能感兴趣的:(人工智能)