学习笔记 NLP里的RNN和LSTM

1. Recurrent neural network

1.1 Elman network

$$h_t={\sigma }_h\left(W_h{x}_t+U_h{h}_{t-1}+b_h\right)$$

$$y_t={\sigma }_y\left(W_y{h}_t+b_y\right)$$

1.2 Jordan network

$$h_t={\sigma }_h\left(W_h{x}_t+U_h{y}_{t-1}+b_h\right)$$

$$y_t={\sigma }_y\left(W_y{h}_t+b_y\right)$$

Variables and functions

• $x_t$ : input vector
• $h_t$ : hidden layer vector
• $y_t$ : output vector
• $W,U$ and $b$ : parameter matrices and vector
• ${\sigma }_h$ and ${\sigma }_y$ : Activation functions

1.3 Bidirectional RNN

2. Long short-term memory

2.1 LSTM with a forget gate

The compact forms of the equations for the forward pass of an LSTM cell with a forget gate are:
$$\begin{array}{rl}{f}_t& ={\sigma }_g\left(W_f{x}_t+U_f{h}_{t-1}+b_f\right)\\ i_t& ={\sigma }_g\left(W_i{x}_t+U_i{h}_{t-1}+b_i\right)\\ o_t& ={\sigma }_g\left(W_o{x}_t+U_o{h}_{t-1}+b_o\right)\\ {\tilde{c}}_t& ={\sigma }_c\left(W_c{x}_t+U_c{h}_{t-1}+b_c\right)\\ c_t& =f_t\circ c_{t-1}+i_t\circ {\tilde{c}}_t\\ h_t& =o_t\circ {\sigma }_h\left(c_t\right)\end{array}$$
where the initial values are $c_0=0$ and $h_0=0$ and the operator o denotes the Hadamard product (element-wise product). The subscript $t$ indexes the time step.
Variables

• $x_t\in {\mathbb{R}}^d$ : input vector to the LSTM unit
• $f_t\in (0,1{)}^h$ : forget gate’s activation vector
• $i_t\in (0,1{)}^h:$ input/update gate’s activation vector
• $o_t\in (0,1{)}^h$ : output gate’s activation vector
• $h_t\in (-1,1{)}^h$ : hidden state vector also known as output vector of the LSTM unit
• ${\tilde{c}}_t\in (-1,1{)}^h:$ cell input activation vector
• $c_t\in {\mathbb{R}}^h$ : cell state vector
• $W\in {\mathbb{R}}^{h\times d},U\in {\mathbb{R}}^{h\times h}$ and $b\in {\mathbb{R}}^h$ : weight matrices and bias vector parameters which need to be learned during training where the superscripts $d$ and $h$ refer to the number of input features and number of hidden units, respectively.

2.2 Peephole LSTM

$$\begin{array}{rl}{f}_t& ={\sigma }_g\left(W_f{x}_t+U_f{c}_{t-1}+b_f\right)\\ i_t& ={\sigma }_g\left(W_i{x}_t+U_i{c}_{t-1}+b_i\right)\\ o_t& ={\sigma }_g\left(W_o{x}_t+U_o{c}_{t-1}+b_o\right)\\ c_t& =f_t\circ c_{t-1}+i_t\circ {\sigma }_c\left(W_c{x}_t+b_c\right)\\ h_t& =o_t\circ {\sigma }_h\left(c_t\right)\end{array}$$

3. training RNN

3.1 Problem

RNN: The error surface is either very flat or very steep → 梯度消失/爆炸 Gradient Vanishing/Exploding

3.2 Techniques

• Clipping the gradients
• Advanced optimization technology
  • NAG
  • RMSprop
• Try LSTM (or other simpler variants)
  • Can deal with gradient vanishing (not gradient explode)
  • Memory and input are added (在RNN中，对于每一个输入，memory会重置)
  • The influence never disappears unless forget gate is closed (No Gradient vanishing, if forget gate is opened.)
• Better initialization
  • Vanilla RNN Initialized with Identity matrix + ReLU activation function [Quoc V. Le, arXiv’15]

参考资料

[1] Recurrent neural network - Wikipedia

[2] Long short-term memory - Wikipedia

[3] Bidirectional Recurrent Neural Networks - Dive into Deep …

[4] 机器学习 李宏毅