【Paper】GRU_LSTM introduce

链接：https://liudongdong1.github.io/2020/07/15/shi-jue-ai/model/gru-lstm/

0. RNN

$$\begin{gathered} h_t=f(x_t,h_{t-1}) \\ {h}_t:=tanh(W_{xh}{x}_t+W_{hh}{h}_{t-1}) \end{gathered}$$

• 计算目标：反向传播时，损失函数$l$ 对$t$ 时刻隐含状态向量$h_t$的偏导。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-qBCwLxYr-1595001102796)(C:/Users/dell/AppData/Roaming/Typora/typora-user-images/image-20200715153213869.png)]

1. GRU

GRU（Gate Recurrent Unit）是循环神经网络（Recurrent Neural Network, RNN）的一种。和LSTM（Long-Short Term Memory）一样，也是为了解决长期记忆和反向传播中的梯度等问题而提出来的。相比LSTM，使用GRU能够达到相当的效果，并且相比之下更容易进行训练，能够很大程度上提高训练效率，因此很多时候会更倾向于使用GRU。

【输入输出结构】

【内部结构】

• r： 控制重置门控；
• z： 为控制更新门控；门控信号越接近1，代表”记忆“下来的数据越多；而越接近0则代表”遗忘“的越多。

【更新表达式】
$$h^t=(1-z)\Theta h^{t-1}+z\Theta h^{\prime }$$

• $(1-z)\Theta h^{t-1}$：表示对原本隐藏状态的选择性“遗忘”。这里的 $1-z$可以想象成遗忘门（forget gate），忘记 $h^{t-1}$维度中一些不重要的信息。
• $z\Theta h^{t-1}$ ： 表示对包含当前节点信息的 $h^{\prime }$进行选择性”记忆“。与上面类似，这里的 $(1-z)$ 同理会忘记 $h^{\prime }$维度中的一些不重要的信息。或者，这里我们更应当看做是对 $h^{\prime }$维度中的某些信息进行选择。
• $h^t=(1-z)\Theta h^{t-1}+z\Theta h^{\prime }$ ：结合上述，这一步的操作就是忘记传递下来的 $h^{t-1}$ 中的某些维度信息，并加入当前节点输入的某些维度信息。

学习于：https://zhuanlan.zhihu.com/p/32481747

2. LSTM

3.LSTM_paper

Paper《Long Short-Term Memory RNN Architectures for Large Scale Acoustic Modeling》

Note:

1. first distribute training of LSTM RNNs using asynchronous stochastic gradient descent optimization on a large cluster of machine.
2. speech database TIMIT ，and the test it on a large vocabulary speech recognition task Google Voice Search Task.
3. 
4. 
5. 
6. how to calculate the total number of parameters and the computational complexity with a moderate number of inputs.
7. Eigen matrix library c++ 矩阵计算库

Paper《Convolutional,Long Short-Term Memory Fully Connected Deep Neural Networks》

Note:

1. ​ CNNs are good at reducing frequency variations, LSTMs are good at temporal modeling, and DNNs are appropriate for mapping features to a more separable space.
2. take advantage of the complementarity of CNNs,LSTMs,DNNs by combining them into one unified architecture,and proposed architecture CLDNN on a variety of large vocabulary tasks.[LVCSR]
3. previous paper train the three models separately and then the ouput were combined through a combination layer.In this we train in a unified structure.
4. 

Paper《Improved Semantic Representations from Tree-Structured LSTM》

Note: author: Kai Sheng Tai stanford.edu

1. models where real valued vectors are used to represent meaning fall into three class:1.Bag-of-Word 2.sequence models 3. tree structured modes

2. Test on two task: semantic relatedness prediction on sentence pairs sentiment classification of sentences drawn from movie reviews

3. available code: https://github.com/stanfordnlp/treelsm ,project : https://nlp.stanford.edu/projects/glove/

4. previous work:

   1. a problem with RNNs with transition functions of this form is that during training components of the gradient vector can gow or decay exponentially over long sequences.exploding or vanishing gradients make it difficult for the RNN to learn long-distance correlations in a sequence.
   2. Bidirectional LSTM (stacked LSTM) allow the hidden state to capture both past and future information,Multilayer LSTM( deep LSTM) let the higher layers capture longer-term dependencies of the input sequences. they only allow for strictly sequential information propagation

5. Datastructure:

   

   

   

   

   

   

   1. different from LSTM is that gating vectors and memory cell updates are dependent on the states of possibly many child units. Tree-LSTM unit contains one forget gate Fjk for each child k. to selective incorporate information from each child.
   2. Classification model:
   3. Semantic Relatedness of Sentence Pairs : given a sentence pair,predict a real-valued similarity score in some range.

Paper《Learning to Forget:Continual Prediction with LSTM》

cited: keyword:

Phenomenon&Challenge:

1. backpropagated error quickly either vanishes or blows up

Chart&Analyse:

Code:

Shortcoming&Confusion:

1. embedded Reber grammar
2. 没怎么看懂公式推导过程

Paper《SPEECH RECOGNITION WITH DEEP RECURRENT NEURAL NETWORKS》

cited: keyword:

Chart&Analyse:

Ft表示遗忘门限，It表示输入门限， ̃Ct表示前一时刻cell状态、Ct表示cell状态（这里就是循环发生的地方），Ot表示输出门限，Ht表示当前单元的输出，Ht-1表示前一时刻单元的输出