自然语言处理教程-注意力模型｜Natural Language Processing with Attention Models

Week 1

1 Neural Machine Translation

1.1 Seq2seq

Shortcomings:

The information bottleneck.

Since seq2seq uses a fixed length memory for the hidden states, long sequences become problematic. This is due to the fact that in traditional seq2seq models, only a fixed amount of information can be passed from the encoder to the decoder no matter how much information is contained in the input sequence.

1.2 Seq2seq Model with Attention

• 将encoder每一个节点的信息都保留，然后把所有的向量加起来，构成了context vector
• （这一步并没有和传统的seq2seq有很大区别）

• Weight some encoders more than others before the point-wise addition.
• Words are more important for the next decoder outputs would have larger weights.
• How these weights are calculated?
• $S_{i-1}$ is the previous hidden state in the decoder, which contains information about the previous words in the output translation.
• You can compare the decoder states with each encoder state to determine the most important inputs.

• $e_{ij}$ is to calculate the alignment. It is the score of how well the inputs around $j$ match the excepted output $i$.

1.3 Queries, Keys, Values, and Attention

• Improving the model performance for larger model sizes and could be seen as a regularization constants.
  

1.4 Teacher Forcing

• Use the ground truth words as decoder inputs instead of the decoder outputs. Even if the model makes a wrong prediction, it pretends as if it made correct one and this can continue. (make training much faster)

1.5 NMT Model with Attention

1.6 BLEU Score

1.6 ROUGE-N Score

1.7 Sampling and Decoding

1.8 Beam Search

1.9 Minimum Bayes Risk

Week 2

1 Text Summarization

1.1 Transformer vs. RNNs

RNN

Seq2Seq Architectures

• The information tends to get lost within the network and vanishing gradients problems arise related to the length of your import sequences.
• LSTMs and GRUs help a little with these problems within those architecture when they try to process very long sequences due to the information bottleneck.

Transformer

• Transformer don’t use RNNs
• Only need attention

1.2 Transformers overview

1.3 Transformer Applications

1.4 Scaled and Dot-Product Attention

1.5 Masked Self Attention

• This type of attention lets you get contextual representations of your words. In other terms, self-attention gives you a representation of the meaning of each word within the sentence.

• Queries cannot attend to the future.

1.6 Multi-head Attention

• In multi-head attention, you apply in parallel the attention mechanism to multiple sets of these matrixes that you can get by transforming the original embeddings.
• In multi-head attention, the number of times that you apply the attention mechanism is the number of heads in the model.
• Using different sets of representations allows your model to learn multiple relationships between the words from the query and key matrices.

• First, you transform each of these matrices into multiple vector spaces. As you saw previously, the number of transformations for each matrix is equal to the number of heads in the model.
• Then, you will apply the scale dot-product attention mechanism to every sets of value, key and query transformations. Where again, the number of sets is equal to the number of heads and the model.
• After that, you concatenate the results from each head in the model into a single matrix.

1.7 Transformer Decoder

1.8 Transformer Summarizer