CMU 11-785 L17 Seq2seq and attention model

Generating Language

Synthesis

• Input: symbols as one-hot vectors
  • Dimensionality of the vector is the size of the 「vocabulary」
  • Projected down to lower-dimensional “embeddings”
• The hidden units are (one or more layers of) LSTM units
• Output at each time: A probability distribution that ideally assigns peak probability to the next word in the sequence
• Divergence

$$\mathrm{Div}({\mathbf{Y}}_{\text{target}}(1\dots T),\mathbf{Y}(1\dots T))=\sum \_t\mathrm{Xent}(\mathbf{Y}\_\text{target}(t),\mathbf{Y}(t))=-\sum \_t\log Y(t,w\_t+1)$$

• Feed the drawn word as the next word in the series
• And draw the next word from the output probability distribution

Beginnings and ends

• A sequence of words by itself does not indicate if it is a complete sentence or not
• To make it explicit, we will add two additional symbols (in addition to the words) to the base vocabulary
  • : Indicates start of a sentence
  • : Indicates end of a sentence
• When do we stop?
  • Continue this process until we draw an
  • Or we decide to terminate generation based on some other criterion

Delayed sequence to sequence

Pseudocode

• Problem: Each word that is output depends only on current hidden state, and not on previous outputs
• The input sequence feeds into a recurrent structure
• The input sequence is terminated by an explicit symbol
  • The hidden activation at the “stores” all information about the sentence
• Subsequently a second RNN uses the hidden activation as initial state to produce a sequence of outputs
  • The output at each time becomes the input at the next time
  • Output production continues until an is produced

Autoencoder

• The recurrent structure that extracts the hidden representation from the input sequence is the encoder
• The recurrent structure that utilizes this representation to produce the output sequence is the decoder

Generating output

• At each time the network produces a probability distribution over words, given the entire input and previous outputs
• At each time a word is drawn from the output distribution

$$P\left(O_1,\dots ,O_L\mid W_1^{in},\dots ,W_N^{in}\right)=y_1^{O_1}{y}_1^{O_2}\dots y_1^{O_L}$$

• The objective of drawing: Produce the most likely output (that ends in an )

$$\underset{O_1,\dots ,O_L}{\mathrm{argmax}}{y}_1^{O_1}{y}_1^{O_2}\dots y_1^{O_L}$$

• How to draw words?
  • Greedy answer
    • Select the most probable word at each time
    • Not good, making a poor choice at any time commits us to a poor future
  • Randomly draw a word at each time according to the output probability distribution
    • Not guaranteed to give you the most likely output
  • Beam search
    • Search multiple choices and prune
    • At each time, retain only the top K scoring forks
    • Terminate: When the current most likely path overall ends in

Train

• In practice, if we apply SGD, we may randomly sample words from the output to actually use for the backprop and update
  • Randomly select training instance: (input, output)
  • Forward pass
  • Randomly select a single output $y(t)$ and corresponding desired output $d(t)$ for backprop
• Trick
  • The input sequence is fed in reverse order
    • This happens both for training and during actual decode
• Problem
  • All the information about the input sequence is embedded into a single vector
  • In reality: All hidden values carry information

Attention model

• Compute a weighted combination of all the hidden outputs into a single vector
  • Weights vary by output time
• Require a time-varying weight that specifies relationship of output time to input time
  • Weights are functions of current output state

$$e_i(t)=g\left({\mathbf{h}}_i,{\mathbf{s}}_{t-1}\right)$$

$$w_i(t)=\frac{\exp \left(e_i(t)\right)}{{\sum }_j\exp \left(e_j(t)\right)}$$

Attention weight

• Typical option for $g()$
  • Inner product
    • $$g\left(\mathbf{h}\_i,\mathbf{s}\_t-1\right)=\mathbf{h}\_i^T\mathbf{s}\_t-1$$
  • Project to the same demension
    • $$g\left({\mathbf{h}}_i,\mathbf{s}\_t-1\right)=\mathbf{h}\_i^T\mathbf{W}\_g\mathbf{s}\_t-1$$
  • Non-linear activation
    • $$g\left(\mathbf{h}\_i,\mathbf{s}\_t-1\right)=v\_g^T\mathbf{t}\mathbf{a}\mathbf{n}\mathbf{h}\left(\mathbf{W}\_g\left[\begin{array}{c}{\mathbf{h}}_i\\ \\ \mathbf{s}\_t-1\end{array}\right]\right)$$
  • MLP
    • $$g\left(\mathbf{h}\_i,\mathbf{s}\_t-1\right)=\mathrm{MLP}\left(\left[\mathbf{h}\_i,\mathbf{s}\_t-1\right]\right)$$

Pseudocode

Train

• Back propagation also updates parameters of the “attention” function
• Trick: Occasionally pass drawn output instead of ground truth, as input
  • Randomly select from output, force network to produce correct word even the prioir word is not correct

variants

• Bidirectional processing of input sequence
  • Neural Machine Translation by Jointly Learning to Align and Translate
• Local attention vs global attention
  • Effective Approaches to Attention-based Neural Machine Translation
• Multihead attention
  • Derive 「value」, and multiple 「keys」 from the encoder
    • $V_i,K_i^l,i=1\dots T,l=1\dots N_{\text{head}}$
  • Derive one or more 「queries」 from decoder
    • $Q_j^l,j=1\dots M,l=1\dots N_{\text{head}}$
  • Each query-key pair gives you one attention distribution
    • And one context vector
    • $a_{j,i}^l=$attention$\left(Q_j^l,K_i^l,i=1\dots T\right),C_j^l={\sum }_i{a}_{j,i}^l{V}_i$
  • Concatenate set of context vectors into one extended context vector
    • $C_j=\left[C_j^1{C}_j^2\dots C_j^{N_{\text{head}}}\right]$
  • Each 「attender」 focuses on a different aspect of the input that’s important for the decode