The Illustrated Transformer 翻译

最近研究NLP，同事推荐了这篇文章，无奈英语水平有限，连续看三行英文就不知所云了。。。无奈怒翻了这篇文章，一边翻译一边研究，要学的东西还很多啊。

The Illustrated Transformer

In the previous post, we looked at Attention – a ubiquitous method in modern deep learning models. Attention is a concept that helped improve the performance of neural machine translation applications. In this post, we will look at The Transformer – a model that uses attention to boost the speed with which these models can be trained. The Transformers outperforms the Google Neural Machine Translation model in specific tasks. The biggest benefit, however, comes from how The Transformer lends itself to parallelization. It is in fact Google Cloud’s recommendation to use The Transformer as a reference model to use their Cloud TPU offering. So let’s try to break the model apart and look at how it functions.

在前一篇文章中，我们讨论了注意力——一种在现代深度学习模型中普遍存在的方法。注意力是一个有助于提高神经机器翻译应用程序性能的概念。在这篇文章中，我们将着眼于Transformer——一个使用注意力来提高这些模型的训练速度的模型。Transformer在特定任务中的性能优于谷歌神经机器翻译模型。然而，最大的好处来自于转换器如何实现并行化。实际上，谷歌Cloud推荐使用Transformer作为参考模型来使用他们的云TPU产品。让我们试着把模型拆开看看它是如何工作的。

The Transformer was proposed in the paper Attention is All You Need. A TensorFlow implementation of it is available as a part of the Tensor2Tensor package. Harvard’s NLP group created a guide annotating the paper with PyTorch implementation. In this post, we will attempt to oversimplify things a bit and introduce the concepts one by one to hopefully make it easier to understand to people without in-depth knowledge of the subject matter.

Transformer的提出是在 Attention is All You Need 论文中。它的TensorFlow实现是tensor2张量包的一部分。哈佛大学的NLP小组创建了一个指南，用PyTorch实现对论文进行注释。在这篇文章中，我们将尝试把事情简单化一点，并逐一介绍概念，希望让没有深入了解主题的人更容易理解。

A High-Level Look

Let’s begin by looking at the model as a single black box. In a machine translation application, it would take a sentence in one language, and output its translation in another.

让我们首先将模型看作一个单独的黑盒。在机器翻译应用程序中，它将使用一种语言的句子，并输出另一种语言的翻译。

 

Popping open that Optimus Prime goodness, we see an encoding component, a decoding component, and connections between them.

看其内部，我们看到一个编码组件，一个解码组件，以及它们之间的联系。

 

The encoding component is a stack of encoders (the paper stacks six of them on top of each other – there’s nothing magical about the number six, one can definitely experiment with other arrangements). The decoding component is a stack of decoders of the same number.

编码组件是一堆编码器(纸上6个编码器叠在一起——数字6没有什么神奇之处，肯定可以尝试其他安排)。解码组件是相同数量的解码器的堆栈。

 

The encoders are all identical in structure (yet they do not share weights). Each one is broken down into two sub-layers:

编码器在结构上都是相同的(但它们不共享权重)。每一个都被分解成两个子层:

 

The encoder’s inputs first flow through a self-attention layer – a layer that helps the encoder look at other words in the input sentence as it encodes a specific word. We’ll look closer at self-attention later in the post.

编码器的输入首先通过一个Self-Attention层——这个层在编码器编码特定单词时帮助编码器查看输入句子中的其他单词。我们将在稍后的文章中更详细地讨论它。

The outputs of the self-attention layer are fed to a feed-forward neural network. The exact same feed-forward network is independently applied to each position.

将self-attention层的输出输入前馈神经网络。完全相同的前馈网络分别应用于每个位置。

The decoder has both those layers, but between them is an attention layer that helps the decoder focus on relevant parts of the input sentence (similar what attention does in seq2seq models).

解码器也有这两个层，但它们之间是一个attention层，帮助解码器将注意力集中于输入句子的相关部分(类似于注意在seq2seq模型中的作用)。

 

Bringing The Tensors Into The Picture

Now that we’ve seen the major components of the model, let’s start to look at the various vectors/tensors and how they flow between these components to turn the input of a trained model into an output.

现在我们已经了解了模型的主要组成部分，让我们开始研究各种向量/张量，以及它们如何在这些组成部分之间流动，从而将经过训练的模型的输入转换为输出。

As is the case in NLP applications in general, we begin by turning each input word into a vector using an embedding algorithm.

与NLP应用程序中的一般情况一样，我们首先使用嵌入算法将每个输入单词转换成一个向量。

 

Each word is embedded into a vector of size 512. We'll represent those vectors with these simple boxes.

每个单词都嵌入到一个大小为512的向量中。我们用这些简单的盒子来表示这些向量。

The embedding only happens in the bottom-most encoder. The abstraction that is common to all the encoders is that they receive a list of vectors each of the size 512 – In the bottom encoder that would be the word embeddings, but in other encoders, it would be the output of the encoder that’s directly below. The size of this list is hyperparameter we can set – basically it would be the length of the longest sentence in our training dataset.

嵌入只发生在最底部的编码器。所有编码器都有一个共同的抽象概念，那就是它们接收到的向量列表，每个向量的大小都是512——在底部的编码器中是词嵌入，但是在其他编码器中，它是下面的编码器的输出。这个列表的大小是我们可以设置的超参数——基本上就是我们的训练数据集中最长句子的长度。

After embedding the words in our input sequence, each of them flows through each of the two layers of the encoder.

在输入序列中嵌入单词之后，每个单词都流经编码器的两层。

 

Here we begin to see one key property of the Transformer, which is that the word in each position flows through its own path in the encoder. There are dependencies between these paths in the self-attention layer. The feed-forward layer does not have those dependencies, however, and thus the various paths can be executed in parallel while flowing through the feed-forward layer.

在这里，我们开始看到Transformer的一个关键属性，即每个位置上的单词在编码器中通过自己的路径流动。在self-attention层中，这些路径之间存在依赖关系。但是，前馈层没有这些依赖项，因此可以在流经前馈层时并行执行各种路径。

Next, we’ll switch up the example to a shorter sentence and we’ll look at what happens in each sub-layer of the encoder.

接下来，我们将把示例转换为一个更短的句子，并查看在编码器的每个子层中发生了什么。

 

Now We’re Encoding!

As we’ve mentioned already, an encoder receives a list of vectors as input. It processes this list by passing these vectors into a ‘self-attention’ layer, then into a feed-forward neural network, then sends out the output upwards to the next encoder.

正如我们已经提到的，编码器接收一组向量作为输入。它通过将这些向量传递到一个“self-attention”层来处理这个列表，然后进入前馈神经网络，然后向上发送输出到下一个编码器。

 

The word at each position passes through a self-encoding process. Then, they each pass through a feed-forward neural network -- the exact same network with each vector flowing through it separately.

每个位置的单词都经过一个自编码过程。然后，它们各自通过前馈神经网络——完全相同的网络，每个向量分别通过它。

 

Self-Attention at a High Level

Don’t be fooled by me throwing around the word “self-attention” like it’s a concept everyone should be familiar with. I had personally never came across the concept until reading the Attention is All You Need paper. Let us distill how it works.

不要被我随便说的“自我关注”这个词所愚弄，好像它是每个人都应该熟悉的概念。我个人从来没有遇到过这个概念，直到阅读这篇论文 Attention is All You Need 。让我们提炼一下它是如何工作的。

Say the following sentence is an input sentence we want to translate:

”The animal didn't cross the street because it was too tired”

假设下面这句话是我们要翻译的输入句:

”The animal didn't cross the street because it was too tired”

What does “it” in this sentence refer to? Is it referring to the street or to the animal? It’s a simple question to a human, but not as simple to an algorithm.

这个句子中的“it”是什么意思?它指的是街道还是动物?这对人类来说是一个简单的问题，但对算法来说就没那么简单了。

When the model is processing the word “it”, self-attention allows it to associate “it” with “animal”.

当模型处理单词“it”时，self-attention将“it”与“animal”联系起来。

As the model processes each word (each position in the input sequence), self attention allows it to look at other positions in the input sequence for clues that can help lead to a better encoding for this word.

当模型处理每个单词(输入序列中的每个位置)时，self - attention允许它查看输入序列中的其他位置，以寻找有助于对该单词进行更好编码的线索。

If you’re familiar with RNNs, think of how maintaining a hidden state allows an RNN to incorporate its representation of previous words/vectors it has processed with the current one it’s processing. Self-attention is the method the Transformer uses to bake the “understanding” of other relevant words into the one we’re currently processing.

如果您熟悉RNN，请考虑如何维护一个隐藏状态，使RNN能够将它处理过的前一个单词/向量的表示与它正在处理的当前单词/向量结合起来。Self-attention是Transformer用来将其他相关单词的“理解”转换成我们正在处理的单词的方法。

 

As we are encoding the word "it" in encoder #5 (the top encoder in the stack), part of the attention mechanism was focusing on "The Animal", and baked a part of its representation into the encoding of "it".

当我们在编码器5中编码“it”(堆栈中最上面的编码器)时，注意力机制的一部分集中在“动物”上，并将其表示的一部分融入到“it”的编码中。

Be sure to check out the Tensor2Tensor notebook where you can load a Transformer model, and examine it using this interactive visualization.

请务必查看"Tensor2Tensor notebook"，在那里您可以加载Transformer模型，并使用这种交互式可视化来检查它。

 

Self-Attention in Detail

Let’s first look at how to calculate self-attention using vectors, then proceed to look at how it’s actually implemented – using matrices.

让我们先看看如何用向量来计算self-attention，然后再看看它是如何实现的——用矩阵。

The first step in calculating self-attention is to create three vectors from each of the encoder’s input vectors (in this case, the embedding of each word). So for each word, we create a Query vector, a Key vector, and a Value vector. These vectors are created by multiplying the embedding by three matrices that we trained during the training process.

计算自我注意的第一步是从编码器的每个输入向量中创建三个向量(在本例中是每个单词的嵌入)。因此，对于每个单词，我们创建一个查询向量、一个键向量和一个值向量。这些向量是通过将嵌入乘以我们在训练过程中训练的三个矩阵得到的。

Notice that these new vectors are smaller in dimension than the embedding vector. Their dimensionality is 64, while the embedding and encoder input/output vectors have dimensionality of 512. They don’t HAVE to be smaller, this is an architecture choice to make the computation of multiheaded attention (mostly) constant.

注意这些新向量的维数比嵌入向量小。其维数为64，而嵌入和编码器的输入/输出向量的维数为512。它们不需要更小，这是一种架构选择，可以使多目标注意力的计算(大部分)保持不变。

 

Multiplying x1 by the WQ weight matrix produces q1, the "query" vector associated with that word. We end up creating a "query", a "key", and a "value" projection of each word in the input sentence.

x1乘以WQ权重矩阵得到q1，即与这个词相关的“查询”向量。我们最终创建了输入句子中每个单词的“查询”、“键”和“值”投影。

What are the “query”, “key”, and “value” vectors? 

什么是“查询”、“键”和“值”向量?

They’re abstractions that are useful for calculating and thinking about attention. Once you proceed with reading how attention is calculated below, you’ll know pretty much all you need to know about the role each of these vectors plays.

它们是对计算和思考注意力很有用的抽象概念。一旦你开始阅读下面计算注意力的方法，你就会知道这些向量所扮演的角色。

The second step in calculating self-attention is to calculate a score. Say we’re calculating the self-attention for the first word in this example, “Thinking”. We need to score each word of the input sentence against this word. The score determines how much focus to place on other parts of the input sentence as we encode a word at a certain position.

计算自我注意的第二步是计算分数。假设我们在计算本例中第一个单词“Thinking”的self-attention。我们需要将输入句子中的每个单词与这个单词进行评分。分数决定了当我们在某个位置编码一个单词时，对输入句子的其他部分的关注程度。

The score is calculated by taking the dot product of the query vector with the key vector of the respective word we’re scoring. So if we’re processing the self-attention for the word in position #1, the first score would be the dot product of q1 and k1. The second score would be the dot product of q1 and k2.

分数是通过查询向量与我们正在评分的单词的key向量的点积计算出来的。如果我们处理位置1的单词的self-attention，第一个分数就是q1和k1的点积。第二个分数是q1和k2的点积。

 

The third and forth steps are to divide the scores by 8 (the square root of the dimension of the key vectors used in the paper – 64. This leads to having more stable gradients. There could be other possible values here, but this is the default), then pass the result through a softmax operation. Softmax normalizes the scores so they’re all positive and add up to 1.

第三步和第四步是将分数除以8(论文中使用的关键向量的维数的平方根- 64)。这导致了更稳定的梯度。这里可能有其他可能的值，但这是默认值)，然后通过softmax操作传递结果。Softmax将分数标准化，使其均为正数，加起来为1。

 

This softmax score determines how much how much each word will be expressed at this position. Clearly the word at this position will have the highest softmax score, but sometimes it’s useful to attend to another word that is relevant to the current word.

softmax分数决定每个单词在这个位置的表达量。显然，这个位置的单词将拥有最高的softmax分数，但有时关注与当前单词相关的另一个单词是有用的。

The fifth step is to multiply each value vector by the softmax score (in preparation to sum them up). The intuition here is to keep intact the values of the word(s) we want to focus on, and drown-out irrelevant words (by multiplying them by tiny numbers like 0.001, for example).

第五步是将每个value向量乘以softmax分数(为求和做准备)。这里直观展示的是保持我们要关注的单词的值，淹没无关单词的值(例如，将它们乘以0.001这样的小数字)。

The sixth step is to sum up the weighted value vectors. This produces the output of the self-attention layer at this position (for the first word).

第六步是加权值向量的求和。这将在这个位置(对于第一个单词)生成self-attention层的输出。

 

That concludes the self-attention calculation. The resulting vector is one we can send along to the feed-forward neural network. In the actual implementation, however, this calculation is done in matrix form for faster processing. So let’s look at that now that we’ve seen the intuition of the calculation on the word level.

这就是自我注意计算的结论。得到的向量是一个我们可以发送到前馈神经网络的向量。然而，在实际实现中，这种计算是以矩阵形式进行的，以便更快地进行处理。现在我们来看看这个我们已经在单词层面上看到了计算的直观展示。

 

Matrix Calculation of Self-Attention

The first step is to calculate the Query, Key, and Value matrices. We do that by packing our embeddings into a matrix X, and multiplying it by the weight matrices we’ve trained (WQ, WK, WV).

第一步是计算query、key和value矩阵。我们把embeddings包装成矩阵X，然后乘以我们训练过的权矩阵(WQ WK WV)。

 

Every row in the X matrix corresponds to a word in the input sentence. We again see the difference in size of the embedding vector (512, or 4 boxes in the figure), and the q/k/v vectors (64, or 3 boxes in the figure)

X矩阵中的每一行对应输入句子中的一个单词。我们再次看到embeddings向量(512，或图中4个框)和q/k/v向量(64，或图中3个框)大小的差异

Finally, since we’re dealing with matrices, we can condense steps two through six in one formula to calculate the outputs of the self-attention layer.

最后，由于我们处理的是矩阵，我们可以将步骤2到步骤6压缩到一个公式中来计算自我注意层的输出。

 

The self-attention calculation in matrix form

矩阵形式的self-attention计算

 

The Beast With Many Heads

The paper further refined the self-attention layer by adding a mechanism called “multi-headed” attention. This improves the performance of the attention layer in two ways:

本文进一步细化了self-attention层，增加了“多头”注意机制。这通过两种方式提高了注意力层的性能:

1. It expands the model’s ability to focus on different positions. Yes, in the example above, z1 contains a little bit of every other encoding, but it could be dominated by the the actual word itself. It would be useful if we’re translating a sentence like “The animal didn’t cross the street because it was too tired”, we would want to know which word “it” refers to.

它扩展了模型关注不同位置的能力。是的，在上面的例子中，z1包含了一些其他编码，但是它可以被实际单词本身所控制。如果我们翻译像“动物没过马路是因为它太累了”这样的句子，它会很有用，我们想知道“它”指的是哪个单词。

2. It gives the attention layer multiple “representation subspaces”. As we’ll see next, with multi-headed attention we have not only one, but multiple sets of Query/Key/Value weight matrices (the Transformer uses eight attention heads, so we end up with eight sets for each encoder/decoder). Each of these sets is randomly initialized. Then, after training, each set is used to project the input embeddings (or vectors from lower encoders/decoders) into a different representation subspace.

它给了attention 层多个“表示子空间”。接下来我们将看到，对于multi-headed ，我们不仅有一个，而且有多个query/key/value权重矩阵集(Transformer 使用8个注意头，因此我们最终为每个编码器/解码器使用8个注意头)。每个集合都是随机初始化的。然后，在训练之后，使用每个集合将输入嵌入(或来自较低编码器/解码器的向量)投影到不同的表示子空间中。

 

With multi-headed attention, we maintain separate Q/K/V weight matrices for each head resulting in different Q/K/V matrices. As we did before, we multiply X by the WQ/WK/WV matrices to produce Q/K/V matrices.

对于多头注意，我们为每个头部维护单独的Q/K/V权重矩阵，从而产生不同的Q/K/V矩阵。和之前一样，我们用X乘以WQ/WK/WV矩阵得到Q/K/V矩阵。

If we do the same self-attention calculation we outlined above, just eight different times with different weight matrices, we end up with eight different Z matrices.

如果我们做同样的self-attention计算，就像上面概述的那样，用不同的权重矩阵做8次不同的计算，我们得到8个不同的Z矩阵。

 

This leaves us with a bit of a challenge. The feed-forward layer is not expecting eight matrices – it’s expecting a single matrix (a vector for each word). So we need a way to condense these eight down into a single matrix.

这给我们留下了一点挑战。前馈层不需要八个矩阵——它只需要一个矩阵(每个单词对应一个向量)。我们需要一种方法把这8个压缩成一个矩阵。

How do we do that? We concat the matrices then multiple them by an additional weights matrix WO.

我们怎么做呢?我们把这些矩阵连起来然后乘以一个额外的权重矩阵。

 

That’s pretty much all there is to multi-headed self-attention. It’s quite a handful of matrices, I realize. Let me try to put them all in one visual so we can look at them in one place

这就是多脑self-attention的全部。我意识到这是相当多的矩阵。我试着把它们放在一个图像中这样我们就能在一个地方看到它们。

 

Now that we have touched upon attention heads, let’s revisit our example from before to see where the different attention heads are focusing as we encode the word “it” in our example sentence:

既然我们已经提到了注意力头部，让我们再来看看之前的例子，看看当我们在例句中编码单词“it”时，不同的注意力头部集中在哪里:

 

As we encode the word "it", one attention head is focusing most on "the animal", while another is focusing on "tired" -- in a sense, the model's representation of the word "it" bakes in some of the representation of both "animal" and "tired".

当我们对“it”这个词进行编码时，一个注意力集中在“动物”上，而另一个注意力集中在“疲倦”上——从某种意义上说，“it”这个词的模型表现在“动物”和“疲倦”两个词的表现中。

If we add all the attention heads to the picture, however, things can be harder to interpret:

然而，如果我们把所有的注意力都集中到这幅图上，事情就很难解释了:

 

 

Representing The Order of The Sequence Using Positional Encoding

用位置编码表示序列的顺序

One thing that’s missing from the model as we have described it so far is a way to account for the order of the words in the input sequence.

到目前为止，我们所描述的模型中缺少的一件事是一种解释输入序列中单词顺序的方法。

To address this, the transformer adds a vector to each input embedding. These vectors follow a specific pattern that the model learns, which helps it determine the position of each word, or the distance between different words in the sequence. The intuition here is that adding these values to the embeddings provides meaningful distances between the embedding vectors once they’re projected into Q/K/V vectors and during dot-product attention.

为了解决这个问题，transformer向每个输入的embedding添加一个向量。这些向量遵循模型学习到的特定模式，这有助于它确定每个单词的位置，或序列中不同单词之间的距离。直觉上，在点积attention时，一旦embeddings被投影到Q/K/V向量中，添加到embeddings中的这些值可以提供embeddings向量之间有意义的距离。

 

To give the model a sense of the order of the words, we add positional encoding vectors -- the values of which follow a specific pattern.

为了给模型一个单词顺序的感觉，我们添加位置编码向量——它们的值遵循特定的模式。

If we assumed the embedding has a dimensionality of 4, the actual positional encodings would look like this:

如果我们假设嵌入的维数是4，那么实际的位置编码应该是这样的:

 

A real example of positional encoding with a toy embedding size of 4

一个真实的位置编码的例子与玩具嵌入大小为4

What might this pattern look like?

这个模式可能是什么样的?

In the following figure, each row corresponds the a positional encoding of a vector. So the first row would be the vector we’d add to the embedding of the first word in an input sequence. Each row contains 512 values – each with a value between 1 and -1. We’ve color-coded them so the pattern is visible.

在下面的图中，每一行对应一个向量的位置编码。所以第一行代表的向量就是我们在输入序列中嵌入第一个单词的向量。每行包含512个值——每个值在1到-1之间。我们用颜色标记了它们，所以模式是可见的。

 

A real example of positional encoding for 20 words (rows) with an embedding size of 512 (columns). You can see that it appears split in half down the center. That's because the values of the left half are generated by one function (which uses sine), and the right half is generated by another function (which uses cosine). They're then concatenated to form each of the positional encoding vectors.

一个实际的位置编码示例，包含20个单词(行)，embedding大小为512(列)。你可以看到它从中间一分为二。这是因为左半部分的值是由一个函数(使用sin)生成的，而右半部分是由另一个函数(使用cos)生成的。然后将它们连接起来形成每个位置编码向量。

The formula for positional encoding is described in the paper (section 3.5). You can see the code for generating positional encodings in get_timing_signal_1d(). This is not the only possible method for positional encoding. It, however, gives the advantage of being able to scale to unseen lengths of sequences (e.g. if our trained model is asked to translate a sentence longer than any of those in our training set).

位置编码的公式在本文(章节3.5)中进行了描述。您可以在get_timing_signal_1d()中看到生成位置编码的代码。这不是唯一可能的位置编码方法。然而，它的优势在于能够扩展到序列的看不见的长度(例如，如果我们训练的模型被要求翻译一个比我们训练集中的任何一个都长的句子)。

 

The Residuals

残差

One detail in the architecture of the encoder that we need to mention before moving on, is that each sub-layer (self-attention, ffnn) in each encoder has a residual connection around it, and is followed by a layer-normalization step.

在继续之前，我们需要提到编码器架构中的一个细节，即每个编码器中的每个子层(self-attention, ffnn)在其周围都有一个残差连接，然后是一个分层规范化步骤。

 

If we’re to visualize the vectors and the layer-norm operation associated with self attention, it would look like this:

如果我们要把这些向量和与自我注意相关的层规范操作形象化，它看起来是这样的:

 

This goes for the sub-layers of the decoder as well. If we’re to think of a Transformer of 2 stacked encoders and decoders, it would look something like this:

这也适用于解码器的子层。如果我们考虑一个由两个堆叠编码器和解码器组成的transformer，它看起来是这样的:

 

 

The Decoder Side

Now that we’ve covered most of the concepts on the encoder side, we basically know how the components of decoders work as well. But let’s take a look at how they work together.

现在我们已经介绍了编码器端的大多数概念，我们基本上也知道解码器的组件是如何工作的。下面让我们来看看它们是如何一起工作的。

The encoder start by processing the input sequence. The output of the top encoder is then transformed into a set of attention vectors K and V. These are to be used by each decoder in its “encoder-decoder attention” layer which helps the decoder focus on appropriate places in the input sequence:

编码器从处理输入序列开始。将顶层编码器的输出转换为一组attention向量K和v，每个编码器在其“编码器-译码器attention”层中使用，帮助译码器在输入序列中适当的位置聚焦:

 

After finishing the encoding phase, we begin the decoding phase. Each step in the decoding phase outputs an element from the output sequence (the English translation sentence in this case).

完成编码阶段后，我们开始解码阶段。解码阶段的每个步骤从输出序列中输出一个元素(在本例中是英语翻译句)。

The following steps repeat the process until a special symbol is reached indicating the transformer decoder has completed its output. The output of each step is fed to the bottom decoder in the next time step, and the decoders bubble up their decoding results just like the encoders did. And just like we did with the encoder inputs, we embed and add positional encoding to those decoder inputs to indicate the position of each word.

下面的步骤重复这个过程，直到到达一个特殊的符号，表明transformer解码器已经完成了输出。每一步的输出在下一次的步骤中被输入到底层解码器，解码器就像编码器一样把它们的解码结果放大。就像我们对编码器输入所做的那样，我们嵌入并添加位置编码到这些解码器输入中来表示每个单词的位置。

 

The self attention layers in the decoder operate in a slightly different way than the one in the encoder:

解码器中的自我注意层的工作方式与编码器中的略有不同:

In the decoder, the self-attention layer is only allowed to attend to earlier positions in the output sequence. This is done by masking future positions (setting them to -inf) before the softmax step in the self-attention calculation.

在解码器中，self-attention层只允许注意输出序列中较早的位置。这是通过在self-attention计算中的softmax步骤之前屏蔽未来位置(将它们设置为-inf)来实现的。

The “Encoder-Decoder Attention” layer works just like multiheaded self-attention, except it creates its Queries matrix from the layer below it, and takes the Keys and Values matrix from the output of the encoder stack.

编码器-解码器注意层的工作原理与多头自注意层类似，只是它从下面的层创建查询矩阵，并从编码器堆栈的输出中获取键和值矩阵。

The Final Linear and Softmax Layer

最后的线性和软最大层

The decoder stack outputs a vector of floats. How do we turn that into a word? That’s the job of the final Linear layer which is followed by a Softmax Layer.

解码器堆栈输出一个浮点向量。我们怎么把它变成一个词呢?这是最后一个线性层的工作，然后是一个Softmax层。

The Linear layer is a simple fully connected neural network that projects the vector produced by the stack of decoders, into a much, much larger vector called a logits vector.

线性层是一个简单的全连接神经网络，它将解码器堆栈产生的向量投影到一个更大的称为逻辑向量的向量中。

Let’s assume that our model knows 10,000 unique English words (our model’s “output vocabulary”) that it’s learned from its training dataset. This would make the logits vector 10,000 cells wide – each cell corresponding to the score of a unique word. That is how we interpret the output of the model followed by the Linear layer.

让我们假设我们的模型从它的训练数据集中知道10,000个独特的英语单词(我们模型的“输出词汇表”)。这将使logits向量宽10,000个单元格—每个单元格对应一个惟一单词的得分。这就是我们解释线性层之后的模型输出的方式。

The softmax layer then turns those scores into probabilities (all positive, all add up to 1.0). The cell with the highest probability is chosen, and the word associated with it is produced as the output for this time step.

softmax层然后将这些分数转换为概率(全部为正数，全部加起来为1.0)。选择概率最大的单元格，并生成与之关联的单词作为此时间步骤的输出。

 

This figure starts from the bottom with the vector produced as the output of the decoder stack. It is then turned into an output word.

此图从底部开始，输出的矢量作为解码器堆栈的输出。然后将其转换为输出字。

Recap Of Training

训练回顾

Now that we’ve covered the entire forward-pass process through a trained Transformer, it would be useful to glance at the intuition of training the model.

既然我们已经介绍了通过训练过的转换器的整个前向传递过程，那么了解一下训练模型的直观了解将是很有用的。

During training, an untrained model would go through the exact same forward pass. But since we are training it on a labeled training dataset, we can compare its output with the actual correct output.

在训练过程中，未训练的模型将经历完全相同的向前传球。但是由于我们是在一个标记的训练数据集上训练它，我们可以将它的输出与实际正确的输出进行比较。

To visualize this, let’s assume our output vocabulary only contains six words(“a”, “am”, “i”, “thanks”, “student”, and “” (short for ‘end of sentence’)).

为了直观地理解这一点，我们假设输出词汇表只包含6个单词(“a”、“am”、“i”、“thanks”、“student”和“”(“end of sentence”的缩写))。

 

The output vocabulary of our model is created in the preprocessing phase before we even begin training.

我们模型的输出词汇表是在我们开始训练之前的预处理阶段创建的。

Once we define our output vocabulary, we can use a vector of the same width to indicate each word in our vocabulary. This also known as one-hot encoding. So for example, we can indicate the word “am” using the following vector:

一旦定义了输出词汇表，就可以使用相同宽度的向量来表示词汇表中的每个单词。这也称为one-hot编码。例如，我们可以用下面的向量来表示am:

 

Example: one-hot encoding of our output vocabulary

示例:输出词汇表的one-hot编码

The Loss Function

损失函数

Say we are training our model. Say it’s our first step in the training phase, and we’re training it on a simple example – translating “merci” into “thanks”.

假设我们正在训练我们的模型。假设这是我们在训练阶段的第一步，我们用一个简单的例子来进行训练——将“merci”翻译成“thanks”。

What this means, is that we want the output to be a probability distribution indicating the word “thanks”. But since this model is not yet trained, that’s unlikely to happen just yet.

这意味着，我们希望输出是一个表示“谢谢”的概率分布。但是由于这个模型还没有经过训练，所以现在还不太可能实现。

 

Since the model's parameters (weights) are all initialized randomly, the (untrained) model produces a probability distribution with arbitrary values for each cell/word. We can compare it with the actual output, then tweak all the model's weights using backpropagation to make the output closer to the desired output.

由于模型的参数(权重)都是随机初始化的，因此(未经训练的)模型生成每个单元格/单词的任意值的概率分布。我们可以将其与实际输出进行比较，然后使用反向传播调整模型的所有权重，使输出更接近所需的输出。

How do you compare two probability distributions? We simply subtract one from the other. For more details, look atcross-entropy and Kullback–Leibler divergence.

如何比较两个概率分布?我们只要把其中一个减去另一个。更多细节，请看交叉熵和库尔巴克-莱布尔散度（KL散度）。

But note that this is an oversimplified example. More realistically, we’ll use a sentence longer than one word. For example – input: “je suis étudiant” and expected output: “i am a student”. What this really means, is that we want our model to successively output probability distributions where:

但是请注意，这是一个过于简化的示例。更实际的情况是，我们将使用一个句子，而不是一个单词。例如，输入:“je suis etudiant”和期望输出:“i am a student”。这真正的意思是，我们想要我们的模型连续输出概率分布，其中:

• Each probability distribution is represented by a vector of width vocab_size (6 in our toy example, but more realistically a number like 3,000 or 10,000)
• The first probability distribution has the highest probability at the cell associated with the word “i”
• The second probability distribution has the highest probability at the cell associated with the word “am”
• And so on, until the fifth output distribution indicates ‘’ symbol, which also has a cell associated with it from the 10,000 element vocabulary.

每个概率分布都由一个宽度为vocab_size的向量表示(在我们的玩具示例中是6，但更实际的数字是3000或10000)

第一个概率分布在与“i”相关的单元格中具有最高的概率

第二个概率分布在与am相关的单元格中具有最高的概率

以此类推，直到第5个输出分布表示' '符号，该符号也有一个来自10,000个元素词汇表的与之关联的单元格。

 

After training the model for enough time on a large enough dataset, we would hope the produced probability distributions would look like this:

在足够大的数据集上训练模型足够长的时间后，我们希望生成的概率分布是这样的:

 

Hopefully upon training, the model would output the right translation we expect. Of course it's no real indication if this phrase was part of the training dataset (see: cross validation). Notice that every position gets a little bit of probability even if it's unlikely to be the output of that time step -- that's a very useful property of softmax which helps the training process.

希望通过训练，该模型能够输出我们期望的正确翻译。当然，如果这个短语是训练数据集的一部分(参见:交叉验证)，这并不是真正的指示。注意，每个位置都有一点概率即使它不太可能是那个时间步长的输出——这是softmax的一个非常有用的属性，它有助于训练过程。

Now, because the model produces the outputs one at a time, we can assume that the model is selecting the word with the highest probability from that probability distribution and throwing away the rest. That’s one way to do it (called greedy decoding). Another way to do it would be to hold on to, say, the top two words (say, ‘I’ and ‘a’ for example), then in the next step, run the model twice: once assuming the first output position was the word ‘I’, and another time assuming the first output position was the word ‘me’, and whichever version produced less error considering both positions #1 and #2 is kept. We repeat this for positions #2 and #3…etc. This method is called “beam search”, where in our example, beam_size was two (because we compared the results after calculating the beams for positions #1 and #2), and top_beams is also two (since we kept two words). These are both hyperparameters that you can experiment with.

现在，因为这个模型每次产生一个输出，我们可以假设这个模型从概率分布中选择概率最大的单词，然后扔掉剩下的。这是一种方法(称为贪婪解码)。另一个方法是坚持,好比,前两个单词(比如“我”和“a”),然后在下一步中,运行模型两次:一次假设第一个输出位置是“I”这个词,而另一个假设第一个输出位置是‘me’这个词,和哪个版本产生更少的错误考虑# 1和# 2保存位置。我们对2号和3号位置重复这个。这种方法称为“beam search”，在我们的示例中，beam_size为2(因为我们在计算位置1和位置2的beam后对结果进行了比较)，top_beam也是2(因为我们保留了两个单词)。这两个都是可以实验的超参数。

Go Forth And Transform

I hope you’ve found this a useful place to start to break the ice with the major concepts of the Transformer. If you want to go deeper, I’d suggest these next steps:

我希望这篇文章能对你有所帮助，让你可以从这里开始了解Transformer的主要概念。如果你想深入了解，我建议你采取以下步骤:

• Read the Attention Is All You Need paper, the Transformer blog post (Transformer: A Novel Neural Network Architecture for Language Understanding), and the Tensor2Tensor announcement.
• 阅读《Attention Is All You Need》这篇文章、Transformer博客文章(Transformer: A Novel Neural Network Architecture for Language Understanding)和Tensor2Tensor announcement。

• Watch Łukasz Kaiser’s talk walking through the model and its details
• 看 Łukasz Kaiser’s talk，讲的是关于模型和细节

• Play with the Jupyter Notebook provided as part of the Tensor2Tensor repo
• 操作一下Jupyter Notebook provided as part of the Tensor2Tensor repo
• Explore the Tensor2Tensor repo.
• 探索 Tensor2Tensor repo.

Acknowledgements

Thanks to Illia Polosukhin, Jakob Uszkoreit, Llion Jones , Lukasz Kaiser, Niki Parmar, and Noam Shazeer for providing feedback on earlier versions of this post.

Please hit me up on Twitter for any corrections or feedback.

Written on June 27, 2018

感谢Illia Polosukhin、Jakob Uszkoreit、Llion Jones、Lukasz Kaiser、Niki Parmar和Noam Shazeer对本文早期版本提供的反馈。

如果有任何更正或反馈，请在Twitter上联系我。

写于2018年6月27日

 

原文： https://jalammar.github.io/illustrated-transformer/

本来是有图片的，粘贴过来就没了。。。