词嵌入和交叉熵｜Word Embedding and Cross Entropy

Word Embedding

• The Embedding layer can be understood as a lookup table that maps from integer indices (which stand for specific words) to dense vectors (their embeddings).
• The dimensionality (or width) of the embedding is a parameter you can experiment with to see what works well for your problem, much in the same way you would experiment with the number of neurons in a Dense layer.
• When you create an Embedding layer, the weights for the embedding are randomly initialized (just like any other layer). During training, they are gradually adjusted via backpropagation. Once trained, the learned word embeddings will roughly encode similarities between words (as they were learned for the specific problem your model is trained on).

Pytorch

tf.keras.layers.Embedding(
    input_dim,
    output_dim,
    input_length=None,
    **kwargs
)

Embedding层有三个重要的参数

• input_dim：输入维度，这个是指词典的大小
• output_dim：输出维度，这个是指embedding的维度
• input_length：这个只是输入序列（数据）的长度，比如长度为10的字符串
• 通常我们先处理好数据，然后调用 embedding 层，所以只需要定义 input_dim 和 output_dim，不需要指定input_length
• The output vectors are not computed from the input using any mathematical operation. Instead, each input integer is used as the index to access a table that contains all possible vectors. That is the reason why you need to specify the size of the vocabulary as the first argument (so the table can be initialized)
• Embedding Layer 可以理解成为一个 lookup table

---

Cross Entropy

• Pytorch 中的 nn.CrossEntropyLoss() 相当于nn.LogSoftmax 和 nn.NLLLoss ，因此，在PyTorch的Cross Entropy Loss之前请勿再使用Softmax方法！

Reference

• https://www.tensorflow.org/text/guide/word_embeddings
• https://stats.stackexchange.com/questions/270546/how-does-keras-embedding-layer-work
• https://blog.csdn.net/geter_CS/article/details/84857220