最简单的方式获取ELMo得到的词向量
Introduction
本文的目的就是用最简单的方式获取 elmo 得到的word representation，看了一些其他人的介绍，其实最后对我有用的就这么多了，我只想要他生成的词向量。
简单介绍一下 elmo：Allen NLP在NAACL2018上的Best paper - Deep contextualized word representations，使用elmo让原有的模型在NLI等Task上效果提升。
那好，直接说怎么得到这个elmo。现在有tf，pytorch，keras各种版本。本文使用的官方给出的elmo片段方式，不用加在模型当中，直接获得词向量的Tensor，因为我只想用他的词向量，训练他的模型又耗时有耗机器。
Environment
首先在conda中新建环境：
conda create -n allennlp python=3.6
接着安装allennlp[保证你电脑里gcc是OK的，编译时需要C++的环境]
pip install allennlp
别断网就OK了，东西有点多，pytorch啥的全套。
然后，下载allennlp给出的训练好的参数和模型
网址：
- https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_options.json
- https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_weights.hdf5
这样方便你重复使用。
Method
下面就是用这两个文件怎么得到词向量了：
from allennlp.commands.elmo import ElmoEmbedder
options_file = "/files/elmo_2x4096_512_2048cnn_2xhighway_options.json"
weight_file = "/files/elmo_2x4096_512_2048cnn_2xhighway_weights.hdf5"
elmo = ElmoEmbedder(options_file, weight_file)
# use batch_to_ids to convert sentences to character ids
context_tokens = [['I', 'love', 'you', '.'], ['Sorry', ',', 'I', 'don', "'t", 'love', 'you', '.']] #references
elmo_embedding, elmo_mask = elmo.batch_to_embeddings(context_tokens)
print(elmo_embedding)
print(elmo_mask)
Result
Embedding:
tensor([[[[ 0.6923, -0.3261, 0.2283, ..., 0.1757, 0.2660, -0.1013],
[-0.7348, -0.0965, -0.1411, ..., -0.3411, 0.3681, 0.5445],
[ 0.3645, -0.1415, -0.0662, ..., 0.1163, 0.1783, -0.7290],
...,
[ 0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000]],
[[-1.1051, -0.4092, -0.4365, ..., -0.6326, 0.4735, -0.2577],
[ 0.0899, -0.4828, -0.5596, ..., 0.4372, 0.3840, -0.7343],
[-0.5538, -0.1473, -0.2441, ..., 0.2551, 0.0873, 0.2774],
...,
[ 0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000]],
[[-3.2634, -0.9448, -0.3199, ..., -1.2070, 0.6930, -0.2016],
[-0.3688, -0.7632, -0.0715, ..., 0.6294, 1.6869, -0.6655],
[-1.0870, -1.4243, -0.2445, ..., 0.0825, 0.5020, 0.2765],
...,
[ 0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000]]],
[[[ 0.5042, -0.6629, -0.0231, ..., -0.3084, -0.9741, -0.7230],
[ 0.1131, 0.1575, 0.1414, ..., 0.3718, -0.1432, -0.0248],
[ 0.6923, -0.3261, 0.2283, ..., 0.1757, 0.2660, -0.1013],
...,
[-0.7348, -0.0965, -0.1411, ..., -0.3411, 0.3681, 0.5445],
[ 0.3645, -0.1415, -0.0662, ..., 0.1163, 0.1783, -0.7290],
[-0.8872, -0.2004, -1.0601, ..., -0.2655, 0.2115, 0.1977]],
[[ 0.1221, -0.7032, 0.0169, ..., -0.3249, -0.4935, -0.4965],
[ 0.3399, -0.4682, 0.1888, ..., -0.0565, 0.1001, -0.0416],
[-0.8135, -0.8491, -0.3264, ..., -0.5674, 0.2638, 0.2006],
...,
[ 0.4460, -0.4475, -0.1583, ..., 0.4372, 0.3840, -0.7343],
[-0.1287, 0.0161, 0.0315, ..., 0.2551, 0.0873, 0.2774],
[-1.2373, -0.3373, 0.1098, ..., -0.0276, -0.0181, 0.0602]],
[[-0.0830, -1.5891, -0.2576, ..., -1.2944, 0.1082, 0.6745],
[-0.0724, -0.7200, 0.1463, ..., 0.6919, 0.9144, -0.1260],
[-2.3460, -1.1714, -0.7065, ..., -1.2885, 0.4679, 0.3800],
...,
[ 0.1246, -0.6929, 0.6330, ..., 0.6294, 1.6869, -0.6655],
[-0.5757, -1.0845, 0.5794, ..., 0.0825, 0.5020, 0.2765],
[-1.2392, -0.6155, -0.9032, ..., 0.0524, -0.0852, 0.0805]]]])
Mask:
tensor([[1, 1, 1, 1, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1, 1, 1]])
Tips
- 实验的输出结果是 2 * 3 * 8 * 1024 的word embedding，都是2、3、8超参数。
- 2是batch_size, 3是两层biLM的输出加一层CNN对character编码的输出, 8是最长list的长度(对齐), 1024是每层输出的维度。
- mask的输出2是batch_size, 8实在最长list的长度, 第一个list有4个tokens,第二个list有8个tokens, 所以对应位置输出1。
References
https://cstsunfu.github.io/2018/06/ELMo/
https://blog.csdn.net/sinat_26917383/article/details/81913790