Encoding.word_ids()

一、Encoding资料

Encoding

二、代码 

from transformers import AutoTokenizer

DOWNLOADED_MODEL_PATH = 'model'
tokenizer = AutoTokenizer.from_pretrained('google/bigbird-roberta-base')
tokenizer.save_pretrained('model')

encoding = tokenizer("i loving you Haarde")

print(encoding['input_ids'])
print(tokenizer.decode(encoding['input_ids']))

print(encoding.word_ids())
print(tokenizer.convert_ids_to_tokens(encoding['input_ids']))

三、输出

[65, 1413, 14543, 446, 9499, 45194, 66]

[CLS] i loving you Haarde[SEP]

[None, 0, 1, 2, 3, 3, None]
['[CLS]', '▁i', '▁loving', '▁you', '▁Ha', 'arde', '[SEP]']

 这一过程使用了wordPiece

参考文献：保姆级教程，用PyTorch和BERT进行命名实体识别