【学习总结】Python transformers AutoTokenizer encode 出现的 101 和 102
1.代码展示:fromtransformersimportAutoTokenizer,AutoModelmodel_name="bert-base-chinese"tokenizer=AutoTokenizer.from_pretrained(model_name)model=AutoModel.from_pretrained(model_name)print(len(tokenizer.voca