ValueError: This tokenizer cannot be instantiated. Please make sure you have `sentencepiece` install

报错信息如下：

ValueError Traceback (most recent call last)

in ()
2
3 model_checkpoint = “Helsinki-NLP/opus-mt-en-zh”
----> 4 translator = pipeline(“translation”, model=model_checkpoint)
5 translator(“Default to expanded threads”)

/usr/local/lib/python3.10/dist-packages/transformers/pipelines/init.py in pipeline(task, model, config, tokenizer, feature_extractor, image_processor, framework, revision, use_fast, use_auth_token, device, device_map, torch_dtype, trust_remote_code, model_kwargs, pipeline_class, **kwargs)
883 tokenizer_kwargs.pop(“torch_dtype”, None)
884
–> 885 tokenizer = AutoTokenizer.from_pretrained(
886 tokenizer_identifier, use_fast=use_fast, _from_pipeline=task, **hub_kwargs, **tokenizer_kwargs
887 )

/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
712 return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
713 else:
–> 714 raise ValueError(
715 "This tokenizer cannot be instantiated. Please make sure you have sentencepiece installed "
716 “in order to use this tokenizer.”

ValueError: This tokenizer cannot be instantiated. Please make sure you have sentencepiece installed in order to use this tokenizer.

解决

pip install transformers[sentencepiece]

如果你使用 jupyter 等编辑器，安装包之后，依然报错，建议重启内核。