text-generation-webui加载codellama报错DLL load failed while importing flash_attn_2_cuda: 找不到指定的模块。
使用text-generation-webui加载codellama，报错：
Traceback (most recent call last):
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\utils\import_utils.py", line 1353, in _get_module
return importlib.import_module("." + module_name, self.__name__)
File "D:\Anaconda\Anaconda\envs\codellama\lib\importlib_init_.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\llama\modeling_llama.py", line 48, in
from flash_attn import flash_attn_func, flash_attn_varlen_func
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\flash_attn_init_.py", line 3, in
from flash_attn.flash_attn_interface import (
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\flash_attn\flash_attn_interface.py", line 8, in
import flash_attn_2_cuda as flash_attn_cuda
ImportError: DLL load failed while importing flash_attn_2_cuda: 找不到指定的模块。
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "E:\模型\text-generation-webui\text-generation-webui\modules\ui_model_menu.py", line 209, in load_model_wrapper
shared.model, shared.tokenizer = load_model(shared.model_name, loader)
File "E:\模型\text-generation-webui\text-generation-webui\modules\models.py", line 85, in load_model
output = load_func_map[loader](model_name)
File "E:\模型\text-generation-webui\text-generation-webui\modules\models.py", line 155, in huggingface_loader
model = LoaderClass.from_pretrained(path_to_model, **params)
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 565, in from_pretrained
model_class = _get_model_class(config, cls._model_mapping)
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 387, in _get_model_class
supported_models = model_mapping[type(config)]
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 740, in getitem
return self._load_attr_from_module(model_type, model_name)
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 754, in _load_attr_from_module
return getattribute_from_module(self._modules[module_name], attr)
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 698, in getattribute_from_module
if hasattr(module, attr):
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\utils\import_utils.py", line 1343, in getattr
module = self._get_module(self._class_to_module[name])
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\utils\import_utils.py", line 1355, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
DLL load failed while importing flash_attn_2_cuda: 找不到指定的模块。一开始排查是以为transformers的版本不对，先确定了transformers的版本，transformers的版本应该大于4.35.0
把transformers升级为4.35.0后仍然报错
接着排查cuda和torch的版本
最后发现是cuda版本与torch版本不匹配
>>> print(torch.version.cuda) # 检查CUDA版本
>>> 11.8控制台运行nvcc --version ：
输出：
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Feb__8_05:53:42_Coordinated_Universal_Time_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0最后解决：
先卸载原本的torch：
pip uninstall torch torchvision torchaudio然后安装12.1的：
pip install torch torchvision torchaudio -f https://download.pytorch.org/whl/cu121/torch_stable.html最后加载成功codellama
