linux Ubuntu Python 3.10 环境报错与解决方案集合
环境配置参考文章：使用Alpaca-Lora基于LLaMA(7B)二十分钟完成微调
1.报错.nvidia/cublas/lib/libcublas.so.11: undefined symbol: cublasLtHSHMatmulAlgoInit, version libcublasLt.so.11
解决方法：
pip uninstall nvidia_cublas_cu11
2.CUDA版本对应不上
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/xxx/anaconda3/envs/lora/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda100.so
/home/xxx/anaconda3/envs/lora/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: /home/xxx/anaconda3/envs/lora did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/home/dell/anaconda3/envs/lora/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/conda/lib')}
warn(msg)
CUDA SETUP: CUDA version lower than 11 are currently not supported for LLM.int8(). You will be only to use 8-bit optimizers and quantization routines!!
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 100
CUDA SETUP: Required library version not found: libbitsandbytes_cuda100.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
4. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via `conda list | grep cuda`.
================================================================================
CUDA SETUP: CUDA 10.0 not supported. Please use a different CUDA version.
CUDA SETUP: Before you try again running bitsandbytes, make sure old CUDA 10.0 versions are uninstalled and removed from $LD_LIBRARY_PATH variables.
CUDA SETUP: Setup Failed!
CUDA SETUP: CUDA 10.0 not supported. Please use a different CUDA version.
CUDA SETUP: Before you try again running bitsandbytes, make sure old CUDA 10.0 versions are uninstalled and removed from $LD_LIBRARY_PATH variables.
Traceback (most recent call last):
File "/home/dell/anaconda3/envs/lora/lib/python3.10/runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/home/dell/anaconda3/envs/lora/lib/python3.10/runpy.py", line 146, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/home/dell/anaconda3/envs/lora/lib/python3.10/runpy.py", line 110, in _get_module_details
__import__(pkg_name)
File "/home/dell/anaconda3/envs/lora/lib/python3.10/site-packages/bitsandbytes/__init__.py", line 6, in
from . import cuda_setup, utils, research
File "/home/dell/anaconda3/envs/lora/lib/python3.10/site-packages/bitsandbytes/research/__init__.py", line 1, in
from . import nn
File "/home/dell/anaconda3/envs/lora/lib/python3.10/site-packages/bitsandbytes/research/nn/__init__.py", line 1, in
from .modules import LinearFP8Mixed, LinearFP8Global
File "/home/dell/anaconda3/envs/lora/lib/python3.10/site-packages/bitsandbytes/research/nn/modules.py", line 8, in
from bitsandbytes.optim import GlobalOptimManager
File "/home/dell/anaconda3/envs/lora/lib/python3.10/site-packages/bitsandbytes/optim/__init__.py", line 6, in
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "/home/dell/anaconda3/envs/lora/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 20, in
raise RuntimeError('''
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:
python -m bitsandbytes
Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
解决方法：
1.下载对应版本的cuda（11.7为例）
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring*.deb
sudo apt-get update
sudo apt-get -y install cuda
2.将cuda版本同步
sudo update-alternatives --display cuda
sudo update-alternatives --config cuda
3.找不到库(以libcusparse.so.11为例)
OSError: libcusparse.so.11: cannot open shared object file: No such file or directory
解决方法：
在 /home/cenghaolong/anaconda3/envs/BIONIC/lib（你建的虚拟环境的 lib 文件夹） 中放入缺少的 libcusparse.so.11 文件。
4. 某个库缺少值cget_col_row_stats
AttributeError: /home/dell/anaconda3/envs/lora/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats
解决方法：
把同一目录下的高版本的.so(如libbitsandbytes_cuda117.so)复制一份重命名为libbitsandbytes_cpu.so之后再放回去，覆盖掉原来的文件：
5.运行时找不到库(libcublas.so.11)：
Traceback (most recent call last):
File "/home/xxx/anaconda3/envs/lora/lib/python3.10/site-packages/torch/__init__.py", line 172, in _load_global_deps
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
File "/home/xxx/anaconda3/envs/lora/lib/python3.10/ctypes/__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libcublas.so.11: cannot open shared object file: No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/xxx/qsj/alpaca-lora-main/generate.py", line 6, in
import torch
File "/home/xxx/anaconda3/envs/lora/lib/python3.10/site-packages/torch/__init__.py", line 217, in
_load_global_deps()
File "/home/xxx/anaconda3/envs/lora/lib/python3.10/site-packages/torch/__init__.py", line 178, in _load_global_deps
_preload_cuda_deps()
File "/home/xxx/anaconda3/envs/lora/lib/python3.10/site-packages/torch/__init__.py", line 158, in _preload_cuda_deps
ctypes.CDLL(cublas_path)
File "/home/xxx/anaconda3/envs/lora/lib/python3.10/ctypes/__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /home/xxx/anaconda3/envs/lora/lib/python3.10/site-packages/nvidia/cublas/lib/libcublas.so.11: cannot open shared object file: No such file or directory
解决方法：
在服务器 find libcudart.so.11.0 存在的地方:
输入查找语句：
cd ~
find -name libcublas.so.11
然后把该库以及其软连接全部 copy 到当前环境需要的 位置
训练7b代码
python finetune.py \
--base_model 'decapoda-research/llama-7b-hf' \
--data_path './ml_100k_instruct_data.json' \
--output_dir './lora-alpaca-ml100k' \
--batch_size 128 \
--micro_batch_size 4 \
--num_epochs 3 \
--learning_rate 1e-4 \
--cutoff_len 512 \
--val_set_size 2000 \
--lora_r 8 \
--lora_alpha 16 \
--lora_dropout 0.05 \
--lora_target_modules '[q_proj,v_proj]' \
--train_on_inputs \
--group_by_length
训练13b代码
python finetune.py \
--base_model 'decapoda-research/llama-13b-hf' \
--data_path './ml_100k_instruct_data.json' \
--output_dir './lora-alpaca-13bml100k' \
--batch_size 128 \
--micro_batch_size 4 \
--num_epochs 3 \
--learning_rate 1e-4 \
--cutoff_len 512 \
--val_set_size 2000 \
--lora_r 8 \
--lora_alpha 16 \
--lora_dropout 0.05 \
--lora_target_modules '[q_proj,v_proj]' \
--train_on_inputs \
--group_by_length
6.使用tensorboard出现ValueError: Duplicate plugins for name projector问题
在terminal运行如下指令打开tensorboard出现错误：
tensorboard --logdir=E:\pythonFiles\files\LLMRank-master\log_tensorboard
解决方法：
安装了重复的tensorboard，需要去删除多余的：
最终保留如下的即可
之后登录本地网站进行测试：
http://localhost:6006/
6.
解决方法：降低cffi版本到1.14.0：
pip install cffi==1.14.0
7.将linux中的一个文件夹中所有的文件复制粘贴到/home/dell目录中：
cp -r 文件夹名称 /home/dell
8.出现报错：TypeError: dispatch_model() got an unexpected keyword argument 'offload_index'
请尝试更新加速包：
pip install accelerate==0.18.0