OSError: (External) CUDA error(222), the provided PTX was compiled with an unsupported toolchain..

使用百度 Paddle 报错：

OSError: (External) CUDA error(222), the provided PTX was compiled with an unsupported toolchain…
[Hint: ‘cudaErrorUnsupportedPtxVersion’. This indicates that the provided PTX was compiled with an unsupported toolchain. The most common reason for this, is the PTXwas generated by a compiler newer than what is supported by the CUDA driver and PTX JIT compiler.] (at …/paddle/fluid/platform/device/gpu/gpu_info.cc:124)

我的情况

ubuntu 20.04
RTX 3090TI
本来的驱动 NVIDIA 470
安装了CUDA 10.1, 11.1, 11.3, 11.4, 11.8, 12.0都不行。

成功解决：NVIDIA driver 535, CUDA 12.2

解决过程

卸载 CUDA

sudo rm -rf /usr/local/cuda*

卸载NVIDIA驱动

检查一下已经安装的NVIDIA驱动，返回为空的话，说明没有驱动，直接跳过这一步

dpkg -l | grep -i nvidia

卸载

sudo apt-get remove --purge '^nvidia-.*'
sudo apt-get remove --purge 'libnvidia-.*'
sudo apt autoremove

再次检查

dpkg -l | grep -i nvidia

安装NVIDIA 535驱动

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-driver-535

重启

sudo reboot

检查驱动是否成功

nvidia-smi

安装 CUDA 12.2

wget https://developer.download.nvidia.com/compute/cuda/12.0.0/local_installers/cuda_12.0.0_525.60.13_linux.run
sudo sh cuda_12.0.0_525.60.13_linux.run

安装cudnn 8.9

百度网盘链接：https://pan.baidu.com/s/18mEvR_RgeMqqju6DR9ixqQ
提取码：fr70

tar -xf cudnn-linux-x86_64-8.9.5.29_cuda12-archive.tar.xz
mv cudnn-linux-x86_64-8.9.5.29_cuda12-archive cuda
sudo cp -P cuda/include/cudnn.h /usr/local/cuda/include
sudo cp -P cuda/lib/libcudnn* /usr/local/cuda/lib64/
sudo chmod a+r /usr/local/cuda/lib64/libcudnn*

检查CUDA

nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:02:13_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0