pynvml.nvml.NVMLError_FunctionNotFound: Function Not Found
在Docker中运行报错：
Traceback (most recent call last):
File "/opt/conda/envs/rapids/lib/python3.8/site-packages/pynvml/nvml.py", line 782, in _nvmlGetFunctionPointer
_nvmlGetFunctionPointer_cache[name] = getattr(nvmlLib, name)
File "/opt/conda/envs/rapids/lib/python3.8/ctypes/__init__.py", line 386, in __getattr__
func = self.__getitem__(name)
File "/opt/conda/envs/rapids/lib/python3.8/ctypes/__init__.py", line 391, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1: undefined symbol: nvmlDeviceGetComputeRunningProcesses_v2
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/rapids/lib/python3.8/site-packages/dask_cuda/initialize.py", line 32, in _create_cuda_context
distributed.comm.ucx.init_once()
File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/comm/ucx.py", line 86, in init_once
pre_existing_cuda_context = has_cuda_context()
File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/diagnostics/nvml.py", line 91, in has_cuda_context
running_processes = pynvml.nvmlDeviceGetComputeRunningProcesses_v2(handle)
File "/opt/conda/envs/rapids/lib/python3.8/site-packages/pynvml/nvml.py", line 2191, in nvmlDeviceGetComputeRunningProcesses_v2
fn = _nvmlGetFunctionPointer("nvmlDeviceGetComputeRunningProcesses_v2")
File "/opt/conda/envs/rapids/lib/python3.8/site-packages/pynvml/nvml.py", line 785, in _nvmlGetFunctionPointer
raise NVMLError(NVML_ERROR_FUNCTION_NOT_FOUND)
pynvml.nvml.NVMLError_FunctionNotFound: Function Not Found
2022-05-16 15:19:14,517 - distributed.preloading - INFO - Run preload setup click command: dask_cuda.initialize
2022-05-16 15:19:14,517 - distributed.worker - INFO - Start worker at: ws://10.233.68.22:39537/
2022-05-16 15:19:14,517 - distributed.worker - INFO - Listening to: ws://10.233.68.22:39537/
2022-05-16 15:19:14,517 - distributed.worker - INFO - dashboard at: 10.233.68.22:35313
2022-05-16 15:19:14,517 - distributed.worker - INFO - Waiting to connect to: ws://launcher-svc-1245231:8786/
2022-05-16 15:19:14,517 - distributed.worker - INFO - -------------------------------------------------
2022-05-16 15:19:14,517 - distributed.worker - INFO - Threads: 1
2022-05-16 15:19:14,517 - distributed.worker - INFO - Memory: 400.00 GiB
2022-05-16 15:19:14,517 - distributed.worker - INFO - Local Directory: /rapids/notebooks/dask-worker-space/worker-ave_m7tw
2022-05-16 15:19:14,517 - distributed.worker - INFO - Starting Worker plugin PreImport-0b003d61-7c5f-4530-bf6f-c95b93c83338
2022-05-16 15:19:14,517 - distributed.worker - INFO - Starting Worker plugin CPUAffinity-a1d437c7-bb5d-408e-a3e0-3120dd6c6a5f
2022-05-16 15:19:14,518 - distributed.worker - INFO - Starting Worker plugin RMMSetup-03e12d8b-4b23-4e0e-9b3c-a79b6b12e7ab
2022-05-16 15:19:14,974 - distributed.worker - INFO - -------------------------------------------------
2022-05-16 15:19:15,025 - distributed.worker - INFO - Registered to: ws://launcher-svc-1245231:8786/
2022-05-16 15:19:15,025 - distributed.worker - INFO - -------------------------------------------------
2022-05-16 15:19:15,026 - distributed.core - INFO - Starting established connection
用nvidia-smi查看当前Cuda版本：
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.199.02 Driver Version: 470.199.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A |
| 35% 33C P8 18W / 220W | 552MiB / 7959MiB | 13% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
参照提示：解决方案
因为Cuda和pynvml库间存在对应关系，要么升级Cuda，要么降级pynvml。
进入python3，查看pynvml版本：
>>> import pynvml
>>> print(pynvml.__version__)
11.5.1
猜想可能是pynvml版本过高与Cuda不匹配导致的，直接通过pip降级pynvml。
pip install pynvml==11.4.1
问题解决。