CUDA 显存释放：GPU无程序运行，仍有部分显存被占用

GPU无程序运行，仍有显存被占用，如图：

root@44668fac5caf:/CUDA# nvidia-smi
Fri Dec  6 05:25:17 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.74       Driver Version: 418.74       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:3B:00.0 Off |                    0 |
| N/A   42C    P0    35W / 250W |   5338MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE...  Off  | 00000000:AF:00.0 Off |                  N/A |
| N/A   41C    P0    35W / 250W |   8241MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

解决：

 通过下面命令查看GPU上所有的程序：

fuser -v /dev/nvidia*

若出现 fuser：command not found,可阅读这篇博客：

Ubuntu：fuser command not found

命令输出结果如下：

root@44668fac5caf:/CUDA# fuser -v /dev/nvidia*
                     USER        PID ACCESS COMMAND
/dev/nvidia-uvm:     root     kernel mount /dev/nvidia-uvm
                     root      202377 F...m cuda
                     root      202562 F...m cuda
/dev/nvidia-uvm-tools:
                     root     kernel mount /dev/nvidia-uvm-tools
/dev/nvidia0:        root     kernel mount /dev/nvidia0
                     root       1037 F.... nvidia-smi
                     root      202377 F...m cuda
                     root      202562 F...m cuda
/dev/nvidia1:        root     kernel mount /dev/nvidia1
                     root      202377 F...m cuda
                     root      202562 F...m cuda
/dev/nvidiactl:      root     kernel mount /dev/nvidiactl
                     root       1037 F.... nvidia-smi
                     root      202377 F...m cuda
                     root      202562 F...m cuda

可以按照下面命令，杀死相关进程：

kill -9 PID