syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
1.问题描述
今天用faster-rcnn跑数据模型报错：
prepared the input data
F0531 13:41:25.938465 12409 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7f23e6c985cd google::LogMessage::Fail()
@ 0x7f23e6c9a433 google::LogMessage::SendToLog()
@ 0x7f23e6c9815b google::LogMessage::Flush()
@ 0x7f23e6c9ae1e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f23e7d89420 caffe::SyncedMemory::to_gpu()
@ 0x7f23e7d883e9 caffe::SyncedMemory::mutable_gpu_data()
@ 0x7f23e7d8a782 caffe::Blob<>::mutable_gpu_data()
@ 0x7f23e7dc5ee8 caffe::BaseConvolutionLayer<>::forward_gpu_gemm()
@ 0x7f23e7f15616 caffe::ConvolutionLayer<>::Forward_gpu()
@ 0x7f23e7ebb7b2 caffe::Net<>::ForwardFromTo()
@ 0x7f23e7ebb906 caffe::Net<>::ForwardPrefilled()
@ 0x7f23e84be176 Detector::Detect()
@ 0x442990 FasterDetect()
@ 0x440de4 PredictContainer2()
@ 0x442dd1 imagedeal()
@ 0x442e67 main
@ 0x7f23e5989830 __libc_start_main
@ 0x4404e9 _start
@ (nil) (unknown)
已放弃 (核心已转储)
2.解决过程
这种错误先看了下cpu内存，top命令查看了下发现内存够用，然后以为是图像数据有问题，或者矩阵运算有内存越界等错误，忙了大半天，没有找出原因，后来研究了下caffe，运算好像是在显示芯片上面运算的（GPU），查看下内存命令：
john@ubun:~/Project/Serverbk$nvidia-smi
Fri May 31 13:41:32 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:65:00.0 On | N/A |
| 0% 58C P2 62W / 250W | 8413MiB / 11169MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1047 G /usr/lib/xorg/Xorg 221MiB |
| 0 1822 G compiz 61MiB |
| 0 2798 C ./Server 189MiB |
| 0 3424 C ./Imagedeal 6981MiB |
| 0 3513 C ./Server 189MiB |
| 0 3563 C ./Imagedeal 189MiB |
| 0 4186 C ./Server 189MiB |
| 0 9778 C ./Server 189MiB |
| 0 10094 G /usr/lib/firefox/firefox 2MiB |
| 0 11819 C ./Imagedeal 189MiB |
+-----------------------------------------------------------------------------+
然后杀了下进程：
john@ubun:~/Project/Serverbk$kill %
[8]+ 已停止 ./Imagedeal
john@ubun:~/Project/Serverbk$
[8]+ 已终止 ./Imagedeal
john@ubun:~/Project/Serverbk$
john@ubun:~/Project/Serverbk$
john@ubun:~/Project/Serverbk$
john@ubun:~/Project/Serverbk$nvidia-smi
Fri May 31 13:49:36 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:65:00.0 On | N/A |
| 0% 51C P0 63W / 250W | 282MiB / 11169MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1047 G /usr/lib/xorg/Xorg 214MiB |
| 0 1822 G compiz 62MiB |
| 0 10094 G /usr/lib/firefox/firefox 2MiB |
+-----------------------------------------------------------------------------+
问题解决！
最后再写个sh脚本，让程序运行前先杀一下后台程序！