【MindSpore-GPU-1.1.0】【LeNet5】训练报cudaHostAlloc failed
MindSpore-GPU-1.1.0
运行环境
Windows 10 家庭中文版 2004 20279.1
WSL2
Ubuntu 18.04
cudatoolkit 1.
cudnn
conda
【操作步骤&问题现象】
python train.py --device_target="GPU"
【日志信息】（可选，上传日志内容或者附件）
============== Starting Training ==============
libnuma: Warning: Cannot read node cpumask from sysfs
numa_sched_setaffinity_v2_int() failed; abort
: Invalid argument
set_mempolicy: Function not implemented
numa_sched_setaffinity_v2_int() failed; abort
: Invalid argument
set_mempolicy: Function not implemented
[ERROR] MD(4311,python):2021-01-01-21:18:15.974.858 [mindspore/ccsrc/minddata/dataset/util/arena.cc:242] Init] cudaHostAlloc failed, ret[2], out of memory
[ERROR] KERNEL(4311,python):2021-01-01-21:23:15.982.517 [mindspore/ccsrc/backend/kernel_compiler/gpu/data/dataset_iterator_kernel.cc:114] ReadDevice] Get data timeout
[ERROR] DEVICE(4311,python):2021-01-01-21:23:15.982.628 [mindspore/ccsrc/runtime/device/gpu/gpu_kernel_runtime.cc:652] LaunchKernelDynamic] Op Error: Launch kernel failed. | Error Number: 0
Traceback (most recent call last):
File "train.py", line 71, in
dataset_sink_mode=args.dataset_sink_mode)
File "/home/bamboo/miniconda3/envs/ms/lib/python3.7/site-packages/mindspore/train/model.py", line 592, in train
sink_size=sink_size)
File "/home/bamboo/miniconda3/envs/ms/lib/python3.7/site-packages/mindspore/train/model.py", line 391, in _train
self._train_dataset_sink_process(epoch, train_dataset, list_callback, cb_params, sink_size)
File "/home/bamboo/miniconda3/envs/ms/lib/python3.7/site-packages/mindspore/train/model.py", line 452, in _train_dataset_sink_process
outputs = self._train_network(*inputs)
File "/home/bamboo/miniconda3/envs/ms/lib/python3.7/site-packages/mindspore/nn/cell.py", line 331, in __call__
out = self.compile_and_run(*inputs)
File "/home/bamboo/miniconda3/envs/ms/lib/python3.7/site-packages/mindspore/nn/cell.py", line 602, in compile_and_run
return _executor(self, *new_inputs, phase=self.phase)
File "/home/bamboo/miniconda3/envs/ms/lib/python3.7/site-packages/mindspore/common/api.py", line 582, in __call__
return self.run(obj, *args, phase=phase)
File "/home/bamboo/miniconda3/envs/ms/lib/python3.7/site-packages/mindspore/common/api.py", line 610, in run
return self._exec_pip(obj, *args, phase=phase_real)
File "/home/bamboo/miniconda3/envs/ms/lib/python3.7/site-packages/mindspore/common/api.py", line 75, in wrapper
results = fn(*arg, **kwargs)
File "/home/bamboo/miniconda3/envs/ms/lib/python3.7/site-packages/mindspore/common/api.py", line 593, in _exec_pip
return self._executor(args_list, phase)
RuntimeError: mindspore/ccsrc/runtime/device/gpu/gpu_kernel_runtime.cc:652 LaunchKernelDynamic] Op Error: Launch kernel failed. | Error Number: 0
解答：
日志中报错提及：out of memory, 应该为内存不够：
[ERROR] MD(4311,python):.../arena.cc:242] Init] cudaHostAlloc failed, ret[2], out of memory
请查下机器上有多少内存，您的gpu机器应该为numa架构，请基于此命令 numastat -cm进行查看 , 正常能看到类似下图结果
当前GPU上运行初始化时，数据模块会尝试申请2G 内存，请结合上述命令分析机器内存是否够用
此外网络部分也会尝试申请内存，请执行上述命令,监控网络运行期间内存的变化，是否已满负荷
$ numastat -cm
Per-node system memory usage(in MBs):
Node 0 Node 1 Total
-------- --------- ----------
MemTotal 12524 12915 25439
MemFree **** **** ****
MemUsed **** **** ***