【debug】报错RuntimeError: CUDA error: an illegal memory access was encountered

目录

 报错信息:

解决:

整理不易,欢迎一键三连!!!


 

介绍:mmseg工程在一台服务器上训练了几十个epoch,在新的服务器上接着resume,已经训练了500个iter,报错RuntimeError: CUDA error: an illegal memory access was encountered

 报错信息:

  File "/data/xx/project/mmsegmentation-master/mmseg/models/losses/accuracy.py", line 49, in accuracy
    correct = correct[:, target != ignore_index]
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: an illegal memory access was encountered

解决:

data = dict(
    samples_per_gpu=16,
    workers_per_gpu=16,
    ......
    )

修改了batch_size 和worker就好了。

整理不易,欢迎一键三连!!!

送你们一条美丽的--分割线--


⛵⛵⭐⭐

你可能感兴趣的:(Python,mmSegmentation,Debug,人工智能,深度学习,机器学习)