RuntimeError: CUDA error: device-side assert triggered [349] CUDA kernel errors might be asynchronou

报错如下：

Traceback (most recent call last):
[338]   File "/yj/code/global_cover_and_landmark/train.py", line 193, in <module>
[339]     main(args)
[340]   File "/yj/code/global_cover_and_landmark/train.py", line 164, in main
[341]     train(args=args, epoch=epoch, trainloader=trainloader, net=net, optimizer=optimizer,
[342]   File "/yj/code/global_cover_and_landmark/train.py", line 59, in train
[343]     total_loss.backward()
[344]   File "/usr/local/lib/python3.8/dist-packages/torch/_tensor.py", line 363, in backward
[345]     torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
[346]   File "/usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py", line 173, in backward
[347]     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[348] RuntimeError: CUDA error: device-side assert triggered
[349] CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
[350] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

解决方法:
在train.py的开头加上两行代码，其实是指定只使用一张卡

import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'

查找解决的时候，发现还有很多其他原因导致的，欢迎大家补充，嘻嘻嘻