【解决问题】Hint: If you want to see a list of allocated tensors when OOM happen

今天运行一个风格迁移神经网络的时候,出现以下问题:

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,64,1669,1200] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[Node: Conv2D_10 = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2D_10-0-TransposeNHWCToNCHW-LayoutOptimizer, encoder/conv1_2/kernel/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Node: clip_by_value/_77 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_487_clip_by_value", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

上网查了以后,说是GPU满了,可以调小 Batch_size来解决,但是我的batch_size本来就只有8,于是下调到4,依然不行。

于是根据其他博主的建议,指定GPU后:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "2"#指定第二块GPU

解决了,但是由于显卡不是很大 所以运行起来特别慢。

参考:

  • 参考博客1
  • 深度学习笔记:windows+tensorflow 指定GPU占用内存(解决gpu爆炸问题)

你可能感兴趣的:(错题集)