Keras/TensorFlow报错CUDA_ERROR_OUT_OF_MEMORY:

用keras的model.predict时报错如下:

failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-03-07 10:24:00.610883: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.60G (3865470464 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-03-07 10:24:00.613876: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.24G (3478923264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-03-07 10:24:00.617918: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.92G (3131030784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-03-07 10:24:00.621197: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.62G (2817927680 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-03-07 10:24:00.624652: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.36G (2536134912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-03-07 10:24:00.627891: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.13G (2282521344 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-03-07 10:24:00.630717: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.11GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-03-07 10:24:00.647626: F tensorflow/stream_executor/cuda/cuda_dnn.cc:231] Check failed: status == CUDNN_STATUS_SUCCESS (7 vs. 0)Failed to set cuDNN stream.

明明显存有8G是够的,但是还说out of memory,于是添加如下代码限制GPU内存占用率

import tensorflow as tf
import keras.backend.tensorflow_backend as KTF

config = tf.ConfigProto()
config.gpu_options.allow_growth = True  # 不全部占满显存, 按需分配
config.gpu_options.per_process_gpu_memory_fraction = 0.6  #限制GPU内存占用率
sess = tf.Session(config=config)
KTF.set_session(sess)  # 设置session

问题解决

你可能感兴趣的:(Keras/TensorFlow报错CUDA_ERROR_OUT_OF_MEMORY:)