GTX 950 内存太小，训练VGG6 Resource exhausted 错误

主要参考这篇文章：利用TensorFlow实现VGG16（http://blog.csdn.net/v1_vivian/article/details/77898652）。用GTX 950 2G RAM训练。直接Resource exhausted，内存太小。

2017-11-28 23:25:44.991463: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2017-11-28 23:25:44.991763: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1031] Found device 0 with properties: 
name: GeForce GTX 950 major: 5 minor: 2 memoryClockRate(GHz): 1.355
pciBusID: 0000:26:00.0
totalMemory: 1.95GiB freeMemory: 1.21GiB
2017-11-28 23:25:44.991779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 950, pci bus id: 0000:26:00.0, compute capability: 5.2)
2017-11-28 23:25:55.865683: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 9.00MiB.  Current allocation summary follows.
2017-11-28 23:25:55.865739: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (256): Total Chunks: 36, Chunks in use: 36. 9.0KiB allocated for chunks. 9.0KiB in use in bin. 1.4KiB client-requested in use in bin.
2017-11-28 23:25:55.865760: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (512): Total Chunks: 5, Chunks in use: 5. 2.5KiB allocated for chunks. 2.5KiB in use in bin. 2.5KiB client-requested in use in bin.
2017-11-28 23:25:55.865771: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1024): Total Chunks: 8, Chunks in use: 8. 8.2KiB allocated for chunks. 8.2KiB in use in bin. 8.0KiB client-requested in use in bin.
2017-11-28 23:25:55.865782: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2048): Total Chunks: 14, Chunks in use: 14. 28.0KiB allocated for chunks. 28.0KiB in use in bin. 28.0KiB client-requested in use in bin.
2017-11-28 23:25:55.865793: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4096): Total Chunks: 2, Chunks in use: 2. 13.5KiB allocated for chunks. 13.5KiB in use in bin. 13.5KiB client-requested in use in bin.
2017-11-28 23:25:55.865803: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-28 23:25:55.865813: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16384): Total Chunks: 3, Chunks in use: 3. 48.0KiB allocated for chunks. 48.0KiB in use in bin. 48.0KiB client-requested in use in bin.
2017-11-28 23:25:55.865824: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (32768): Total Chunks: 1, Chunks in use: 1. 32.0KiB allocated for chunks. 32.0KiB in use in bin. 32.0KiB client-requested in use in bin.
2017-11-28 23:25:55.865833: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-28 23:25:55.865844: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (131072): Total Chunks: 2, Chunks in use: 2. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2017-11-28 23:25:55.865854: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (262144): Total Chunks: 2, Chunks in use: 2. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2017-11-28 23:25:55.865865: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (524288): Total Chunks: 2, Chunks in use: 2. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2017-11-28 23:25:55.865875: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1048576): Total Chunks: 2, Chunks in use: 2. 2.25MiB allocated for chunks. 2.25MiB in use in bin. 2.25MiB client-requested in use in bin.
2017-11-28 23:25:55.865885: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2097152): Total Chunks: 4, Chunks in use: 4. 9.00MiB allocated for chunks. 9.00MiB in use in bin. 9.00MiB client-requested in use in bin.
2017-11-28 23:25:55.865895: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4194304): Total Chunks: 2, Chunks in use: 2. 9.00MiB allocated for chunks. 9.00MiB in use in bin. 9.00MiB client-requested in use in bin.
2017-11-28 23:25:55.865905: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8388608): Total Chunks: 9, Chunks in use: 9. 81.00MiB allocated for chunks. 81.00MiB in use in bin. 81.00MiB client-requested in use in bin.
2017-11-28 23:25:55.865914: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-28 23:25:55.865922: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-28 23:25:55.865933: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (67108864): Total Chunks: 1, Chunks in use: 1. 64.00MiB allocated for chunks. 64.00MiB in use in bin. 64.00MiB client-requested in use in bin.
2017-11-28 23:25:55.865942: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-28 23:25:55.865953: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (268435456): Total Chunks: 2, Chunks in use: 2. 841.64MiB allocated for chunks. 841.64MiB in use in bin. 784.00MiB client-requested in use in bin.
2017-11-28 23:25:55.865962: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin for 9.00MiB was 8.00MiB, Chunk State: 
2017-11-28 23:25:55.865972: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503180000 of size 1280
2017-11-28 23:25:55.865980: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503180500 of size 256
2017-11-28 23:25:55.865987: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503180600 of size 256
2017-11-28 23:25:55.865994: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503180700 of size 256
2017-11-28 23:25:55.866001: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503180800 of size 256
2017-11-28 23:25:55.866008: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503180900 of size 256
2017-11-28 23:25:55.866014: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503180a00 of size 256
2017-11-28 23:25:55.866021: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503180b00 of size 256
2017-11-28 23:25:55.866029: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503180c00 of size 512
2017-11-28 23:25:55.866036: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503180e00 of size 256
2017-11-28 23:25:55.866043: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503180f00 of size 256
2017-11-28 23:25:55.866050: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503181000 of size 256
2017-11-28 23:25:55.866057: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503181100 of size 256
2017-11-28 23:25:55.866064: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503181200 of size 1024
2017-11-28 23:25:55.866071: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503181600 of size 256
2017-11-28 23:25:55.866078: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503181700 of size 256
2017-11-28 23:25:55.866085: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503181800 of size 256
2017-11-28 23:25:55.866092: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503181900 of size 256
2017-11-28 23:25:55.866099: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503181a00 of size 256
2017-11-28 23:25:55.866106: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503181b00 of size 2048
2017-11-28 23:25:55.866113: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503182300 of size 256
2017-11-28 23:25:55.866120: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503182400 of size 256
2017-11-28 23:25:55.866126: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503182500 of size 2048
2017-11-28 23:25:55.866133: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503182d00 of size 256
2017-11-28 23:25:55.866140: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503182e00 of size 256
2017-11-28 23:25:55.866147: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503182f00 of size 256
2017-11-28 23:25:55.866154: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x503183000 of size 411041792
2017-11-28 23:25:55.866161: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b983000 of size 256
2017-11-28 23:25:55.866168: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b983100 of size 256
2017-11-28 23:25:55.866175: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b983200 of size 256
2017-11-28 23:25:55.866182: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b983300 of size 16384
2017-11-28 23:25:55.866189: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b987300 of size 256
2017-11-28 23:25:55.866196: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b987400 of size 256
2017-11-28 23:25:55.866203: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b987500 of size 256
2017-11-28 23:25:55.866210: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b987600 of size 256
2017-11-28 23:25:55.866216: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b987700 of size 256
2017-11-28 23:25:55.866223: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b987800 of size 256
2017-11-28 23:25:55.866230: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b987900 of size 256
2017-11-28 23:25:55.866237: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b987a00 of size 256
2017-11-28 23:25:55.866244: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b987b00 of size 6912
2017-11-28 23:25:55.866251: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b989600 of size 256
2017-11-28 23:25:55.866259: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b989700 of size 147456
2017-11-28 23:25:55.866266: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b9ad700 of size 16384
2017-11-28 23:25:55.866273: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b9b1700 of size 256
2017-11-28 23:25:55.866280: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b9b1800 of size 294912
2017-11-28 23:25:55.866287: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b9f9800 of size 512
2017-11-28 23:25:55.866293: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51b9f9a00 of size 589824
2017-11-28 23:25:55.866299: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51ba89a00 of size 512
2017-11-28 23:25:55.866305: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51ba89c00 of size 1179648
2017-11-28 23:25:55.866312: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51bba9c00 of size 67108864
2017-11-28 23:25:55.866317: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51fba9c00 of size 1024
2017-11-28 23:25:55.866324: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51fbaa000 of size 2359296
2017-11-28 23:25:55.866330: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51fdea000 of size 1024
2017-11-28 23:25:55.866336: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51fdea400 of size 2359296
2017-11-28 23:25:55.866342: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x52002a400 of size 1024
2017-11-28 23:25:55.866348: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x52002a800 of size 4718592
2017-11-28 23:25:55.866354: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5204aa800 of size 16384
2017-11-28 23:25:55.866359: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5204ae800 of size 2048
2017-11-28 23:25:55.866366: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5204af000 of size 9437184
2017-11-28 23:25:55.866372: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x520daf000 of size 2048
2017-11-28 23:25:55.866378: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x520daf800 of size 9437184
2017-11-28 23:25:55.866383: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5216af800 of size 2048
2017-11-28 23:25:55.866389: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5216b0000 of size 9437184
2017-11-28 23:25:55.866396: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x521fb0000 of size 32768
2017-11-28 23:25:55.866402: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x521fb8000 of size 2048
2017-11-28 23:25:55.866407: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x521fb8800 of size 9437184
2017-11-28 23:25:55.866413: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5228b8800 of size 2048
2017-11-28 23:25:55.866419: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5228b9000 of size 9437184
2017-11-28 23:25:55.866425: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5231b9000 of size 6912
2017-11-28 23:25:55.866431: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5231bab00 of size 256
2017-11-28 23:25:55.866437: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5231bac00 of size 147456
2017-11-28 23:25:55.866442: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5231dec00 of size 256
2017-11-28 23:25:55.866448: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5231ded00 of size 294912
2017-11-28 23:25:55.866454: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x523226d00 of size 512
2017-11-28 23:25:55.866460: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x523226f00 of size 589824
2017-11-28 23:25:55.866466: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5232b6f00 of size 512
2017-11-28 23:25:55.866472: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5232b7100 of size 1179648
2017-11-28 23:25:55.866477: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5233d7100 of size 1024
2017-11-28 23:25:55.866484: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5233d7500 of size 2359296
2017-11-28 23:25:55.866490: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x523617500 of size 1024
2017-11-28 23:25:55.866495: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x523617900 of size 2359296
2017-11-28 23:25:55.866501: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x523857900 of size 1024
2017-11-28 23:25:55.866507: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x523857d00 of size 4718592
2017-11-28 23:25:55.866513: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x523cd7d00 of size 2048
2017-11-28 23:25:55.866519: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x523cd8500 of size 9437184
2017-11-28 23:25:55.866525: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5245d8500 of size 2048
2017-11-28 23:25:55.866530: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5245d8d00 of size 2048
2017-11-28 23:25:55.866536: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5245d9500 of size 9437184
2017-11-28 23:25:55.866542: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x524ed9500 of size 2048
2017-11-28 23:25:55.866548: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x524ed9d00 of size 2048
2017-11-28 23:25:55.866554: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x524eda500 of size 9437184
2017-11-28 23:25:55.866560: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5257da500 of size 2048
2017-11-28 23:25:55.866566: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5257dad00 of size 9437184
2017-11-28 23:25:55.866571: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5260dad00 of size 2048
2017-11-28 23:25:55.866577: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5260db500 of size 471485184
2017-11-28 23:25:55.866584: I tensorflow/core/common_runtime/bfc_allocator.cc:676]      Summary of in-use Chunks by size: 
2017-11-28 23:25:55.866592: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 36 Chunks of size 256 totalling 9.0KiB
2017-11-28 23:25:55.866599: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 5 Chunks of size 512 totalling 2.5KiB
2017-11-28 23:25:55.866606: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 7 Chunks of size 1024 totalling 7.0KiB
2017-11-28 23:25:55.866613: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1280 totalling 1.2KiB
2017-11-28 23:25:55.866620: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 14 Chunks of size 2048 totalling 28.0KiB
2017-11-28 23:25:55.866627: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 6912 totalling 13.5KiB
2017-11-28 23:25:55.866634: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 16384 totalling 48.0KiB
2017-11-28 23:25:55.866641: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 32768 totalling 32.0KiB
2017-11-28 23:25:55.866648: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 147456 totalling 288.0KiB
2017-11-28 23:25:55.866655: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 294912 totalling 576.0KiB
2017-11-28 23:25:55.866662: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 589824 totalling 1.12MiB
2017-11-28 23:25:55.866669: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 1179648 totalling 2.25MiB
2017-11-28 23:25:55.866676: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 2359296 totalling 9.00MiB
2017-11-28 23:25:55.866683: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 4718592 totalling 9.00MiB
2017-11-28 23:25:55.866690: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 9 Chunks of size 9437184 totalling 81.00MiB
2017-11-28 23:25:55.866697: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 67108864 totalling 64.00MiB
2017-11-28 23:25:55.866704: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 411041792 totalling 392.00MiB
2017-11-28 23:25:55.866712: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 471485184 totalling 449.64MiB
2017-11-28 23:25:55.866719: I tensorflow/core/common_runtime/bfc_allocator.cc:683] Sum Total of in-use chunks: 1009.00MiB
2017-11-28 23:25:55.866729: I tensorflow/core/common_runtime/bfc_allocator.cc:685] Stats: 
Limit:                  1058013184
InUse:                  1058013184
MaxInUse:               1058013184
NumAllocs:                      95
MaxAllocSize:            471485184


2017-11-28 23:25:55.866740: W tensorflow/core/common_runtime/bfc_allocator.cc:277] ***********************************************************************************************xxxxx
2017-11-28 23:25:55.866758: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[3,3,512,512]