WARNING:tensorflow:From D:\GitHub\CapsNet-Tensorflow\utils.py:89: slice_input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(tuple(tensor_list)).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.
WARNING:tensorflow:From C:\Miniconda3\lib\site-packages\tensorflow\python\training\input.py:372: range_input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.range(limit).shuffle(limit).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.
WARNING:tensorflow:From C:\Miniconda3\lib\site-packages\tensorflow\python\training\input.py:318: input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.
WARNING:tensorflow:From C:\Miniconda3\lib\site-packages\tensorflow\python\training\input.py:188: limit_epochs (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensors(tensor).repeat(num_epochs)`.
WARNING:tensorflow:From C:\Miniconda3\lib\site-packages\tensorflow\python\training\input.py:197: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From C:\Miniconda3\lib\site-packages\tensorflow\python\training\input.py:197: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From D:\GitHub\CapsNet-Tensorflow\utils.py:94: shuffle_batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.shuffle(min_after_dequeue).batch(batch_size)`.
WARNING:tensorflow:From D:\GitHub\CapsNet-Tensorflow\main.py:118: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-03-09 16:03:25.423321: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-03-09 16:03:26.161740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce 940MX major: 5 minor: 0 memoryClockRate(GHz): 1.2415
pciBusID: 0000:06:00.0
totalMemory: 2.00GiB freeMemory: 1.65GiB
2019-03-09 16:03:26.177190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-03-09 16:03:27.553913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-09 16:03:27.554403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2019-03-09 16:03:27.554761: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2019-03-09 16:03:27.555263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1401 MB memory) -> physical GPU (device: 0, name: GeForce 940MX, pci bus id: 0000:06:00.0, compute capability: 5.0)
Note: all of results will be saved to directory: results
Training for epoch 0/50:
0%| | 0/429 [00:00
2019-03-09 16:05:24.438309: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 544.02MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-03-09 16:05:34.442486: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 720.00MiB. Current allocation summary follows.
2019-03-09 16:05:34.443922: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 45, Chunks in use: 43. 11.3KiB allocated for chunks. 10.8KiB in use in bin. 172B client-requested in use in bin.
2019-03-09 16:05:34.444819: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 4, Chunks in use: 4. 3.0KiB allocated for chunks. 3.0KiB in use in bin. 2.5KiB client-requested in use in bin.
2019-03-09 16:05:34.445645: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 10, Chunks in use: 9. 10.8KiB allocated for chunks. 9.3KiB in use in bin. 9.0KiB client-requested in use in bin.
2019-03-09 16:05:34.446490: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 11, Chunks in use: 8. 31.0KiB allocated for chunks. 21.0KiB in use in bin. 20.3KiB client-requested in use in bin.
2019-03-09 16:05:34.447777: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 11, Chunks in use: 10. 50.0KiB allocated for chunks. 46.0KiB in use in bin. 46.0KiB client-requested in use in bin.
2019-03-09 16:05:34.448726: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 2, Chunks in use: 2. 19.5KiB allocated for chunks. 19.5KiB in use in bin. 10.0KiB client-requested in use in bin.
2019-03-09 16:05:34.449553: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-09 16:05:34.450159: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-09 16:05:34.450895: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 4, Chunks in use: 4. 324.0KiB allocated for chunks. 324.0KiB in use in bin. 324.0KiB client-requested in use in bin.
2019-03-09 16:05:34.451656: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-09 16:05:34.452333: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 7, Chunks in use: 7. 2.43MiB allocated for chunks. 2.43MiB in use in bin. 2.40MiB client-requested in use in bin.
2019-03-09 16:05:34.453096: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 4, Chunks in use: 3. 2.25MiB allocated for chunks. 1.69MiB in use in bin. 1.69MiB client-requested in use in bin.
2019-03-09 16:05:34.453704: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-09 16:05:34.454382: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 7, Chunks in use: 7. 17.19MiB allocated for chunks. 17.19MiB in use in bin. 17.19MiB client-requested in use in bin.
2019-03-09 16:05:34.455384: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 7, Chunks in use: 7. 35.50MiB allocated for chunks. 35.50MiB in use in bin. 34.56MiB client-requested in use in bin.
2019-03-09 16:05:34.456283: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 3, Chunks in use: 2. 32.62MiB allocated for chunks. 18.54MiB in use in bin. 11.25MiB client-requested in use in bin.
2019-03-09 16:05:34.457115: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 3, Chunks in use: 2. 69.75MiB allocated for chunks. 40.50MiB in use in bin. 40.50MiB client-requested in use in bin.
2019-03-09 16:05:34.458949: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 4, Chunks in use: 3. 170.20MiB allocated for chunks. 120.12MiB in use in bin. 90.50MiB client-requested in use in bin.
2019-03-09 16:05:34.460676: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 1, Chunks in use: 0. 91.19MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-09 16:05:34.461385: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-09 16:05:34.462028: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 3, Chunks in use: 1. 979.86MiB allocated for chunks. 256.00MiB in use in bin. 164.49MiB client-requested in use in bin.
2019-03-09 16:05:34.462787: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 720.00MiB was 256.00MiB, Chunk State:
2019-03-09 16:05:34.463166: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 361.43MiB | Requested Size: 137.62MiB | in_use: 0, prev: Size: 50.00MiB | Requested Size: 50.00MiB | in_use: 1
2019-03-09 16:05:34.463779: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 362.43MiB | Requested Size: 272.08MiB | in_use: 0
2019-03-09 16:05:34.464216: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000701240000 of size 1024
2019-03-09 16:05:34.464609: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000701240400 of size 82944
2019-03-09 16:05:34.464917: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000701254800 of size 256
2019-03-09 16:05:34.465358: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000701254900 of size 256
2019-03-09 16:05:34.465739: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000701254A00 of size 2048
2019-03-09 16:05:34.466167: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000701255200 of size 327680
2019-03-09 16:05:34.466652: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A5200 of size 256
2019-03-09 16:05:34.467041: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A5300 of size 256
2019-03-09 16:05:34.467404: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A5400 of size 4096
2019-03-09 16:05:34.467775: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A6400 of size 256
2019-03-09 16:05:34.468091: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A6500 of size 256
2019-03-09 16:05:34.468436: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A6600 of size 3328
2019-03-09 16:05:34.468677: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A7300 of size 256
2019-03-09 16:05:34.469020: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A7400 of size 256
2019-03-09 16:05:34.469309: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A7500 of size 256
2019-03-09 16:05:34.469609: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A7600 of size 768
2019-03-09 16:05:34.469951: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A7900 of size 256
2019-03-09 16:05:34.470237: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A7A00 of size 256
2019-03-09 16:05:34.470606: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A7B00 of size 1024
2019-03-09 16:05:34.471040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A7F00 of size 256
2019-03-09 16:05:34.471417: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A8000 of size 256
2019-03-09 16:05:34.471780: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A8100 of size 256
2019-03-09 16:05:34.472159: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A8200 of size 256
2019-03-09 16:05:34.472564: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A8300 of size 1280
2019-03-09 16:05:34.472892: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A8800 of size 1024
2019-03-09 16:05:34.473227: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A8C00 of size 1024
2019-03-09 16:05:34.473575: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A9000 of size 1024
2019-03-09 16:05:34.473877: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012A9400 of size 82944
2019-03-09 16:05:34.474360: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012BD800 of size 82944
2019-03-09 16:05:34.475280: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012D1C00 of size 82944
2019-03-09 16:05:34.476246: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012E6000 of size 2048
2019-03-09 16:05:34.477194: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012E6800 of size 2048
2019-03-09 16:05:34.478319: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012E7000 of size 2048
2019-03-09 16:05:34.478973: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007012E7800 of size 362496
2019-03-09 16:05:34.479279: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000701340000 of size 2097152
2019-03-09 16:05:34.479688: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000701540000 of size 4194304
2019-03-09 16:05:34.480062: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000701940000 of size 8388608
2019-03-09 16:05:34.480415: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000702140000 of size 33554432
2019-03-09 16:05:34.480800: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000704140000 of size 327680
2019-03-09 16:05:34.481155: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000704190000 of size 327680
2019-03-09 16:05:34.481562: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007041E0000 of size 4096
2019-03-09 16:05:34.481928: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007041E1000 of size 4096
2019-03-09 16:05:34.482296: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007041E2000 of size 4096
2019-03-09 16:05:34.482570: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007041E3000 of size 2097152
2019-03-09 16:05:34.482913: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007043E3000 of size 2097152
2019-03-09 16:05:34.483311: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007045E3000 of size 2097152
2019-03-09 16:05:34.483660: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007047E3000 of size 3328
2019-03-09 16:05:34.483992: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007047E3D00 of size 3328
2019-03-09 16:05:34.484321: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007047E4A00 of size 3328
2019-03-09 16:05:34.484651: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007047E5700 of size 3211264
2019-03-09 16:05:34.485036: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000704AF5700 of size 3211264
2019-03-09 16:05:34.485599: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000704E05700 of size 3211264
2019-03-09 16:05:34.486360: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000705115700 of size 5898240
2019-03-09 16:05:34.486971: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007056B5700 of size 11053312
2019-03-09 16:05:34.487386: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000706140000 of size 5898240
2019-03-09 16:05:34.488009: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007066E0000 of size 768
2019-03-09 16:05:34.488208: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007066E0300 of size 768
2019-03-09 16:05:34.488665: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007066E0600 of size 768
2019-03-09 16:05:34.489025: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007066E0900 of size 1024
2019-03-09 16:05:34.489362: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007066E0D00 of size 1024
2019-03-09 16:05:34.489727: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007066E1100 of size 1024
2019-03-09 16:05:34.490035: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007066E1500 of size 21233664
2019-03-09 16:05:34.490370: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000707B21500 of size 39971584
2019-03-09 16:05:34.492432: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070A140000 of size 21233664
2019-03-09 16:05:34.493677: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B580000 of size 256
2019-03-09 16:05:34.494214: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B580100 of size 256
2019-03-09 16:05:34.494574: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B580200 of size 256
2019-03-09 16:05:34.494978: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B580300 of size 256
2019-03-09 16:05:34.495397: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B580400 of size 256
2019-03-09 16:05:34.495763: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B580500 of size 256
2019-03-09 16:05:34.496331: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B580600 of size 256
2019-03-09 16:05:34.497030: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B580700 of size 256
2019-03-09 16:05:34.497489: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B580800 of size 5898240
2019-03-09 16:05:34.497828: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB20800 of size 256
2019-03-09 16:05:34.498182: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB20900 of size 256
2019-03-09 16:05:34.498531: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB20A00 of size 256
2019-03-09 16:05:34.498853: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB20B00 of size 256
2019-03-09 16:05:34.499230: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 000000070BB20C00 of size 256
2019-03-09 16:05:34.499496: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB20D00 of size 256
2019-03-09 16:05:34.499847: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB20E00 of size 256
2019-03-09 16:05:34.500137: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB20F00 of size 256
2019-03-09 16:05:34.500473: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB21000 of size 256
2019-03-09 16:05:34.500842: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB21100 of size 256
2019-03-09 16:05:34.501168: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB21200 of size 256
2019-03-09 16:05:34.501473: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB21300 of size 256
2019-03-09 16:05:34.501683: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 000000070BB21400 of size 1536
2019-03-09 16:05:34.502130: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB21A00 of size 256
2019-03-09 16:05:34.502492: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB21B00 of size 256
2019-03-09 16:05:34.502805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB21C00 of size 256
2019-03-09 16:05:34.503209: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB21D00 of size 256
2019-03-09 16:05:34.503565: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 000000070BB21E00 of size 4096
2019-03-09 16:05:34.503935: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB22E00 of size 256
2019-03-09 16:05:34.504189: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB22F00 of size 256
2019-03-09 16:05:34.504523: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 000000070BB23000 of size 256
2019-03-09 16:05:34.504858: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB23100 of size 9984
2019-03-09 16:05:34.505208: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB25800 of size 5120
2019-03-09 16:05:34.505577: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB26C00 of size 256
2019-03-09 16:05:34.505947: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB26D00 of size 256
2019-03-09 16:05:34.506235: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 000000070BB26E00 of size 3328
2019-03-09 16:05:34.506568: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB27B00 of size 256
2019-03-09 16:05:34.507033: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 000000070BB27C00 of size 3584
2019-03-09 16:05:34.509006: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070BB28A00 of size 5898240
2019-03-09 16:05:34.510021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 000000070C0C8A00 of size 3328
2019-03-09 16:05:34.511004: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070C0C9700 of size 9984
2019-03-09 16:05:34.511460: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070C0CBE00 of size 401408
2019-03-09 16:05:34.511852: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070C12DE00 of size 401408
2019-03-09 16:05:34.512178: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070C18FE00 of size 4718592
2019-03-09 16:05:34.512538: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 000000070C60FE00 of size 95617536
2019-03-09 16:05:34.512877: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000712340000 of size 268435456
2019-03-09 16:05:34.513286: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0000000722340000 of size 30670848
2019-03-09 16:05:34.513689: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000724080000 of size 4718592
2019-03-09 16:05:34.514096: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000724500000 of size 589824
2019-03-09 16:05:34.514486: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0000000724590000 of size 589824
2019-03-09 16:05:34.514770: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000724620000 of size 589824
2019-03-09 16:05:34.515220: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007246B0000 of size 589824
2019-03-09 16:05:34.515612: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0000000724740000 of size 14763008
2019-03-09 16:05:34.516007: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000725554400 of size 5120
2019-03-09 16:05:34.516337: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000725555800 of size 401408
2019-03-09 16:05:34.516623: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007255B7800 of size 5120
2019-03-09 16:05:34.516941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007255B8C00 of size 5120
2019-03-09 16:05:34.517254: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007255BA000 of size 5120
2019-03-09 16:05:34.517574: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007255BB400 of size 5120
2019-03-09 16:05:34.517988: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 00000007255BC800 of size 52511744
2019-03-09 16:05:34.518514: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000007287D0C00 of size 52428800
2019-03-09 16:05:34.519085: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 000000072B9D0C00 of size 378991616
2019-03-09 16:05:34.519497: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0000000742F00000 of size 380033280
2019-03-09 16:05:34.519932: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size:
2019-03-09 16:05:34.520411: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 43 Chunks of size 256 totalling 10.8KiB
2019-03-09 16:05:34.520854: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 768 totalling 3.0KiB
2019-03-09 16:05:34.521255: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 8 Chunks of size 1024 totalling 8.0KiB
2019-03-09 16:05:34.521664: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.3KiB
2019-03-09 16:05:34.521995: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 2048 totalling 8.0KiB
2019-03-09 16:05:34.522357: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 3328 totalling 13.0KiB
2019-03-09 16:05:34.522790: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 4096 totalling 16.0KiB
2019-03-09 16:05:34.523127: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 5120 totalling 30.0KiB
2019-03-09 16:05:34.523564: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 9984 totalling 19.5KiB
2019-03-09 16:05:34.524054: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 82944 totalling 324.0KiB
2019-03-09 16:05:34.524972: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 327680 totalling 960.0KiB
2019-03-09 16:05:34.525991: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 362496 totalling 354.0KiB
2019-03-09 16:05:34.526690: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 401408 totalling 1.15MiB
2019-03-09 16:05:34.527049: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 589824 totalling 1.69MiB
2019-03-09 16:05:34.527723: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 2097152 totalling 8.00MiB
2019-03-09 16:05:34.528635: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 3211264 totalling 9.19MiB
2019-03-09 16:05:34.529142: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4194304 totalling 4.00MiB
2019-03-09 16:05:34.529622: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 4718592 totalling 9.00MiB
2019-03-09 16:05:34.530052: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 5898240 totalling 22.50MiB
2019-03-09 16:05:34.530379: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 8388608 totalling 8.00MiB
2019-03-09 16:05:34.530715: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 11053312 totalling 10.54MiB
2019-03-09 16:05:34.531099: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 21233664 totalling 40.50MiB
2019-03-09 16:05:34.531472: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 33554432 totalling 32.00MiB
2019-03-09 16:05:34.531828: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 39971584 totalling 38.12MiB
2019-03-09 16:05:34.532223: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 52428800 totalling 50.00MiB
2019-03-09 16:05:34.532598: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 268435456 totalling 256.00MiB
2019-03-09 16:05:34.532953: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 492.39MiB
2019-03-09 16:05:34.533288: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit: 1469503897
InUse: 516309504
MaxInUse: 1469498880
NumAllocs: 17067
MaxAllocSize: 380033280
2019-03-09 16:05:34.534297: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *********x***______*************xxxxxx_***__*****___________________________________________________
2019-03-09 16:05:34.534939: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at tile_ops.cc:124 : Resource exhausted: OOM when allocating tensor with shape[128,1152,160,8,1] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
---------------------
修改一下config.py，将
flags.DEFINE_integer('batch_size', 128, 'batch size') 降为64，还是不行，降到32就可以了。
参考文献
1、在TensorFlow中使用共享GPU内存？ https://codeday.me/bug/20190203/575208.html
2、CUDA学习笔记(6) 共享内存与全局内存、https://blog.csdn.net/q583956932/article/details/78872146
3、Resource exhausted: OOM when allocating tensor with shape[2304,384] Traceback (most recent call last): #1993 https://github.com/tensorflow/models/issues/1993