TF学习——TF之TFOD：基于TFOD AP训练ssd_mobilenet预模型+faster_rcnn_inception_resnet_v2_模型训练过程(TensorBoard监控)全记录

TF学习——TF之TFOD：基于TFOD AP训练ssd_mobilenet预模型+faster_rcnn_inception_resnet_v2_模型训练过程(TensorBoard监控)全记录

 

目录

训练日志结果

TensorBoard监控结果

训练过程全记录

---

 

 

 

 

训练日志结果

 

TensorBoard监控结果

1、LearningRate

2、Loss

3、Batch

4、Global step

5、Queue

 

 

训练过程全记录

开始

Instructions for updating:
Please switch to tf.train.get_or_create_global_step
W0929 16:09:53.723011 19388 variables_helper.py:141] Variable [SecondStageBoxPredictor/BoxEncodingPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[360]], model variable shape: [[80]]. This variable will not be initialized from the checkpoint.
W0929 16:09:53.723512 19388 variables_helper.py:141] Variable [SecondStageBoxPredictor/BoxEncodingPredictor/biases/Momentum] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[360]], model variable shape: [[80]]. This variable will not be initialized from the checkpoint.
W0929 16:09:53.725018 19388 variables_helper.py:141] Variable [SecondStageBoxPredictor/BoxEncodingPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1536, 360]], model variable shape: [[1536, 80]]. This variable will not be initialized from the checkpoint.
W0929 16:09:53.725520 19388 variables_helper.py:141] Variable [SecondStageBoxPredictor/BoxEncodingPredictor/weights/Momentum] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1536, 360]], model variable shape: [[1536, 80]]. This variable will not be initialized from the checkpoint.
W0929 16:09:53.729529 19388 variables_helper.py:141] Variable [SecondStageBoxPredictor/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[91]], model variable shape: [[21]]. This variable will not be initialized from the checkpoint.
W0929 16:09:53.730031 19388 variables_helper.py:141] Variable [SecondStageBoxPredictor/ClassPredictor/biases/Momentum] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[91]], model variable shape: [[21]]. This variable will not be initialized from the checkpoint.
W0929 16:09:53.731035 19388 variables_helper.py:141] Variable [SecondStageBoxPredictor/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1536, 91]], model variable shape: [[1536, 21]]. This variable will not be initialized from the checkpoint.
W0929 16:09:53.731536 19388 variables_helper.py:141] Variable [SecondStageBoxPredictor/ClassPredictor/weights/Momentum] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1536, 91]], model variable shape: [[1536, 21]]. This variable will not be initialized from the checkpoint.
W0929 16:09:53.733070 19388 variables_helper.py:144] Variable [global_step] is not available in checkpoint
WARNING:tensorflow:From F:\Program Files\Python\Python36\Lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py:737: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.

继续16:10

Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2018-09-29 16:10:02.301976: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
INFO:tensorflow:Restoring parameters from voc/pretrained/model.ckpt
I0929 16:10:11.780281 19388 tf_logging.py:115] Restoring parameters from voc/pretrained/model.ckpt
INFO:tensorflow:Running local_init_op.
I0929 16:10:18.535013 19388 tf_logging.py:115] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0929 16:10:19.400116 19388 tf_logging.py:115] Done running local_init_op.
INFO:tensorflow:Starting Session.
I0929 16:10:40.838994 19388 tf_logging.py:115] Starting Session.
INFO:tensorflow:Saving checkpoint to path voc/train_dir/model.ckpt
I0929 16:10:41.317768 19576 tf_logging.py:115] Saving checkpoint to path voc/train_dir/model.ckpt
INFO:tensorflow:Starting Queues.
I0929 16:10:41.328797 19388 tf_logging.py:115] Starting Queues.
INFO:tensorflow:global_step/sec: 0
I0929 16:10:54.108844 10952 tf_logging.py:159] global_step/sec: 0
INFO:tensorflow:Recording summary at step 0.
I0929 16:11:07.153036  1076 tf_logging.py:115] Recording summary at step 0.
INFO:tensorflow:global step 1: loss = 5.2855 (96.556 sec/step)
I0929 16:12:20.200682 19388 tf_logging.py:115] global step 1: loss = 5.2855 (96.556 sec/step)
INFO:tensorflow:global_step/sec: 0.00848223
I0929 16:12:52.110595 10952 tf_logging.py:159] global_step/sec: 0.00848223

第1、4、6、9步:16:13

INFO:tensorflow:Recording summary at step 1.
I0929 16:13:06.314384  1076 tf_logging.py:115] Recording summary at step 1.
INFO:tensorflow:global step 2: loss = 4.8016 (53.369 sec/step)
I0929 16:13:13.988794 19388 tf_logging.py:115] global step 2: loss = 4.8016 (53.369 sec/step)
INFO:tensorflow:global step 3: loss = 4.7835 (45.608 sec/step)
I0929 16:13:59.598836 19388 tf_logging.py:115] global step 3: loss = 4.7835 (45.608 sec/step)
INFO:tensorflow:global step 4: loss = 4.8204 (46.679 sec/step)
I0929 16:14:46.279041 19388 tf_logging.py:115] global step 4: loss = 4.8204 (46.679 sec/step)
INFO:tensorflow:global_step/sec: 0.0251812
I0929 16:14:51.139468 10952 tf_logging.py:159] global_step/sec: 0.0251812
INFO:tensorflow:Recording summary at step 4.
I0929 16:15:04.911759  1076 tf_logging.py:115] Recording summary at step 4.
INFO:tensorflow:global step 5: loss = 4.8758 (49.236 sec/step)
I0929 16:15:35.515572 19388 tf_logging.py:115] global step 5: loss = 4.8758 (49.236 sec/step)
INFO:tensorflow:global step 6: loss = 3.2228 (40.198 sec/step)
I0929 16:16:15.715026 19388 tf_logging.py:115] global step 6: loss = 3.2228 (40.198 sec/step)
INFO:tensorflow:global_step/sec: 0.0166681
I0929 16:16:51.128734 10952 tf_logging.py:159] global_step/sec: 0.0166681
INFO:tensorflow:Recording summary at step 6.
I0929 16:17:01.526894  1076 tf_logging.py:115] Recording summary at step 6.
INFO:tensorflow:global step 7: loss = 4.9633 (53.262 sec/step)
I0929 16:17:08.979240 19388 tf_logging.py:115] global step 7: loss = 4.9633 (53.262 sec/step)
INFO:tensorflow:global step 8: loss = 4.2235 (47.167 sec/step)
I0929 16:17:56.148062 19388 tf_logging.py:115] global step 8: loss = 4.2235 (47.167 sec/step)
INFO:tensorflow:global step 9: loss = 3.0626 (41.395 sec/step)
I0929 16:18:37.543740 19388 tf_logging.py:115] global step 9: loss = 3.0626 (41.395 sec/step)
INFO:tensorflow:global_step/sec: 0.0249998
I0929 16:18:51.130415 10952 tf_logging.py:159] global_step/sec: 0.0249998
INFO:tensorflow:Recording summary at step 9.
I0929 16:19:06.285729  1076 tf_logging.py:115] Recording summary at step 9.
INFO:tensorflow:global step 10: loss = 3.9423 (53.277 sec/step)
I0929 16:19:30.821868 19388 tf_logging.py:115] global step 10: loss = 3.9423 (53.277 sec/step)
INFO:tensorflow:global step 11: loss = 3.8488 (40.221 sec/step)
I0929 16:20:11.045604 19388 tf_logging.py:115] global step 11: loss = 3.8488 (40.221 sec/step)
INFO:tensorflow:Saving checkpoint to path voc/train_dir/model.ckpt
I0929 16:20:41.343273 19576 tf_logging.py:115] Saving checkpoint to path voc/train_dir/model.ckpt
INFO:tensorflow:global_step/sec: 0.0165616
I0929 16:20:51.891337 10952 tf_logging.py:159] global_step/sec: 0.0165616

第11~24步:16:21

INFO:tensorflow:Recording summary at step 11.
I0929 16:21:03.727980  1076 tf_logging.py:115] Recording summary at step 11.
INFO:tensorflow:global step 12: loss = 2.6101 (54.090 sec/step)
I0929 16:21:05.137731 19388 tf_logging.py:115] global step 12: loss = 2.6101 (54.090 sec/step)
INFO:tensorflow:global step 13: loss = 3.6712 (46.100 sec/step)
I0929 16:21:51.240856 19388 tf_logging.py:115] global step 13: loss = 3.6712 (46.100 sec/step)
INFO:tensorflow:global step 14: loss = 1.9164 (44.411 sec/step)
I0929 16:22:35.656246 19388 tf_logging.py:115] global step 14: loss = 1.9164 (44.411 sec/step)
INFO:tensorflow:global_step/sec: 0.0251609
I0929 16:22:51.124389 10952 tf_logging.py:159] global_step/sec: 0.0251609
INFO:tensorflow:Recording summary at step 14.
I0929 16:23:03.518480  1076 tf_logging.py:115] Recording summary at step 14.
INFO:tensorflow:global step 15: loss = 0.9196 (51.261 sec/step)
I0929 16:23:26.920724 19388 tf_logging.py:115] global step 15: loss = 0.9196 (51.261 sec/step)
INFO:tensorflow:global step 16: loss = 3.1620 (41.240 sec/step)
I0929 16:24:08.163412 19388 tf_logging.py:115] global step 16: loss = 3.1620 (41.240 sec/step)
INFO:tensorflow:global_step/sec: 0.0166543
I0929 16:24:51.213029 10952 tf_logging.py:159] global_step/sec: 0.0166543
INFO:tensorflow:Recording summary at step 16.
I0929 16:24:59.248900  1076 tf_logging.py:115] Recording summary at step 16.
INFO:tensorflow:global step 17: loss = 2.7205 (51.696 sec/step)
I0929 16:24:59.862766 19388 tf_logging.py:115] global step 17: loss = 2.7205 (51.696 sec/step)
INFO:tensorflow:global step 18: loss = 1.7422 (41.517 sec/step)
I0929 16:25:41.382816 19388 tf_logging.py:115] global step 18: loss = 1.7422 (41.517 sec/step)
INFO:tensorflow:global step 19: loss = 0.3852 (42.130 sec/step)
I0929 16:26:23.515615 19388 tf_logging.py:115] global step 19: loss = 0.3852 (42.130 sec/step)
INFO:tensorflow:global_step/sec: 0.0250171
I0929 16:26:51.131010 10952 tf_logging.py:159] global_step/sec: 0.0250171
INFO:tensorflow:Recording summary at step 19.
I0929 16:27:05.333280  1076 tf_logging.py:115] Recording summary at step 19.
INFO:tensorflow:global step 20: loss = 3.4916 (50.366 sec/step)
I0929 16:27:13.883975 19388 tf_logging.py:115] global step 20: loss = 3.4916 (50.366 sec/step)
INFO:tensorflow:global step 21: loss = 0.5574 (39.555 sec/step)
I0929 16:27:53.443298 19388 tf_logging.py:115] global step 21: loss = 0.5574 (39.555 sec/step)
INFO:tensorflow:global step 22: loss = 2.8081 (39.814 sec/step)
I0929 16:28:33.260167 19388 tf_logging.py:115] global step 22: loss = 2.8081 (39.814 sec/step)
INFO:tensorflow:global_step/sec: 0.0249801
I0929 16:28:51.226652 10952 tf_logging.py:159] global_step/sec: 0.0249801
INFO:tensorflow:Recording summary at step 22.
I0929 16:28:57.415109  1076 tf_logging.py:115] Recording summary at step 22.
INFO:tensorflow:global step 23: loss = 2.1274 (45.275 sec/step)
I0929 16:29:18.536998 19388 tf_logging.py:115] global step 23: loss = 2.1274 (45.275 sec/step)
INFO:tensorflow:global step 24: loss = 2.8762 (38.967 sec/step)
I0929 16:29:57.505466 19388 tf_logging.py:115] global step 24: loss = 2.8762 (38.967 sec/step)
INFO:tensorflow:Saving checkpoint to path voc/train_dir/model.ckpt
I0929 16:30:41.327496 19576 tf_logging.py:115] Saving checkpoint to path voc/train_dir/model.ckpt
INFO:tensorflow:Recording summary at step 24.
I0929 16:30:59.247197  1076 tf_logging.py:115] Recording summary at step 24.
INFO:tensorflow:global step 25: loss = 2.3282 (45.570 sec/step)
I0929 16:31:02.410132 19388 tf_logging.py:115] global step 25: loss = 2.3282 (45.570 sec/step)
INFO:tensorflow:global step 26: loss = 2.4746 (34.973 sec/step)
I0929 16:32:17.320121 19388 tf_logging.py:115] global step 26: loss = 2.4746 (34.973 sec/step)

经过……步骤

 

第168~184步:18:02

INFO:tensorflow:Recording summary at step 168.
I0929 18:02:57.965489  1076 tf_logging.py:115] Recording summary at step 168.
INFO:tensorflow:global step 169: loss = 1.0579 (39.345 sec/step)
I0929 18:03:30.128316 19388 tf_logging.py:115] global step 169: loss = 1.0579 (39.345 sec/step)
INFO:tensorflow:global step 170: loss = 1.8678 (34.187 sec/step)
I0929 18:04:04.328794 19388 tf_logging.py:115] global step 170: loss = 1.8678 (34.187 sec/step)
INFO:tensorflow:global step 171: loss = 1.3184 (35.753 sec/step)
I0929 18:04:40.097321 19388 tf_logging.py:115] global step 171: loss = 1.3184 (35.753 sec/step)
INFO:tensorflow:Recording summary at step 171.
I0929 18:05:01.107349  1076 tf_logging.py:115] Recording summary at step 171.
INFO:tensorflow:global step 172: loss = 1.8373 (42.644 sec/step)
I0929 18:05:22.745107 19388 tf_logging.py:115] global step 172: loss = 1.8373 (42.644 sec/step)
INFO:tensorflow:global step 173: loss = 0.6442 (34.338 sec/step)
I0929 18:05:57.096073 19388 tf_logging.py:115] global step 173: loss = 0.6442 (34.338 sec/step)
INFO:tensorflow:global step 174: loss = 2.1237 (36.256 sec/step)
I0929 18:06:33.718968 19388 tf_logging.py:115] global step 174: loss = 2.1237 (36.256 sec/step)
INFO:tensorflow:Recording summary at step 174.
I0929 18:06:59.115108  1076 tf_logging.py:115] Recording summary at step 174.
INFO:tensorflow:global step 175: loss = 2.0013 (42.911 sec/step)
I0929 18:07:16.632529 19388 tf_logging.py:115] global step 175: loss = 2.0013 (42.911 sec/step)
INFO:tensorflow:global step 176: loss = 0.3421 (33.903 sec/step)
I0929 18:07:50.547195 19388 tf_logging.py:115] global step 176: loss = 0.3421 (33.903 sec/step)
INFO:tensorflow:global step 177: loss = 3.3426 (31.519 sec/step)
I0929 18:08:22.122741 19388 tf_logging.py:115] global step 177: loss = 3.3426 (31.519 sec/step)
INFO:tensorflow:Recording summary at step 177.
I0929 18:08:58.772549  1076 tf_logging.py:115] Recording summary at step 177.
INFO:tensorflow:global step 178: loss = 1.3222 (41.238 sec/step)
I0929 18:09:03.364229 19388 tf_logging.py:115] global step 178: loss = 1.3222 (41.238 sec/step)
INFO:tensorflow:global step 179: loss = 0.4027 (33.960 sec/step)
I0929 18:09:37.338058 19388 tf_logging.py:115] global step 179: loss = 0.4027 (33.960 sec/step)
INFO:tensorflow:global step 180: loss = 0.6667 (34.785 sec/step)
I0929 18:10:12.152817 19388 tf_logging.py:115] global step 180: loss = 0.6667 (34.785 sec/step)
INFO:tensorflow:Saving checkpoint to path voc/train_dir/model.ckpt
I0929 18:10:41.401597 19576 tf_logging.py:115] Saving checkpoint to path voc/train_dir/model.ckpt
INFO:tensorflow:global step 181: loss = 1.0127 (36.835 sec/step)
I0929 18:10:49.260302 19388 tf_logging.py:115] global step 181: loss = 1.0127 (36.835 sec/step)
INFO:tensorflow:Recording summary at step 181.
I0929 18:11:01.773863  1076 tf_logging.py:115] Recording summary at step 181.
INFO:tensorflow:global step 182: loss = 0.4533 (42.206 sec/step)
I0929 18:11:31.493520 19388 tf_logging.py:115] global step 182: loss = 0.4533 (42.206 sec/step)
INFO:tensorflow:global step 183: loss = 0.3991 (33.720 sec/step)
I0929 18:12:05.214678 19388 tf_logging.py:115] global step 183: loss = 0.3991 (33.720 sec/step)
INFO:tensorflow:global step 184: loss = 1.1574 (33.347 sec/step)
I0929 18:12:38.562483 19388 tf_logging.py:115] global step 184: loss = 1.1574 (33.347 sec/step)
INFO:tensorflow:Recording summary at step 184.
I0929 18:13:00.190217  1076 tf_logging.py:115] Recording summary at step 184.
INFO:tensorflow:global step 185: loss = 2.0064 (41.449 sec/step)
I0929 18:13:20.013966 19388 tf_logging.py:115] global step 185: loss = 2.0064 (41.449 sec/step)
INFO:tensorflow:global step 186: loss = 1.4978 (38.620 sec/step)
I0929 18:13:58.636997 19388 tf_logging.py:115] global step 186: loss = 1.4978 (38.620 sec/step)
INFO:tensorflow:global step 187: loss = 0.2496 (36.551 sec/step)
I0929 18:14:35.188669 19388 tf_logging.py:115] global step 187: loss = 0.2496 (36.551 sec/step)

第190~202步:18:17

INFO:tensorflow:Recording summary at step 190.
I0929 18:17:01.592503  1076 tf_logging.py:115] Recording summary at step 190.
INFO:tensorflow:global step 191: loss = 1.1224 (43.398 sec/step)
I0929 18:17:22.084096 19388 tf_logging.py:115] global step 191: loss = 1.1224 (43.398 sec/step)
INFO:tensorflow:global step 192: loss = 0.8829 (35.666 sec/step)
I0929 18:17:57.750644 19388 tf_logging.py:115] global step 192: loss = 0.8829 (35.666 sec/step)
INFO:tensorflow:global step 193: loss = 2.1432 (38.027 sec/step)
I0929 18:18:35.782682 19388 tf_logging.py:115] global step 193: loss = 2.1432 (38.027 sec/step)
INFO:tensorflow:Recording summary at step 193.
I0929 18:19:02.559637  1076 tf_logging.py:115] Recording summary at step 193.
INFO:tensorflow:global step 194: loss = 1.1090 (46.321 sec/step)
I0929 18:19:22.108624 19388 tf_logging.py:115] global step 194: loss = 1.1090 (46.321 sec/step)
INFO:tensorflow:global step 195: loss = 2.5213 (34.215 sec/step)
I0929 18:19:56.324908 19388 tf_logging.py:115] global step 195: loss = 2.5213 (34.215 sec/step)
INFO:tensorflow:global step 196: loss = 2.8718 (35.261 sec/step)
I0929 18:20:31.586679 19388 tf_logging.py:115] global step 196: loss = 2.8718 (35.261 sec/step)
INFO:tensorflow:Saving checkpoint to path voc/train_dir/model.ckpt
I0929 18:20:41.404536 19576 tf_logging.py:115] Saving checkpoint to path voc/train_dir/model.ckpt
INFO:tensorflow:Recording summary at step 196.
I0929 18:21:09.099129  1076 tf_logging.py:115] Recording summary at step 196.
INFO:tensorflow:global step 197: loss = 0.6111 (48.939 sec/step)
I0929 18:21:20.526573 19388 tf_logging.py:115] global step 197: loss = 0.6111 (48.939 sec/step)
INFO:tensorflow:global step 198: loss = 1.5971 (35.082 sec/step)
I0929 18:21:55.609520 19388 tf_logging.py:115] global step 198: loss = 1.5971 (35.082 sec/step)
INFO:tensorflow:global step 199: loss = 2.2728 (35.880 sec/step)
I0929 18:22:31.491082 19388 tf_logging.py:115] global step 199: loss = 2.2728 (35.880 sec/step)
INFO:tensorflow:Recording summary at step 199.
I0929 18:23:00.506135  1076 tf_logging.py:115] Recording summary at step 199.
INFO:tensorflow:global step 200: loss = 0.7774 (42.992 sec/step)
I0929 18:23:14.495092 19388 tf_logging.py:115] global step 200: loss = 0.7774 (42.992 sec/step)
INFO:tensorflow:global step 201: loss = 1.0625 (33.835 sec/step)
I0929 18:23:48.330853 19388 tf_logging.py:115] global step 201: loss = 1.0625 (33.835 sec/step)
INFO:tensorflow:global step 202: loss = 1.9644 (33.608 sec/step)
I0929 18:24:21.939998 19388 tf_logging.py:115] global step 202: loss = 1.9644 (33.608 sec/step)
INFO:tensorflow:Recording summary at step 202.
I0929 18:25:03.530977  1076 tf_logging.py:115] Recording summary at step 202.
INFO:tensorflow:global step 203: loss = 2.2303 (47.871 sec/step)
I0929 18:25:09.812503 19388 tf_logging.py:115] global step 203: loss = 2.2303 (47.871 sec/step)