一次 Kaggle CIFAR-10 提交

最近在B站上听李沐老师的动手学MXNET，CIFAR-10的图片识别是其中一个作业，虽然不是同步无法参与互动，但还是很有意思自己跑了一个结果出来。
100 epochs, 得分 0.91540
用服务器跑了一晚上，晚六点开始，我看submission文件是早上六点多生成的。
下面是训练过程：

epoch 1, loss 2.204299, train acc 0.244175, valid acc 0.378600, time 00:30:50, lr 0.1
epoch 2, loss 1.600064, train acc 0.410218, valid acc 0.467200, time 00:13:54, lr 0.1
epoch 3, loss 1.375437, train acc 0.499958, valid acc 0.520400, time 00:05:18, lr 0.1
epoch 4, loss 1.195475, train acc 0.572562, valid acc 0.559400, time 00:05:02, lr 0.1
epoch 5, loss 1.034070, train acc 0.635535, valid acc 0.577400, time 00:01:55, lr 0.1
epoch 6, loss 0.867561, train acc 0.694792, valid acc 0.629600, time 00:03:19, lr 0.1
epoch 7, loss 0.742619, train acc 0.743184, valid acc 0.649000, time 00:04:09, lr 0.1
epoch 8, loss 0.665055, train acc 0.766949, valid acc 0.764200, time 00:04:33, lr 0.1
epoch 9, loss 0.612949, train acc 0.788229, valid acc 0.757000, time 00:05:10, lr 0.1
epoch 10, loss 0.570910, train acc 0.801652, valid acc 0.709800, time 00:05:36, lr 0.1
epoch 11, loss 0.541980, train acc 0.812310, valid acc 0.731000, time 00:04:58, lr 0.1
epoch 12, loss 0.527184, train acc 0.818737, valid acc 0.732400, time 00:09:45, lr 0.1
epoch 13, loss 0.502743, train acc 0.826682, valid acc 0.677200, time 00:10:27, lr 0.1
epoch 14, loss 0.487677, train acc 0.831901, valid acc 0.769000, time 00:14:10, lr 0.1
epoch 15, loss 0.480422, train acc 0.834746, valid acc 0.749000, time 00:18:06, lr 0.1
epoch 16, loss 0.463393, train acc 0.839267, valid acc 0.695400, time 00:14:28, lr 0.1
epoch 17, loss 0.450438, train acc 0.843605, valid acc 0.745000, time 00:15:18, lr 0.1
epoch 18, loss 0.446354, train acc 0.846981, valid acc 0.781400, time 00:10:00, lr 0.1
epoch 19, loss 0.438400, train acc 0.848877, valid acc 0.753400, time 00:06:12, lr 0.1
epoch 20, loss 0.427498, train acc 0.853124, valid acc 0.795600, time 00:04:02, lr 0.1
epoch 21, loss 0.415870, train acc 0.856630, valid acc 0.770600, time 00:01:52, lr 0.1
epoch 22, loss 0.417540, train acc 0.856411, valid acc 0.671000, time 00:03:19, lr 0.1
epoch 23, loss 0.411857, train acc 0.857999, valid acc 0.765800, time 00:03:45, lr 0.1
epoch 24, loss 0.405953, train acc 0.861846, valid acc 0.760000, time 00:04:13, lr 0.1
epoch 25, loss 0.393419, train acc 0.864623, valid acc 0.776200, time 00:05:58, lr 0.1
epoch 26, loss 0.398574, train acc 0.862620, valid acc 0.758200, time 00:07:47, lr 0.1
epoch 27, loss 0.385781, train acc 0.867693, valid acc 0.719400, time 00:09:25, lr 0.1
epoch 28, loss 0.380056, train acc 0.869086, valid acc 0.767600, time 00:15:36, lr 0.1
epoch 29, loss 0.383758, train acc 0.868014, valid acc 0.766000, time 00:14:09, lr 0.1
epoch 30, loss 0.381887, train acc 0.869000, valid acc 0.725600, time 00:16:40, lr 0.1

可以看出，30轮过后就进入了瓶颈期，没有很大的提升。

epoch 31, loss 0.376606, train acc 0.871400, valid acc 0.807600, time 00:21:15, lr 0.1
epoch 32, loss 0.369865, train acc 0.872201, valid acc 0.816000, time 00:37:23, lr 0.1
epoch 33, loss 0.371696, train acc 0.873708, valid acc 0.802200, time 00:13:18, lr 0.1
epoch 34, loss 0.364968, train acc 0.873335, valid acc 0.765400, time 00:01:31, lr 0.1
epoch 35, loss 0.366952, train acc 0.873779, valid acc 0.779200, time 00:01:13, lr 0.1
epoch 36, loss 0.366652, train acc 0.873261, valid acc 0.829000, time 00:01:13, lr 0.1
epoch 37, loss 0.361195, train acc 0.874598, valid acc 0.767800, time 00:01:12, lr 0.1
epoch 38, loss 0.362409, train acc 0.876509, valid acc 0.789200, time 00:01:13, lr 0.1
epoch 39, loss 0.356273, train acc 0.878046, valid acc 0.778600, time 00:01:13, lr 0.1
epoch 40, loss 0.355038, train acc 0.877851, valid acc 0.756200, time 00:01:12, lr 0.1
epoch 41, loss 0.354369, train acc 0.878926, valid acc 0.787200, time 00:01:11, lr 0.1
epoch 42, loss 0.351924, train acc 0.878882, valid acc 0.738800, time 00:01:12, lr 0.1
epoch 43, loss 0.346999, train acc 0.880763, valid acc 0.820200, time 00:01:14, lr 0.1
epoch 44, loss 0.353424, train acc 0.880356, valid acc 0.771600, time 00:01:13, lr 0.1
epoch 45, loss 0.352883, train acc 0.878142, valid acc 0.771200, time 00:01:13, lr 0.1
epoch 46, loss 0.348928, train acc 0.880894, valid acc 0.777600, time 00:01:12, lr 0.1
epoch 47, loss 0.342978, train acc 0.882386, valid acc 0.782400, time 00:01:13, lr 0.1
epoch 48, loss 0.351075, train acc 0.880243, valid acc 0.800200, time 00:01:13, lr 0.1
epoch 49, loss 0.344487, train acc 0.882830, valid acc 0.802000, time 00:01:13, lr 0.1
epoch 50, loss 0.351551, train acc 0.877875, valid acc 0.821000, time 00:01:11, lr 0.1
epoch 51, loss 0.348850, train acc 0.880507, valid acc 0.836800, time 00:01:11, lr 0.1
epoch 52, loss 0.339322, train acc 0.883866, valid acc 0.830200, time 00:01:12, lr 0.1
epoch 53, loss 0.344748, train acc 0.882013, valid acc 0.723200, time 00:01:10, lr 0.1
epoch 54, loss 0.340964, train acc 0.883656, valid acc 0.748400, time 00:01:12, lr 0.1
epoch 55, loss 0.337965, train acc 0.884970, valid acc 0.807400, time 00:01:11, lr 0.1
epoch 56, loss 0.334912, train acc 0.885570, valid acc 0.790600, time 00:01:11, lr 0.1
epoch 57, loss 0.342623, train acc 0.880420, valid acc 0.796200, time 00:01:10, lr 0.1
epoch 58, loss 0.337697, train acc 0.883533, valid acc 0.839400, time 00:01:19, lr 0.1
epoch 59, loss 0.339286, train acc 0.884267, valid acc 0.765200, time 00:01:15, lr 0.1
epoch 60, loss 0.341186, train acc 0.883875, valid acc 0.763400, time 00:01:14, lr 0.1
epoch 61, loss 0.340064, train acc 0.882748, valid acc 0.830000, time 00:01:38, lr 0.1
epoch 62, loss 0.330042, train acc 0.886166, valid acc 0.764000, time 00:01:12, lr 0.1
epoch 63, loss 0.333694, train acc 0.885387, valid acc 0.786200, time 00:05:27, lr 0.1
epoch 64, loss 0.335942, train acc 0.884561, valid acc 0.855400, time 00:06:42, lr 0.1
epoch 65, loss 0.331788, train acc 0.886615, valid acc 0.807200, time 00:01:12, lr 0.1
epoch 66, loss 0.330144, train acc 0.886238, valid acc 0.783200, time 00:01:11, lr 0.1
epoch 67, loss 0.331106, train acc 0.885730, valid acc 0.771200, time 00:01:11, lr 0.1
epoch 68, loss 0.336561, train acc 0.883646, valid acc 0.825200, time 00:01:10, lr 0.1
epoch 69, loss 0.327757, train acc 0.887357, valid acc 0.841000, time 00:01:10, lr 0.1
epoch 70, loss 0.327064, train acc 0.889017, valid acc 0.828800, time 00:01:10, lr 0.1
epoch 71, loss 0.330027, train acc 0.887185, valid acc 0.798600, time 00:01:11, lr 0.1
epoch 72, loss 0.334022, train acc 0.885005, valid acc 0.741000, time 00:01:10, lr 0.1
epoch 73, loss 0.327411, train acc 0.888383, valid acc 0.772400, time 00:01:10, lr 0.1
epoch 74, loss 0.333187, train acc 0.885986, valid acc 0.773600, time 00:01:11, lr 0.1
epoch 75, loss 0.328935, train acc 0.886452, valid acc 0.793000, time 00:01:10, lr 0.1
epoch 76, loss 0.324520, train acc 0.887898, valid acc 0.816200, time 00:01:10, lr 0.1
epoch 77, loss 0.331574, train acc 0.885750, valid acc 0.820000, time 00:01:10, lr 0.1
epoch 78, loss 0.328412, train acc 0.887108, valid acc 0.745000, time 00:01:11, lr 0.1
epoch 79, loss 0.327722, train acc 0.887071, valid acc 0.768800, time 00:01:10, lr 0.1
epoch 80, loss 0.323882, train acc 0.889089, valid acc 0.769000, time 00:01:10, lr 0.1

80轮过后，learning rate 减小到原来的0.1，立马就有效果。

epoch 81, loss 0.209426, train acc 0.930134, valid acc 0.891000, time 00:01:09, lr 0.010000000000000002
epoch 82, loss 0.132424, train acc 0.956496, valid acc 0.888000, time 00:01:10, lr 0.010000000000000002
epoch 83, loss 0.107024, train acc 0.964713, valid acc 0.895600, time 00:01:10, lr 0.010000000000000002
epoch 84, loss 0.094055, train acc 0.968370, valid acc 0.906800, time 00:01:10, lr 0.010000000000000002
epoch 85, loss 0.082123, train acc 0.972920, valid acc 0.903400, time 00:01:09, lr 0.010000000000000002
epoch 86, loss 0.075913, train acc 0.974997, valid acc 0.905400, time 00:01:09, lr 0.010000000000000002
epoch 87, loss 0.067130, train acc 0.979041, valid acc 0.909000, time 00:01:10, lr 0.010000000000000002
epoch 88, loss 0.060554, train acc 0.980878, valid acc 0.913000, time 00:01:09, lr 0.010000000000000002
epoch 89, loss 0.053192, train acc 0.983902, valid acc 0.903200, time 00:01:09, lr 0.010000000000000002
epoch 90, loss 0.048942, train acc 0.985184, valid acc 0.900800, time 00:01:09, lr 0.010000000000000002
epoch 91, loss 0.045887, train acc 0.985739, valid acc 0.905800, time 00:01:09, lr 0.010000000000000002
epoch 92, loss 0.043544, train acc 0.986755, valid acc 0.903800, time 00:01:09, lr 0.010000000000000002

但几个epoch之后也没有明显提升了。

而且train acc 和valid acc差距越来越大，明显是overfitting了。

epoch 93, loss 0.039191, train acc 0.987793, valid acc 0.911000, time 00:01:09, lr 0.010000000000000002
epoch 94, loss 0.038024, train acc 0.988614, valid acc 0.907800, time 00:01:08, lr 0.010000000000000002
epoch 95, loss 0.037767, train acc 0.988348, valid acc 0.905800, time 00:01:08, lr 0.010000000000000002
epoch 96, loss 0.034426, train acc 0.989729, valid acc 0.904000, time 00:01:08, lr 0.010000000000000002
epoch 97, loss 0.032643, train acc 0.989924, valid acc 0.910200, time 00:01:09, lr 0.010000000000000002
epoch 98, loss 0.030057, train acc 0.991083, valid acc 0.910600, time 00:01:08, lr 0.010000000000000002
epoch 99, loss 0.030856, train acc 0.990589, valid acc 0.907200, time 00:01:09, lr 0.010000000000000002
epoch 100, loss 0.031544, train acc 0.990417, valid acc 0.911200, time 00:01:09, lr 0.010000000000000002