cs231n-2017-assignments2-TensorFlow.ipynb 心得体会

新手菜鸟级别，大神请绕道

今天较好的完成了cs231n-2017-assignments2-TensorFlow.ipynb中的最后一个部分：构建一个model对cifar数据集进行训练，今天的精度在train上达到0.98，在val上达到0.82，明天继续进行优化。

具体代码见TensorFlow.ipynb

以下仅贴出model：

def my_model(X,y,is_training):
    
    # [conv-relu-conv-relu-pool]  out=14x14
    conv1 = tf.layers.conv2d(X,128,kernel_size=[3,3],strides=(1,1),activation=tf.nn.relu)
    ba1   = tf.layers.batch_normalization(conv1,training=is_training)
    conv2 = tf.layers.conv2d(ba1,256,[3,3],activation=tf.nn.relu)
    ba2   = tf.layers.batch_normalization(conv2,training=is_training)
    pool1 = tf.layers.max_pooling2d(ba2,pool_size=[2,2],strides=2)
    #[conv-relu-conv-relu-pool]  out=5x5
    conv3 = tf.layers.conv2d(pool1,512,[3,3],activation=tf.nn.relu)
    ba3   = tf.layers.batch_normalization(conv3,training=is_training)
    conv4 = tf.layers.conv2d(ba3,256,[3,3],activation=tf.nn.relu)
    ba4   = tf.layers.batch_normalization(conv4,training=is_training)
    pool2 = tf.layers.max_pooling2d(ba4,pool_size=[2,2],strides=2)
    #[dense-relu]x2 layer
    pool2_flat = tf.reshape(pool2,[-1,5*5*256])
    dense1 =tf.layers.dense(pool2_flat,units=512,activation=tf.nn.relu)
    ba5 = tf.layers.batch_normalization(dense1,center=False,scale=False,training=is_training)
    dropout1 = tf.layers.dropout(ba5,training=is_training)
    dense2 = tf.layers.dense(dropout1,units=128,activation=tf.nn.relu)
    ba6 = tf.layers.batch_normalization(dense2,center=False,scale=False,training=is_training)
    dropout2 = tf.layers.dropout(ba6,training=is_training)
    #logit out
    logits = tf.layers.dense(dropout2,units=10)
    return logits

tf.reset_default_graph()

X = tf.placeholder(tf.float32, [None, 32, 32, 3])
y = tf.placeholder(tf.int64, [None])
is_training = tf.placeholder(tf.bool)

y_out = my_model(X,y,is_training)
total_loss= tf.losses.softmax_cross_entropy(tf.one_hot(y,10),y_out)+tf.losses.get_regularization_loss()
mean_loss = tf.reduce_mean(total_loss)
optimizer = tf.train.RMSPropOptimizer(1e-3,decay=0.90,momentum=0.1)

# batch normalization in tensorflow requires this extra dependency
extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(extra_update_ops):
    train_step = optimizer.minimize(mean_loss)

心得体会：

1.模型很小时，即使是在train时，也很容易过拟合：几个epoch之后train_acc开始下降

2.模型很大时，为解决过拟合问题，在dense层加入了dropout，在loss中加入了regularization

3.没有batchnorm时，val_acc只能勉强到0.7，加入batchnorm后，直接到了0.8。这里要注意conv后接spatial batchnorm；而dense后接vanilla batchnorm

4.经常要计算conv的输出size，较为麻烦，编了个python的小程序，还挺方便

5.遇到问题，多查tensorflow官网