神经网络-实现学习率随迭代次数变化

1.需要变化的学习率的原因:经历一段时间的学习之后,我们距离目标值越来越近,此时如果学习率太大,则会造成训练在最优值附近来回波动,这时候我们就需要减少学习率

2.实现:学习率是在Optimizer中使用,我们每次迭代通过tf.assign修改学习率

3.note:    tf.assign(ref, value, validate_shape=None, use_locking=None, name=None)

将value赋值给ref,其中：ref 必须是tf.Variable创建的tensor，如果ref=tf.constant()会报错！

以下是详细的实现代码:

#3-3 MNIst数据集使用drop-out提升准确率,消除部分过拟合

import tensorflow as tf;

import numpy as np;

from tensorflow.examples.tutorials.mnist import input_data

#原始数据

mnist =input_data.read_data_sets( "MNIST_data", one_hot = True)

x =tf.placeholder(tf.float32, shape =[ None, 784])

y =tf.placeholder(tf.float32, shape =[ None, 10])

drop_out =tf.placeholder(tf.float32)

learning_rate =tf.Variable( 0.2, dtype =tf.float32) #每次迭代修改学习率

batch_size = 100;

n_batch =mnist.train.num_examples //batch_size #获取共有多少批次

#创建神经网络计算图

w1 =tf.Variable(tf.truncated_normal( shape =[ 784, 1000], stddev = 0.1))

b1 =tf.Variable(tf.constant( 0.1, shape =[ 1000]))

layer1 =tf.nn.relu(tf.matmul(x,w1) +b1)

layer1 =tf.nn.dropout(layer1,drop_out)

w2 =tf.Variable(tf.truncated_normal( shape =[ 1000, 2000], stddev = 0.1))

b2 =tf.Variable(tf.constant( 0.1, shape =[ 2000]))

layer2 =tf.nn.relu(tf.matmul(layer1,w2) +b2)

layer2 =tf.nn.dropout(layer2,drop_out)

w3 =tf.Variable(tf.truncated_normal( shape =[ 2000, 10], stddev = 0.1))

b3 =tf.Variable(tf.constant( 0.1, shape =[ 10]))

layer3 =tf.nn.relu(tf.matmul(layer2,w3) +b3)

layer3 =tf.nn.dropout(layer3,drop_out)

prediction =tf.nn.softmax(layer3)

#定义损失函数

loss =tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits( logits =prediction, labels =y))

train_step =tf.train.GradientDescentOptimizer( learning_rate).minimize(loss)

currect_rate =tf.reduce_mean(tf.cast(tf.equal(tf.arg_max(y, 1),tf.arg_max(prediction, 1)),tf.float32))

init =tf.global_variables_initializer()

with tf.Session() as sess:

sess.run(init)

for epoch in range( 1221):

    sess.run(tf.assign(learning_rate, 0.2 *( 0.95 **epoch)))

    for batch in range(n_batch):

        x_batch,y_batch =mnist.train.next_batch(batch_size)

        sess.run(train_step, feed_dict ={x:x_batch,y:y_batch,drop_out: 0.7})

    acc =sess.run(currect_rate, feed_dict ={x:mnist.test.images,y:mnist.test.labels,drop_out: 1.0})

    acc_train =sess.run(currect_rate, feed_dict ={x:mnist.train.images,y:mnist.train.labels,drop_out: 1.0})

    print( "epoch: {0} ,acc: {1} ,train_acc: {1} ".format(epoch,acc,acc_train))