【手把手TensorFlow】三、神经网络搭建完整框架+MNIST数据集实践

前景回顾：
【手把手TensorFlow】一、从开始使用TensorFlow到弄清楚“搭建神经网络套路”
【手把手TensorFlow】二、神经网络优化

神经网络搭建完整框架：

1. 前向传播

定义参数w和偏置b，定义从输入到输出的网络结构

#前向传播过程
def forward(x,regularizer):
	w=get_weight()
	b=get_bias()
	y=
def get_weight(shape,regularizer):
	...
def get_bias(shape):
	...

2. 反向传播

反向传播过程完成网络参数的优化

def backward(mnist):
	x = tf.placeholder(dtype, shape)
	y_= tf.placeholder(dtype, shape)
	y=forward()
	global_step=...
	loss=...
	train_step=tf.train.GradientDescentOptimizer(learning_rate)
					.minmize(loss,global_step=global_step)
	#实例化saver，保存模型
	saver=tf.train.Saver()
	with tf.Session() as sess:
		#初始化模型参数
		tf.initialize_all_variables().run()
		#训练模型
		for i in range(STEPS):
			sess.run(train_step , feed_dict={x:  , y_:  })
			if i% 轮数==0:
				print
				saver.save()

3. 正则化，指数衰减，滑动平均方法的设置

正则化项：

在前向传播过程中设置正则化参数regularization为1时，表明反向传播过程中虚化模型参数时，需加入正则化项。
首先，在forward.py中加入：

if  regularizer != None: 
	tf.add_to_collection('losses',
			tf.contrib.layers.12_regularizer(regularizer)(w))

其次，要在backword.py中加入：

#交叉熵+softmax
ce = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y 
								,labels=tf.argmax(y_,1))
#均值
cem=tf.reduce_mean(ce)
#损失函数
loss=cem + tf.add_n(tf.get_collection('losses'))

tf.nn.sparse_softmax_cross_entropy_with_logits()
表示Softmax()函数和交叉熵一起使用。

指数衰减学习率

指数衰减学习率使模型接近收敛时学习率下降，使训练后期不会有太大波动。
在反向传播backward.py中加入：

learning_rate=tf.train.exponential_decay(
	LEARNING_RATE_BASE,
	global_step,
	LEARNING_RATE_STEP,LEARNING_RATE_DECAY,
	staircase=True
)

LEARNING_RATE_STEP表示多少轮后更新一次学习率。
LEARNING_RATE_DECAY为指数衰减率。
LEARNING_RATE_BASE为学习率基数。
staircase=True表示取整数，False表示取平滑曲线。

滑动平均

滑动平均记录一段时间中所有参数w和b各自的平均值，使模型在测试集上表现的更加健壮。
需要在backword.py中加入：

ema = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY,global_step)
ema_op = ema.applay(tf.trainable_variables())
with tf.control_dependencies([train_step,ema_op]):
	train_op=tf.no_op(name='train')

LEARNING_RATE_DECAY为指数衰减率。

4. 测试过程

测试工程用于验证神经网络的性能，结构如下：
①模型验证函数

def test(mnist):
	with tf.Graph().as_default() as g:
		#占位
		x= tf.placeholder(dtype,shape)
		y_=tf.placeholder(dtype,shape)
		#前向传播得到预测结果y
		y= mnist_forward.forward(x,None)
		#实例化可还原欢动平均的saver
		ema= tf.train.ExponentialMovingAverage(欢动衰减率)
		ema_restore = ema.variables_to_restore()
		saver = tf.train.Saver(ema_restore)
		#计算正确率
		correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_ , 1))
		accuracy = tf.reduce_mean(tf.cast(correct_prediction,tf.float32))
	
		while(True):
			with tf.Session() as sess:
				#加载训练好的模型
				ckpt=tf.train.get_checkpoint_state(存储路径)
				#如果已有ckpt模型则恢复
				if ckpt and ckpt.model_checkpoint_path:
					#恢复会话
					saver.restore(sess,ckpt.model_checkpoint_path)
					#恢复轮数
					global_step=ckpt.model_checkpoint_path.split('/')[-1].split('-')[-1]
					#计算准确率
					accuracy_score=sess.run(accuracy, feed_dict={x:测试数据, y_:测试标签})
					#打印提示
					print("After %s training step(s) , test_accuracy=%g" (global_step,accuracy_score))
				else:
					print('No checkpoint file found')
					return

其次，需要定制main()函数

def main():
	#加载测试集
	mnist=input_data.read_data_sets("./data/",one_hot=True)
	#调用定义好的测试函数test()
	test(mnist)
if __name__ == '__main__':
	main()

通过对测试数据的预测得到的准确率，从而判断出训练出的神经网络模型的性能好坏。

MNIST数据集实验：

forward . py

#coding:utf-8
import tensorflow as tf

INPUT_NODE = 784
OUTPUT_NODE = 10
LAYER1_NODE = 500

tf.reset_default_graph()

#设置参数w
def get_weight_variable(shape, regularizer):
    weights = tf.get_variable("weights", shape, initializer=tf.truncated_normal_initializer(stddev=0.1))
    if regularizer != None: tf.add_to_collection('losses', regularizer(weights))
    return weights

#定义前向传播
def forward(input_tensor, regularizer):
    with tf.variable_scope('layer1'):

        weights = get_weight_variable([INPUT_NODE, LAYER1_NODE], regularizer)
        biases = tf.get_variable("biases", [LAYER1_NODE], initializer=tf.constant_initializer(0.0))
        layer1 = tf.nn.relu(tf.matmul(input_tensor, weights) + biases)

    with tf.variable_scope('layer2'):
        weights = get_weight_variable([LAYER1_NODE, OUTPUT_NODE], regularizer)
        biases = tf.get_variable("biases", [OUTPUT_NODE], initializer=tf.constant_initializer(0.0))
        layer2 = tf.matmul(layer1, weights) + biases

    return layer2

backward. py

#coding:utf-8
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
import mnist_forward
import os

BATCH_SIZE = 100 
LEARNING_RATE_BASE = 0.8
LEARNING_RATE_DECAY = 0.99
REGULARIZATION_RATE = 0.0001
TRAINING_STEPS = 30000
MOVING_AVERAGE_DECAY = 0.99 
MODEL_SAVE_PATH = "MNIST_model/"
MODEL_NAME = "mnist_model"

def train(mnist):
    # 定义输入输出placeholder。
    x = tf.placeholder(tf.float32, [None, mnist_forward.INPUT_NODE], name='x-input')
    y_ = tf.placeholder(tf.float32, [None, mnist_forward.OUTPUT_NODE], name='y-input')

    regularizer = tf.contrib.layers.l2_regularizer(REGULARIZATION_RATE)
    y = mnist_forward.forward(x, regularizer)
    global_step = tf.Variable(0, trainable=False)
    
    # 定义损失函数、学习率、滑动平均操作以及训练过程。
    variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step)
    variables_averages_op = variable_averages.apply(tf.trainable_variables())
    cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1))
    cross_entropy_mean = tf.reduce_mean(cross_entropy)
    loss = cross_entropy_mean + tf.add_n(tf.get_collection('losses'))
    learning_rate = tf.train.exponential_decay(
        LEARNING_RATE_BASE,
        global_step,
        mnist.train.num_examples / BATCH_SIZE, LEARNING_RATE_DECAY,
        staircase=True)
    train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)
    with tf.control_dependencies([train_step, variables_averages_op]):
        train_op = tf.no_op(name='train')
        
    # 初始化TensorFlow持久化类。
    saver = tf.train.Saver()
    with tf.Session() as sess:
        tf.global_variables_initializer().run()
        
        #断点续训
        ckpt=tf.train.get_checkpoint_state(MODEL_SAVE_PATH)
        if ckpt and ckpt.model_checkpoint_path:
            saver.restore(sess,ckpt.model_checkpoint_path)

        for i in range(TRAINING_STEPS):
            xs, ys = mnist.train.next_batch(BATCH_SIZE)
            _, loss_value, step = sess.run([train_op, loss, global_step], feed_dict={x: xs, y_: ys})
            if i % 1000 == 0:
                print("After %d training step(s), loss on training batch is %g." % (step, loss_value))
                saver.save(sess, os.path.join(MODEL_SAVE_PATH, MODEL_NAME), global_step=global_step)
				
def main(argv=None):
    mnist = input_data.read_data_sets("../../../datasets/MNIST_data", one_hot=True)
    train(mnist)

if __name__ == '__main__':
    main()				

test .py

#coding:utf-8
import time
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
import mnist_forward
import mnist_backward

#每隔10秒钟加载一次新生成的模型进行测试
EVAL_INTERVAL_SECS = 10
#测试
def evaluate(mnist):
    with tf.Graph().as_default() as g:
        x = tf.placeholder(tf.float32, [None, mnist_forward.INPUT_NODE], name='x-input')
        y_ = tf.placeholder(tf.float32, [None, mnist_forward.OUTPUT_NODE], name='y-input')
        validate_feed = {x: mnist.validation.images, y_: mnist.validation.labels}

        y = mnist_forward.forward(x, None)
        correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

        #加载移动平均值
        variable_averages = tf.train.ExponentialMovingAverage(mnist_backward.MOVING_AVERAGE_DECAY)
        variables_to_restore = variable_averages.variables_to_restore()
        saver = tf.train.Saver(variables_to_restore)

        while True:
            with tf.Session() as sess:
                ckpt = tf.train.get_checkpoint_state(mnist_backward.MODEL_SAVE_PATH)
                if ckpt and ckpt.model_checkpoint_path:
                    saver.restore(sess, ckpt.model_checkpoint_path)
                    global_step = ckpt.model_checkpoint_path.split('/')[-1].split('-')[-1]
                    accuracy_score = sess.run(accuracy, feed_dict=validate_feed)
                    print("After %s training step(s), validation accuracy = %g" % (global_step, accuracy_score))
                else:
                    print('No checkpoint file found')
                    return
            time.sleep(EVAL_INTERVAL_SECS)
			
def main(argv=None):
    mnist = input_data.read_data_sets("../../../datasets/MNIST_data", one_hot=True)
    evaluate(mnist)

if __name__ == '__main__':
    main()			

搭建神经网络步骤最后总结：
1.前向传播
2.反向传播
3.正则化，指数衰减，滑动平均方法的设置（正则化参数在前向传播和反向传播中都加入，其它在反向传播中加入）
4.测试过程

遇到的BUG

1. Variable has existed/does not exist ,Did you mean to set reuse=True/None?
   解决：
   错误原因：自动保存了上一次的变量，导致变量名重复。
   解决方法：在forward.py开头加入tf.reset_default_graph()
   已加入。

2. 如果断电，需要断点处续训
   解决方法：加入下面通用的代码，保证下次从上次训练结束处开始训练。

        #断点续训
        ckpt=tf.train.get_checkpoint_state(MODEL_SAVE_PATH)
        if ckpt and ckpt.model_checkpoint_path:
            saver.restore(sess,ckpt.model_checkpoint_path)

参考文献：

1. 中国MOOC《tensorflow学习笔记》By 北京大学 曹健老师
2. ValueError: Variable rnn/basic_rnn_cell/kernel already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope?