Tensorflow学习笔记8：多层感知器
多层感知器
此代码用于培训Multi Layer Perceptron架构，其中输入将被转发到包含一些隐藏层的网络。
训练
训练：
可以使用train.sh bash脚本文件使用以下命令运行训练：
./train.sh
bash脚本如下：
python train_mlp.py \
--batch_size = 512 \
--max_num_checkpoint = 10 \
--num_classes = 10 \
--num_epochs = 1 \
--initial_learning_rate = 0.001 \
--num_epochs_per_decay = 1 \
--is_training =真\
--allow_soft_placement = True \
--fine_tuning =假\
--online_test =真\
--log_device_placement =假
帮助：
为了实现运行以下命令的输入参数是什么，建议：
python train_mlp.py --help
其中train_mlp.py是运行培训的主要文件。上述命令的结果如下：
--train_dir TRAIN_DIR
写入事件日志的目录。
--checkpoint_dir CHECKPOINT_DIR
写入检查点的目录。
--max_num_checkpoint MAX_NUM_CHECKPOINT
TensorFlow的最大检查点数
保持。
--num_classes NUM_CLASSES
要部署的模型克隆数。
--batch_size BATCH_SIZE
要部署的模型克隆数。
--num_epochs NUM_EPOCHS
时代数的训练。
--initial_learning_rate INITIAL_LEARNING_RATE
初始学习率。
--learning_rate_decay_factor LEARNING_RATE_DECAY_FACTOR
学习率衰减因子。
--num_epochs_per_decay NUM_EPOCHS_PER_DECAY
时代传递到衰减学习率的数量。
--is_training [IS_TRAINING]
训练/测试。
--fine_tuning [FINE_TUNING]
是否需要微调？。
--online_test [ONLINE_TEST]
是否需要微调？。
--allow_soft_placement [ALLOW_SOFT_PLACEMENT]如果没有，则
自动将变量放在CPU上
GPU支持。
--log_device_placement [LOG_DEVICE_PLACEMENT]
演示哪些变量在哪个设备上。
评估
将使用以下命令使用evaluation.sh bash脚本文件运行评估：
./evaluation.sh
源代码
evaluation.sh
# Run training.
python test_classifier.py \
--batch_size=512 \
--allow_soft_placement
test_classifier.py
from __future__ import print_function
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
import numpy as np
import os
import sys
######################################
######### Necessary Flags ############
######################################
tf.app.flags.DEFINE_string(
'test_dir', os.path.dirname(os.path.abspath(__file__)) + '/test_logs',
'Directory where event logs are written to.')
tf.app.flags.DEFINE_string(
'checkpoint_dir',
os.path.dirname(os.path.abspath(__file__)) + '/checkpoints',
'Directory where checkpoints are written to.')
tf.app.flags.DEFINE_integer('num_classes', 10,
'Number of model clones to deploy.')
tf.app.flags.DEFINE_integer('batch_size', np.power(2, 9),
'Number of model clones to deploy.')
#########################################
########## status flags #################
#########################################
tf.app.flags.DEFINE_boolean('is_training', False,
'Training/Testing.')
tf.app.flags.DEFINE_boolean('allow_soft_placement', True,
'Automatically put the variables on CPU if there is no GPU support.')
tf.app.flags.DEFINE_boolean('log_device_placement', False,
'Demonstrate which variables are on what device.')
# Store all elemnts in FLAG structure!
FLAGS = tf.app.flags.FLAGS
################################################
################# handling errors!##############
################################################
if not os.path.isabs(FLAGS.checkpoint_dir):
raise ValueError('You must assign absolute path for --checkpoint_dir')
##########################################
####### Load and Organize Data ###########
##########################################
'''
In this part the input must be prepared.
1 - The MNIST data will be downloaded.
2 - The images and labels for both training and testing will be extracted.
3 - The prepared data format(?,784) is different by the appropriate image shape(?,28,28,1) which needs
to be fed to the CNN architecture. So it needs to be reshaped.
'''
# Download and get MNIST dataset(available in tensorflow.contrib.learn.python.learn.datasets.mnist)
# It checks and download MNIST if it's not already downloaded then extract it.
# The 'reshape' is True by default to extract feature vectors but we set it to false to we get the original images.
mnist = input_data.read_data_sets("MNIST_data/", reshape=True, one_hot=True)
# Dimentionality of train
dimensionality = mnist.train.images.shape
# Dimensions
num_train_samples = dimensionality[0]
num_features = dimensionality[1]
#######################################
########## Defining Graph ############
#######################################
graph = tf.Graph()
with graph.as_default():
###################################
########### Parameters ############
###################################
# global step
global_step = tf.Variable(0, name="global_step", trainable=False)
###############################################
########### Defining place holders ############
###############################################
image_place = tf.placeholder(tf.float32, shape=([None, num_features]), name='image')
label_place = tf.placeholder(tf.float32, shape=([None, FLAGS.num_classes]), name='gt')
dropout_param = tf.placeholder(tf.float32)
##################################################
########### Model + loss + accuracy ##############
##################################################
# MODEL(MPL with two hidden layer)
# LAYER-1
net = tf.contrib.layers.fully_connected(inputs=image_place, num_outputs=250, scope='fc-1')
# LAYER-2
net = tf.contrib.layers.fully_connected(inputs=net, num_outputs=250, scope='fc-2')
# SOFTMAX
logits_last = tf.contrib.layers.fully_connected(inputs=net, num_outputs=FLAGS.num_classes, scope='fc-3')
# Define loss
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits_last, labels=label_place))
# Accuracy
# Evaluate the model
pred_classifier = tf.equal(tf.argmax(logits_last, 1), tf.argmax(label_place, 1))
# Accuracy calculation
accuracy = tf.reduce_mean(tf.cast(pred_classifier, tf.float32))
###############################################
############ Define Sammaries #################
###############################################
# Image summaries(draw three random images from data in both training and testing phases)
# The image summaries is only cerated for train summaries and it get three random images from the training set.
arr = np.random.randint(mnist.test.images.shape[0], size=(3,))
tf.summary.image('images', mnist.test.images[arr], max_outputs=3,
collections=['per_epoch_train'])
# Summaries for loss and accuracy
tf.summary.scalar("loss", loss, collections=['test'])
tf.summary.scalar("accuracy", accuracy, collections=['test'])
tf.summary.scalar("global_step", global_step, collections=['test'])
# Merge all summaries together.
summary_test_op = tf.summary.merge_all('test')
########################################################
############ # Defining the tensors list ###############
########################################################
# tensors_key = ['loss', 'accuracy', 'global_step', 'image_place', 'label_place',
# 'summary_test_op']
# tensors_values = [loss, accuracy, global_step, image_place, label_place, summary_test_op]
# tensors = dict(zip(tensors_key, tensors_values))
############################################
############ Run the Session ###############
############################################
session_conf = tf.ConfigProto(
allow_soft_placement=FLAGS.allow_soft_placement,
log_device_placement=FLAGS.log_device_placement)
sess = tf.Session(graph=graph, config=session_conf)
with sess.as_default():
# The saver op.
saver = tf.train.Saver()
# Initialize all variables
sess.run(tf.global_variables_initializer())
###################################################################
########## Defining the summary writers for test ###########
###################################################################
test_summary_dir = os.path.join(FLAGS.test_dir, "summaries", "test")
test_summary_writer = tf.summary.FileWriter(test_summary_dir)
test_summary_writer.add_graph(sess.graph)
# The prefix for checkpoint files
checkpoint_prefix = 'model'
# Restoring the saved weights.
saver.restore(sess, os.path.join(FLAGS.checkpoint_dir, checkpoint_prefix))
print("Model restored...")
###################################################################
########## Run the training and loop over the batches #############
###################################################################
total_batch_test = int(mnist.test.images.shape[0] / FLAGS.batch_size)
# go through the batches
test_accuracy = 0
for batch_num in range(total_batch_test):
#################################################
########## Get the training batches #############
#################################################
start_idx = batch_num * FLAGS.batch_size
end_idx = (batch_num + 1) * FLAGS.batch_size
# Fit training using batch data
test_batch_data, test_batch_label = mnist.test.images[start_idx:end_idx], mnist.test.labels[
start_idx:end_idx]
########################################
########## Run the session #############
########################################
# Run session and Calculate batch loss and accuracy
# When the tensor tensors['global_step'] is evaluated, it will be incremented by one.
test_batch_accuracy, batch_loss, test_summaries, test_step = sess.run(
[accuracy, loss, summary_test_op,
global_step],
feed_dict={image_place: test_batch_data,
label_place: test_batch_label})
test_accuracy += test_batch_accuracy
########################################
########## Write summaries #############
########################################
# Write the summaries
test_summary_writer.add_summary(test_summaries, global_step=test_step)
# # Write the specific summaries for training phase.
# train_summary_writer.add_summary(train_image_summary, global_step=training_step)
#################################################
########## Plot the progressive bar #############
#################################################
print("Batch " + str(batch_num + 1) + ", Testing Loss= " + \
"{:.5f}".format(test_batch_accuracy))
######################################################################
########## Calculate the accuracy for the whole test set #############
######################################################################
test_accuracy_total = test_accuracy / float(total_batch_test)
print("Total Test Accuracy= " + \
"{:.5f}".format(test_accuracy_total))
train.sh
# Run training.
python train_mlp.py \
--batch_size=512 \
--max_num_checkpoint=10 \
--num_classes=10 \
--num_epochs=1 \
--initial_learning_rate=0.001 \
--num_epochs_per_decay=1 \
--is_training=True \
--allow_soft_placement=True \
--fine_tuning=False \
--online_test=True \
--log_device_placement=False
train_mlp.py
from __future__ import print_function
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
import numpy as np
import os
######################################
######### Necessary Flags ############
######################################
tf.app.flags.DEFINE_string(
'train_root', os.path.dirname(os.path.abspath(__file__)) + '/train_logs',
'Directory where event logs are written to.')
tf.app.flags.DEFINE_string(
'checkpoint_root',
os.path.dirname(os.path.abspath(__file__)) + '/checkpoints',
'Directory where checkpoints are written to.')
tf.app.flags.DEFINE_integer('max_num_checkpoint', 10,
'Maximum number of checkpoints that TensorFlow will keep.')
tf.app.flags.DEFINE_integer('num_classes', 10,
'Number of model clones to deploy.')
tf.app.flags.DEFINE_integer('batch_size', np.power(2, 7),
'Number of model clones to deploy.')
tf.app.flags.DEFINE_integer('num_epochs', 5,
'Number of epochs for training.')
##########################################
######## Learning rate flags #############
##########################################
tf.app.flags.DEFINE_float('initial_learning_rate', 0.001, 'Initial learning rate.')
tf.app.flags.DEFINE_float(
'learning_rate_decay_factor', 0.95, 'Learning rate decay factor.')
tf.app.flags.DEFINE_float(
'num_epochs_per_decay', 1, 'Number of epoch pass to decay learning rate.')
#########################################
########## status flags #################
#########################################
tf.app.flags.DEFINE_boolean('is_training', False,
'Training/Testing.')
tf.app.flags.DEFINE_boolean('fine_tuning', False,
'Fine tuning is desired or not?.')
tf.app.flags.DEFINE_boolean('online_test', True,
'Fine tuning is desired or not?.')
tf.app.flags.DEFINE_boolean('allow_soft_placement', True,
'Automatically put the variables on CPU if there is no GPU support.')
tf.app.flags.DEFINE_boolean('log_device_placement', False,
'Demonstrate which variables are on what device.')
# Store all elemnts in FLAG structure!
FLAGS = tf.app.flags.FLAGS
################################################
################# handling errors!##############
################################################
if not os.path.isabs(FLAGS.train_root):
raise ValueError('You must assign absolute path for --train_root')
if not os.path.isabs(FLAGS.checkpoint_root):
raise ValueError('You must assign absolute path for --checkpoint_root')
##########################################
####### Load and Organize Data ###########
##########################################
'''
In this part the input must be prepared.
1 - The MNIST data will be downloaded.
2 - The images and labels for both training and testing will be extracted.
3 - The prepared data format(?,784) is different by the appropriate image shape(?,28,28,1) which needs
to be fed to the CNN architecture. So it needs to be reshaped.
'''
# Download and get MNIST dataset(available in tensorflow.contrib.learn.python.learn.datasets.mnist)
# It checks and download MNIST if it's not already downloaded then extract it.
# The 'reshape' is True by default to extract feature vectors but we set it to false to we get the original images.
mnist = input_data.read_data_sets("MNIST_data/", reshape=True, one_hot=True)
train_data = mnist.train.images
train_label = mnist.train.labels
test_data = mnist.test.images
test_label = mnist.test.labels
# # The 'input.provide_data' is provided to organize any custom dataset which has specific characteristics.
# data = input.provide_data(mnist)
# Dimentionality of train
dimensionality_train = train_data.shape
# Dimensions
num_train_samples = dimensionality_train[0]
num_features = dimensionality_train[1]
#######################################
########## Defining Graph ############
#######################################
graph = tf.Graph()
with graph.as_default():
###################################
########### Parameters ############
###################################
# global step
global_step = tf.Variable(0, name="global_step", trainable=False)
# learning rate policy
decay_steps = int(num_train_samples / FLAGS.batch_size *
FLAGS.num_epochs_per_decay)
learning_rate = tf.train.exponential_decay(FLAGS.initial_learning_rate,
global_step,
decay_steps,
FLAGS.learning_rate_decay_factor,
staircase=True,
name='exponential_decay_learning_rate')
###############################################
########### Defining place holders ############
###############################################
image_place = tf.placeholder(tf.float32, shape=([None, num_features]), name='image')
label_place = tf.placeholder(tf.float32, shape=([None, FLAGS.num_classes]), name='gt')
dropout_param = tf.placeholder(tf.float32)
##################################################
########### Model + Loss + Accuracy ##############
##################################################
# MODEL(MPL with two hidden layer)
# LAYER-1
net = tf.contrib.layers.fully_connected(inputs=image_place, num_outputs=250, scope='fc-1')
# LAYER-2
net = tf.contrib.layers.fully_connected(inputs=net, num_outputs=250, scope='fc-2')
# SOFTMAX
logits_pre_softmax = tf.contrib.layers.fully_connected(inputs=net, num_outputs=FLAGS.num_classes, scope='fc-3')
# Define loss
softmax_loss = tf.reduce_mean(
tf.nn.softmax_cross_entropy_with_logits(logits=logits_pre_softmax, labels=label_place))
# Accuracy
accuracy = tf.reduce_mean(
tf.cast(tf.equal(tf.argmax(logits_pre_softmax, 1), tf.argmax(label_place, 1)), tf.float32))
#############################################
########### training operation ##############
#############################################
# Define optimizer by its default values
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
# 'train_op' is a operation that is run for gradient update on parameters.
# Each execution of 'train_op' is a training step.
# By passing 'global_step' to the optimizer, each time that the 'train_op' is run, Tensorflow
# update the 'global_step' and increment it by one!
# gradient update.
with tf.name_scope('train_scope'):
grads = optimizer.compute_gradients(softmax_loss)
train_op = optimizer.apply_gradients(grads, global_step=global_step)
###############################################
############ Define Sammaries #################
###############################################
# Summaries for loss and accuracy
tf.summary.scalar("loss", softmax_loss, collections=['train', 'test'])
tf.summary.scalar("accuracy", accuracy, collections=['train', 'test'])
tf.summary.scalar("global_step", global_step, collections=['train'])
tf.summary.scalar("learning_rate", learning_rate, collections=['train'])
# Merge all summaries together.
summary_train_op = tf.summary.merge_all('train')
summary_test_op = tf.summary.merge_all('test')
############################################
############ Run the Session ###############
############################################
session_conf = tf.ConfigProto(
allow_soft_placement=FLAGS.allow_soft_placement,
log_device_placement=FLAGS.log_device_placement)
sess = tf.Session(graph=graph, config=session_conf)
with sess.as_default():
# Run the saver.
# 'max_to_keep' flag determines the maximum number of models that the tensorflow save and keep. default by TensorFlow = 5.
saver = tf.train.Saver(max_to_keep=FLAGS.max_num_checkpoint)
# Initialize all variables
sess.run(tf.global_variables_initializer())
###################################################
############ Training / Evaluation ###############
###################################################
# The prefix for checkpoint files
checkpoint_prefix = 'model'
###################################################################
########## Defining the summary writers for train/test ###########
###################################################################
train_summary_dir = os.path.join(FLAGS.train_root, "summaries", "train")
train_summary_writer = tf.summary.FileWriter(train_summary_dir)
train_summary_writer.add_graph(sess.graph)
test_summary_dir = os.path.join(FLAGS.train_root, "summaries", "test")
test_summary_writer = tf.summary.FileWriter(test_summary_dir)
test_summary_writer.add_graph(sess.graph)
# If fine-tuning flag in 'True' the model will be restored.
if FLAGS.fine_tuning:
saver.restore(sess, os.path.join(FLAGS.checkpoint_root, checkpoint_prefix))
print("Model restored for fine-tuning...")
###################################################################
########## Run the training and loop over the batches #############
###################################################################
for epoch in range(FLAGS.num_epochs):
total_batch_training = int(train_data.shape[0] / FLAGS.batch_size)
# go through the batches
for batch_num in range(total_batch_training):
#################################################
########## Get the training batches #############
#################################################
start_idx = batch_num * FLAGS.batch_size
end_idx = (batch_num + 1) * FLAGS.batch_size
# Fit training using batch data
train_batch_data, train_batch_label = train_data[start_idx:end_idx], train_label[
start_idx:end_idx]
########################################
########## Run the session #############
########################################
# Run optimization op (backprop) and Calculate batch loss and accuracy
# When the tensor tensors['global_step'] is evaluated, it will be incremented by one.
batch_loss, _, train_summaries, training_step = sess.run(
[softmax_loss, train_op,
summary_train_op,
global_step],
feed_dict={image_place: train_batch_data,
label_place: train_batch_label,
dropout_param: 0.5})
########################################
########## Write summaries #############
########################################
# Write the summaries
train_summary_writer.add_summary(train_summaries, global_step=training_step)
# # Write the specific summaries for training phase.
# train_summary_writer.add_summary(train_image_summary, global_step=training_step)
#################################################
########## Plot the progressive bar #############
#################################################
print("Epoch #" + str(epoch + 1) + ", Train Loss=" + \
"{:.3f}".format(batch_loss))
#####################################################
########## Evaluation on the test data #############
#####################################################
if FLAGS.online_test:
# WARNING: In this evaluation the whole test data is fed. In case the test data is huge this implementation
# may lead to memory error. In presense of large testing samples, batch evaluation on testing is
# recommended as in the training phase.
test_accuracy_epoch, test_summaries = sess.run(
[accuracy, summary_test_op],
feed_dict={image_place: test_data,
label_place: test_label,
dropout_param: 1.})
print("Test Accuracy= " + \
"{:.4f}".format(test_accuracy_epoch))
###########################################################
########## Write the summaries for test phase #############
###########################################################
# Returning the value of global_step if necessary
current_step = tf.train.global_step(sess, global_step)
# Add the couter of global step for proper scaling between train and test summuries.
test_summary_writer.add_summary(test_summaries, global_step=current_step)
###########################################################
############ Saving the model checkpoint ##################
###########################################################
# # The model will be saved when the training is done.
# Create the path for saving the checkpoints.
if not os.path.exists(FLAGS.checkpoint_root):
os.makedirs(FLAGS.checkpoint_root)
# save the model
save_path = saver.save(sess, os.path.join(FLAGS.checkpoint_root, checkpoint_prefix))
print("Model saved in file: %s" % save_path)
############################################################################
########## Run the session for pur evaluation on the test data #############
############################################################################
# The prefix for checkpoint files
checkpoint_prefix = 'model'
# Restoring the saved weights.
saver.restore(sess, os.path.join(FLAGS.checkpoint_root, checkpoint_prefix))
print("Model restored...")
# Evaluation of the model
total_test_accuracy = sess.run(accuracy, feed_dict={
image_place: test_data,
label_place: test_label,
dropout_param: 1.})
print("Final Test Accuracy is %.2f" % total_test_accuracy)
train_mlp.ipynb
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from __future__ import print_function\n",
"from tensorflow.examples.tutorials.mnist import input_data\n",
"import tensorflow as tf\n",
"import numpy as np\n",
"import os"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"######################################\n",
"######### Necessary Flags ############\n",
"######################################\n",
"\n",
"num_classes = 10\n",
"batch_size = 128\n",
"num_epochs = 10"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"##########################################\n",
"######## Learning rate flags #############\n",
"##########################################\n",
"initial_learning_rate = 0.001\n",
"learning_rate_decay_factor = 0.95\n",
"num_epochs_per_decay = 1"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#########################################\n",
"########## status flags #################\n",
"#########################################\n",
"is_training = False\n",
"fine_tuning = False\n",
"online_test = True\n",
"allow_soft_placement = True\n",
"log_device_placement = False"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
"Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
"Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
"Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
]
}
],
"source": [
"##########################################\n",
"####### Load and Organize Data ###########\n",
"##########################################\n",
"'''\n",
"In this part the input must be prepared.\n",
"\n",
" 1 - The MNIST data will be downloaded.\n",
" 2 - The images and labels for both training and testing will be extracted.\n",
" 3 - The prepared data format(?,784) is different by the appropriate image shape(?,28,28,1) which needs\n",
" to be fed to the CNN architecture. So it needs to be reshaped.\n",
"\n",
"'''\n",
"\n",
"# Download and get MNIST dataset(available in tensorflow.contrib.learn.python.learn.datasets.mnist)\n",
"# It checks and download MNIST if it's not already downloaded then extract it.\n",
"# The 'reshape' is True by default to extract feature vectors but we set it to false to we get the original images.\n",
"mnist = input_data.read_data_sets(\"MNIST_data/\", reshape=True, one_hot=True)\n",
"train_data = mnist.train.images\n",
"train_label = mnist.train.labels\n",
"test_data = mnist.test.images\n",
"test_label = mnist.test.labels\n",
"\n",
"# # The 'input.provide_data' is provided to organize any custom dataset which has specific characteristics.\n",
"# data = input.provide_data(mnist)\n",
"\n",
"# Dimentionality of train\n",
"dimensionality_train = train_data.shape\n",
"\n",
"# Dimensions\n",
"num_train_samples = dimensionality_train[0]\n",
"num_features = dimensionality_train[1]"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch #1, Train Loss=0.248\n",
"Test Accuracy= 0.8565\n",
"Epoch #2, Train Loss=0.242\n",
"Test Accuracy= 0.8610\n",
"Epoch #3, Train Loss=0.240\n",
"Test Accuracy= 0.8676\n",
"Epoch #4, Train Loss=0.236\n",
"Test Accuracy= 0.8737\n",
"Epoch #5, Train Loss=0.235\n",
"Test Accuracy= 0.8740\n",
"Epoch #6, Train Loss=0.234\n",
"Test Accuracy= 0.8753\n",
"Epoch #7, Train Loss=0.234\n",
"Test Accuracy= 0.8756\n",
"Epoch #8, Train Loss=0.244\n",
"Test Accuracy= 0.8766\n",
"Epoch #9, Train Loss=0.234\n",
"Test Accuracy= 0.8765\n",
"Epoch #10, Train Loss=0.236\n",
"Test Accuracy= 0.8786\n",
"Final Test Accuracy is 0.88\n"
]
}
],
"source": [
"#######################################\n",
"########## Defining Graph ############\n",
"#######################################\n",
"\n",
"graph = tf.Graph()\n",
"with graph.as_default():\n",
" ###################################\n",
" ########### Parameters ############\n",
" ###################################\n",
"\n",
" # global step\n",
" global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n",
"\n",
" # learning rate policy\n",
" decay_steps = int(num_train_samples / batch_size *\n",
" num_epochs_per_decay)\n",
" learning_rate = tf.train.exponential_decay(initial_learning_rate,\n",
" global_step,\n",
" decay_steps,\n",
" learning_rate_decay_factor,\n",
" staircase=True,\n",
" name='exponential_decay_learning_rate')\n",
"\n",
" ###############################################\n",
" ########### Defining place holders ############\n",
" ###############################################\n",
" image_place = tf.placeholder(tf.float32, shape=([None, num_features]), name='image')\n",
" label_place = tf.placeholder(tf.float32, shape=([None, num_classes]), name='gt')\n",
" dropout_param = tf.placeholder(tf.float32)\n",
"\n",
" ##################################################\n",
" ########### Model + Loss + Accuracy ##############\n",
" ##################################################\n",
"\n",
" # MODEL(MPL with two hidden layer)\n",
"\n",
" # LAYER-1\n",
" net = tf.contrib.layers.fully_connected(inputs=image_place, num_outputs=250, scope='fc-1')\n",
"\n",
" # LAYER-2\n",
" net = tf.contrib.layers.fully_connected(inputs=net, num_outputs=250, scope='fc-2')\n",
"\n",
" # SOFTMAX\n",
" logits_pre_softmax = tf.contrib.layers.fully_connected(inputs=net, num_outputs=num_classes, scope='fc-3')\n",
"\n",
" # Define loss\n",
" softmax_loss = tf.reduce_mean(\n",
" tf.nn.softmax_cross_entropy_with_logits(logits=logits_pre_softmax, labels=label_place))\n",
"\n",
" # Accuracy\n",
" accuracy = tf.reduce_mean(\n",
" tf.cast(tf.equal(tf.argmax(logits_pre_softmax, 1), tf.argmax(label_place, 1)), tf.float32))\n",
"\n",
" #############################################\n",
" ########### training operation ##############\n",
" #############################################\n",
"\n",
" # Define optimizer by its default values\n",
" optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
"\n",
" # 'train_op' is a operation that is run for gradient update on parameters.\n",
" # Each execution of 'train_op' is a training step.\n",
" # By passing 'global_step' to the optimizer, each time that the 'train_op' is run, Tensorflow\n",
" # update the 'global_step' and increment it by one!\n",
"\n",
" # gradient update.\n",
" with tf.name_scope('train_scope'):\n",
" grads = optimizer.compute_gradients(softmax_loss)\n",
" train_op = optimizer.apply_gradients(grads, global_step=global_step)\n",
"\n",
" ###############################################\n",
" ############ Define Sammaries #################\n",
" ###############################################\n",
"\n",
" # Summaries for loss and accuracy\n",
" tf.summary.scalar(\"loss\", softmax_loss, collections=['train', 'test'])\n",
" tf.summary.scalar(\"accuracy\", accuracy, collections=['train', 'test'])\n",
" tf.summary.scalar(\"global_step\", global_step, collections=['train'])\n",
" tf.summary.scalar(\"learning_rate\", learning_rate, collections=['train'])\n",
"\n",
" # Merge all summaries together.\n",
" summary_train_op = tf.summary.merge_all('train')\n",
" summary_test_op = tf.summary.merge_all('test')\n",
"\n",
" ############################################\n",
" ############ Run the Session ###############\n",
" ############################################\n",
" session_conf = tf.ConfigProto(\n",
" allow_soft_placement=allow_soft_placement,\n",
" log_device_placement=log_device_placement)\n",
" sess = tf.Session(graph=graph, config=session_conf)\n",
"\n",
" with sess.as_default():\n",
"\n",
" # Initialize all variables\n",
" sess.run(tf.global_variables_initializer())\n",
"\n",
" ###################################################################\n",
" ########## Run the training and loop over the batches #############\n",
" ###################################################################\n",
" for epoch in range(num_epochs):\n",
" total_batch_training = int(train_data.shape[0] / batch_size)\n",
"\n",
" # go through the batches\n",
" for batch_num in range(total_batch_training):\n",
" #################################################\n",
" ########## Get the training batches #############\n",
" #################################################\n",
"\n",
" start_idx = batch_num * batch_size\n",
" end_idx = (batch_num + 1) * batch_size\n",
"\n",
" # Fit training using batch data\n",
" train_batch_data, train_batch_label = train_data[start_idx:end_idx], train_label[\n",
" start_idx:end_idx]\n",
"\n",
" ########################################\n",
" ########## Run the session #############\n",
" ########################################\n",
"\n",
" # Run optimization op (backprop) and Calculate batch loss and accuracy\n",
" # When the tensor tensors['global_step'] is evaluated, it will be incremented by one.\n",
" batch_loss, _, training_step = sess.run(\n",
" [softmax_loss, train_op, global_step],\n",
" feed_dict={image_place: train_batch_data,\n",
" label_place: train_batch_label,\n",
" dropout_param: 0.5})\n",
"\n",
"\n",
" #################################################\n",
" ########## Plot the progressive bar #############\n",
" #################################################\n",
"\n",
" print(\"Epoch #\" + str(epoch + 1) + \", Train Loss=\" + \\\n",
" \"{:.3f}\".format(batch_loss))\n",
"\n",
" #####################################################\n",
" ########## Evaluation on the test data #############\n",
" #####################################################\n",
"\n",
" if online_test:\n",
" # WARNING: In this evaluation the whole test data is fed. In case the test data is huge this implementation\n",
" # may lead to memory error. In presense of large testing samples, batch evaluation on testing is\n",
" # recommended as in the training phase.\n",
" test_accuracy_epoch, test_summaries = sess.run(\n",
" [accuracy, summary_test_op],\n",
" feed_dict={image_place: test_data,\n",
" label_place: test_label,\n",
" dropout_param: 1.})\n",
" print(\"Test Accuracy= \" + \\\n",
" \"{:.4f}\".format(test_accuracy_epoch))\n",
"\n",
" ###########################################################\n",
" ########## Write the summaries for test phase #############\n",
" ###########################################################\n",
"\n",
" # Returning the value of global_step if necessary\n",
" current_step = tf.train.global_step(sess, global_step)\n",
"\n",
"\n",
" # Evaluation of the model\n",
" total_test_accuracy = sess.run(accuracy, feed_dict={\n",
" image_place: test_data,\n",
" label_place: test_label,\n",
" dropout_param: 1.})\n",
"\n",
" print(\"Final Test Accuracy is %.2f\" % total_test_accuracy)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}