深度学习·TensorFlow实现CNNs处理MNIST数据集
1. CNNs基本原理
CNNs即Convolutional Neural Networks多层卷积神经网络，专门用于图像识别
生物仿生学原理：
该网络结构模仿了人体识别图像的过程：
1 瞳孔摄入物体的像素信息
2 大脑皮层一些细胞进行初步处理：如发现物体的边缘和方向
3 抽象判断物体的形状(如是圆形或方形)
4 进一步抽象判断(如是气球或木块)
简而言之，高层是低层的抽象化表示，具体到神经网络中计算权重的过程即是每层的特征抽取过程
在实现模型之前，我们先来了解CNNs的最基本的两个基本概念：卷积和池化
我们所熟悉的一般BPNN(或是MLP)处理图像信息的过程是：先将像素信息处理成一维数组，然后进行前向传播计算权重和误差，当达到输出层后再将误差反向传播并更新每一层的权重。但这样的处理方式面临着一个问题：当处理图像是RGB 256 500*500像素时，每个神经元3*500*500=750000的计算量还要加上神网络的层数和每层的深度，此时模型的计算量将非常巨大，所以卷积和池化简单来说就起着降低特征维度和简化计算的作用
卷积
基本原理:
卷积即是我们使用卷积核(一个二维矩阵)去按照一定步长和边距扫描(矩阵点乘)原始图像的像素矩阵，得到一个相对初始像素矩阵特征更清晰也更抽象的矩阵(也称为特征平面)的过程
但与BPNN的前后神经元全连接不同，CNN只是部分连接，由全局感知变为局部感知从而实现了降维的目的
个人理解：一般来说图像的像素矩阵在使用神经网络处理之前是先将原始的2维数据转换为1维的向量才方便计算，而卷积的作用就是在模型中逐渐把1维的向量变得立体起来，通过不同的卷积核将数据进行分块处理，比如在识别汽车时一个卷积核只识别反光镜，而另一个卷积核只识别轮胎等等
池化
基本原理：
又成为子采样，是特殊的卷积(相当于卷积核元素都是1的矩阵)，用于对卷积之后特征矩阵进行降维，一般有两种类型：
1 最大池化，即只取矩阵中元素的最大值
2 平均池化，即取矩阵中所有元素的平均值
还有结构概念，即经过一系列的卷积和池化作用后，得到的结果仍然是矩阵，我们就需要在最后的处理步骤通过一个结构转化为输出所需要的向量
除此之外还有误差反向传播的过程，要涉及池化层和卷积层的向上传播略微复杂在此就先不展开说明了
2. 建立多重卷积神经网络
封装权重和偏置量的初始化
# initialize weight
# use ReLU neuron, so initialize to small positive value to void
# output forever 0
def weight_variable(shape):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
def bias_variable(shape):
initial = tf.constant(0.1, shape=shape)
return tf.Variable(initial)注解：
- 因为接下来要使用ReLU为神经元间的激活函数，所以初始值不能为0
- tf.truncated_normal()表示输出来自截断正态分布(即限制变量x的取值范围的一种分布)的随机值，其中
- shape是1维整数张量，表示输出张量的大小
- stddev表示在截断前正态分布的标准偏移量
- tf.constant()创建一个常量增量，第一个参数是张量中每个元素的初始值
封装卷积和池化
# Revolution and pooling, vanilla version
# Revolution
# stride size: 1
# padding size: 0
# Pooling
# 2 * 2 max pooling
def conv2d(x, W):
return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def max_pool_2x2(x):
return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
strides=[1, 2, 2, 1], padding='SAME')注解：
- 使用Vanilla版本，即卷积步长为1，边距为0；池化使用基本的2x2矩阵做最大池化
- tf.nn.conv2d()表示对给定的4维的输入和卷积核张量计算2维的卷积，其中，
- 四个维度分别表示[批数，高，宽，频道数]
- 第1、2个参数分别为输入和卷积核(即权重)
- strides参数为1个长为4的1维张量，表示在每个维度每次滑动的步长，即本例中每个维度的移动步长都是1
- padding参数表示当滑动时剩下的列数不够一次滑动时(例如设定滑动步长是2，但是总列数只有5列，就必然导致了最后一次滑动时只剩下一列)的处理方式，’SAME’表示不够的列以0填充，而’VALID’表示停止当前滑动并舍弃多余的列
- tf.nn.max_pool()即对输入进行最大池化，其中，
- 第1个参数是输入
- ksize参数是有4个整形数值的列表或元组表示池化窗口的大小，而[1, 2, 2, 1]表示使用2x2的矩阵进行池化，批数和频道数都为1表示不对二者做池化
- strides和padding参数与tf.nn.conv2d()相同
进行第1次卷积
# 1st convolution
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])
x_image = tf.reshape(x, [-1, 28, 28, 1])
h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)注解：
- W_conv1为5x5x1x32的权重张量作为第1层卷积核(各个维度我们可以理解为卷积核是5x5的矩阵，然后输入通道的数目为1(灰度图)，输出通道数目为32)，即每个5x5的卷积核会得到32个特征
- b_conv1 为32组偏置值
- x_image为[-1, 28, 28, 1]将原始的1维向量转换为28x28的矩阵，其中值得一提的是tf.reshape()中的第二个参数shape中当传入值-1时表示表示在该维度使用适应值(可以理解为可使每次变换大小都有效的值，比如1x4矩阵可以变换成2x2矩阵，但是不能变为2*3，若使用[4 -1]就自动变为4x1矩阵，此时-1变为1)，与placeholder中的None异曲同工
- h_conv1为使用线性修正激活函数将从输入层提取的特征进行转化，得到的特征大小为[-1, 28, 28, 32]
- h_pool1为将上个步骤得到的矩阵(特征平面)进行降维，得到[-1, 14, 14, 32]特征
第2层卷积
# 2nd convolution
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)因为过程类似，不再赘述
- 再卷积后得到的特征矩阵为[-1, 14, 14, 32]
- 再池化后得到的特征矩阵为[-1, 7, 7, 64]
密集连接层(即全连接层)
# Dense link layer
W_fc1 = weight_variable([7 * 7 * 64, 1024])
b_fc1 = bias_variable([1024])
h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)注解：
- 由于经过两次卷积池化后，图片的大小已变为7x7，则下一层使用有1024个神经元的全连接层，使得便于处理整个图片(至于全连接层的神经元个数为什么取1024，暂时找不到相应的解释)
- 此外需要注意的是计算权重时使用了线性回归算法
dropout操作
原理：
为了减少模型的过拟合，在网络的训练过程中，按照一定的概率将神经网络单元暂时丢弃(为什么这样做能减少过拟合，请看)
# Add dropout
# to minus overfit
keep_prob = tf.placeholder('float')
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)注解：
- 首先使用1个占位符来表示1个神经单元的输出在dropout操作中能保持不变的概率，即不会被暂时舍弃
- tf.nn.dropout()除了能屏蔽神经元的输出外，还能自动处理神经元输出值的单位
输出层
# Output layer
# add softmax layer
W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])
y_conv = tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)注解：
- 也就是上文提到的结构所起到的作用，将卷积的结果转换为所需的输出向量
- 此外需注意的是，本层的激励函数使用的softmax来计算特征属于某个标签的概率
3. 训练和评估
# Use more comlicated optimizer ADAM to perform gradientdescent
# add keep_prob argument into feed_list to contral size of dropout
cross_entropy = -tf.reduce_sum(y_ * tf.log(y_conv))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'))
sess.run(tf.global_variables_initializer())
for i in range(20000):
batch = mnist.train.next_batch(50)
if i % 100 == 0:
train_accuracy = accuracy.eval(feed_dict={
x: batch[0], y_: batch[1], keep_prob: 1.0
})
print('step %d, training accuracy %g' % (i, train_accuracy))
train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
print('test accuracy %g' % accuracy.eval(feed_dict={
x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0
}))注解：
- 本次训练使用的代价函数仍是交叉熵函数，但梯度下降算法变为ADAM(即随机梯度算法的扩展，具体请看)，模型评估指标仍使用准确率
- 使用每批50张图片的速度进行训练，并将dropout的参数keep_prob放入feed_list中一起进行传递
- 训练时使每次训练的dropout保留概率为0.5，此外每训练100次评估一下当前训练效果(keep_prob设置为1表示不舍弃保留全部神经元)
- 最后打印出的训练结果(略长完整版附在文章末尾)：
step 0, training accuracy 0.02
step 100, training accuracy 0.84
step 200, training accuracy 0.86
…
step 19700, training accuracy 1
step 19800, training accuracy 1
step 19900, training accuracy 1
test accuracy 0.9919
由此可见CNN模型最终得到的效果要好于简单的单层softmax模型，但不得不提的一点是训练过程很耗时。
另外通过观察完整训练过程后我们可以发现在训练14100次后模型效果才基本趋于稳定，因此要想模型效果好足够训练时间确是必不可少的。
参考博客：
0. TensorFlow中文社区MNIST进阶
1. 卷积神经网络CNN（一）基本概念、卷积
2. 卷积神经网络CNN（二）池化、结构、误差反向传播
3. 深入MNIST
4. 理解dropout
5. 深度学习最常用的学习算法：Adam优化算法
附表1
step 0, training accuracy 0.02
step 100, training accuracy 0.84
step 200, training accuracy 0.86
step 300, training accuracy 0.98
step 400, training accuracy 0.96
step 500, training accuracy 0.96
step 600, training accuracy 0.94
step 700, training accuracy 0.98
step 800, training accuracy 0.94
step 900, training accuracy 0.98
step 1000, training accuracy 0.94
step 1100, training accuracy 0.98
step 1200, training accuracy 1
step 1300, training accuracy 1
step 1400, training accuracy 0.98
step 1500, training accuracy 0.96
step 1600, training accuracy 1
step 1700, training accuracy 1
step 1800, training accuracy 1
step 1900, training accuracy 0.98
step 2000, training accuracy 0.92
step 2100, training accuracy 0.98
step 2200, training accuracy 0.98
step 2300, training accuracy 0.94
step 2400, training accuracy 0.98
step 2500, training accuracy 0.98
step 2600, training accuracy 0.98
step 2700, training accuracy 1
step 2800, training accuracy 0.98
step 2900, training accuracy 0.96
step 3000, training accuracy 0.98
step 3100, training accuracy 0.96
step 3200, training accuracy 1
step 3300, training accuracy 0.98
step 3400, training accuracy 0.98
step 3500, training accuracy 0.98
step 3600, training accuracy 1
step 3700, training accuracy 0.98
step 3800, training accuracy 1
step 3900, training accuracy 0.96
step 4000, training accuracy 0.98
step 4100, training accuracy 0.98
step 4200, training accuracy 0.98
step 4300, training accuracy 0.98
step 4400, training accuracy 1
step 4500, training accuracy 0.98
step 4600, training accuracy 1
step 4700, training accuracy 1
step 4800, training accuracy 1
step 4900, training accuracy 1
step 5000, training accuracy 0.98
step 5100, training accuracy 0.96
step 5200, training accuracy 1
step 5300, training accuracy 1
step 5400, training accuracy 1
step 5500, training accuracy 0.96
step 5600, training accuracy 1
step 5700, training accuracy 1
step 5800, training accuracy 1
step 5900, training accuracy 1
step 6000, training accuracy 1
step 6100, training accuracy 1
step 6200, training accuracy 0.96
step 6300, training accuracy 1
step 6400, training accuracy 0.98
step 6500, training accuracy 1
step 6600, training accuracy 1
step 6700, training accuracy 0.98
step 6800, training accuracy 1
step 6900, training accuracy 0.98
step 7000, training accuracy 0.98
step 7100, training accuracy 1
step 7200, training accuracy 1
step 7300, training accuracy 0.98
step 7400, training accuracy 1
step 7500, training accuracy 1
step 7600, training accuracy 0.98
step 7700, training accuracy 0.98
step 7800, training accuracy 0.98
step 7900, training accuracy 1
step 8000, training accuracy 1
step 8100, training accuracy 1
step 8200, training accuracy 1
step 8300, training accuracy 1
step 8400, training accuracy 0.98
step 8500, training accuracy 1
step 8600, training accuracy 1
step 8700, training accuracy 1
step 8800, training accuracy 0.98
step 8900, training accuracy 1
step 9000, training accuracy 1
step 9100, training accuracy 1
step 9200, training accuracy 1
step 9300, training accuracy 0.98
step 9400, training accuracy 1
step 9500, training accuracy 0.98
step 9600, training accuracy 0.98
step 9700, training accuracy 1
step 9800, training accuracy 1
step 9900, training accuracy 0.98
step 10000, training accuracy 1
step 10100, training accuracy 0.98
step 10200, training accuracy 1
step 10300, training accuracy 0.98
step 10400, training accuracy 1
step 10500, training accuracy 1
step 10600, training accuracy 1
step 10700, training accuracy 0.98
step 10800, training accuracy 1
step 10900, training accuracy 0.98
step 11000, training accuracy 1
step 11100, training accuracy 1
step 11200, training accuracy 1
step 11300, training accuracy 1
step 11400, training accuracy 1
step 11500, training accuracy 0.96
step 11600, training accuracy 1
step 11700, training accuracy 1
step 11800, training accuracy 1
step 11900, training accuracy 0.98
step 12000, training accuracy 1
step 12100, training accuracy 1
step 12200, training accuracy 1
step 12300, training accuracy 1
step 12400, training accuracy 1
step 12500, training accuracy 1
step 12600, training accuracy 1
step 12700, training accuracy 1
step 12800, training accuracy 0.96
step 12900, training accuracy 1
step 13000, training accuracy 1
step 13100, training accuracy 1
step 13200, training accuracy 1
step 13300, training accuracy 1
step 13400, training accuracy 1
step 13500, training accuracy 1
step 13600, training accuracy 1
step 13700, training accuracy 1
step 13800, training accuracy 1
step 13900, training accuracy 1
step 14000, training accuracy 0.98
step 14100, training accuracy 0.98
step 14200, training accuracy 1
step 14300, training accuracy 1
step 14400, training accuracy 1
step 14500, training accuracy 1
step 14600, training accuracy 1
step 14700, training accuracy 1
step 14800, training accuracy 1
step 14900, training accuracy 1
step 15000, training accuracy 1
step 15100, training accuracy 1
step 15200, training accuracy 1
step 15300, training accuracy 1
step 15400, training accuracy 1
step 15500, training accuracy 1
step 15600, training accuracy 1
step 15700, training accuracy 1
step 15800, training accuracy 1
step 15900, training accuracy 1
step 16000, training accuracy 1
step 16100, training accuracy 1
step 16200, training accuracy 1
step 16300, training accuracy 1
step 16400, training accuracy 1
step 16500, training accuracy 1
step 16600, training accuracy 1
step 16700, training accuracy 1
step 16800, training accuracy 1
step 16900, training accuracy 1
step 17000, training accuracy 1
step 17100, training accuracy 1
step 17200, training accuracy 1
step 17300, training accuracy 1
step 17400, training accuracy 1
step 17500, training accuracy 1
step 17600, training accuracy 1
step 17700, training accuracy 1
step 17800, training accuracy 1
step 17900, training accuracy 1
step 18000, training accuracy 0.98
step 18100, training accuracy 1
step 18200, training accuracy 1
step 18300, training accuracy 1
step 18400, training accuracy 1
step 18500, training accuracy 1
step 18600, training accuracy 1
step 18700, training accuracy 1
step 18800, training accuracy 1
step 18900, training accuracy 1
step 19000, training accuracy 1
step 19100, training accuracy 1
step 19200, training accuracy 1
step 19300, training accuracy 1
step 19400, training accuracy 1
step 19500, training accuracy 1
step 19600, training accuracy 1
step 19700, training accuracy 1
step 19800, training accuracy 1
step 19900, training accuracy 1
test accuracy 0.9919