TensorFlow学习记录-- 5.用lstm对手写数字进行识别(待修改，差增加rnn以及lstm的知识)

待修改

二 Rnn_mnist

1 整体解释

整个结构为每次将batch=500，图片大小为28×28的数据,对于每张图片来说，每次一列一列的读入数据，即分成28个时间序列每次28个送入rnn网络进行训练，将最后一个时间点的输出output[-1]作为最终输出，其中隐藏层 里面有 n_hidden=256个特征数，即最后输出out的特征的数量也是这么多，一般取最后一个out作为最终输出，最后一个out[-1]是这里是256维数，然后在加一个全链接网络w，b再训练作为预测的输出。具体请看下面代码。
其中batch的大小和n_hidden的大小都可以设置，我设置batch=500，n_hidden=256最终的正确率居然可以达到0.992188

2 运行结果

Iter 5000, Minibatch Loss= 1.608084, Training Accuracy= 0.47200
Iter 10000, Minibatch Loss= 1.334855, Training Accuracy= 0.53200
Iter 15000, Minibatch Loss= 0.802162, Training Accuracy= 0.74000
Iter 20000, Minibatch Loss= 0.737002, Training Accuracy= 0.77000
Iter 25000, Minibatch Loss= 0.760078, Training Accuracy= 0.74400
Iter 30000, Minibatch Loss= 0.571749, Training Accuracy= 0.82200
Iter 35000, Minibatch Loss= 0.428172, Training Accuracy= 0.85800
Iter 40000, Minibatch Loss= 0.354639, Training Accuracy= 0.88400
Iter 45000, Minibatch Loss= 0.420507, Training Accuracy= 0.84800
Iter 50000, Minibatch Loss= 0.275887, Training Accuracy= 0.89800
Iter 55000, Minibatch Loss= 0.261437, Training Accuracy= 0.93800
Iter 60000, Minibatch Loss= 0.287925, Training Accuracy= 0.89800
Iter 65000, Minibatch Loss= 0.267201, Training Accuracy= 0.93600
Iter 70000, Minibatch Loss= 0.193925, Training Accuracy= 0.94000
Iter 75000, Minibatch Loss= 0.161271, Training Accuracy= 0.94600
Iter 80000, Minibatch Loss= 0.161922, Training Accuracy= 0.94600
Iter 85000, Minibatch Loss= 0.161221, Training Accuracy= 0.94600
Iter 90000, Minibatch Loss= 0.112243, Training Accuracy= 0.95800
Iter 95000, Minibatch Loss= 0.185963, Training Accuracy= 0.94000
Optimization Finished!
Testing Accuracy: 0.992188

3 代码

# -*- coding: utf-8 -*-
'''
A Recurrent Neural Network (LSTM) implementation example using TensorFlow library.
This example is using the MNIST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)
Long Short Term Memory paper: http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf
Author: Aymeric Damien
Project: https://github.com/aymericdamien/TensorFlow-Examples/
'''

from __future__ import print_function

import tensorflow as tf
from tensorflow.python.ops import rnn, rnn_cell
import numpy as np

# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("data", one_hot=True)

'''
To classify images using a recurrent neural network, we consider every image
row as a sequence of pixels. Because MNIST image shape is 28*28px, we will then
handle 28 sequences of 28 steps for every sample.
'''

# Parameters
learning_rate = 0.001
training_iters = 100000
batch_size = 500
display_step = 10

# Network Parameters
n_input = 28 # MNIST data input (img shape: 28*28)
n_steps = 28 # timesteps
n_hidden = 256 # hidden layer num of features
n_classes = 10 # MNIST total classes (0-9 digits)

# tf Graph input
x = tf.placeholder("float", [None, n_steps, n_input])
y = tf.placeholder("float", [None, n_classes])

# Define weights
weights = {
    'out': tf.Variable(tf.random_normal([n_hidden, n_classes]))
}
biases = {
    'out': tf.Variable(tf.random_normal([n_classes]))
}


def RNN(x, weights, biases):

    # Prepare data shape to match `rnn` function requirements
    # Current data input shape: (batch_size, n_steps, n_input)
    # Required shape: 'n_steps' tensors list of shape (batch_size, n_input)
    #最开始，x为(batch,28,28)
    # Permuting batch_size and n_steps
    x = tf.transpose(x, [1, 0, 2])
    # 现在变为 (28,batch,28)
    # Reshaping to (n_steps*batch_size, n_input)
    x = tf.reshape(x, [-1, n_input])
    # 现在变为 (28*batch,28)
    # Split to get a list of 'n_steps' tensors of shape (batch_size, n_input)
    x = tf.split(0, n_steps, x)
    # 如今再拆成28个时间序列送进lstm,如今的样子如下：
    '''
    [<tf.Tensor 'split:0' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:1' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:2' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:3' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:4' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:5' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:6' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:7' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:8' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:9' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:10' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:11' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:12' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:13' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:14' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:15' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:16' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:17' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:18' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:19' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:20' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:21' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:22' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:23' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:24' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:25' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:26' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:27' shape=(?, 28) dtype=float32>]

    '''
    # Define a lstm cell with tensorflow
    lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)

    # Get lstm cell output
    # outputs和states的shape都为(?,256)这个样子
    outputs, states = rnn.rnn(lstm_cell, x, dtype=tf.float32)
    '''outputs是这个样子的28*(?,256),256是隐藏层的个数
    [<tf.Tensor 'RNN/BasicLSTMCell/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_1/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_2/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_3/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_4/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_5/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_6/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_7/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_8/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_9/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_10/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_11/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_12/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_13/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_14/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_15/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_16/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_17/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_18/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_19/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_20/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_21/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_22/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_23/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_24/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_25/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_26/mul_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'RNN/BasicLSTMCell_27/mul_2:0' shape=(?, 256) dtype=float32>]
下面是states的样子：
LSTMStateTuple(c=<tf.Tensor 'RNN/BasicLSTMCell_27/add_2:0' shape=(?, 256) dtype=float32>, h=<tf.Tensor 'RNN/BasicLSTMCell_27/mul_2:0' shape=(?, 256) dtype=float32>)

    '''


    # Linear activation, using rnn inner loop last output
    return tf.matmul(outputs[-1], weights['out']) + biases['out']

pred = RNN(x, weights, biases)

# Define loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

# Evaluate model
correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

# Initializing the variables
init = tf.initialize_all_variables()

# Launch the graph
with tf.Session() as sess:
    sess.run(init)
    step = 1
    # Keep training until reach max iterations
    while step * batch_size < training_iters:
        batch_x, batch_y = mnist.train.next_batch(batch_size)
        # Reshape data to get 28 seq of 28 elements
        batch_x = batch_x.reshape((batch_size, n_steps, n_input))
        # Run optimization op (backprop)
        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
        if step % display_step == 0:
            # Calculate batch accuracy
            acc = sess.run(accuracy, feed_dict={x: batch_x, y: batch_y})
            # Calculate batch loss
            loss = sess.run(cost, feed_dict={x: batch_x, y: batch_y})
            print("Iter " + str(step*batch_size) + ", Minibatch Loss= " + \
                  "{:.6f}".format(loss) + ", Training Accuracy= " + \
                  "{:.5f}".format(acc))
        step += 1
    print("Optimization Finished!")

    # Calculate accuracy for 128 mnist test images
    test_len = 128
    test_data = mnist.test.images[:test_len].reshape((-1, n_steps, n_input))
    test_label = mnist.test.labels[:test_len]
    print("Testing Accuracy:", \
        sess.run(accuracy, feed_dict={x: test_data, y: test_label}))

4 pdb调试结果

> /home/huxiang/tools/tensorflow/tensorflow/examples/tutorials/mnist/rnn_mnist.py(57)RNN()
-> x = tf.transpose(x, [1, 0, 2])
(Pdb) l
 52         # Prepare data shape to match `rnn` function requirements
 53         # Current data input shape: (batch_size, n_steps, n_input)
 54         # Required shape: 'n_steps' tensors list of shape (batch_size, n_input)
 55         pdb.set_trace()
 56         # Permuting batch_size and n_steps
 57  ->     x = tf.transpose(x, [1, 0, 2])
 58         # Reshaping to (n_steps*batch_size, n_input)
 59         x = tf.reshape(x, [-1, n_input])
 60         # Split to get a list of 'n_steps' tensors of shape (batch_size, n_input)
 61         x = tf.split(0, n_steps, x)
 62     
(Pdb) p x #最开始，x是 (batch,28,28)
<tf.Tensor 'Placeholder:0' shape=(?, 28, 28) dtype=float32>
(Pdb) n
> /home/huxiang/tools/tensorflow/tensorflow/examples/tutorials/mnist/rnn_mnist.py(59)RNN()
-> x = tf.reshape(x, [-1, n_input])
(Pdb) p x
<tf.Tensor 'transpose:0' shape=(28, ?, 28) dtype=float32>
(Pdb) n
> /home/huxiang/tools/tensorflow/tensorflow/examples/tutorials/mnist/rnn_mnist.py(61)RNN()
-> x = tf.split(0, n_steps, x)
(Pdb) p x
<tf.Tensor 'Reshape:0' shape=(?, 28) dtype=float32>
(Pdb) n
> /home/huxiang/tools/tensorflow/tensorflow/examples/tutorials/mnist/rnn_mnist.py(64)RNN()
-> lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)
(Pdb) p x
[<tf.Tensor 'split:0' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:1' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:2' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:3' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:4' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:5' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:6' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:7' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:8' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:9' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:10' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:11' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:12' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:13' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:14' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:15' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:16' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:17' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:18' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:19' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:20' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:21' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:22' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:23' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:24' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:25' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:26' shape=(?, 28) dtype=float32>, <tf.Tensor 'split:27' shape=(?, 28) dtype=float32>]