Tensorflow Fine-Tuning 的一些说明

一般训练好的模型可以直接拿来用，或者再原来的基础上增加新的layer。http://tflearn.org/getting_started/#fine-tuning
现在以代码说明:

cifar_10.py

# -*- coding: utf-8 -*-

""" Convolutional network applied to CIFAR-10 dataset classification task.
References:
    Learning Multiple Layers of Features from Tiny Images, A. Krizhevsky, 2009.
Links:
    [CIFAR-10 Dataset](https://www.cs.toronto.edu/~kriz/cifar.html)
"""
from __future__ import division, print_function, absolute_import

import tflearn
from tflearn.data_utils import shuffle, to_categorical
from tflearn.layers.core import input_data, dropout, fully_connected
from tflearn.layers.conv import conv_2d, max_pool_2d
from tflearn.layers.estimator import regression
from tflearn.data_preprocessing import ImagePreprocessing
from tflearn.data_augmentation import ImageAugmentation

# Data loading and preprocessing
from tflearn.datasets import cifar10
(X, Y), (X_test, Y_test) = cifar10.load_data(dirname="./cifar-10/")
X, Y = shuffle(X, Y)
Y = to_categorical(Y,10)
Y_test = to_categorical(Y_test,10)

# Real-time data preprocessing
img_prep = ImagePreprocessing()
img_prep.add_featurewise_zero_center()
img_prep.add_featurewise_stdnorm()

# Real-time data augmentation
img_aug = ImageAugmentation()
img_aug.add_random_flip_leftright()
img_aug.add_random_rotation(max_angle=25.)

# Convolutional network building
network = input_data(shape=[None, 32, 32, 3],
                     data_preprocessing=img_prep,
                     data_augmentation=img_aug)
network = conv_2d(network, 32, 3, activation='relu')
network = max_pool_2d(network, 2)
network = conv_2d(network, 64, 3, activation='relu')
network = conv_2d(network, 64, 3, activation='relu')
network = max_pool_2d(network, 2)
network = fully_connected(network, 512, activation='relu')
network = dropout(network, 0.5)
network = fully_connected(network, 10, activation='softmax')
network = regression(network, optimizer='adam',
                     loss='categorical_crossentropy',
                     learning_rate=0.001)

# Train using classifier
model = tflearn.DNN(network, tensorboard_verbose=0)
model.fit(X, Y, n_epoch=50, shuffle=True, validation_set=(X_test, Y_test),
          show_metric=True, batch_size=96, run_id='cifar10_cnn')
model.save('./cifar10_cnn')

具体的模型可以参照代码中的网址，有pdf文件可以下载。
代码中的卷积层都没有显式的定义name，TFLearn的处理是以layer命名
Conv2D,Conv2D_1,Conv2D_2,FullyConnected,FullyConnected_1,这样的。
训练过程中的输出。

Training Step: 26016  | total loss: 0.38068 | time: 227.672s
| Adam | epoch: 050 | loss: 0.38068 - acc: 0.8713 -- iter: 46752/50000
Training Step: 26017  | total loss: 0.38373 | time: 228.142s
| Adam | epoch: 050 | loss: 0.38373 - acc: 0.8696 -- iter: 46848/50000
Training Step: 26018  | total loss: 0.38030 | time: 228.612s
| Adam | epoch: 050 | loss: 0.38030 - acc: 0.8680 -- iter: 46944/50000
Training Step: 26019  | total loss: 0.38134 | time: 229.079s
| Adam | epoch: 050 | loss: 0.38134 - acc: 0.8677 -- iter: 47040/50000
Training Step: 26020  | total loss: 0.40366 | time: 229.530s
| Adam | epoch: 050 | loss: 0.40366 - acc: 0.8611 -- iter: 47136/50000
Training Step: 26021  | total loss: 0.38866 | time: 230.000s
| Adam | epoch: 050 | loss: 0.38866 - acc: 0.8656 -- iter: 47232/50000
Training Step: 26022  | total loss: 0.40954 | time: 230.477s
| Adam | epoch: 050 | loss: 0.40954 - acc: 0.8624 -- iter: 47328/50000
Training Step: 26023  | total loss: 0.40813 | time: 230.933s
| Adam | epoch: 050 | loss: 0.40813 - acc: 0.8605 -- iter: 47424/50000
Training Step: 26024  | total loss: 0.40759 | time: 231.406s
| Adam | epoch: 050 | loss: 0.40759 - acc: 0.8630 -- iter: 47520/50000
Training Step: 26025  | total loss: 0.40606 | time: 231.858s
| Adam | epoch: 050 | loss: 0.40606 - acc: 0.8621 -- iter: 47616/50000
Training Step: 26026  | total loss: 0.39454 | time: 232.324s
| Adam | epoch: 050 | loss: 0.39454 - acc: 0.8634 -- iter: 47712/50000
Training Step: 26027  | total loss: 0.39360 | time: 232.824s
| Adam | epoch: 050 | loss: 0.39360 - acc: 0.8615 -- iter: 47808/50000
Training Step: 26028  | total loss: 0.37261 | time: 233.300s
| Adam | epoch: 050 | loss: 0.37261 - acc: 0.8691 -- iter: 47904/50000
Training Step: 26029  | total loss: 0.37792 | time: 233.772s
| Adam | epoch: 050 | loss: 0.37792 - acc: 0.8665 -- iter: 48000/50000
Training Step: 26030  | total loss: 0.36900 | time: 234.250s
| Adam | epoch: 050 | loss: 0.36900 - acc: 0.8674 -- iter: 48096/50000
Training Step: 26031  | total loss: 0.35785 | time: 234.717s
| Adam | epoch: 050 | loss: 0.35785 - acc: 0.8713 -- iter: 48192/50000
Training Step: 26032  | total loss: 0.36420 | time: 235.180s
| Adam | epoch: 050 | loss: 0.36420 - acc: 0.8675 -- iter: 48288/50000
Training Step: 26033  | total loss: 0.35347 | time: 235.628s
| Adam | epoch: 050 | loss: 0.35347 - acc: 0.8693 -- iter: 48384/50000
Training Step: 26034  | total loss: 0.38350 | time: 236.098s
| Adam | epoch: 050 | loss: 0.38350 - acc: 0.8657 -- iter: 48480/50000
Training Step: 26035  | total loss: 0.37958 | time: 236.576s
| Adam | epoch: 050 | loss: 0.37958 - acc: 0.8676 -- iter: 48576/50000
Training Step: 26036  | total loss: 0.36793 | time: 237.037s
| Adam | epoch: 050 | loss: 0.36793 - acc: 0.8705 -- iter: 48672/50000
Training Step: 26037  | total loss: 0.35811 | time: 237.501s
| Adam | epoch: 050 | loss: 0.35811 - acc: 0.8740 -- iter: 48768/50000
Training Step: 26038  | total loss: 0.36710 | time: 237.976s
| Adam | epoch: 050 | loss: 0.36710 - acc: 0.8700 -- iter: 48864/50000
Training Step: 26039  | total loss: 0.36583 | time: 238.435s
| Adam | epoch: 050 | loss: 0.36583 - acc: 0.8715 -- iter: 48960/50000
Training Step: 26040  | total loss: 0.37000 | time: 238.890s
| Adam | epoch: 050 | loss: 0.37000 - acc: 0.8708 -- iter: 49056/50000
Training Step: 26041  | total loss: 0.36465 | time: 239.348s
| Adam | epoch: 050 | loss: 0.36465 - acc: 0.8723 -- iter: 49152/50000
Training Step: 26042  | total loss: 0.35985 | time: 239.821s
| Adam | epoch: 050 | loss: 0.35985 - acc: 0.8767 -- iter: 49248/50000
Training Step: 26043  | total loss: 0.36621 | time: 240.288s
| Adam | epoch: 050 | loss: 0.36621 - acc: 0.8776 -- iter: 49344/50000
Training Step: 26044  | total loss: 0.37555 | time: 240.752s
| Adam | epoch: 050 | loss: 0.37555 - acc: 0.8742 -- iter: 49440/50000
Training Step: 26045  | total loss: 0.37014 | time: 241.216s
| Adam | epoch: 050 | loss: 0.37014 - acc: 0.8753 -- iter: 49536/50000
Training Step: 26046  | total loss: 0.37784 | time: 241.675s
| Adam | epoch: 050 | loss: 0.37784 - acc: 0.8743 -- iter: 49632/50000
Training Step: 26047  | total loss: 0.37061 | time: 242.141s
| Adam | epoch: 050 | loss: 0.37061 - acc: 0.8785 -- iter: 49728/50000
Training Step: 26048  | total loss: 0.37322 | time: 242.633s
| Adam | epoch: 050 | loss: 0.37322 - acc: 0.8792 -- iter: 49824/50000
Training Step: 26049  | total loss: 0.37036 | time: 243.098s
| Adam | epoch: 050 | loss: 0.37036 - acc: 0.8798 -- iter: 49920/50000
Training Step: 26050  | total loss: 0.37210 | time: 260.038s
| Adam | epoch: 050 | loss: 0.37210 - acc: 0.8825 | val_loss: 0.67619 - val_acc: 0.8143 -- iter: 50000/50000

现在FineTuning，下面是fineTuning代码

#encoding:utf-8
#@Time : 2017/12/8 15:46
#@Author : JackNiu
import tflearn
from tflearn.layers.core import input_data, dropout, fully_connected
from tflearn.layers.conv import conv_2d, max_pool_2d
from tflearn.layers.estimator import regression
from tflearn.data_preprocessing import ImagePreprocessing
from tflearn.data_augmentation import ImageAugmentation
from tflearn.data_utils import shuffle, to_categorical

# Data loading and preprocessing
from tflearn.datasets import cifar10
(X, Y), (X_test, Y_test) = cifar10.load_data(dirname="./cifar-10/")
X, Y = shuffle(X, Y)
Y = to_categorical(Y,10)
Y_test = to_categorical(Y_test,10)

# Real-time data preprocessing
img_prep = ImagePreprocessing()
img_prep.add_featurewise_zero_center()
img_prep.add_featurewise_stdnorm()

# Real-time data augmentation
img_aug = ImageAugmentation()
img_aug.add_random_flip_leftright()
img_aug.add_random_rotation(max_angle=25.)



# Redefinition of convnet_cifar10 network
network = input_data(shape=[None, 32, 32, 3])
network = conv_2d(network, 32, 3, activation='relu')
network = max_pool_2d(network, 2)
network = dropout(network, 0.75)
network = conv_2d(network, 64, 3, activation='relu')
network = conv_2d(network, 64, 3, activation='relu')
network = max_pool_2d(network, 2)
network = dropout(network, 0.5)


network = fully_connected(network, 512, activation='relu')
network = dropout(network, 0.5)
# Finetuning Softmax layer (Setting restore=False to not restore its weights)
# 最后一层全连接层
softmax = fully_connected(network, 10, activation='softmax', restore=False)
regression = regression(softmax, optimizer='adam',
                        loss='categorical_crossentropy',
                        learning_rate=0.001)

model = tflearn.DNN(regression, checkpoint_path='./tmp/fine/model_finetuning',
                    max_checkpoints=3, tensorboard_verbose=0)
# Load pre-existing model, restoring all weights, except softmax layer ones
# 下载之前存在的模型，恢复所有的权重，除了最后一层的权重

model.load('cifar10_cnn')
# 下载已经训练好的模型，那么这里的之前定义的权重是怎么一一对应到的？ 是根据名字

# 获取模型的第一层conv 的w和b
print(tflearn.variables.get_all_trainable_variable())
conv1= tflearn.variables.get_layer_variables_by_name("Conv2D")
print("Dense1 layer weights:")
print(conv1)
print(model.get_weights(conv1[0])) # dense1.W

# Start finetuning
model.fit(X, Y, n_epoch=10, validation_set=0.1, shuffle=True,
          show_metric=True, batch_size=64, snapshot_step=200,
          snapshot_epoch=False, run_id='model_finetuning')

model.save('model_finetuning.tflearn')

会加载cifai10_cnn 中的权重以b，但不会加载最后一个全连接层的权重。

[<tf.Variable 'Conv2D/W:0' shape=(3, 3, 3, 32) dtype=float32_ref>, <tf.Variable 'Conv2D/b:0' shape=(32,) dtype=float32_ref>, <tf.Variable 'Conv2D_1/W:0' shape=(3, 3, 32, 64) dtype=float32_ref>, <tf.Variable 'Conv2D_1/b:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'Conv2D_2/W:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'Conv2D_2/b:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'FullyConnected/W:0' shape=(4096, 512) dtype=float32_ref>, <tf.Variable 'FullyConnected/b:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'FullyConnected_1/W:0' shape=(512, 10) dtype=float32_ref>, <tf.Variable 'FullyConnected_1/b:0' shape=(10,) dtype=float32_ref>]

Dense1 layer weights:
[<tf.Variable 'Conv2D/W:0' shape=(3, 3, 3, 32) dtype=float32_ref>, <tf.Variable 'Conv2D/b:0' shape=(32,) dtype=float32_ref>]

[[[[  2.27525160e-01  -8.02579746e-02  -1.49709478e-01   1.95076019e-01
     -2.86777616e-01  -1.84236169e-01   8.42581168e-02  -3.03720593e-01
      2.49673024e-01  -3.37609112e-01   2.24997491e-01   9.04519483e-02
     -3.34136724e-01  -2.22282767e-01   4.01710495e-02   1.44447014e-01
     -2.03410551e-01  -5.28300963e-02   8.31742585e-02   3.66900163e-03
      2.77024060e-01   7.51862824e-02   2.45897129e-01  -2.23820955e-01
      1.32494599e-01  -2.48694703e-01   2.52264559e-01   8.77561271e-02
     -2.69655794e-01   7.40043744e-02   3.05063482e-02  -8.41584057e-02]...

在格式上要和已经训练的模型进行对应， 也不能在新的模型中显式定义name，要不然找不到同一个variable

具体代码参考: http://tflearn.org/getting_started/#fine-tuning
cifar10_cnn 模型文件: http://download.csdn.net/download/u010651137/10153505
可以直接下载。