六步法实现LeNet(1998)、AlexNet(2012)、VGGNet(2014)、InceptionNet(2014)、ResNet(2015)
LeNet由Yann LeCun于1998年提出，卷积神经网络开篇之作
在统计卷积神经网络层数时，一般只统计卷积计算层和全连接计算层，其余操作可以认为是卷积计算层的附属。LeNet卷积神经网络共有5层，包括2层卷积层加上3层全连接层
import tensorflow as tf
import os
import numpy as np
from matplotlib import pyplot as plt
from tensorflow.keras.layers import Conv2D, BatchNormalization, Activation, MaxPool2D, Dropout, Flatten, Dense
from tensorflow.keras import Model
np.set_printoptions(threshold=np.inf)
cifar10 = tf.keras.datasets.cifar10
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
class LeNet5(Model):
def __init__(self):
super(LeNet5, self).__init__()
self.c1 = Conv2D(filters=6, kernel_size=(5, 5),
activation='sigmoid')
self.p1 = MaxPool2D(pool_size=(2, 2), strides=2)
self.c2 = Conv2D(filters=16, kernel_size=(5, 5),
activation='sigmoid')
self.p2 = MaxPool2D(pool_size=(2, 2), strides=2)
self.flatten = Flatten()
self.f1 = Dense(120, activation='sigmoid')
self.f2 = Dense(84, activation='sigmoid')
self.f3 = Dense(10, activation='softmax')
def call(self, x):
x = self.c1(x)
x = self.p1(x)
x = self.c2(x)
x = self.p2(x)
x = self.flatten(x)
x = self.f1(x)
x = self.f2(x)
y = self.f3(x)
return y
model = LeNet5()
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['sparse_categorical_accuracy'])
checkpoint_save_path = "./checkpoint/LeNet5.ckpt"
if os.path.exists(checkpoint_save_path + '.index'):
print('-------------load the model-----------------')
model.load_weights(checkpoint_save_path)
cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_save_path,
save_weights_only=True,
save_best_only=True)
history = model.fit(x_train, y_train, batch_size=32, epochs=5, validation_data=(x_test, y_test), validation_freq=1,
callbacks=[cp_callback])
model.summary()
# print(model.trainable_variables)
file = open('./weights.txt', 'w')
for v in model.trainable_variables:
file.write(str(v.name) + '\n')
file.write(str(v.shape) + '\n')
file.write(str(v.numpy()) + '\n')
file.close()
############################################### show ###############################################
# 显示训练集和验证集的acc和loss曲线
acc = history.history['sparse_categorical_accuracy']
val_acc = history.history['val_sparse_categorical_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
plt.subplot(1, 2, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.title('Training and Validation Accuracy')
plt.legend()
plt.subplot(1, 2, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.title('Training and Validation Loss')
plt.legend()
plt.show()
Epoch 1/5
1563/1563 [==============================] - 12s 8ms/step - loss: 2.0554 - sparse_categorical_accuracy: 0.2283 - val_loss: 1.8637 - val_sparse_categorical_accuracy: 0.3102
Epoch 2/5
1563/1563 [==============================] - 12s 8ms/step - loss: 1.7815 - sparse_categorical_accuracy: 0.3454 - val_loss: 1.6746 - val_sparse_categorical_accuracy: 0.3800
Epoch 3/5
1563/1563 [==============================] - 12s 7ms/step - loss: 1.6476 - sparse_categorical_accuracy: 0.3935 - val_loss: 1.5669 - val_sparse_categorical_accuracy: 0.4154
Epoch 4/5
1563/1563 [==============================] - 12s 8ms/step - loss: 1.5415 - sparse_categorical_accuracy: 0.4346 - val_loss: 1.5874 - val_sparse_categorical_accuracy: 0.4130
Epoch 5/5
1563/1563 [==============================] - 7s 4ms/step - loss: 1.4767 - sparse_categorical_accuracy: 0.4622 - val_loss: 1.4263 - val_sparse_categorical_accuracy: 0.4744
Model: "le_net5"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) multiple 456
_________________________________________________________________
max_pooling2d (MaxPooling2D) multiple 0
_________________________________________________________________
conv2d_1 (Conv2D) multiple 2416
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 multiple 0
_________________________________________________________________
flatten (Flatten) multiple 0
_________________________________________________________________
dense (Dense) multiple 48120
_________________________________________________________________
dense_1 (Dense) multiple 10164
_________________________________________________________________
dense_2 (Dense) multiple 850
=================================================================
Total params: 62,006
Trainable params: 62,006
Non-trainable params: 0
_________________________________________________________________
output_1_1.png
AlexNet网络诞生于2012年，当年ImageNet竞赛的冠军，Top5错误率为16.4%
使用Relu激活函数，使用Dropout缓解过拟合。共有8层，5层卷积层加3层全连接层
import tensorflow as tf
import os
import numpy as np
from matplotlib import pyplot as plt
from tensorflow.keras.layers import Conv2D, BatchNormalization, Activation, MaxPool2D, Dropout, Flatten, Dense
from tensorflow.keras import Model
np.set_printoptions(threshold=np.inf)
cifar10 = tf.keras.datasets.cifar10
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
class AlexNet8(Model):
def __init__(self):
super(AlexNet8, self).__init__()
self.c1 = Conv2D(filters=96, kernel_size=(3, 3))
self.b1 = BatchNormalization()
self.a1 = Activation('relu')
self.p1 = MaxPool2D(pool_size=(3, 3), strides=2)
self.c2 = Conv2D(filters=256, kernel_size=(3, 3))
self.b2 = BatchNormalization()
self.a2 = Activation('relu')
self.p2 = MaxPool2D(pool_size=(3, 3), strides=2)
self.c3 = Conv2D(filters=384, kernel_size=(3, 3), padding='same',
activation='relu')
self.c4 = Conv2D(filters=384, kernel_size=(3, 3), padding='same',
activation='relu')
self.c5 = Conv2D(filters=256, kernel_size=(3, 3), padding='same',
activation='relu')
self.p3 = MaxPool2D(pool_size=(3, 3), strides=2)
self.flatten = Flatten()
self.f1 = Dense(2048, activation='relu')
self.d1 = Dropout(0.5)
self.f2 = Dense(2048, activation='relu')
self.d2 = Dropout(0.5)
self.f3 = Dense(10, activation='softmax')
def call(self, x):
x = self.c1(x)
x = self.b1(x)
x = self.a1(x)
x = self.p1(x)
x = self.c2(x)
x = self.b2(x)
x = self.a2(x)
x = self.p2(x)
x = self.c3(x)
x = self.c4(x)
x = self.c5(x)
x = self.p3(x)
x = self.flatten(x)
x = self.f1(x)
x = self.d1(x)
x = self.f2(x)
x = self.d2(x)
y = self.f3(x)
return y
model = AlexNet8()
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['sparse_categorical_accuracy'])
checkpoint_save_path = "./checkpoint/AlexNet8.ckpt"
if os.path.exists(checkpoint_save_path + '.index'):
print('-------------load the model-----------------')
model.load_weights(checkpoint_save_path)
cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_save_path,
save_weights_only=True,
save_best_only=True)
history = model.fit(x_train, y_train, batch_size=32, epochs=5, validation_data=(x_test, y_test), validation_freq=1,
callbacks=[cp_callback])
model.summary()
# print(model.trainable_variables)
file = open('./weights.txt', 'w')
for v in model.trainable_variables:
file.write(str(v.name) + '\n')
file.write(str(v.shape) + '\n')
file.write(str(v.numpy()) + '\n')
file.close()
############################################### show ###############################################
# 显示训练集和验证集的acc和loss曲线
acc = history.history['sparse_categorical_accuracy']
val_acc = history.history['val_sparse_categorical_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
plt.subplot(1, 2, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.title('Training and Validation Accuracy')
plt.legend()
plt.subplot(1, 2, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.title('Training and Validation Loss')
plt.legend()
plt.show()
Epoch 1/5
1563/1563 [==============================] - 88s 57ms/step - loss: 1.6349 - sparse_categorical_accuracy: 0.3951 - val_loss: 1.5745 - val_sparse_categorical_accuracy: 0.4296
Epoch 2/5
1563/1563 [==============================] - 89s 57ms/step - loss: 1.2938 - sparse_categorical_accuracy: 0.5426 - val_loss: 1.6495 - val_sparse_categorical_accuracy: 0.4515
Epoch 3/5
1563/1563 [==============================] - 90s 57ms/step - loss: 1.1614 - sparse_categorical_accuracy: 0.5939 - val_loss: 1.3964 - val_sparse_categorical_accuracy: 0.5416
Epoch 4/5
1563/1563 [==============================] - 90s 57ms/step - loss: 1.0716 - sparse_categorical_accuracy: 0.6314 - val_loss: 1.1833 - val_sparse_categorical_accuracy: 0.5871
Epoch 5/5
1563/1563 [==============================] - 89s 57ms/step - loss: 1.0096 - sparse_categorical_accuracy: 0.6524 - val_loss: 1.0562 - val_sparse_categorical_accuracy: 0.6396
Model: "alex_net8"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_2 (Conv2D) multiple 2688
_________________________________________________________________
batch_normalization (BatchNo multiple 384
_________________________________________________________________
activation (Activation) multiple 0
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 multiple 0
_________________________________________________________________
conv2d_3 (Conv2D) multiple 221440
_________________________________________________________________
batch_normalization_1 (Batch multiple 1024
_________________________________________________________________
activation_1 (Activation) multiple 0
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 multiple 0
_________________________________________________________________
conv2d_4 (Conv2D) multiple 885120
_________________________________________________________________
conv2d_5 (Conv2D) multiple 1327488
_________________________________________________________________
conv2d_6 (Conv2D) multiple 884992
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 multiple 0
_________________________________________________________________
flatten_1 (Flatten) multiple 0
_________________________________________________________________
dense_3 (Dense) multiple 2099200
_________________________________________________________________
dropout (Dropout) multiple 0
_________________________________________________________________
dense_4 (Dense) multiple 4196352
_________________________________________________________________
dropout_1 (Dropout) multiple 0
_________________________________________________________________
dense_5 (Dense) multiple 20490
=================================================================
Total params: 9,639,178
Trainable params: 9,638,474
Non-trainable params: 704
_________________________________________________________________
output_3_1.png
VGGNet网络诞生于2014年，当年ImageNet竞赛的冠军，Top5错误率减小到7.3%
VGGNet使用小尺寸卷积核，在减少参数的同时，提高了识别准确率，VGG网络规整，非常适合硬件加速。共计16层结构，包括13层卷积操作和3层全连接操作。通过增加卷积核的个数，增加特征图的深度，保持了信息的承载能力
import tensorflow as tf
import os
import numpy as np
from matplotlib import pyplot as plt
from tensorflow.keras.layers import Conv2D, BatchNormalization, Activation, MaxPool2D, Dropout, Flatten, Dense
from tensorflow.keras import Model
np.set_printoptions(threshold=np.inf)
cifar10 = tf.keras.datasets.cifar10
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
class VGG16(Model):
def __init__(self):
super(VGG16, self).__init__()
self.c1 = Conv2D(filters=64, kernel_size=(3, 3), padding='same') # 卷积层1
self.b1 = BatchNormalization() # BN层1
self.a1 = Activation('relu') # 激活层1
self.c2 = Conv2D(filters=64, kernel_size=(3, 3), padding='same', )
self.b2 = BatchNormalization() # BN层1
self.a2 = Activation('relu') # 激活层1
self.p1 = MaxPool2D(pool_size=(2, 2), strides=2, padding='same')
self.d1 = Dropout(0.2) # dropout层
self.c3 = Conv2D(filters=128, kernel_size=(3, 3), padding='same')
self.b3 = BatchNormalization() # BN层1
self.a3 = Activation('relu') # 激活层1
self.c4 = Conv2D(filters=128, kernel_size=(3, 3), padding='same')
self.b4 = BatchNormalization() # BN层1
self.a4 = Activation('relu') # 激活层1
self.p2 = MaxPool2D(pool_size=(2, 2), strides=2, padding='same')
self.d2 = Dropout(0.2) # dropout层
self.c5 = Conv2D(filters=256, kernel_size=(3, 3), padding='same')
self.b5 = BatchNormalization() # BN层1
self.a5 = Activation('relu') # 激活层1
self.c6 = Conv2D(filters=256, kernel_size=(3, 3), padding='same')
self.b6 = BatchNormalization() # BN层1
self.a6 = Activation('relu') # 激活层1
self.c7 = Conv2D(filters=256, kernel_size=(3, 3), padding='same')
self.b7 = BatchNormalization()
self.a7 = Activation('relu')
self.p3 = MaxPool2D(pool_size=(2, 2), strides=2, padding='same')
self.d3 = Dropout(0.2)
self.c8 = Conv2D(filters=512, kernel_size=(3, 3), padding='same')
self.b8 = BatchNormalization() # BN层1
self.a8 = Activation('relu') # 激活层1
self.c9 = Conv2D(filters=512, kernel_size=(3, 3), padding='same')
self.b9 = BatchNormalization() # BN层1
self.a9 = Activation('relu') # 激活层1
self.c10 = Conv2D(filters=512, kernel_size=(3, 3), padding='same')
self.b10 = BatchNormalization()
self.a10 = Activation('relu')
self.p4 = MaxPool2D(pool_size=(2, 2), strides=2, padding='same')
self.d4 = Dropout(0.2)
self.c11 = Conv2D(filters=512, kernel_size=(3, 3), padding='same')
self.b11 = BatchNormalization() # BN层1
self.a11 = Activation('relu') # 激活层1
self.c12 = Conv2D(filters=512, kernel_size=(3, 3), padding='same')
self.b12 = BatchNormalization() # BN层1
self.a12 = Activation('relu') # 激活层1
self.c13 = Conv2D(filters=512, kernel_size=(3, 3), padding='same')
self.b13 = BatchNormalization()
self.a13 = Activation('relu')
self.p5 = MaxPool2D(pool_size=(2, 2), strides=2, padding='same')
self.d5 = Dropout(0.2)
self.flatten = Flatten()
self.f1 = Dense(512, activation='relu')
self.d6 = Dropout(0.2)
self.f2 = Dense(512, activation='relu')
self.d7 = Dropout(0.2)
self.f3 = Dense(10, activation='softmax')
def call(self, x):
x = self.c1(x)
x = self.b1(x)
x = self.a1(x)
x = self.c2(x)
x = self.b2(x)
x = self.a2(x)
x = self.p1(x)
x = self.d1(x)
x = self.c3(x)
x = self.b3(x)
x = self.a3(x)
x = self.c4(x)
x = self.b4(x)
x = self.a4(x)
x = self.p2(x)
x = self.d2(x)
x = self.c5(x)
x = self.b5(x)
x = self.a5(x)
x = self.c6(x)
x = self.b6(x)
x = self.a6(x)
x = self.c7(x)
x = self.b7(x)
x = self.a7(x)
x = self.p3(x)
x = self.d3(x)
x = self.c8(x)
x = self.b8(x)
x = self.a8(x)
x = self.c9(x)
x = self.b9(x)
x = self.a9(x)
x = self.c10(x)
x = self.b10(x)
x = self.a10(x)
x = self.p4(x)
x = self.d4(x)
x = self.c11(x)
x = self.b11(x)
x = self.a11(x)
x = self.c12(x)
x = self.b12(x)
x = self.a12(x)
x = self.c13(x)
x = self.b13(x)
x = self.a13(x)
x = self.p5(x)
x = self.d5(x)
x = self.flatten(x)
x = self.f1(x)
x = self.d6(x)
x = self.f2(x)
x = self.d7(x)
y = self.f3(x)
return y
model = VGG16()
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['sparse_categorical_accuracy'])
checkpoint_save_path = "./checkpoint/VGG16.ckpt"
if os.path.exists(checkpoint_save_path + '.index'):
print('-------------load the model-----------------')
model.load_weights(checkpoint_save_path)
cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_save_path,
save_weights_only=True,
save_best_only=True)
history = model.fit(x_train, y_train, batch_size=32, epochs=5, validation_data=(x_test, y_test), validation_freq=1,
callbacks=[cp_callback])
model.summary()
# print(model.trainable_variables)
file = open('./weights.txt', 'w')
for v in model.trainable_variables:
file.write(str(v.name) + '\n')
file.write(str(v.shape) + '\n')
file.write(str(v.numpy()) + '\n')
file.close()
############################################### show ###############################################
# 显示训练集和验证集的acc和loss曲线
acc = history.history['sparse_categorical_accuracy']
val_acc = history.history['val_sparse_categorical_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
plt.subplot(1, 2, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.title('Training and Validation Accuracy')
plt.legend()
plt.subplot(1, 2, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.title('Training and Validation Loss')
plt.legend()
plt.show()
Epoch 1/5
1563/1563 [==============================] - 243s 156ms/step - loss: 1.8934 - sparse_categorical_accuracy: 0.2371 - val_loss: 1.8404 - val_sparse_categorical_accuracy: 0.3229
Epoch 2/5
1563/1563 [==============================] - 242s 155ms/step - loss: 1.4783 - sparse_categorical_accuracy: 0.4276 - val_loss: 1.4761 - val_sparse_categorical_accuracy: 0.4665
Epoch 3/5
1563/1563 [==============================] - 242s 155ms/step - loss: 1.1642 - sparse_categorical_accuracy: 0.5907 - val_loss: 1.1908 - val_sparse_categorical_accuracy: 0.5724
Epoch 4/5
1563/1563 [==============================] - 242s 155ms/step - loss: 0.9775 - sparse_categorical_accuracy: 0.6679 - val_loss: 1.0298 - val_sparse_categorical_accuracy: 0.6609
Epoch 5/5
1563/1563 [==============================] - 243s 155ms/step - loss: 0.8571 - sparse_categorical_accuracy: 0.7136 - val_loss: 0.8510 - val_sparse_categorical_accuracy: 0.7127
Model: "vg_g16"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_7 (Conv2D) multiple 1792
_________________________________________________________________
batch_normalization_2 (Batch multiple 256
_________________________________________________________________
activation_2 (Activation) multiple 0
_________________________________________________________________
conv2d_8 (Conv2D) multiple 36928
_________________________________________________________________
batch_normalization_3 (Batch multiple 256
_________________________________________________________________
activation_3 (Activation) multiple 0
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 multiple 0
_________________________________________________________________
dropout_2 (Dropout) multiple 0
_________________________________________________________________
conv2d_9 (Conv2D) multiple 73856
_________________________________________________________________
batch_normalization_4 (Batch multiple 512
_________________________________________________________________
activation_4 (Activation) multiple 0
_________________________________________________________________
conv2d_10 (Conv2D) multiple 147584
_________________________________________________________________
batch_normalization_5 (Batch multiple 512
_________________________________________________________________
activation_5 (Activation) multiple 0
_________________________________________________________________
max_pooling2d_6 (MaxPooling2 multiple 0
_________________________________________________________________
dropout_3 (Dropout) multiple 0
_________________________________________________________________
conv2d_11 (Conv2D) multiple 295168
_________________________________________________________________
batch_normalization_6 (Batch multiple 1024
_________________________________________________________________
activation_6 (Activation) multiple 0
_________________________________________________________________
conv2d_12 (Conv2D) multiple 590080
_________________________________________________________________
batch_normalization_7 (Batch multiple 1024
_________________________________________________________________
activation_7 (Activation) multiple 0
_________________________________________________________________
conv2d_13 (Conv2D) multiple 590080
_________________________________________________________________
batch_normalization_8 (Batch multiple 1024
_________________________________________________________________
activation_8 (Activation) multiple 0
_________________________________________________________________
max_pooling2d_7 (MaxPooling2 multiple 0
_________________________________________________________________
dropout_4 (Dropout) multiple 0
_________________________________________________________________
conv2d_14 (Conv2D) multiple 1180160
_________________________________________________________________
batch_normalization_9 (Batch multiple 2048
_________________________________________________________________
activation_9 (Activation) multiple 0
_________________________________________________________________
conv2d_15 (Conv2D) multiple 2359808
_________________________________________________________________
batch_normalization_10 (Batc multiple 2048
_________________________________________________________________
activation_10 (Activation) multiple 0
_________________________________________________________________
conv2d_16 (Conv2D) multiple 2359808
_________________________________________________________________
batch_normalization_11 (Batc multiple 2048
_________________________________________________________________
activation_11 (Activation) multiple 0
_________________________________________________________________
max_pooling2d_8 (MaxPooling2 multiple 0
_________________________________________________________________
dropout_5 (Dropout) multiple 0
_________________________________________________________________
conv2d_17 (Conv2D) multiple 2359808
_________________________________________________________________
batch_normalization_12 (Batc multiple 2048
_________________________________________________________________
activation_12 (Activation) multiple 0
_________________________________________________________________
conv2d_18 (Conv2D) multiple 2359808
_________________________________________________________________
batch_normalization_13 (Batc multiple 2048
_________________________________________________________________
activation_13 (Activation) multiple 0
_________________________________________________________________
conv2d_19 (Conv2D) multiple 2359808
_________________________________________________________________
batch_normalization_14 (Batc multiple 2048
_________________________________________________________________
activation_14 (Activation) multiple 0
_________________________________________________________________
max_pooling2d_9 (MaxPooling2 multiple 0
_________________________________________________________________
dropout_6 (Dropout) multiple 0
_________________________________________________________________
flatten_2 (Flatten) multiple 0
_________________________________________________________________
dense_6 (Dense) multiple 262656
_________________________________________________________________
dropout_7 (Dropout) multiple 0
_________________________________________________________________
dense_7 (Dense) multiple 262656
_________________________________________________________________
dropout_8 (Dropout) multiple 0
_________________________________________________________________
dense_8 (Dense) multiple 5130
=================================================================
Total params: 15,262,026
Trainable params: 15,253,578
Non-trainable params: 8,448
_________________________________________________________________
output_5_1.png
InceptionNet网络诞生于2014年，当年ImageNet竞赛的冠军，Top5错误率为6.67%
InceptionNet引入了Inception结构块，在同一层网络内使用了不同尺寸的卷积核，提高了模型感知力，提取不同尺寸的特征。使用了批标准化，缓解了梯度消失。InceptionNet的核心是它的基本单元Inception结构块
Inception结构块包含四个分支，分别经过
InceptionNet共有十层网络结构
import tensorflow as tf
import os
import numpy as np
from matplotlib import pyplot as plt
from tensorflow.keras.layers import Conv2D, BatchNormalization, Activation, MaxPool2D, Dropout, Flatten, Dense, \
GlobalAveragePooling2D
from tensorflow.keras import Model
np.set_printoptions(threshold=np.inf)
cifar10 = tf.keras.datasets.cifar10
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
class ConvBNRelu(Model):
def __init__(self, ch, kernelsz=3, strides=1, padding='same'):
super(ConvBNRelu, self).__init__()
self.model = tf.keras.models.Sequential([
Conv2D(ch, kernelsz, strides=strides, padding=padding),
BatchNormalization(),
Activation('relu')
])
def call(self, x):
x = self.model(x, training=False) #在training=False时，BN通过整个训练集计算均值、方差去做批归一化，training=True时，通过当前batch的均值、方差去做批归一化。推理时 training=False效果好
return x
class InceptionBlk(Model):
def __init__(self, ch, strides=1):
super(InceptionBlk, self).__init__()
self.ch = ch
self.strides = strides
self.c1 = ConvBNRelu(ch, kernelsz=1, strides=strides)
self.c2_1 = ConvBNRelu(ch, kernelsz=1, strides=strides)
self.c2_2 = ConvBNRelu(ch, kernelsz=3, strides=1)
self.c3_1 = ConvBNRelu(ch, kernelsz=1, strides=strides)
self.c3_2 = ConvBNRelu(ch, kernelsz=5, strides=1)
self.p4_1 = MaxPool2D(3, strides=1, padding='same')
self.c4_2 = ConvBNRelu(ch, kernelsz=1, strides=strides)
def call(self, x):
x1 = self.c1(x)
x2_1 = self.c2_1(x)
x2_2 = self.c2_2(x2_1)
x3_1 = self.c3_1(x)
x3_2 = self.c3_2(x3_1)
x4_1 = self.p4_1(x)
x4_2 = self.c4_2(x4_1)
# concat along axis=channel
x = tf.concat([x1, x2_2, x3_2, x4_2], axis=3)
return x
class Inception10(Model):
def __init__(self, num_blocks, num_classes, init_ch=16, **kwargs):
super(Inception10, self).__init__(**kwargs)
self.in_channels = init_ch
self.out_channels = init_ch
self.num_blocks = num_blocks
self.init_ch = init_ch
self.c1 = ConvBNRelu(init_ch)
self.blocks = tf.keras.models.Sequential()
for block_id in range(num_blocks):
for layer_id in range(2):
if layer_id == 0:
block = InceptionBlk(self.out_channels, strides=2)
else:
block = InceptionBlk(self.out_channels, strides=1)
self.blocks.add(block)
# enlarger out_channels per block
self.out_channels *= 2
self.p1 = GlobalAveragePooling2D()
self.f1 = Dense(num_classes, activation='softmax')
def call(self, x):
x = self.c1(x)
x = self.blocks(x)
x = self.p1(x)
y = self.f1(x)
return y
model = Inception10(num_blocks=2, num_classes=10)
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['sparse_categorical_accuracy'])
checkpoint_save_path = "./checkpoint/Inception10.ckpt"
if os.path.exists(checkpoint_save_path + '.index'):
print('-------------load the model-----------------')
model.load_weights(checkpoint_save_path)
cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_save_path,
save_weights_only=True,
save_best_only=True)
history = model.fit(x_train, y_train, batch_size=32, epochs=5, validation_data=(x_test, y_test), validation_freq=1,
callbacks=[cp_callback])
model.summary()
# print(model.trainable_variables)
file = open('./weights.txt', 'w')
for v in model.trainable_variables:
file.write(str(v.name) + '\n')
file.write(str(v.shape) + '\n')
file.write(str(v.numpy()) + '\n')
file.close()
############################################### show ###############################################
# 显示训练集和验证集的acc和loss曲线
acc = history.history['sparse_categorical_accuracy']
val_acc = history.history['val_sparse_categorical_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
plt.subplot(1, 2, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.title('Training and Validation Accuracy')
plt.legend()
plt.subplot(1, 2, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.title('Training and Validation Loss')
plt.legend()
plt.show()
Epoch 1/5
1563/1563 [==============================] - 53s 34ms/step - loss: 1.6564 - sparse_categorical_accuracy: 0.3820 - val_loss: 1.3811 - val_sparse_categorical_accuracy: 0.4933
Epoch 2/5
1563/1563 [==============================] - 52s 34ms/step - loss: 1.2522 - sparse_categorical_accuracy: 0.5450 - val_loss: 1.1182 - val_sparse_categorical_accuracy: 0.6035
Epoch 3/5
1563/1563 [==============================] - 53s 34ms/step - loss: 1.0752 - sparse_categorical_accuracy: 0.6155 - val_loss: 1.0111 - val_sparse_categorical_accuracy: 0.6402
Epoch 4/5
1563/1563 [==============================] - 54s 34ms/step - loss: 0.9630 - sparse_categorical_accuracy: 0.6549 - val_loss: 0.8991 - val_sparse_categorical_accuracy: 0.6791
Epoch 5/5
1563/1563 [==============================] - 53s 34ms/step - loss: 0.8731 - sparse_categorical_accuracy: 0.6925 - val_loss: 0.9316 - val_sparse_categorical_accuracy: 0.6712
Model: "inception10"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv_bn_relu (ConvBNRelu) multiple 512
_________________________________________________________________
sequential_1 (Sequential) multiple 119616
_________________________________________________________________
global_average_pooling2d (Gl multiple 0
_________________________________________________________________
dense_9 (Dense) multiple 1290
=================================================================
Total params: 121,418
Trainable params: 120,234
Non-trainable params: 1,184
_________________________________________________________________
output_7_1.png
ResNet网络诞生于2015年，当年ImageNet竞赛的冠军，Top5错误率为3.57%
ResNet提出了层间残差跳连，引入了前方信息，缓解梯度消失，使神经网络层数增加成为可能，作者何凯明。何凯明发现56层卷积网络的错误率要高于20层卷积网络的错误率，他认为单纯堆叠神经网络层数会使神经网络模型退化，以至于后边的特征丢失了前边特征的原本模样。于是他用了一根跳连线，将前边的特征直接接到了后边，使这里的输出结果为
import tensorflow as tf
import os
import numpy as np
from matplotlib import pyplot as plt
from tensorflow.keras.layers import Conv2D, BatchNormalization, Activation, MaxPool2D, Dropout, Flatten, Dense
from tensorflow.keras import Model
np.set_printoptions(threshold=np.inf)
cifar10 = tf.keras.datasets.cifar10
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
class ResnetBlock(Model):
def __init__(self, filters, strides=1, residual_path=False):
super(ResnetBlock, self).__init__()
self.filters = filters
self.strides = strides
self.residual_path = residual_path
self.c1 = Conv2D(filters, (3, 3), strides=strides, padding='same', use_bias=False)
self.b1 = BatchNormalization()
self.a1 = Activation('relu')
self.c2 = Conv2D(filters, (3, 3), strides=1, padding='same', use_bias=False)
self.b2 = BatchNormalization()
# residual_path为True时，对输入进行下采样，即用1x1的卷积核做卷积操作，保证x能和F(x)维度相同，顺利相加
if residual_path:
self.down_c1 = Conv2D(filters, (1, 1), strides=strides, padding='same', use_bias=False)
self.down_b1 = BatchNormalization()
self.a2 = Activation('relu')
def call(self, inputs):
residual = inputs # residual等于输入值本身，即residual=x
# 将输入通过卷积、BN层、激活层，计算F(x)
x = self.c1(inputs)
x = self.b1(x)
x = self.a1(x)
x = self.c2(x)
y = self.b2(x)
if self.residual_path:
residual = self.down_c1(inputs)
residual = self.down_b1(residual)
out = self.a2(y + residual) # 最后输出的是两部分的和，即F(x)+x或F(x)+Wx,再过激活函数
return out
class ResNet18(Model):
def __init__(self, block_list, initial_filters=64): # block_list表示每个block有几个卷积层
super(ResNet18, self).__init__()
self.num_blocks = len(block_list) # 共有几个block
self.block_list = block_list
self.out_filters = initial_filters
self.c1 = Conv2D(self.out_filters, (3, 3), strides=1, padding='same', use_bias=False)
self.b1 = BatchNormalization()
self.a1 = Activation('relu')
self.blocks = tf.keras.models.Sequential()
# 构建ResNet网络结构
for block_id in range(len(block_list)): # 第几个resnet block
for layer_id in range(block_list[block_id]): # 第几个卷积层
if block_id != 0 and layer_id == 0: # 对除第一个block以外的每个block的输入进行下采样
block = ResnetBlock(self.out_filters, strides=2, residual_path=True)
else:
block = ResnetBlock(self.out_filters, residual_path=False)
self.blocks.add(block) # 将构建好的block加入resnet
self.out_filters *= 2 # 下一个block的卷积核数是上一个block的2倍
self.p1 = tf.keras.layers.GlobalAveragePooling2D()
self.f1 = tf.keras.layers.Dense(10, activation='softmax', kernel_regularizer=tf.keras.regularizers.l2())
def call(self, inputs):
x = self.c1(inputs)
x = self.b1(x)
x = self.a1(x)
x = self.blocks(x)
x = self.p1(x)
y = self.f1(x)
return y
model = ResNet18([2, 2, 2, 2])
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['sparse_categorical_accuracy'])
checkpoint_save_path = "./checkpoint/ResNet18.ckpt"
if os.path.exists(checkpoint_save_path + '.index'):
print('-------------load the model-----------------')
model.load_weights(checkpoint_save_path)
cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_save_path,
save_weights_only=True,
save_best_only=True)
history = model.fit(x_train, y_train, batch_size=32, epochs=5, validation_data=(x_test, y_test), validation_freq=1,
callbacks=[cp_callback])
model.summary()
# print(model.trainable_variables)
file = open('./weights.txt', 'w')
for v in model.trainable_variables:
file.write(str(v.name) + '\n')
file.write(str(v.shape) + '\n')
file.write(str(v.numpy()) + '\n')
file.close()
############################################### show ###############################################
# 显示训练集和验证集的acc和loss曲线
acc = history.history['sparse_categorical_accuracy']
val_acc = history.history['val_sparse_categorical_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
plt.subplot(1, 2, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.title('Training and Validation Accuracy')
plt.legend()
plt.subplot(1, 2, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.title('Training and Validation Loss')
plt.legend()
plt.show()
Epoch 1/5
1563/1563 [==============================] - 271s 173ms/step - loss: 1.3384 - sparse_categorical_accuracy: 0.5604 - val_loss: 3.0897 - val_sparse_categorical_accuracy: 0.3352
Epoch 2/5
1563/1563 [==============================] - 275s 176ms/step - loss: 0.8018 - sparse_categorical_accuracy: 0.7368 - val_loss: 1.0366 - val_sparse_categorical_accuracy: 0.6766
Epoch 3/5
1563/1563 [==============================] - 277s 177ms/step - loss: 0.6195 - sparse_categorical_accuracy: 0.7967 - val_loss: 0.8154 - val_sparse_categorical_accuracy: 0.7282
Epoch 4/5
1563/1563 [==============================] - 276s 177ms/step - loss: 0.4936 - sparse_categorical_accuracy: 0.8412 - val_loss: 0.8613 - val_sparse_categorical_accuracy: 0.7247
Epoch 5/5
1563/1563 [==============================] - 277s 177ms/step - loss: 0.3918 - sparse_categorical_accuracy: 0.8753 - val_loss: 0.6585 - val_sparse_categorical_accuracy: 0.7954
Model: "res_net18"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_45 (Conv2D) multiple 1728
_________________________________________________________________
batch_normalization_40 (Batc multiple 256
_________________________________________________________________
activation_40 (Activation) multiple 0
_________________________________________________________________
sequential_26 (Sequential) multiple 11176448
_________________________________________________________________
global_average_pooling2d_1 ( multiple 0
_________________________________________________________________
dense_10 (Dense) multiple 5130
=================================================================
Total params: 11,183,562
Trainable params: 11,173,962
Non-trainable params: 9,600
_________________________________________________________________
output_9_1.png
五个经典卷积神经网络总结
- LeNet：卷积网络开篇之作，共享卷积核，减少网络参数
- AlexNet：使用Relu激活函数，提升训练速度；使用Dropout，缓解过拟合
- VGGNet：小尺寸卷积核减少参数，网络结构规整，适合并行加速
- InceptionNet：一层内使用不同尺寸卷积核，提升感知力。使用批标准化，缓解梯度消失
- ResNet：层间残差跳连，引入前方信息，缓解模型退化，使神经网络层数加深成为可能