【神经网络】(14) MnasNet 代码复现，网络解析，附Tensorflow完整代码
各位同学好，今天和大家分享一下如何使用 Tensorflow 复现谷歌轻量化神经网络 MnasNet
通常而言，移动端（手机）和终端（安防监控、无人驾驶）上的设备计算能力有限，无法搭载庞大的神经网络模型。我们需要减少模型参数量、减小计算量、更少的内存访问量、更少的能耗。MobileNet、ShuffleNet 等轻量化网络就非常适合于一些计算能力受限的设备，通过牺牲少量准确率来换取更快的运算速度。
在之前的章节中，我介绍了一些经典的轻量化神经网络模型。本篇需要用到MobileNetV1、V2的相关知识，建议大家先学习一下这两个网络。
MobileNetV1：https://blog.csdn.net/dgvv4/article/details/123415708
MobileNetV2：https://blog.csdn.net/dgvv4/article/details/123417739
1. MobileNet 核心知识回顾
由于 MnasNet 是介于 MobileNetV2 和 MobileNetV3 之间的网络，因此开始之前一定要熟悉 MobileNet 的一些概念。为了能帮助大家理解 MnasNet 网络模型，先简单复习一下 MobileNet 系列网络的核心部件。
1.1 深度可分离卷积
深度可分离卷积（depthwise separable convolution）可理解为：由 深度卷积（depthwise convolution）和 逐点卷积（pointwise convolution）构成。
深度卷积处理长宽方向的信息。输入图像有多少个通道就有多少个卷积核，每个通道都用自己对应的卷积核生成一张对应的特征图。输入和输出的通道数相同
逐点卷积处理跨通道方向的信息。采用的是普通卷积的方法，只不过卷积核size只有1*1。一个卷积核负责所有的通道，滑动过程中对应像素值相乘再相加，输出特征图个数由卷积核个数决定。
1.2 逆残差结构
MobileNetV2 使用了逆转残差模块。图片输入，先使用1x1卷积升维；然后在高维空间下使用深度卷积；最后使用1x1卷积降维，降维时使用线性激活函数。当步长为 2（下采样阶段）时，没有残差连接。当步长为 1（基本模块）时，残差连接输入和输出的特征图。
2. MnasNet 网络模型
2.1 模型创新点
（1）MnasNet 运用了多目标优化函数，兼顾了速度和精度。
真实的手机推理时间作为速度，准确度作为精度，构建多目标的优化函数，如下。T 是一个硬指标，规定的时间（如80ms内完成预测）；LAT(m) 代表预测时间；w 总是小于0，表示预测时间越长计算结果越大
若 
，横轴是延误，纵轴是目标函数。小于硬指标时
，目标函数的值和延误无关，只与准确率有关。超过硬指标时
，目标函数急剧减少，此时目标函数和延误有关。超过硬指标就会被惩罚，生成的模型更加集中。
若
，不管是低于硬指标还是超过硬指标，目标函数和延误的关系是平滑的，不会出现严厉的惩罚。模型搜索空间更多，搜索到更多样的速度和精度的权衡
（2）分层的神经架构搜索空间
将一个卷积神经网络模型分层7个Block，每一个Block内部的结构是一样的，但Block之间的结构不一样，使得我们能设计网络不同部位的深层和浅层的模型结构。
采用强化学习探索网络最优的卷积方式、卷积核大小、SE注意力机制、跨层连接方式、每个Block中的层数。
使得不同Block的结构是多样的，使网络具有多样性。
2.2 模型评价
论文中将其与MobileNet做比较，速度上 MnasNet 比 MobileNet 快1.8倍；在不同的延误下，MnasNet 的准确率都要比 MobileNet 的准确率要高
在不同的宽度超参数和分辨率超参数下，MnasNet 的性能都超过 MobileNet
3. 代码复现
3.1 网络结构
网络结构如下图所示。紫色卷积块是MobileNetV1的深度可分离卷积块，绿色卷积块是MobileNetV2的逆残差结构模块，红色卷积块是添加了SE注意力机制后的逆残差结构模块。
其中，MBConv6代表逆残差连接模块中1*1卷积上升通道数为原图像通道数的6倍；MBConv6模块右侧的 x2 代表该模块重复两次，先进行1次下采样(stride=2)，再进行一次基本模块(stride=1)；有的模块需要下采样有的不需要，搭载网络时需要注意。
3.2 搭建各个卷积模块
（1）标准卷积块
一个标准卷积块由 卷积层Conv + 批标准化层BN + 激活函数Relu 组成
#（1）标准卷积模块
def conv_block(input_tensor, filters, kernel_size, stride):
# 普通卷积+BN+激活
x = layers.Conv2D(filters = filters, # 卷积核个数
kernel_size = kernel_size, # 卷积核size
strides = stride, # 步长
use_bias = False, # 有BN层就不要偏置
padding = 'same')(input_tensor) # 步长=1时特征图size不变，步长=2时特征图长宽减半
x = layers.BatchNormalization()(x) # 批标准化
x = layers.ReLU()(x) # relu激活函数
return x # 如果activation=False可以直接输出结果（2）深度可分离卷积块
一个深度可分离卷积块由 一个深度卷积 和 一个1*1逐点卷积 组成
#（2）深度卷积
def depthwise_conv_block(input_tensor, kernel_size, stride):
# 深度卷积只处理长宽方向的空间信息，输入输出的通道数相同
x = layers.DepthwiseConv2D(kernel_size = kernel_size, # 卷积核size
strides = stride, # 步长
use_bias = False, # 有BN层不要偏置
padding = 'same')(input_tensor) # stride=1卷积过程中size不变
x = layers.BatchNormalization()(x) # 批标准化
x = layers.ReLU()(x) # 激活函数
return x # 返回深度卷积的特征图，个数保持不变
#（3）逐点卷积
def pointwise_conv_block(input_tensor, filters):
# 1*1卷积只负责通道方向的信息融合，一个卷积核输出一张特征图
x = layers.Conv2D(filters = filters, # 卷积核个数，即输出特征图个数
kernel_size = (1,1), # 卷积核size=1*1，不处理长宽方向的信息
strides = 1, # 步长=1卷积过程中特征图size不变
padding = 'same', # 卷积过程中size不变
use_bias = False)(input_tensor) # 有BN层就不要偏置
x = layers.BatchNormalization()(x) # 批标准化
# 不使用relu激活函数，线性激活
return x
#（4）深度可分离卷积 == 深度卷积 + 逐点卷积
def sep_conv_block(input_tensor, kernel_size, stride, filters):
# 深度卷积，处理长宽方向的空间信息，不关心跨通道信息
x = depthwise_conv_block(input_tensor, kernel_size, stride)
# 逐点卷积，处理跨通道信息，跨层信息交融
x = pointwise_conv_block(x, filters)
return x # 返回深度可分离卷积输出特征图（3）逆转残差结构模块
先1*1卷积升维；然后在高维空间下深度卷积；最后1*1卷积降维（线性激活函数）。当步长=1并且输出和输出特征图的shape相同，才能将输入和输出残差连接。
#（5）深度可分离卷积的逆转残差模块
# 1x1标准卷积升维N倍，然后深度卷积，再1x1逐点卷积降维
def inverted_res_block(input_tensor, expansion, kernel_size, stride, out_channel):
# keras.backend.int_shape得到图像的shape，这里只需要最后一个维度即通道维度的大小
in_channel = keras.backend.int_shape(input_tensor)[-1]
# 调用自定义的标准卷积函数，上升通道数
x = conv_block(input_tensor, # 输入特征图
kernel_size = (1,1), # 卷积核size
filters = in_channel*expansion, # 通道上升为原来的expansion倍
stride = 1)
# 调用自定义的深度卷积函数
x = depthwise_conv_block(x, kernel_size=kernel_size, stride=stride)
# 调用自定义的逐点卷积函数，下降通道数
x = pointwise_conv_block(x, filters = out_channel) # out_channel输出特征图数量
# 如果步长=1，并且输入和输出的shape相同时，输入和输出残差连接
if stride == 1 and input_tensor.shape==x.shape:
output = layers.Add()([input_tensor, x])
return output
# 如果步长=2，直接输出逐点卷积后的结果
else:
return x（4）添加了SE注意力机制后的逆残差结构模块
先1*1卷积升维；然后在高维空间下深度卷积；再通过SE注意力机制；最后1*1卷积降维（使用线性激活函数）。输入和输出shape相同时，用残差连接
#（7）定义压缩和激活方法SE
def squeeze_excitation(input_tensor):
inputs = input_tensor # 将特征图复制一份
squeeze = inputs.shape[-1]/2 # 将特征图在通道维度上平分成两份，即压缩量为原通道的1/2
excitation = inputs.shape[-1] # 通道上升到原通道数大小
# 如：[416,416,24]==>[None,24]
x = layers.GlobalAveragePooling2D()(input_tensor) # 全局平均池化
# 如：[None,24]==>[None,12]
x = layers.Dense(squeeze)(x) # 全连接层，通道数减半
# 激活函数，shape不变
x = layers.ReLU()(x)
# 如：[None,12]==>[None,24]
x = layers.Dense(excitation)(x) # 全连接层，通道数回升至原来
# 激活函数，shape不变
x = tf.nn.sigmoid(x)
# 如：[None,24]==>[1,1,24]
x = layers.Reshape(target_shape = (1,1,excitation))(x)
# [416,416,24]*[1,1,24]==>[416,416,24]
output = inputs * x # 点乘，元素之间相乘，shape不变
return output
#（8）应用压缩和激活方法后的深度可分离卷积的逆转残差模块
def inverted_se_res_block(input_tensor, expansion, kernel_size, stride, out_channel):
# 就比inverted_res_block多了一个SE层，其他都一样
# 得到输出特征图的通道数量
in_channel = keras.backend.int_shape(input_tensor)[-1]
# 1*1标准卷积模块，通道数上升expansion倍
x = conv_block(input_tensor, filters=in_channel*expansion,
kernel_size=(1,1), stride=1)
# 深度卷积模块，输出的特征图的通道数不变
x = depthwise_conv_block(x, kernel_size, stride)
# SE模块
x = squeeze_excitation(x)
# 逐点卷积，1*1卷积下降通道数
x = pointwise_conv_block(x, filters=out_channel)
# 如果步长=1，并且输入和输出的shape相同时，需要残差连接输入和输出
if stride == 1 and input_tensor.shape==x.shape:
output = layers.Add()([input_tensor, x])
return output
# 如果步长=2，直接输出逐点卷积结果
else:
return x3.3 完整代码展示
根据3.1网络模型结构图堆叠网络各层，结合上面的解释，代码中每行都有注释，有疑问的可在评论区留言。
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, Model
#（1）标准卷积模块
def conv_block(input_tensor, filters, kernel_size, stride):
# 普通卷积+BN+激活
x = layers.Conv2D(filters = filters, # 卷积核个数
kernel_size = kernel_size, # 卷积核size
strides = stride, # 步长
use_bias = False, # 有BN层就不要偏置
padding = 'same')(input_tensor) # 步长=1时特征图size不变，步长=2时特征图长宽减半
x = layers.BatchNormalization()(x) # 批标准化
x = layers.ReLU()(x) # relu激活函数
return x # 如果activation=False可以直接输出结果
#（2）深度卷积
def depthwise_conv_block(input_tensor, kernel_size, stride):
# 深度卷积只处理长宽方向的空间信息，输入输出的通道数相同
x = layers.DepthwiseConv2D(kernel_size = kernel_size, # 卷积核size
strides = stride, # 步长
use_bias = False, # 有BN层不要偏置
padding = 'same')(input_tensor) # stride=1卷积过程中size不变
x = layers.BatchNormalization()(x) # 批标准化
x = layers.ReLU()(x) # 激活函数
return x # 返回深度卷积的特征图，个数保持不变
#（3）逐点卷积
def pointwise_conv_block(input_tensor, filters):
# 1*1卷积只负责通道方向的信息融合，一个卷积核输出一张特征图
x = layers.Conv2D(filters = filters, # 卷积核个数，即输出特征图个数
kernel_size = (1,1), # 卷积核size=1*1，不处理长宽方向的信息
strides = 1, # 步长=1卷积过程中特征图size不变
padding = 'same', # 卷积过程中size不变
use_bias = False)(input_tensor) # 有BN层就不要偏置
x = layers.BatchNormalization()(x) # 批标准化
# 不使用relu激活函数，线性激活
return x
#（4）深度可分离卷积 == 深度卷积 + 逐点卷积
def sep_conv_block(input_tensor, kernel_size, stride, filters):
# 深度卷积，处理长宽方向的空间信息，不关心跨通道信息
x = depthwise_conv_block(input_tensor, kernel_size, stride)
# 逐点卷积，处理跨通道信息，跨层信息交融
x = pointwise_conv_block(x, filters)
return x # 返回深度可分离卷积输出特征图
#（5）深度可分离卷积的逆转残差模块
# 1x1标准卷积升维N倍，然后深度卷积，再1x1逐点卷积降维
def inverted_res_block(input_tensor, expansion, kernel_size, stride, out_channel):
# keras.backend.int_shape得到图像的shape，这里只需要最后一个维度即通道维度的大小
in_channel = keras.backend.int_shape(input_tensor)[-1]
# 调用自定义的标准卷积函数，上升通道数
x = conv_block(input_tensor, # 输入特征图
kernel_size = (1,1), # 卷积核size
filters = in_channel*expansion, # 通道上升为原来的expansion倍
stride = 1)
# 调用自定义的深度卷积函数
x = depthwise_conv_block(x, kernel_size=kernel_size, stride=stride)
# 调用自定义的逐点卷积函数，下降通道数
x = pointwise_conv_block(x, filters = out_channel) # out_channel输出特征图数量
# 如果步长=1，并且输入和输出的shape相同时，输入和输出残差连接
if stride == 1 and input_tensor.shape==x.shape:
output = layers.Add()([input_tensor, x])
return output
# 如果步长=2，直接输出逐点卷积后的结果
else:
return x
#（6）一个MBConv模块是由一个下采样模块(stride=2)和若干个基本模块(stride=1)组成
def MBConv(input_tensor, expansion, kernel_size, filters, stride, num):
# 一个下采样模块，也可能不需要下采样
x = inverted_res_block(input_tensor, expansion, kernel_size, stride, out_channel=filters)
# num-1个基本模块。num代表整个MBConv模块包含几个inverted_res_block模块
for _ in range(1, num):
x = inverted_res_block(x, expansion, kernel_size, stride=1, out_channel=filters)
return x # 返回MBConv卷积块的特征图
#（7）定义压缩和激活方法SE
def squeeze_excitation(input_tensor):
inputs = input_tensor # 将特征图复制一份
squeeze = inputs.shape[-1]/2 # 将特征图在通道维度上平分成两份，即压缩量为原通道的1/2
excitation = inputs.shape[-1] # 通道上升到原通道数大小
# 如：[416,416,24]==>[None,24]
x = layers.GlobalAveragePooling2D()(input_tensor) # 全局平均池化
# 如：[None,24]==>[None,12]
x = layers.Dense(squeeze)(x) # 全连接层，通道数减半
# 激活函数，shape不变
x = layers.ReLU()(x)
# 如：[None,12]==>[None,24]
x = layers.Dense(excitation)(x) # 全连接层，通道数回升至原来
# 激活函数，shape不变
x = tf.nn.sigmoid(x)
# 如：[None,24]==>[1,1,24]
x = layers.Reshape(target_shape = (1,1,excitation))(x)
# [416,416,24]*[1,1,24]==>[416,416,24]
output = inputs * x # 点乘，元素之间相乘，shape不变
return output
#（8）应用压缩和激活方法后的深度可分离卷积的逆转残差模块
def inverted_se_res_block(input_tensor, expansion, kernel_size, stride, out_channel):
# 就比inverted_res_block多了一个SE层，其他都一样
# 得到输出特征图的通道数量
in_channel = keras.backend.int_shape(input_tensor)[-1]
# 1*1标准卷积模块，通道数上升expansion倍
x = conv_block(input_tensor, filters=in_channel*expansion,
kernel_size=(1,1), stride=1)
# 深度卷积模块，输出的特征图的通道数不变
x = depthwise_conv_block(x, kernel_size, stride)
# SE模块
x = squeeze_excitation(x)
# 逐点卷积，1*1卷积下降通道数
x = pointwise_conv_block(x, filters=out_channel)
# 如果步长=1，并且输入和输出的shape相同时，需要残差连接输入和输出
if stride == 1 and input_tensor.shape==x.shape:
output = layers.Add()([input_tensor, x])
return output
# 如果步长=2，直接输出逐点卷积结果
else:
return x
#（9）一个MBConv_SE模块是由一个下采样模块(stride=2)和若干个基本模块(stride=1)组成
def MBConv_SE(input_tensor, expansion, kernel_size, filters, stride, num):
# 一个下采样模块，也可能不需要下采样
x = inverted_se_res_block(input_tensor, expansion, kernel_size, stride, out_channel=filters)
# num-1个基本模块。num代表整个MBConv模块包含几个inverted_res_block模块
for _ in range(1, num):
x = inverted_se_res_block(x, expansion, kernel_size, stride=1, out_channel=filters)
return x # 返回MBConv_SE卷积块的特征图
#（10）搭建主干网络
def MnasNet(input_shape, classes):
# 构建网络输入tensor
inputs = keras.Input(shape=input_shape)
# [224,224,3]==>[112,112,32]
x = conv_block(inputs, 32, kernel_size=(3,3), stride=2)
# [112,112,32]==>[112,112,16]
x = sep_conv_block(x, kernel_size=(3,3), stride=1, filters=16)
# [112,112,16]==>[56,56,24]
x = MBConv(x, expansion=6, kernel_size=(3,3), filters=24, stride=2, num=2)
# [56,56,24]==>[28,28,40]
x = MBConv_SE(x, expansion=3, kernel_size=(5,5), filters=40, stride=2, num=3)
# [28,28,40]==>[14,14,80]
x = MBConv(x, expansion=6, kernel_size=(3,3), filters=80, stride=2, num=4)
# [14,14,80]==>[14,14,112]
x = MBConv_SE(x, expansion=6, kernel_size=(3,3), filters=112, stride=1, num=2)
# [14,14,112]==>[7,7,160]
x = MBConv_SE(x, expansion=6, kernel_size=(5,5), filters=160, stride=2, num=3)
# [7,7,160]==>[7,7,320]
x = MBConv(x, expansion=6, kernel_size=(3,3), filters=320, stride=1, num=1)
# 再进行一次标准卷积 [7,7,320]==>[7,7,1280]
x = conv_block(x, filters=1280, kernel_size=(1,1), stride=1)
# [7,7,1280]==>[None,1280]
x = layers.GlobalAveragePooling2D()(x)
# [None,1280]==>[None,1000]
logits = layers.Dense(classes)(x) # 为了网络稳定，训练时再使用Softmax函数
# 完成网络构架
model = Model(inputs, logits)
return model
#（11）接收网络模型
if __name__ == '__main__':
model = MnasNet(input_shape=[224,224,3], classes=1000) # 给出输入图像shape和分类数
# 查看网络结构
model.summary()3.4 模型架构展示
通过model.summary()查看网络框架，参数量六百万
Model: "model"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 224, 224, 3) 0
__________________________________________________________________________________________________
conv2d (Conv2D) (None, 112, 112, 32) 864 input_1[0][0]
__________________________________________________________________________________________________
batch_normalization (BatchNorma (None, 112, 112, 32) 128 conv2d[0][0]
__________________________________________________________________________________________________
re_lu (ReLU) (None, 112, 112, 32) 0 batch_normalization[0][0]
__________________________________________________________________________________________________
depthwise_conv2d (DepthwiseConv (None, 112, 112, 32) 288 re_lu[0][0]
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 112, 112, 32) 128 depthwise_conv2d[0][0]
__________________________________________________________________________________________________
re_lu_1 (ReLU) (None, 112, 112, 32) 0 batch_normalization_1[0][0]
__________________________________________________________________________________________________
conv2d_1 (Conv2D) (None, 112, 112, 16) 512 re_lu_1[0][0]
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 112, 112, 16) 64 conv2d_1[0][0]
__________________________________________________________________________________________________
conv2d_2 (Conv2D) (None, 112, 112, 96) 1536 batch_normalization_2[0][0]
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 112, 112, 96) 384 conv2d_2[0][0]
__________________________________________________________________________________________________
re_lu_2 (ReLU) (None, 112, 112, 96) 0 batch_normalization_3[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_1 (DepthwiseCo (None, 56, 56, 96) 864 re_lu_2[0][0]
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 56, 56, 96) 384 depthwise_conv2d_1[0][0]
__________________________________________________________________________________________________
re_lu_3 (ReLU) (None, 56, 56, 96) 0 batch_normalization_4[0][0]
__________________________________________________________________________________________________
conv2d_3 (Conv2D) (None, 56, 56, 24) 2304 re_lu_3[0][0]
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 56, 56, 24) 96 conv2d_3[0][0]
__________________________________________________________________________________________________
conv2d_4 (Conv2D) (None, 56, 56, 144) 3456 batch_normalization_5[0][0]
__________________________________________________________________________________________________
batch_normalization_6 (BatchNor (None, 56, 56, 144) 576 conv2d_4[0][0]
__________________________________________________________________________________________________
re_lu_4 (ReLU) (None, 56, 56, 144) 0 batch_normalization_6[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_2 (DepthwiseCo (None, 56, 56, 144) 1296 re_lu_4[0][0]
__________________________________________________________________________________________________
batch_normalization_7 (BatchNor (None, 56, 56, 144) 576 depthwise_conv2d_2[0][0]
__________________________________________________________________________________________________
re_lu_5 (ReLU) (None, 56, 56, 144) 0 batch_normalization_7[0][0]
__________________________________________________________________________________________________
conv2d_5 (Conv2D) (None, 56, 56, 24) 3456 re_lu_5[0][0]
__________________________________________________________________________________________________
batch_normalization_8 (BatchNor (None, 56, 56, 24) 96 conv2d_5[0][0]
__________________________________________________________________________________________________
conv2d_6 (Conv2D) (None, 56, 56, 72) 1728 batch_normalization_8[0][0]
__________________________________________________________________________________________________
batch_normalization_9 (BatchNor (None, 56, 56, 72) 288 conv2d_6[0][0]
__________________________________________________________________________________________________
re_lu_6 (ReLU) (None, 56, 56, 72) 0 batch_normalization_9[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_3 (DepthwiseCo (None, 28, 28, 72) 1800 re_lu_6[0][0]
__________________________________________________________________________________________________
batch_normalization_10 (BatchNo (None, 28, 28, 72) 288 depthwise_conv2d_3[0][0]
__________________________________________________________________________________________________
re_lu_7 (ReLU) (None, 28, 28, 72) 0 batch_normalization_10[0][0]
__________________________________________________________________________________________________
global_average_pooling2d (Globa (None, 72) 0 re_lu_7[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, 36) 2628 global_average_pooling2d[0][0]
__________________________________________________________________________________________________
re_lu_8 (ReLU) (None, 36) 0 dense[0][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, 72) 2664 re_lu_8[0][0]
__________________________________________________________________________________________________
tf.math.sigmoid (TFOpLambda) (None, 72) 0 dense_1[0][0]
__________________________________________________________________________________________________
reshape (Reshape) (None, 1, 1, 72) 0 tf.math.sigmoid[0][0]
__________________________________________________________________________________________________
tf.math.multiply (TFOpLambda) (None, 28, 28, 72) 0 re_lu_7[0][0]
reshape[0][0]
__________________________________________________________________________________________________
conv2d_7 (Conv2D) (None, 28, 28, 40) 2880 tf.math.multiply[0][0]
__________________________________________________________________________________________________
batch_normalization_11 (BatchNo (None, 28, 28, 40) 160 conv2d_7[0][0]
__________________________________________________________________________________________________
conv2d_8 (Conv2D) (None, 28, 28, 120) 4800 batch_normalization_11[0][0]
__________________________________________________________________________________________________
batch_normalization_12 (BatchNo (None, 28, 28, 120) 480 conv2d_8[0][0]
__________________________________________________________________________________________________
re_lu_9 (ReLU) (None, 28, 28, 120) 0 batch_normalization_12[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_4 (DepthwiseCo (None, 28, 28, 120) 3000 re_lu_9[0][0]
__________________________________________________________________________________________________
batch_normalization_13 (BatchNo (None, 28, 28, 120) 480 depthwise_conv2d_4[0][0]
__________________________________________________________________________________________________
re_lu_10 (ReLU) (None, 28, 28, 120) 0 batch_normalization_13[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_1 (Glo (None, 120) 0 re_lu_10[0][0]
__________________________________________________________________________________________________
dense_2 (Dense) (None, 60) 7260 global_average_pooling2d_1[0][0]
__________________________________________________________________________________________________
re_lu_11 (ReLU) (None, 60) 0 dense_2[0][0]
__________________________________________________________________________________________________
dense_3 (Dense) (None, 120) 7320 re_lu_11[0][0]
__________________________________________________________________________________________________
tf.math.sigmoid_1 (TFOpLambda) (None, 120) 0 dense_3[0][0]
__________________________________________________________________________________________________
reshape_1 (Reshape) (None, 1, 1, 120) 0 tf.math.sigmoid_1[0][0]
__________________________________________________________________________________________________
tf.math.multiply_1 (TFOpLambda) (None, 28, 28, 120) 0 re_lu_10[0][0]
reshape_1[0][0]
__________________________________________________________________________________________________
conv2d_9 (Conv2D) (None, 28, 28, 40) 4800 tf.math.multiply_1[0][0]
__________________________________________________________________________________________________
batch_normalization_14 (BatchNo (None, 28, 28, 40) 160 conv2d_9[0][0]
__________________________________________________________________________________________________
conv2d_10 (Conv2D) (None, 28, 28, 120) 4800 batch_normalization_14[0][0]
__________________________________________________________________________________________________
batch_normalization_15 (BatchNo (None, 28, 28, 120) 480 conv2d_10[0][0]
__________________________________________________________________________________________________
re_lu_12 (ReLU) (None, 28, 28, 120) 0 batch_normalization_15[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_5 (DepthwiseCo (None, 28, 28, 120) 3000 re_lu_12[0][0]
__________________________________________________________________________________________________
batch_normalization_16 (BatchNo (None, 28, 28, 120) 480 depthwise_conv2d_5[0][0]
__________________________________________________________________________________________________
re_lu_13 (ReLU) (None, 28, 28, 120) 0 batch_normalization_16[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_2 (Glo (None, 120) 0 re_lu_13[0][0]
__________________________________________________________________________________________________
dense_4 (Dense) (None, 60) 7260 global_average_pooling2d_2[0][0]
__________________________________________________________________________________________________
re_lu_14 (ReLU) (None, 60) 0 dense_4[0][0]
__________________________________________________________________________________________________
dense_5 (Dense) (None, 120) 7320 re_lu_14[0][0]
__________________________________________________________________________________________________
tf.math.sigmoid_2 (TFOpLambda) (None, 120) 0 dense_5[0][0]
__________________________________________________________________________________________________
reshape_2 (Reshape) (None, 1, 1, 120) 0 tf.math.sigmoid_2[0][0]
__________________________________________________________________________________________________
tf.math.multiply_2 (TFOpLambda) (None, 28, 28, 120) 0 re_lu_13[0][0]
reshape_2[0][0]
__________________________________________________________________________________________________
conv2d_11 (Conv2D) (None, 28, 28, 40) 4800 tf.math.multiply_2[0][0]
__________________________________________________________________________________________________
batch_normalization_17 (BatchNo (None, 28, 28, 40) 160 conv2d_11[0][0]
__________________________________________________________________________________________________
conv2d_12 (Conv2D) (None, 28, 28, 240) 9600 batch_normalization_17[0][0]
__________________________________________________________________________________________________
batch_normalization_18 (BatchNo (None, 28, 28, 240) 960 conv2d_12[0][0]
__________________________________________________________________________________________________
re_lu_15 (ReLU) (None, 28, 28, 240) 0 batch_normalization_18[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_6 (DepthwiseCo (None, 14, 14, 240) 2160 re_lu_15[0][0]
__________________________________________________________________________________________________
batch_normalization_19 (BatchNo (None, 14, 14, 240) 960 depthwise_conv2d_6[0][0]
__________________________________________________________________________________________________
re_lu_16 (ReLU) (None, 14, 14, 240) 0 batch_normalization_19[0][0]
__________________________________________________________________________________________________
conv2d_13 (Conv2D) (None, 14, 14, 80) 19200 re_lu_16[0][0]
__________________________________________________________________________________________________
batch_normalization_20 (BatchNo (None, 14, 14, 80) 320 conv2d_13[0][0]
__________________________________________________________________________________________________
conv2d_14 (Conv2D) (None, 14, 14, 480) 38400 batch_normalization_20[0][0]
__________________________________________________________________________________________________
batch_normalization_21 (BatchNo (None, 14, 14, 480) 1920 conv2d_14[0][0]
__________________________________________________________________________________________________
re_lu_17 (ReLU) (None, 14, 14, 480) 0 batch_normalization_21[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_7 (DepthwiseCo (None, 14, 14, 480) 4320 re_lu_17[0][0]
__________________________________________________________________________________________________
batch_normalization_22 (BatchNo (None, 14, 14, 480) 1920 depthwise_conv2d_7[0][0]
__________________________________________________________________________________________________
re_lu_18 (ReLU) (None, 14, 14, 480) 0 batch_normalization_22[0][0]
__________________________________________________________________________________________________
conv2d_15 (Conv2D) (None, 14, 14, 80) 38400 re_lu_18[0][0]
__________________________________________________________________________________________________
batch_normalization_23 (BatchNo (None, 14, 14, 80) 320 conv2d_15[0][0]
__________________________________________________________________________________________________
conv2d_16 (Conv2D) (None, 14, 14, 480) 38400 batch_normalization_23[0][0]
__________________________________________________________________________________________________
batch_normalization_24 (BatchNo (None, 14, 14, 480) 1920 conv2d_16[0][0]
__________________________________________________________________________________________________
re_lu_19 (ReLU) (None, 14, 14, 480) 0 batch_normalization_24[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_8 (DepthwiseCo (None, 14, 14, 480) 4320 re_lu_19[0][0]
__________________________________________________________________________________________________
batch_normalization_25 (BatchNo (None, 14, 14, 480) 1920 depthwise_conv2d_8[0][0]
__________________________________________________________________________________________________
re_lu_20 (ReLU) (None, 14, 14, 480) 0 batch_normalization_25[0][0]
__________________________________________________________________________________________________
conv2d_17 (Conv2D) (None, 14, 14, 80) 38400 re_lu_20[0][0]
__________________________________________________________________________________________________
batch_normalization_26 (BatchNo (None, 14, 14, 80) 320 conv2d_17[0][0]
__________________________________________________________________________________________________
conv2d_18 (Conv2D) (None, 14, 14, 480) 38400 batch_normalization_26[0][0]
__________________________________________________________________________________________________
batch_normalization_27 (BatchNo (None, 14, 14, 480) 1920 conv2d_18[0][0]
__________________________________________________________________________________________________
re_lu_21 (ReLU) (None, 14, 14, 480) 0 batch_normalization_27[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_9 (DepthwiseCo (None, 14, 14, 480) 4320 re_lu_21[0][0]
__________________________________________________________________________________________________
batch_normalization_28 (BatchNo (None, 14, 14, 480) 1920 depthwise_conv2d_9[0][0]
__________________________________________________________________________________________________
re_lu_22 (ReLU) (None, 14, 14, 480) 0 batch_normalization_28[0][0]
__________________________________________________________________________________________________
conv2d_19 (Conv2D) (None, 14, 14, 80) 38400 re_lu_22[0][0]
__________________________________________________________________________________________________
batch_normalization_29 (BatchNo (None, 14, 14, 80) 320 conv2d_19[0][0]
__________________________________________________________________________________________________
conv2d_20 (Conv2D) (None, 14, 14, 480) 38400 batch_normalization_29[0][0]
__________________________________________________________________________________________________
batch_normalization_30 (BatchNo (None, 14, 14, 480) 1920 conv2d_20[0][0]
__________________________________________________________________________________________________
re_lu_23 (ReLU) (None, 14, 14, 480) 0 batch_normalization_30[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_10 (DepthwiseC (None, 14, 14, 480) 4320 re_lu_23[0][0]
__________________________________________________________________________________________________
batch_normalization_31 (BatchNo (None, 14, 14, 480) 1920 depthwise_conv2d_10[0][0]
__________________________________________________________________________________________________
re_lu_24 (ReLU) (None, 14, 14, 480) 0 batch_normalization_31[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_3 (Glo (None, 480) 0 re_lu_24[0][0]
__________________________________________________________________________________________________
dense_6 (Dense) (None, 240) 115440 global_average_pooling2d_3[0][0]
__________________________________________________________________________________________________
re_lu_25 (ReLU) (None, 240) 0 dense_6[0][0]
__________________________________________________________________________________________________
dense_7 (Dense) (None, 480) 115680 re_lu_25[0][0]
__________________________________________________________________________________________________
tf.math.sigmoid_3 (TFOpLambda) (None, 480) 0 dense_7[0][0]
__________________________________________________________________________________________________
reshape_3 (Reshape) (None, 1, 1, 480) 0 tf.math.sigmoid_3[0][0]
__________________________________________________________________________________________________
tf.math.multiply_3 (TFOpLambda) (None, 14, 14, 480) 0 re_lu_24[0][0]
reshape_3[0][0]
__________________________________________________________________________________________________
conv2d_21 (Conv2D) (None, 14, 14, 112) 53760 tf.math.multiply_3[0][0]
__________________________________________________________________________________________________
batch_normalization_32 (BatchNo (None, 14, 14, 112) 448 conv2d_21[0][0]
__________________________________________________________________________________________________
conv2d_22 (Conv2D) (None, 14, 14, 672) 75264 batch_normalization_32[0][0]
__________________________________________________________________________________________________
batch_normalization_33 (BatchNo (None, 14, 14, 672) 2688 conv2d_22[0][0]
__________________________________________________________________________________________________
re_lu_26 (ReLU) (None, 14, 14, 672) 0 batch_normalization_33[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_11 (DepthwiseC (None, 14, 14, 672) 6048 re_lu_26[0][0]
__________________________________________________________________________________________________
batch_normalization_34 (BatchNo (None, 14, 14, 672) 2688 depthwise_conv2d_11[0][0]
__________________________________________________________________________________________________
re_lu_27 (ReLU) (None, 14, 14, 672) 0 batch_normalization_34[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_4 (Glo (None, 672) 0 re_lu_27[0][0]
__________________________________________________________________________________________________
dense_8 (Dense) (None, 336) 226128 global_average_pooling2d_4[0][0]
__________________________________________________________________________________________________
re_lu_28 (ReLU) (None, 336) 0 dense_8[0][0]
__________________________________________________________________________________________________
dense_9 (Dense) (None, 672) 226464 re_lu_28[0][0]
__________________________________________________________________________________________________
tf.math.sigmoid_4 (TFOpLambda) (None, 672) 0 dense_9[0][0]
__________________________________________________________________________________________________
reshape_4 (Reshape) (None, 1, 1, 672) 0 tf.math.sigmoid_4[0][0]
__________________________________________________________________________________________________
tf.math.multiply_4 (TFOpLambda) (None, 14, 14, 672) 0 re_lu_27[0][0]
reshape_4[0][0]
__________________________________________________________________________________________________
conv2d_23 (Conv2D) (None, 14, 14, 112) 75264 tf.math.multiply_4[0][0]
__________________________________________________________________________________________________
batch_normalization_35 (BatchNo (None, 14, 14, 112) 448 conv2d_23[0][0]
__________________________________________________________________________________________________
conv2d_24 (Conv2D) (None, 14, 14, 672) 75264 batch_normalization_35[0][0]
__________________________________________________________________________________________________
batch_normalization_36 (BatchNo (None, 14, 14, 672) 2688 conv2d_24[0][0]
__________________________________________________________________________________________________
re_lu_29 (ReLU) (None, 14, 14, 672) 0 batch_normalization_36[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_12 (DepthwiseC (None, 7, 7, 672) 16800 re_lu_29[0][0]
__________________________________________________________________________________________________
batch_normalization_37 (BatchNo (None, 7, 7, 672) 2688 depthwise_conv2d_12[0][0]
__________________________________________________________________________________________________
re_lu_30 (ReLU) (None, 7, 7, 672) 0 batch_normalization_37[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_5 (Glo (None, 672) 0 re_lu_30[0][0]
__________________________________________________________________________________________________
dense_10 (Dense) (None, 336) 226128 global_average_pooling2d_5[0][0]
__________________________________________________________________________________________________
re_lu_31 (ReLU) (None, 336) 0 dense_10[0][0]
__________________________________________________________________________________________________
dense_11 (Dense) (None, 672) 226464 re_lu_31[0][0]
__________________________________________________________________________________________________
tf.math.sigmoid_5 (TFOpLambda) (None, 672) 0 dense_11[0][0]
__________________________________________________________________________________________________
reshape_5 (Reshape) (None, 1, 1, 672) 0 tf.math.sigmoid_5[0][0]
__________________________________________________________________________________________________
tf.math.multiply_5 (TFOpLambda) (None, 7, 7, 672) 0 re_lu_30[0][0]
reshape_5[0][0]
__________________________________________________________________________________________________
conv2d_25 (Conv2D) (None, 7, 7, 160) 107520 tf.math.multiply_5[0][0]
__________________________________________________________________________________________________
batch_normalization_38 (BatchNo (None, 7, 7, 160) 640 conv2d_25[0][0]
__________________________________________________________________________________________________
conv2d_26 (Conv2D) (None, 7, 7, 960) 153600 batch_normalization_38[0][0]
__________________________________________________________________________________________________
batch_normalization_39 (BatchNo (None, 7, 7, 960) 3840 conv2d_26[0][0]
__________________________________________________________________________________________________
re_lu_32 (ReLU) (None, 7, 7, 960) 0 batch_normalization_39[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_13 (DepthwiseC (None, 7, 7, 960) 24000 re_lu_32[0][0]
__________________________________________________________________________________________________
batch_normalization_40 (BatchNo (None, 7, 7, 960) 3840 depthwise_conv2d_13[0][0]
__________________________________________________________________________________________________
re_lu_33 (ReLU) (None, 7, 7, 960) 0 batch_normalization_40[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_6 (Glo (None, 960) 0 re_lu_33[0][0]
__________________________________________________________________________________________________
dense_12 (Dense) (None, 480) 461280 global_average_pooling2d_6[0][0]
__________________________________________________________________________________________________
re_lu_34 (ReLU) (None, 480) 0 dense_12[0][0]
__________________________________________________________________________________________________
dense_13 (Dense) (None, 960) 461760 re_lu_34[0][0]
__________________________________________________________________________________________________
tf.math.sigmoid_6 (TFOpLambda) (None, 960) 0 dense_13[0][0]
__________________________________________________________________________________________________
reshape_6 (Reshape) (None, 1, 1, 960) 0 tf.math.sigmoid_6[0][0]
__________________________________________________________________________________________________
tf.math.multiply_6 (TFOpLambda) (None, 7, 7, 960) 0 re_lu_33[0][0]
reshape_6[0][0]
__________________________________________________________________________________________________
conv2d_27 (Conv2D) (None, 7, 7, 160) 153600 tf.math.multiply_6[0][0]
__________________________________________________________________________________________________
batch_normalization_41 (BatchNo (None, 7, 7, 160) 640 conv2d_27[0][0]
__________________________________________________________________________________________________
conv2d_28 (Conv2D) (None, 7, 7, 960) 153600 batch_normalization_41[0][0]
__________________________________________________________________________________________________
batch_normalization_42 (BatchNo (None, 7, 7, 960) 3840 conv2d_28[0][0]
__________________________________________________________________________________________________
re_lu_35 (ReLU) (None, 7, 7, 960) 0 batch_normalization_42[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_14 (DepthwiseC (None, 7, 7, 960) 24000 re_lu_35[0][0]
__________________________________________________________________________________________________
batch_normalization_43 (BatchNo (None, 7, 7, 960) 3840 depthwise_conv2d_14[0][0]
__________________________________________________________________________________________________
re_lu_36 (ReLU) (None, 7, 7, 960) 0 batch_normalization_43[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_7 (Glo (None, 960) 0 re_lu_36[0][0]
__________________________________________________________________________________________________
dense_14 (Dense) (None, 480) 461280 global_average_pooling2d_7[0][0]
__________________________________________________________________________________________________
re_lu_37 (ReLU) (None, 480) 0 dense_14[0][0]
__________________________________________________________________________________________________
dense_15 (Dense) (None, 960) 461760 re_lu_37[0][0]
__________________________________________________________________________________________________
tf.math.sigmoid_7 (TFOpLambda) (None, 960) 0 dense_15[0][0]
__________________________________________________________________________________________________
reshape_7 (Reshape) (None, 1, 1, 960) 0 tf.math.sigmoid_7[0][0]
__________________________________________________________________________________________________
tf.math.multiply_7 (TFOpLambda) (None, 7, 7, 960) 0 re_lu_36[0][0]
reshape_7[0][0]
__________________________________________________________________________________________________
conv2d_29 (Conv2D) (None, 7, 7, 160) 153600 tf.math.multiply_7[0][0]
__________________________________________________________________________________________________
batch_normalization_44 (BatchNo (None, 7, 7, 160) 640 conv2d_29[0][0]
__________________________________________________________________________________________________
conv2d_30 (Conv2D) (None, 7, 7, 960) 153600 batch_normalization_44[0][0]
__________________________________________________________________________________________________
batch_normalization_45 (BatchNo (None, 7, 7, 960) 3840 conv2d_30[0][0]
__________________________________________________________________________________________________
re_lu_38 (ReLU) (None, 7, 7, 960) 0 batch_normalization_45[0][0]
__________________________________________________________________________________________________
depthwise_conv2d_15 (DepthwiseC (None, 7, 7, 960) 8640 re_lu_38[0][0]
__________________________________________________________________________________________________
batch_normalization_46 (BatchNo (None, 7, 7, 960) 3840 depthwise_conv2d_15[0][0]
__________________________________________________________________________________________________
re_lu_39 (ReLU) (None, 7, 7, 960) 0 batch_normalization_46[0][0]
__________________________________________________________________________________________________
conv2d_31 (Conv2D) (None, 7, 7, 320) 307200 re_lu_39[0][0]
__________________________________________________________________________________________________
batch_normalization_47 (BatchNo (None, 7, 7, 320) 1280 conv2d_31[0][0]
__________________________________________________________________________________________________
conv2d_32 (Conv2D) (None, 7, 7, 1280) 409600 batch_normalization_47[0][0]
__________________________________________________________________________________________________
batch_normalization_48 (BatchNo (None, 7, 7, 1280) 5120 conv2d_32[0][0]
__________________________________________________________________________________________________
re_lu_40 (ReLU) (None, 7, 7, 1280) 0 batch_normalization_48[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_8 (Glo (None, 1280) 0 re_lu_40[0][0]
__________________________________________________________________________________________________
dense_16 (Dense) (None, 1000) 1281000 global_average_pooling2d_8[0][0]
==================================================================================================
Total params: 6,679,396
Trainable params: 6,645,908
Non-trainable params: 33,488
__________________________________________________________________________________________________