【神经网络】(14) MnasNet 代码复现,网络解析,附Tensorflow完整代码

各位同学好,今天和大家分享一下如何使用 Tensorflow 复现谷歌轻量化神经网络 MnasNet 

通常而言,移动端(手机)和终端(安防监控、无人驾驶)上的设备计算能力有限,无法搭载庞大的神经网络模型。我们需要减少模型参数量、减小计算量、更少的内存访问量、更少的能耗MobileNet、ShuffleNet 等轻量化网络就非常适合于一些计算能力受限的设备,通过牺牲少量准确率来换取更快的运算速度。

在之前的章节中,我介绍了一些经典的轻量化神经网络模型。本篇需要用到MobileNetV1、V2的相关知识,建议大家先学习一下这两个网络

MobileNetV1:https://blog.csdn.net/dgvv4/article/details/123415708

MobileNetV2:https://blog.csdn.net/dgvv4/article/details/123417739


1. MobileNet 核心知识回顾

由于 MnasNet 是介于 MobileNetV2 和 MobileNetV3 之间的网络,因此开始之前一定要熟悉 MobileNet 的一些概念。为了能帮助大家理解 MnasNet 网络模型,先简单复习一下 MobileNet 系列网络的核心部件。

1.1 深度可分离卷积

深度可分离卷积(depthwise separable convolution)可理解为:由 深度卷积(depthwise convolution)和 逐点卷积(pointwise convolution)构成。

深度卷积处理长宽方向的信息。输入图像有多少个通道就有多少个卷积核,每个通道都用自己对应的卷积核生成一张对应的特征图。输入和输出的通道数相同

逐点卷积处理跨通道方向的信息。采用的是普通卷积的方法,只不过卷积核size只有1*1一个卷积核负责所有的通道,滑动过程中对应像素值相乘再相加,输出特征图个数由卷积核个数决定。

【神经网络】(14) MnasNet 代码复现,网络解析,附Tensorflow完整代码_第1张图片


1.2 逆残差结构

MobileNetV2 使用了逆转残差模块。图片输入,先使用1x1卷积升维;然后在高维空间下使用深度卷积;最后使用1x1卷积降维降维时使用线性激活函数。当步长为 2(下采样阶段)时,没有残差连接。当步长为 1(基本模块)时,残差连接输入和输出的特征图

【神经网络】(14) MnasNet 代码复现,网络解析,附Tensorflow完整代码_第2张图片


 2. MnasNet 网络模型

2.1 模型创新点

(1)MnasNet 运用了多目标优化函数,兼顾了速度和精度。

真实的手机推理时间作为速度,准确度作为精度,构建多目标的优化函数,如下。T 是一个硬指标,规定的时间(如80ms内完成预测);LAT(m) 代表预测时间;w 总是小于0,表示预测时间越长计算结果越大

【神经网络】(14) MnasNet 代码复现,网络解析,附Tensorflow完整代码_第3张图片

若 \alpha =0, \beta =-1,横轴是延误,纵轴是目标函数。小于硬指标时w=\alpha,目标函数的值和延误无关,只与准确率有关。超过硬指标时w=\beta,目标函数急剧减少,此时目标函数和延误有关。超过硬指标就会被惩罚,生成的模型更加集中。

\alpha =\beta =-0.07,不管是低于硬指标还是超过硬指标,目标函数和延误的关系是平滑的,不会出现严厉的惩罚。模型搜索空间更多,搜索到更多样的速度和精度的权衡

【神经网络】(14) MnasNet 代码复现,网络解析,附Tensorflow完整代码_第4张图片

 (2)分层的神经架构搜索空间

将一个卷积神经网络模型分层7个Block,每一个Block内部的结构是一样的,但Block之间的结构不一样,使得我们能设计网络不同部位的深层和浅层的模型结构。

采用强化学习探索网络最优的卷积方式、卷积核大小、SE注意力机制、跨层连接方式、每个Block中的层数

使得不同Block的结构是多样的,使网络具有多样性。

【神经网络】(14) MnasNet 代码复现,网络解析,附Tensorflow完整代码_第5张图片


2.2 模型评价

论文中将其与MobileNet做比较,速度上 MnasNet 比 MobileNet 快1.8倍在不同的延误下,MnasNet 的准确率都要比 MobileNet 的准确率要高

【神经网络】(14) MnasNet 代码复现,网络解析,附Tensorflow完整代码_第6张图片

不同的宽度超参数和分辨率超参数下,MnasNet 的性能都超过 MobileNet

【神经网络】(14) MnasNet 代码复现,网络解析,附Tensorflow完整代码_第7张图片


3. 代码复现

3.1 网络结构

网络结构如下图所示。紫色卷积块是MobileNetV1的深度可分离卷积块,绿色卷积块是MobileNetV2的逆残差结构模块,红色卷积块是添加了SE注意力机制后的逆残差结构模块。

其中,MBConv6代表逆残差连接模块中1*1卷积上升通道数为原图像通道数的6倍;MBConv6模块右侧的 x2 代表该模块重复两次先进行1次下采样(stride=2),再进行一次基本模块(stride=1);有的模块需要下采样有的不需要,搭载网络时需要注意。

【神经网络】(14) MnasNet 代码复现,网络解析,附Tensorflow完整代码_第8张图片


3.2 搭建各个卷积模块

(1)标准卷积块

一个标准卷积块由 卷积层Conv + 批标准化层BN + 激活函数Relu 组成

#(1)标准卷积模块
def conv_block(input_tensor, filters, kernel_size, stride):

    # 普通卷积+BN+激活
    x = layers.Conv2D(filters = filters,  # 卷积核个数
                      kernel_size = kernel_size,  # 卷积核size
                      strides = stride,  # 步长
                      use_bias = False,  # 有BN层就不要偏置
                      padding = 'same')(input_tensor)  # 步长=1时特征图size不变,步长=2时特征图长宽减半

    x = layers.BatchNormalization()(x)  # 批标准化

    x = layers.ReLU()(x)  # relu激活函数
    
    return x  # 如果activation=False可以直接输出结果

(2)深度可分离卷积块

一个深度可分离卷积块由 一个深度卷积 和 一个1*1逐点卷积 组成

#(2)深度卷积
def depthwise_conv_block(input_tensor, kernel_size, stride):

    # 深度卷积只处理长宽方向的空间信息,输入输出的通道数相同
    x = layers.DepthwiseConv2D(kernel_size = kernel_size,  # 卷积核size
                               strides = stride,  # 步长
                               use_bias = False,  # 有BN层不要偏置
                               padding = 'same')(input_tensor)  # stride=1卷积过程中size不变

    x = layers.BatchNormalization()(x)  # 批标准化

    x = layers.ReLU()(x)  # 激活函数

    return x  # 返回深度卷积的特征图,个数保持不变


#(3)逐点卷积
def pointwise_conv_block(input_tensor, filters):

    # 1*1卷积只负责通道方向的信息融合,一个卷积核输出一张特征图
    x = layers.Conv2D(filters = filters,  # 卷积核个数,即输出特征图个数
                      kernel_size = (1,1),  # 卷积核size=1*1,不处理长宽方向的信息
                      strides = 1,  # 步长=1卷积过程中特征图size不变
                      padding = 'same',  # 卷积过程中size不变
                      use_bias = False)(input_tensor)  # 有BN层就不要偏置
    
    x = layers.BatchNormalization()(x)  # 批标准化

    # 不使用relu激活函数,线性激活
    return x


#(4)深度可分离卷积 == 深度卷积 + 逐点卷积
def sep_conv_block(input_tensor, kernel_size, stride, filters):

    # 深度卷积,处理长宽方向的空间信息,不关心跨通道信息
    x = depthwise_conv_block(input_tensor, kernel_size, stride)
    # 逐点卷积,处理跨通道信息,跨层信息交融
    x = pointwise_conv_block(x, filters)

    return x  # 返回深度可分离卷积输出特征图

(3)逆转残差结构模块

先1*1卷积升维;然后在高维空间下深度卷积;最后1*1卷积降维(线性激活函数)。当步长=1并且输出和输出特征图的shape相同,才能将输入和输出残差连接。

#(5)深度可分离卷积的逆转残差模块
# 1x1标准卷积升维N倍,然后深度卷积,再1x1逐点卷积降维
def inverted_res_block(input_tensor, expansion, kernel_size, stride, out_channel):
    
    # keras.backend.int_shape得到图像的shape,这里只需要最后一个维度即通道维度的大小
    in_channel = keras.backend.int_shape(input_tensor)[-1]

    # 调用自定义的标准卷积函数,上升通道数
    x = conv_block(input_tensor,  # 输入特征图
                   kernel_size = (1,1),  # 卷积核size
                   filters = in_channel*expansion,  # 通道上升为原来的expansion倍
                   stride = 1)
    
    # 调用自定义的深度卷积函数
    x = depthwise_conv_block(x, kernel_size=kernel_size, stride=stride)

    # 调用自定义的逐点卷积函数,下降通道数
    x = pointwise_conv_block(x, filters = out_channel)  # out_channel输出特征图数量

    # 如果步长=1,并且输入和输出的shape相同时,输入和输出残差连接
    if stride == 1 and input_tensor.shape==x.shape:
        output = layers.Add()([input_tensor, x])
        return output
    
    # 如果步长=2,直接输出逐点卷积后的结果
    else:
        return x

(4)添加了SE注意力机制后的逆残差结构模块

先1*1卷积升维;然后在高维空间下深度卷积;再通过SE注意力机制;最后1*1卷积降维(使用线性激活函数)。输入和输出shape相同时,用残差连接

#(7)定义压缩和激活方法SE
def squeeze_excitation(input_tensor):

    inputs = input_tensor   # 将特征图复制一份
    squeeze = inputs.shape[-1]/2  # 将特征图在通道维度上平分成两份,即压缩量为原通道的1/2
    excitation = inputs.shape[-1]  # 通道上升到原通道数大小
    
    # 如:[416,416,24]==>[None,24]
    x = layers.GlobalAveragePooling2D()(input_tensor)  # 全局平均池化
    # 如:[None,24]==>[None,12]
    x = layers.Dense(squeeze)(x)  # 全连接层,通道数减半
    # 激活函数,shape不变
    x = layers.ReLU()(x)
    # 如:[None,12]==>[None,24]
    x = layers.Dense(excitation)(x)  # 全连接层,通道数回升至原来
    # 激活函数,shape不变
    x = tf.nn.sigmoid(x)
    # 如:[None,24]==>[1,1,24]
    x = layers.Reshape(target_shape = (1,1,excitation))(x)
    # [416,416,24]*[1,1,24]==>[416,416,24]
    output = inputs * x  # 点乘,元素之间相乘,shape不变

    return output


#(8)应用压缩和激活方法后的深度可分离卷积的逆转残差模块
def inverted_se_res_block(input_tensor, expansion, kernel_size, stride, out_channel):
    # 就比inverted_res_block多了一个SE层,其他都一样

    # 得到输出特征图的通道数量
    in_channel = keras.backend.int_shape(input_tensor)[-1]

    # 1*1标准卷积模块,通道数上升expansion倍
    x = conv_block(input_tensor, filters=in_channel*expansion, 
                   kernel_size=(1,1), stride=1)

    # 深度卷积模块,输出的特征图的通道数不变
    x = depthwise_conv_block(x, kernel_size, stride)

    # SE模块
    x = squeeze_excitation(x)

    # 逐点卷积,1*1卷积下降通道数
    x = pointwise_conv_block(x, filters=out_channel)

    # 如果步长=1,并且输入和输出的shape相同时,需要残差连接输入和输出
    if stride == 1 and input_tensor.shape==x.shape:
        output = layers.Add()([input_tensor, x])
        return output
    
    # 如果步长=2,直接输出逐点卷积结果
    else:
        return x

3.3 完整代码展示

根据3.1网络模型结构图堆叠网络各层,结合上面的解释,代码中每行都有注释,有疑问的可在评论区留言。

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, Model

#(1)标准卷积模块
def conv_block(input_tensor, filters, kernel_size, stride):

    # 普通卷积+BN+激活
    x = layers.Conv2D(filters = filters,  # 卷积核个数
                      kernel_size = kernel_size,  # 卷积核size
                      strides = stride,  # 步长
                      use_bias = False,  # 有BN层就不要偏置
                      padding = 'same')(input_tensor)  # 步长=1时特征图size不变,步长=2时特征图长宽减半

    x = layers.BatchNormalization()(x)  # 批标准化

    x = layers.ReLU()(x)  # relu激活函数
    
    return x  # 如果activation=False可以直接输出结果


#(2)深度卷积
def depthwise_conv_block(input_tensor, kernel_size, stride):

    # 深度卷积只处理长宽方向的空间信息,输入输出的通道数相同
    x = layers.DepthwiseConv2D(kernel_size = kernel_size,  # 卷积核size
                               strides = stride,  # 步长
                               use_bias = False,  # 有BN层不要偏置
                               padding = 'same')(input_tensor)  # stride=1卷积过程中size不变

    x = layers.BatchNormalization()(x)  # 批标准化

    x = layers.ReLU()(x)  # 激活函数

    return x  # 返回深度卷积的特征图,个数保持不变


#(3)逐点卷积
def pointwise_conv_block(input_tensor, filters):

    # 1*1卷积只负责通道方向的信息融合,一个卷积核输出一张特征图
    x = layers.Conv2D(filters = filters,  # 卷积核个数,即输出特征图个数
                      kernel_size = (1,1),  # 卷积核size=1*1,不处理长宽方向的信息
                      strides = 1,  # 步长=1卷积过程中特征图size不变
                      padding = 'same',  # 卷积过程中size不变
                      use_bias = False)(input_tensor)  # 有BN层就不要偏置
    
    x = layers.BatchNormalization()(x)  # 批标准化

    # 不使用relu激活函数,线性激活
    return x


#(4)深度可分离卷积 == 深度卷积 + 逐点卷积
def sep_conv_block(input_tensor, kernel_size, stride, filters):

    # 深度卷积,处理长宽方向的空间信息,不关心跨通道信息
    x = depthwise_conv_block(input_tensor, kernel_size, stride)
    # 逐点卷积,处理跨通道信息,跨层信息交融
    x = pointwise_conv_block(x, filters)

    return x  # 返回深度可分离卷积输出特征图


#(5)深度可分离卷积的逆转残差模块
# 1x1标准卷积升维N倍,然后深度卷积,再1x1逐点卷积降维
def inverted_res_block(input_tensor, expansion, kernel_size, stride, out_channel):
    
    # keras.backend.int_shape得到图像的shape,这里只需要最后一个维度即通道维度的大小
    in_channel = keras.backend.int_shape(input_tensor)[-1]

    # 调用自定义的标准卷积函数,上升通道数
    x = conv_block(input_tensor,  # 输入特征图
                   kernel_size = (1,1),  # 卷积核size
                   filters = in_channel*expansion,  # 通道上升为原来的expansion倍
                   stride = 1)
    
    # 调用自定义的深度卷积函数
    x = depthwise_conv_block(x, kernel_size=kernel_size, stride=stride)

    # 调用自定义的逐点卷积函数,下降通道数
    x = pointwise_conv_block(x, filters = out_channel)  # out_channel输出特征图数量

    # 如果步长=1,并且输入和输出的shape相同时,输入和输出残差连接
    if stride == 1 and input_tensor.shape==x.shape:
        output = layers.Add()([input_tensor, x])
        return output
    
    # 如果步长=2,直接输出逐点卷积后的结果
    else:
        return x


#(6)一个MBConv模块是由一个下采样模块(stride=2)和若干个基本模块(stride=1)组成
def MBConv(input_tensor, expansion, kernel_size, filters, stride, num):

    # 一个下采样模块,也可能不需要下采样
    x = inverted_res_block(input_tensor, expansion, kernel_size, stride, out_channel=filters)

    # num-1个基本模块。num代表整个MBConv模块包含几个inverted_res_block模块
    for _ in range(1, num):
        x = inverted_res_block(x, expansion, kernel_size, stride=1, out_channel=filters)

    return x  # 返回MBConv卷积块的特征图


#(7)定义压缩和激活方法SE
def squeeze_excitation(input_tensor):

    inputs = input_tensor   # 将特征图复制一份
    squeeze = inputs.shape[-1]/2  # 将特征图在通道维度上平分成两份,即压缩量为原通道的1/2
    excitation = inputs.shape[-1]  # 通道上升到原通道数大小
    
    # 如:[416,416,24]==>[None,24]
    x = layers.GlobalAveragePooling2D()(input_tensor)  # 全局平均池化
    # 如:[None,24]==>[None,12]
    x = layers.Dense(squeeze)(x)  # 全连接层,通道数减半
    # 激活函数,shape不变
    x = layers.ReLU()(x)
    # 如:[None,12]==>[None,24]
    x = layers.Dense(excitation)(x)  # 全连接层,通道数回升至原来
    # 激活函数,shape不变
    x = tf.nn.sigmoid(x)
    # 如:[None,24]==>[1,1,24]
    x = layers.Reshape(target_shape = (1,1,excitation))(x)
    # [416,416,24]*[1,1,24]==>[416,416,24]
    output = inputs * x  # 点乘,元素之间相乘,shape不变

    return output


#(8)应用压缩和激活方法后的深度可分离卷积的逆转残差模块
def inverted_se_res_block(input_tensor, expansion, kernel_size, stride, out_channel):
    # 就比inverted_res_block多了一个SE层,其他都一样

    # 得到输出特征图的通道数量
    in_channel = keras.backend.int_shape(input_tensor)[-1]

    # 1*1标准卷积模块,通道数上升expansion倍
    x = conv_block(input_tensor, filters=in_channel*expansion, 
                   kernel_size=(1,1), stride=1)

    # 深度卷积模块,输出的特征图的通道数不变
    x = depthwise_conv_block(x, kernel_size, stride)

    # SE模块
    x = squeeze_excitation(x)

    # 逐点卷积,1*1卷积下降通道数
    x = pointwise_conv_block(x, filters=out_channel)

    # 如果步长=1,并且输入和输出的shape相同时,需要残差连接输入和输出
    if stride == 1 and input_tensor.shape==x.shape:
        output = layers.Add()([input_tensor, x])
        return output
    
    # 如果步长=2,直接输出逐点卷积结果
    else:
        return x


#(9)一个MBConv_SE模块是由一个下采样模块(stride=2)和若干个基本模块(stride=1)组成
def MBConv_SE(input_tensor, expansion, kernel_size, filters, stride, num):

    # 一个下采样模块,也可能不需要下采样
    x = inverted_se_res_block(input_tensor, expansion, kernel_size, stride, out_channel=filters)

    # num-1个基本模块。num代表整个MBConv模块包含几个inverted_res_block模块
    for _ in range(1, num):
        x = inverted_se_res_block(x, expansion, kernel_size, stride=1, out_channel=filters)

    return x  # 返回MBConv_SE卷积块的特征图


#(10)搭建主干网络
def MnasNet(input_shape, classes):

    # 构建网络输入tensor
    inputs = keras.Input(shape=input_shape)

    # [224,224,3]==>[112,112,32]
    x = conv_block(inputs, 32, kernel_size=(3,3), stride=2)
    # [112,112,32]==>[112,112,16]
    x = sep_conv_block(x, kernel_size=(3,3), stride=1, filters=16)
    # [112,112,16]==>[56,56,24]
    x = MBConv(x, expansion=6, kernel_size=(3,3), filters=24, stride=2, num=2)
    # [56,56,24]==>[28,28,40]
    x = MBConv_SE(x, expansion=3, kernel_size=(5,5), filters=40, stride=2, num=3)
    # [28,28,40]==>[14,14,80]
    x = MBConv(x, expansion=6, kernel_size=(3,3), filters=80, stride=2, num=4)
    # [14,14,80]==>[14,14,112]
    x = MBConv_SE(x, expansion=6, kernel_size=(3,3), filters=112, stride=1, num=2)
    # [14,14,112]==>[7,7,160]
    x = MBConv_SE(x, expansion=6, kernel_size=(5,5), filters=160, stride=2, num=3)
    # [7,7,160]==>[7,7,320]
    x = MBConv(x, expansion=6, kernel_size=(3,3), filters=320, stride=1, num=1)
    
    # 再进行一次标准卷积 [7,7,320]==>[7,7,1280]
    x = conv_block(x, filters=1280, kernel_size=(1,1), stride=1)
    # [7,7,1280]==>[None,1280]
    x = layers.GlobalAveragePooling2D()(x)
    # [None,1280]==>[None,1000]
    logits = layers.Dense(classes)(x)  # 为了网络稳定,训练时再使用Softmax函数

    # 完成网络构架
    model = Model(inputs, logits)

    return model


#(11)接收网络模型
if __name__ == '__main__':

    model = MnasNet(input_shape=[224,224,3], classes=1000)  # 给出输入图像shape和分类数

    # 查看网络结构
    model.summary()

3.4 模型架构展示

通过model.summary()查看网络框架,参数量六百万

Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 224, 224, 3) 0                                            
__________________________________________________________________________________________________
conv2d (Conv2D)                 (None, 112, 112, 32) 864         input_1[0][0]                    
__________________________________________________________________________________________________
batch_normalization (BatchNorma (None, 112, 112, 32) 128         conv2d[0][0]                     
__________________________________________________________________________________________________
re_lu (ReLU)                    (None, 112, 112, 32) 0           batch_normalization[0][0]        
__________________________________________________________________________________________________
depthwise_conv2d (DepthwiseConv (None, 112, 112, 32) 288         re_lu[0][0]                      
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 112, 112, 32) 128         depthwise_conv2d[0][0]           
__________________________________________________________________________________________________
re_lu_1 (ReLU)                  (None, 112, 112, 32) 0           batch_normalization_1[0][0]      
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 112, 112, 16) 512         re_lu_1[0][0]                    
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 112, 112, 16) 64          conv2d_1[0][0]                   
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, 112, 112, 96) 1536        batch_normalization_2[0][0]      
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 112, 112, 96) 384         conv2d_2[0][0]                   
__________________________________________________________________________________________________
re_lu_2 (ReLU)                  (None, 112, 112, 96) 0           batch_normalization_3[0][0]      
__________________________________________________________________________________________________
depthwise_conv2d_1 (DepthwiseCo (None, 56, 56, 96)   864         re_lu_2[0][0]                    
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 56, 56, 96)   384         depthwise_conv2d_1[0][0]         
__________________________________________________________________________________________________
re_lu_3 (ReLU)                  (None, 56, 56, 96)   0           batch_normalization_4[0][0]      
__________________________________________________________________________________________________
conv2d_3 (Conv2D)               (None, 56, 56, 24)   2304        re_lu_3[0][0]                    
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 56, 56, 24)   96          conv2d_3[0][0]                   
__________________________________________________________________________________________________
conv2d_4 (Conv2D)               (None, 56, 56, 144)  3456        batch_normalization_5[0][0]      
__________________________________________________________________________________________________
batch_normalization_6 (BatchNor (None, 56, 56, 144)  576         conv2d_4[0][0]                   
__________________________________________________________________________________________________
re_lu_4 (ReLU)                  (None, 56, 56, 144)  0           batch_normalization_6[0][0]      
__________________________________________________________________________________________________
depthwise_conv2d_2 (DepthwiseCo (None, 56, 56, 144)  1296        re_lu_4[0][0]                    
__________________________________________________________________________________________________
batch_normalization_7 (BatchNor (None, 56, 56, 144)  576         depthwise_conv2d_2[0][0]         
__________________________________________________________________________________________________
re_lu_5 (ReLU)                  (None, 56, 56, 144)  0           batch_normalization_7[0][0]      
__________________________________________________________________________________________________
conv2d_5 (Conv2D)               (None, 56, 56, 24)   3456        re_lu_5[0][0]                    
__________________________________________________________________________________________________
batch_normalization_8 (BatchNor (None, 56, 56, 24)   96          conv2d_5[0][0]                   
__________________________________________________________________________________________________
conv2d_6 (Conv2D)               (None, 56, 56, 72)   1728        batch_normalization_8[0][0]      
__________________________________________________________________________________________________
batch_normalization_9 (BatchNor (None, 56, 56, 72)   288         conv2d_6[0][0]                   
__________________________________________________________________________________________________
re_lu_6 (ReLU)                  (None, 56, 56, 72)   0           batch_normalization_9[0][0]      
__________________________________________________________________________________________________
depthwise_conv2d_3 (DepthwiseCo (None, 28, 28, 72)   1800        re_lu_6[0][0]                    
__________________________________________________________________________________________________
batch_normalization_10 (BatchNo (None, 28, 28, 72)   288         depthwise_conv2d_3[0][0]         
__________________________________________________________________________________________________
re_lu_7 (ReLU)                  (None, 28, 28, 72)   0           batch_normalization_10[0][0]     
__________________________________________________________________________________________________
global_average_pooling2d (Globa (None, 72)           0           re_lu_7[0][0]                    
__________________________________________________________________________________________________
dense (Dense)                   (None, 36)           2628        global_average_pooling2d[0][0]   
__________________________________________________________________________________________________
re_lu_8 (ReLU)                  (None, 36)           0           dense[0][0]                      
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 72)           2664        re_lu_8[0][0]                    
__________________________________________________________________________________________________
tf.math.sigmoid (TFOpLambda)    (None, 72)           0           dense_1[0][0]                    
__________________________________________________________________________________________________
reshape (Reshape)               (None, 1, 1, 72)     0           tf.math.sigmoid[0][0]            
__________________________________________________________________________________________________
tf.math.multiply (TFOpLambda)   (None, 28, 28, 72)   0           re_lu_7[0][0]                    
                                                                 reshape[0][0]                    
__________________________________________________________________________________________________
conv2d_7 (Conv2D)               (None, 28, 28, 40)   2880        tf.math.multiply[0][0]           
__________________________________________________________________________________________________
batch_normalization_11 (BatchNo (None, 28, 28, 40)   160         conv2d_7[0][0]                   
__________________________________________________________________________________________________
conv2d_8 (Conv2D)               (None, 28, 28, 120)  4800        batch_normalization_11[0][0]     
__________________________________________________________________________________________________
batch_normalization_12 (BatchNo (None, 28, 28, 120)  480         conv2d_8[0][0]                   
__________________________________________________________________________________________________
re_lu_9 (ReLU)                  (None, 28, 28, 120)  0           batch_normalization_12[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_4 (DepthwiseCo (None, 28, 28, 120)  3000        re_lu_9[0][0]                    
__________________________________________________________________________________________________
batch_normalization_13 (BatchNo (None, 28, 28, 120)  480         depthwise_conv2d_4[0][0]         
__________________________________________________________________________________________________
re_lu_10 (ReLU)                 (None, 28, 28, 120)  0           batch_normalization_13[0][0]     
__________________________________________________________________________________________________
global_average_pooling2d_1 (Glo (None, 120)          0           re_lu_10[0][0]                   
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 60)           7260        global_average_pooling2d_1[0][0] 
__________________________________________________________________________________________________
re_lu_11 (ReLU)                 (None, 60)           0           dense_2[0][0]                    
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 120)          7320        re_lu_11[0][0]                   
__________________________________________________________________________________________________
tf.math.sigmoid_1 (TFOpLambda)  (None, 120)          0           dense_3[0][0]                    
__________________________________________________________________________________________________
reshape_1 (Reshape)             (None, 1, 1, 120)    0           tf.math.sigmoid_1[0][0]          
__________________________________________________________________________________________________
tf.math.multiply_1 (TFOpLambda) (None, 28, 28, 120)  0           re_lu_10[0][0]                   
                                                                 reshape_1[0][0]                  
__________________________________________________________________________________________________
conv2d_9 (Conv2D)               (None, 28, 28, 40)   4800        tf.math.multiply_1[0][0]         
__________________________________________________________________________________________________
batch_normalization_14 (BatchNo (None, 28, 28, 40)   160         conv2d_9[0][0]                   
__________________________________________________________________________________________________
conv2d_10 (Conv2D)              (None, 28, 28, 120)  4800        batch_normalization_14[0][0]     
__________________________________________________________________________________________________
batch_normalization_15 (BatchNo (None, 28, 28, 120)  480         conv2d_10[0][0]                  
__________________________________________________________________________________________________
re_lu_12 (ReLU)                 (None, 28, 28, 120)  0           batch_normalization_15[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_5 (DepthwiseCo (None, 28, 28, 120)  3000        re_lu_12[0][0]                   
__________________________________________________________________________________________________
batch_normalization_16 (BatchNo (None, 28, 28, 120)  480         depthwise_conv2d_5[0][0]         
__________________________________________________________________________________________________
re_lu_13 (ReLU)                 (None, 28, 28, 120)  0           batch_normalization_16[0][0]     
__________________________________________________________________________________________________
global_average_pooling2d_2 (Glo (None, 120)          0           re_lu_13[0][0]                   
__________________________________________________________________________________________________
dense_4 (Dense)                 (None, 60)           7260        global_average_pooling2d_2[0][0] 
__________________________________________________________________________________________________
re_lu_14 (ReLU)                 (None, 60)           0           dense_4[0][0]                    
__________________________________________________________________________________________________
dense_5 (Dense)                 (None, 120)          7320        re_lu_14[0][0]                   
__________________________________________________________________________________________________
tf.math.sigmoid_2 (TFOpLambda)  (None, 120)          0           dense_5[0][0]                    
__________________________________________________________________________________________________
reshape_2 (Reshape)             (None, 1, 1, 120)    0           tf.math.sigmoid_2[0][0]          
__________________________________________________________________________________________________
tf.math.multiply_2 (TFOpLambda) (None, 28, 28, 120)  0           re_lu_13[0][0]                   
                                                                 reshape_2[0][0]                  
__________________________________________________________________________________________________
conv2d_11 (Conv2D)              (None, 28, 28, 40)   4800        tf.math.multiply_2[0][0]         
__________________________________________________________________________________________________
batch_normalization_17 (BatchNo (None, 28, 28, 40)   160         conv2d_11[0][0]                  
__________________________________________________________________________________________________
conv2d_12 (Conv2D)              (None, 28, 28, 240)  9600        batch_normalization_17[0][0]     
__________________________________________________________________________________________________
batch_normalization_18 (BatchNo (None, 28, 28, 240)  960         conv2d_12[0][0]                  
__________________________________________________________________________________________________
re_lu_15 (ReLU)                 (None, 28, 28, 240)  0           batch_normalization_18[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_6 (DepthwiseCo (None, 14, 14, 240)  2160        re_lu_15[0][0]                   
__________________________________________________________________________________________________
batch_normalization_19 (BatchNo (None, 14, 14, 240)  960         depthwise_conv2d_6[0][0]         
__________________________________________________________________________________________________
re_lu_16 (ReLU)                 (None, 14, 14, 240)  0           batch_normalization_19[0][0]     
__________________________________________________________________________________________________
conv2d_13 (Conv2D)              (None, 14, 14, 80)   19200       re_lu_16[0][0]                   
__________________________________________________________________________________________________
batch_normalization_20 (BatchNo (None, 14, 14, 80)   320         conv2d_13[0][0]                  
__________________________________________________________________________________________________
conv2d_14 (Conv2D)              (None, 14, 14, 480)  38400       batch_normalization_20[0][0]     
__________________________________________________________________________________________________
batch_normalization_21 (BatchNo (None, 14, 14, 480)  1920        conv2d_14[0][0]                  
__________________________________________________________________________________________________
re_lu_17 (ReLU)                 (None, 14, 14, 480)  0           batch_normalization_21[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_7 (DepthwiseCo (None, 14, 14, 480)  4320        re_lu_17[0][0]                   
__________________________________________________________________________________________________
batch_normalization_22 (BatchNo (None, 14, 14, 480)  1920        depthwise_conv2d_7[0][0]         
__________________________________________________________________________________________________
re_lu_18 (ReLU)                 (None, 14, 14, 480)  0           batch_normalization_22[0][0]     
__________________________________________________________________________________________________
conv2d_15 (Conv2D)              (None, 14, 14, 80)   38400       re_lu_18[0][0]                   
__________________________________________________________________________________________________
batch_normalization_23 (BatchNo (None, 14, 14, 80)   320         conv2d_15[0][0]                  
__________________________________________________________________________________________________
conv2d_16 (Conv2D)              (None, 14, 14, 480)  38400       batch_normalization_23[0][0]     
__________________________________________________________________________________________________
batch_normalization_24 (BatchNo (None, 14, 14, 480)  1920        conv2d_16[0][0]                  
__________________________________________________________________________________________________
re_lu_19 (ReLU)                 (None, 14, 14, 480)  0           batch_normalization_24[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_8 (DepthwiseCo (None, 14, 14, 480)  4320        re_lu_19[0][0]                   
__________________________________________________________________________________________________
batch_normalization_25 (BatchNo (None, 14, 14, 480)  1920        depthwise_conv2d_8[0][0]         
__________________________________________________________________________________________________
re_lu_20 (ReLU)                 (None, 14, 14, 480)  0           batch_normalization_25[0][0]     
__________________________________________________________________________________________________
conv2d_17 (Conv2D)              (None, 14, 14, 80)   38400       re_lu_20[0][0]                   
__________________________________________________________________________________________________
batch_normalization_26 (BatchNo (None, 14, 14, 80)   320         conv2d_17[0][0]                  
__________________________________________________________________________________________________
conv2d_18 (Conv2D)              (None, 14, 14, 480)  38400       batch_normalization_26[0][0]     
__________________________________________________________________________________________________
batch_normalization_27 (BatchNo (None, 14, 14, 480)  1920        conv2d_18[0][0]                  
__________________________________________________________________________________________________
re_lu_21 (ReLU)                 (None, 14, 14, 480)  0           batch_normalization_27[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_9 (DepthwiseCo (None, 14, 14, 480)  4320        re_lu_21[0][0]                   
__________________________________________________________________________________________________
batch_normalization_28 (BatchNo (None, 14, 14, 480)  1920        depthwise_conv2d_9[0][0]         
__________________________________________________________________________________________________
re_lu_22 (ReLU)                 (None, 14, 14, 480)  0           batch_normalization_28[0][0]     
__________________________________________________________________________________________________
conv2d_19 (Conv2D)              (None, 14, 14, 80)   38400       re_lu_22[0][0]                   
__________________________________________________________________________________________________
batch_normalization_29 (BatchNo (None, 14, 14, 80)   320         conv2d_19[0][0]                  
__________________________________________________________________________________________________
conv2d_20 (Conv2D)              (None, 14, 14, 480)  38400       batch_normalization_29[0][0]     
__________________________________________________________________________________________________
batch_normalization_30 (BatchNo (None, 14, 14, 480)  1920        conv2d_20[0][0]                  
__________________________________________________________________________________________________
re_lu_23 (ReLU)                 (None, 14, 14, 480)  0           batch_normalization_30[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_10 (DepthwiseC (None, 14, 14, 480)  4320        re_lu_23[0][0]                   
__________________________________________________________________________________________________
batch_normalization_31 (BatchNo (None, 14, 14, 480)  1920        depthwise_conv2d_10[0][0]        
__________________________________________________________________________________________________
re_lu_24 (ReLU)                 (None, 14, 14, 480)  0           batch_normalization_31[0][0]     
__________________________________________________________________________________________________
global_average_pooling2d_3 (Glo (None, 480)          0           re_lu_24[0][0]                   
__________________________________________________________________________________________________
dense_6 (Dense)                 (None, 240)          115440      global_average_pooling2d_3[0][0] 
__________________________________________________________________________________________________
re_lu_25 (ReLU)                 (None, 240)          0           dense_6[0][0]                    
__________________________________________________________________________________________________
dense_7 (Dense)                 (None, 480)          115680      re_lu_25[0][0]                   
__________________________________________________________________________________________________
tf.math.sigmoid_3 (TFOpLambda)  (None, 480)          0           dense_7[0][0]                    
__________________________________________________________________________________________________
reshape_3 (Reshape)             (None, 1, 1, 480)    0           tf.math.sigmoid_3[0][0]          
__________________________________________________________________________________________________
tf.math.multiply_3 (TFOpLambda) (None, 14, 14, 480)  0           re_lu_24[0][0]                   
                                                                 reshape_3[0][0]                  
__________________________________________________________________________________________________
conv2d_21 (Conv2D)              (None, 14, 14, 112)  53760       tf.math.multiply_3[0][0]         
__________________________________________________________________________________________________
batch_normalization_32 (BatchNo (None, 14, 14, 112)  448         conv2d_21[0][0]                  
__________________________________________________________________________________________________
conv2d_22 (Conv2D)              (None, 14, 14, 672)  75264       batch_normalization_32[0][0]     
__________________________________________________________________________________________________
batch_normalization_33 (BatchNo (None, 14, 14, 672)  2688        conv2d_22[0][0]                  
__________________________________________________________________________________________________
re_lu_26 (ReLU)                 (None, 14, 14, 672)  0           batch_normalization_33[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_11 (DepthwiseC (None, 14, 14, 672)  6048        re_lu_26[0][0]                   
__________________________________________________________________________________________________
batch_normalization_34 (BatchNo (None, 14, 14, 672)  2688        depthwise_conv2d_11[0][0]        
__________________________________________________________________________________________________
re_lu_27 (ReLU)                 (None, 14, 14, 672)  0           batch_normalization_34[0][0]     
__________________________________________________________________________________________________
global_average_pooling2d_4 (Glo (None, 672)          0           re_lu_27[0][0]                   
__________________________________________________________________________________________________
dense_8 (Dense)                 (None, 336)          226128      global_average_pooling2d_4[0][0] 
__________________________________________________________________________________________________
re_lu_28 (ReLU)                 (None, 336)          0           dense_8[0][0]                    
__________________________________________________________________________________________________
dense_9 (Dense)                 (None, 672)          226464      re_lu_28[0][0]                   
__________________________________________________________________________________________________
tf.math.sigmoid_4 (TFOpLambda)  (None, 672)          0           dense_9[0][0]                    
__________________________________________________________________________________________________
reshape_4 (Reshape)             (None, 1, 1, 672)    0           tf.math.sigmoid_4[0][0]          
__________________________________________________________________________________________________
tf.math.multiply_4 (TFOpLambda) (None, 14, 14, 672)  0           re_lu_27[0][0]                   
                                                                 reshape_4[0][0]                  
__________________________________________________________________________________________________
conv2d_23 (Conv2D)              (None, 14, 14, 112)  75264       tf.math.multiply_4[0][0]         
__________________________________________________________________________________________________
batch_normalization_35 (BatchNo (None, 14, 14, 112)  448         conv2d_23[0][0]                  
__________________________________________________________________________________________________
conv2d_24 (Conv2D)              (None, 14, 14, 672)  75264       batch_normalization_35[0][0]     
__________________________________________________________________________________________________
batch_normalization_36 (BatchNo (None, 14, 14, 672)  2688        conv2d_24[0][0]                  
__________________________________________________________________________________________________
re_lu_29 (ReLU)                 (None, 14, 14, 672)  0           batch_normalization_36[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_12 (DepthwiseC (None, 7, 7, 672)    16800       re_lu_29[0][0]                   
__________________________________________________________________________________________________
batch_normalization_37 (BatchNo (None, 7, 7, 672)    2688        depthwise_conv2d_12[0][0]        
__________________________________________________________________________________________________
re_lu_30 (ReLU)                 (None, 7, 7, 672)    0           batch_normalization_37[0][0]     
__________________________________________________________________________________________________
global_average_pooling2d_5 (Glo (None, 672)          0           re_lu_30[0][0]                   
__________________________________________________________________________________________________
dense_10 (Dense)                (None, 336)          226128      global_average_pooling2d_5[0][0] 
__________________________________________________________________________________________________
re_lu_31 (ReLU)                 (None, 336)          0           dense_10[0][0]                   
__________________________________________________________________________________________________
dense_11 (Dense)                (None, 672)          226464      re_lu_31[0][0]                   
__________________________________________________________________________________________________
tf.math.sigmoid_5 (TFOpLambda)  (None, 672)          0           dense_11[0][0]                   
__________________________________________________________________________________________________
reshape_5 (Reshape)             (None, 1, 1, 672)    0           tf.math.sigmoid_5[0][0]          
__________________________________________________________________________________________________
tf.math.multiply_5 (TFOpLambda) (None, 7, 7, 672)    0           re_lu_30[0][0]                   
                                                                 reshape_5[0][0]                  
__________________________________________________________________________________________________
conv2d_25 (Conv2D)              (None, 7, 7, 160)    107520      tf.math.multiply_5[0][0]         
__________________________________________________________________________________________________
batch_normalization_38 (BatchNo (None, 7, 7, 160)    640         conv2d_25[0][0]                  
__________________________________________________________________________________________________
conv2d_26 (Conv2D)              (None, 7, 7, 960)    153600      batch_normalization_38[0][0]     
__________________________________________________________________________________________________
batch_normalization_39 (BatchNo (None, 7, 7, 960)    3840        conv2d_26[0][0]                  
__________________________________________________________________________________________________
re_lu_32 (ReLU)                 (None, 7, 7, 960)    0           batch_normalization_39[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_13 (DepthwiseC (None, 7, 7, 960)    24000       re_lu_32[0][0]                   
__________________________________________________________________________________________________
batch_normalization_40 (BatchNo (None, 7, 7, 960)    3840        depthwise_conv2d_13[0][0]        
__________________________________________________________________________________________________
re_lu_33 (ReLU)                 (None, 7, 7, 960)    0           batch_normalization_40[0][0]     
__________________________________________________________________________________________________
global_average_pooling2d_6 (Glo (None, 960)          0           re_lu_33[0][0]                   
__________________________________________________________________________________________________
dense_12 (Dense)                (None, 480)          461280      global_average_pooling2d_6[0][0] 
__________________________________________________________________________________________________
re_lu_34 (ReLU)                 (None, 480)          0           dense_12[0][0]                   
__________________________________________________________________________________________________
dense_13 (Dense)                (None, 960)          461760      re_lu_34[0][0]                   
__________________________________________________________________________________________________
tf.math.sigmoid_6 (TFOpLambda)  (None, 960)          0           dense_13[0][0]                   
__________________________________________________________________________________________________
reshape_6 (Reshape)             (None, 1, 1, 960)    0           tf.math.sigmoid_6[0][0]          
__________________________________________________________________________________________________
tf.math.multiply_6 (TFOpLambda) (None, 7, 7, 960)    0           re_lu_33[0][0]                   
                                                                 reshape_6[0][0]                  
__________________________________________________________________________________________________
conv2d_27 (Conv2D)              (None, 7, 7, 160)    153600      tf.math.multiply_6[0][0]         
__________________________________________________________________________________________________
batch_normalization_41 (BatchNo (None, 7, 7, 160)    640         conv2d_27[0][0]                  
__________________________________________________________________________________________________
conv2d_28 (Conv2D)              (None, 7, 7, 960)    153600      batch_normalization_41[0][0]     
__________________________________________________________________________________________________
batch_normalization_42 (BatchNo (None, 7, 7, 960)    3840        conv2d_28[0][0]                  
__________________________________________________________________________________________________
re_lu_35 (ReLU)                 (None, 7, 7, 960)    0           batch_normalization_42[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_14 (DepthwiseC (None, 7, 7, 960)    24000       re_lu_35[0][0]                   
__________________________________________________________________________________________________
batch_normalization_43 (BatchNo (None, 7, 7, 960)    3840        depthwise_conv2d_14[0][0]        
__________________________________________________________________________________________________
re_lu_36 (ReLU)                 (None, 7, 7, 960)    0           batch_normalization_43[0][0]     
__________________________________________________________________________________________________
global_average_pooling2d_7 (Glo (None, 960)          0           re_lu_36[0][0]                   
__________________________________________________________________________________________________
dense_14 (Dense)                (None, 480)          461280      global_average_pooling2d_7[0][0] 
__________________________________________________________________________________________________
re_lu_37 (ReLU)                 (None, 480)          0           dense_14[0][0]                   
__________________________________________________________________________________________________
dense_15 (Dense)                (None, 960)          461760      re_lu_37[0][0]                   
__________________________________________________________________________________________________
tf.math.sigmoid_7 (TFOpLambda)  (None, 960)          0           dense_15[0][0]                   
__________________________________________________________________________________________________
reshape_7 (Reshape)             (None, 1, 1, 960)    0           tf.math.sigmoid_7[0][0]          
__________________________________________________________________________________________________
tf.math.multiply_7 (TFOpLambda) (None, 7, 7, 960)    0           re_lu_36[0][0]                   
                                                                 reshape_7[0][0]                  
__________________________________________________________________________________________________
conv2d_29 (Conv2D)              (None, 7, 7, 160)    153600      tf.math.multiply_7[0][0]         
__________________________________________________________________________________________________
batch_normalization_44 (BatchNo (None, 7, 7, 160)    640         conv2d_29[0][0]                  
__________________________________________________________________________________________________
conv2d_30 (Conv2D)              (None, 7, 7, 960)    153600      batch_normalization_44[0][0]     
__________________________________________________________________________________________________
batch_normalization_45 (BatchNo (None, 7, 7, 960)    3840        conv2d_30[0][0]                  
__________________________________________________________________________________________________
re_lu_38 (ReLU)                 (None, 7, 7, 960)    0           batch_normalization_45[0][0]     
__________________________________________________________________________________________________
depthwise_conv2d_15 (DepthwiseC (None, 7, 7, 960)    8640        re_lu_38[0][0]                   
__________________________________________________________________________________________________
batch_normalization_46 (BatchNo (None, 7, 7, 960)    3840        depthwise_conv2d_15[0][0]        
__________________________________________________________________________________________________
re_lu_39 (ReLU)                 (None, 7, 7, 960)    0           batch_normalization_46[0][0]     
__________________________________________________________________________________________________
conv2d_31 (Conv2D)              (None, 7, 7, 320)    307200      re_lu_39[0][0]                   
__________________________________________________________________________________________________
batch_normalization_47 (BatchNo (None, 7, 7, 320)    1280        conv2d_31[0][0]                  
__________________________________________________________________________________________________
conv2d_32 (Conv2D)              (None, 7, 7, 1280)   409600      batch_normalization_47[0][0]     
__________________________________________________________________________________________________
batch_normalization_48 (BatchNo (None, 7, 7, 1280)   5120        conv2d_32[0][0]                  
__________________________________________________________________________________________________
re_lu_40 (ReLU)                 (None, 7, 7, 1280)   0           batch_normalization_48[0][0]     
__________________________________________________________________________________________________
global_average_pooling2d_8 (Glo (None, 1280)         0           re_lu_40[0][0]                   
__________________________________________________________________________________________________
dense_16 (Dense)                (None, 1000)         1281000     global_average_pooling2d_8[0][0] 
==================================================================================================
Total params: 6,679,396
Trainable params: 6,645,908
Non-trainable params: 33,488
__________________________________________________________________________________________________

你可能感兴趣的:(TensorFlow神经网络,tensorflow,神经网络,深度学习,python,cnn)