单目图像深度估计算法-FastDepth

基于深度学习的单目深度估计在近几年是比较热门的研究方向之一,MIT的Diana Wofk等人在ICRA 2019上提出了一种用于嵌入式系统的深度估计算法FastDepth,在保证准确率的情况下,大大提高了模型的计算效率。

论文:FastDepth: Fast Monocular Depth Estimation on Embedded Systems
Offical Pytorch:https://github.com/dwofk/fast-depth

方法

模型

模型的整体结构比较简单,采用了Encoder-Decoder的架构。Encoder部分采用了MobileNet模型提取到7x7x1024的特征;Decoder部分采用了5次上采样,中间三次上采样结果通过Skip Connections的方法分别与Encoder部分的特征进行了特征融合,为了减小上采样部分的通道特征,还使用了5x5的卷积来降维;最后使用1*1的卷积得到深度图。

单目图像深度估计算法-FastDepth_第1张图片
Model

使用Keras实现基本的FastDepth模型:

from keras.layers import Conv2D, UpSampling2D, SeparableConv2D, BatchNormalization, Activation, add
from keras.models import Model
from keras.applications.mobilenet import MobileNet

     
class FastDepth:
    def __init__(self):
        self.build_net()

    def _SDWConv(self, filtres, kernel):
        def f(x):
            x = SeparableConv2D(filtres, kernel, padding='same')(x)
            x = BatchNormalization()(x)
            x = Activation('relu')(x)

            return x
        return f

    def _encoder(self):
        self.MN = MobileNet(input_shape=(224, 224, 3),
                            weights=None,
                            include_top='False')

        # 7*7*1024
        latent = self.MN.get_layer('conv_pw_13_relu').output

        return latent

    def _decoder(self, x):
        # 14*14*512
        x1 = self._SDWConv(512, (5, 5))(x)
        x1 = UpSampling2D()(x1)

        # 28*28*256
        x2 = self._SDWConv(256, (5, 5))(x1)
        x2 = UpSampling2D()(x2)
        s2 = self.MN.get_layer('conv_pw_5_relu').output
        x2 = add([x2, s2])

        # 56*56*128
        x3 = self._SDWConv(128, (5, 5))(x2)
        x3 = UpSampling2D()(x3)
        s3 = self.MN.get_layer('conv_pw_3_relu').output
        x3 = add([x3, s3])

        # 112*112*64
        x4 = self._SDWConv(64, (5, 5))(x3)
        x4 = UpSampling2D()(x4)
        s4 = self.MN.get_layer('conv_pw_1_relu').output
        x4 = add([x4, s4])

        # 224*224*32
        x5 = self._SDWConv(32, (5, 5))(x4)
        x5 = UpSampling2D()(x5)

        return x5

    def build_net(self):
        latent = self._encoder()
        out = self._decoder(latent)
        out_dense = Conv2D(1, (1, 1))(out)

        self.model = Model(inputs=self.MN.input, outputs=out_dense)


if __name__ == '__main__':
    net = FastDepth()
    net.model.summary()

Decoder部分的结构如下所示:

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
 省略MobileNet...
__________________________________________________________________________________________________  
conv_pw_13_relu (ReLU)          (None, 7, 7, 1024)   0           conv_pw_13_bn[0][0]              
__________________________________________________________________________________________________
separable_conv2d_3 (SeparableCo (None, 7, 7, 512)    550400      conv_pw_13_relu[0][0]            
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 7, 7, 512)    2048        separable_conv2d_3[0][0]         
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 7, 7, 512)    0           batch_normalization_1[0][0]      
__________________________________________________________________________________________________
up_sampling2d_3 (UpSampling2D)  (None, 14, 14, 512)  0           activation_1[0][0]               
__________________________________________________________________________________________________
separable_conv2d_4 (SeparableCo (None, 14, 14, 256)  144128      up_sampling2d_3[0][0]            
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 14, 14, 256)  1024        separable_conv2d_4[0][0]         
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 14, 14, 256)  0           batch_normalization_2[0][0]      
__________________________________________________________________________________________________
up_sampling2d_4 (UpSampling2D)  (None, 28, 28, 256)  0           activation_2[0][0]               
__________________________________________________________________________________________________
add_2 (Add)                     (None, 28, 28, 256)  0           up_sampling2d_4[0][0]            
                                                                 conv_pw_5_relu[0][0]             
__________________________________________________________________________________________________
separable_conv2d_5 (SeparableCo (None, 28, 28, 128)  39296       add_2[0][0]                      
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 28, 28, 128)  512         separable_conv2d_5[0][0]         
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 28, 28, 128)  0           batch_normalization_3[0][0]      
__________________________________________________________________________________________________
up_sampling2d_5 (UpSampling2D)  (None, 56, 56, 128)  0           activation_3[0][0]               
__________________________________________________________________________________________________
add_3 (Add)                     (None, 56, 56, 128)  0           up_sampling2d_5[0][0]            
                                                                 conv_pw_3_relu[0][0]             
__________________________________________________________________________________________________
separable_conv2d_6 (SeparableCo (None, 56, 56, 64)   11456       add_3[0][0]                      
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 56, 56, 64)   256         separable_conv2d_6[0][0]         
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 56, 56, 64)   0           batch_normalization_4[0][0]      
__________________________________________________________________________________________________
up_sampling2d_6 (UpSampling2D)  (None, 112, 112, 64) 0           activation_4[0][0]               
__________________________________________________________________________________________________
add_4 (Add)                     (None, 112, 112, 64) 0           up_sampling2d_6[0][0]            
                                                                 conv_pw_1_relu[0][0]             
__________________________________________________________________________________________________
separable_conv2d_7 (SeparableCo (None, 112, 112, 32) 3680        add_4[0][0]                      
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 112, 112, 32) 128         separable_conv2d_7[0][0]         
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 112, 112, 32) 0           batch_normalization_5[0][0]      
__________________________________________________________________________________________________
up_sampling2d_7 (UpSampling2D)  (None, 224, 224, 32) 0           activation_5[0][0]               
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 224, 224, 1)  33          up_sampling2d_7[0][0]            
==================================================================================================
Total params: 3,981,825
Trainable params: 3,957,953
Non-trainable params: 23,872
__________________________________________________________________________________________________

网络裁剪

为了减小模型体积,提高运算效率,使得模型更适用于嵌入式设备,使用NetAdapt算法对FastDepth进行了裁剪。

单目图像深度估计算法-FastDepth_第2张图片
NerAdapt

实验结果

模型在NYU Depth V2 dataset上进行了训练,基本实验结果如下图所示。可以看出论文提出的FastDepth算法相较当前准确率最高的算法低了4%,但是运算速度有着大幅提升,因此特别适用于嵌入式设备。

单目图像深度估计算法-FastDepth_第3张图片
GPU
单目图像深度估计算法-FastDepth_第4张图片
error.png

下图是深度估计的可视化效果:

单目图像深度估计算法-FastDepth_第5张图片
vis

下图是不同方法下Encoder和Decoder部分的运算效率和准确率,可以看出论文提出的方法运算速度非常快,而且Depthwise、Skip Connections和网络裁剪这三个技巧可以大幅提高运算效率而且对准确率的影响比较小。

单目图像深度估计算法-FastDepth_第6张图片
speed

你可能感兴趣的:(单目图像深度估计算法-FastDepth)