ConvFormer: Closing the Gap Between CNN and Vision Transformers

ConvFormer: Closing the Gap Between CNN and Vision Transformers

提出一个新颖的注意力机制：MCA，并且在此基础上提出一个神经网络：ConvFormer

Motivation

Generally, both CNN and ViT have their pros and cons.
CNN 和 ViT 各有优缺点
To this end, there is a trend to take the merits of both CNNs and ViTs by migrating desired properties of ViTs to CNNs, including the general architecture design, long-range dependency, and data specificity provided by the attention mechanism.
为此，有一个同时考虑CNN 和 ViT 的所需要特征的趋势，包括架构设计，长距离以来，数据特异性（？）
因此作者他们提出MCA，MCA combines small and large kernel sizes for difffferent resolution patterns, which is proved to improve accuracy and effiffifficiency in (Tan and Le, 2019b).

Method

• MCA的框架
  
  Expand Conv: 将输入的通道增加
  DConv: depth-wise convolution
  中间的1*1 conv：为了残差相加，将通道减少到与原输入一样