Transformer系列：Shunted self-attention (CVPR2022 oral)

文章地址：https://arxiv.org/abs/2111.15193

1. Motivation

ViT的每层特征的感受野大小是相似的，导致无法处理多尺度目标大小的任务。

2. Contribution

提出SSA，将attention head分组，每组负责不同的attention granularity，来处理hybrid-scale attention

3. Methods

 3.1 Shunted transformer block

Shunted self-attention: multi-head self-attention中不同head的key和value采用不同的下采样率

Data-specific feedforward layers: 在原先的两层point-wise convolution的中间多加了一层depth-wise convolution作为residual branch

 

 3.2  Patch embedding

相比于ViT中直接将图像裁剪成16 * 16 non-overlapp patch的方式，用convolution来产生token sequence更好。

 3.3 Architecture Details

 

4. Experiments

 4.1 Image classification

Dataset：ImageNet-1K

Optimizer： AdamW

Epoch： 300

Batch size：1024

Learning rate：1*10^-3，cosine learning rate decay

Data augmentation: follow DeiT, including random cropping, random flipping, label smoothing, Mixup, CutMix and random erasing

Results: our model is the first transformer based model that achieves comparable results with EfficientNet

 4.2 Object Detection and Instance Segmentation

Dataset: COCO2017

Setting: pretraining on ImageNet-1K and fine-tuning on COCO, using Mask R-CNN

Optimizer： AdamW

Epoch : 1× with 12 epochs and 3× with 36 epochs.

Image size: In 1× schedule, the shorter side of the input image will be resize to 800 while keeping the longer side no more than 1333. In 3× schedule, we take multi-scale training strategy of resizing the shorter size between 480 to 800

Batch size：16

Learning rate：1*10^-4

4.3 Semantic Segmentation

Dataset: ADE20K

Optimizer： AdamW

Framework : UperNet and Semantic FPN

Epoch : 160K iterations for UperNet , 80K for  Semantic FPN

Learning rate：for UperNet , 6*10^-5 with 1500 iteration warmup at the begining of training and linear learning rate decay; for Semantic FPN, 0.0001

Data augmentation: for UperNet , random flipping, random scaling and random photo-metric distortion.

4.4 Ablation Study