kaggle论文阅读

Swin Transformer

https://arxiv.org/abs/2103.14030

优点

1. brings greater efficiency with linear complexity by limiting self-attention computation to non-overlapping local windows while also providing a cross-window bridge. Lower complexity means lower latency and less resource, thus allowing a more efficient implementation in general hardware and handling high resolution image.

2. hierarchical feature maps, solve the problem that image scale can vary which not exists in nlp because tokens are almost fixed.

3. speed-accuracy tradeoff method , but linear complexity compared to ViT’s quadratic.

4. to make a backbone to cv as transformer to nlp

5. a step toward It author’s belief that a unified architecture across computer vision and natural language processing could benefit both fields

6. 对edge small windows cyclic shift 替代了small window padding，没有多余的窗口，提高了效率。 而且是均匀、对称的。

   

7. 在某些模型（MLP-Mixer）上同时提高速度和降低内存

缺点

1. 作者的unified model（多模态）构想没有实现，这篇论文的价值只是个技术验证作用
2. pytorch built-in function is not well-optimized
3. 丢弃了nlp transformer的global attention，肯定有损失

缺点&改进

这是作者的图

其中边角这块使用的是cyclic shift算的

根据卷积核的思想，图片的特征主要是跟相邻的pixel有关，绿圈的几个区块算attention权值都挺小，意义不大。去掉试试？由于是hierachy结构，去掉也许能直接省掉1/4的时间，感觉精度可能受的影响较小?

即只对右图中的三个蓝色部分进行计算

而且去掉之后四个角还是有办法过来计算相互之间的attention的

可能过来的代价比较大