SAM（空间注意力机制）

---

SAM-Model(pytorch版本)

---

• 1.一 论文导读
• 2.二 论文精读
• 3.三 代码实现
• 4.四 问题思索

《An Empirical Study of Spatial Attention Mechanisms in Deep Networks》
—空间注意力机制
作者：Xizhou Zhu, Dazhi Cheng,etc.
单位：微软亚研
发表会议及时间：arXiv 2019

论文地址
相关代码

Submission history
From: Jifeng Dai [view email]
[v1] Thu, 11 Apr 2019 17:58:37 UTC (696 KB)

一 论文导读

二 论文精读

---

• Abstract

注意机制已成为深度神经网络中流行的组成部分，但很少有研究探讨不同的影响因素以及从这些因素计算注意力的方法如何影响性能。

为了更好地理解注意力机制，我们提供了一项经验研究，可以消除广义注意力公式中的各种空间注意力元素，包括占主导地位的Transformer注意力以及流行的可变形卷积和动态卷积模块。

在各种应用中进行的研究得出了有关深层网络中空间注意力的重要发现，其中一些与传统理解背道而驰。例如，我们发现Transformer注意中的查询和关键内容比较对于自我注意而言是微不足道的，但对于注意编码器-解码器至关重要。变形卷积与仅关键内容的显着性的适当组合可以实现最佳的自我注意效率-效率折衷。

我们的结果表明，注意力机制的设计存在很大的改进空间。

---

Attention mechanisms have become a popular component in deep neural networks, yet there has been little examination of how different influencing factors and methods for computing attention from these factors affect performance. Toward a better general understanding of attention mechanisms, we present an empirical study that ablates various spatial attention elements within a generalized attention formulation, encompassing the dominant Transformer attention as well as the prevalent deformable convolution and dynamic convolution modules. Conducted on a variety of applications, the study yields significant findings about spatial attention in deep networks, some of which run counter to conventional understanding. For example, we find that the query and key content comparison in Transformer attention is negligible for self-attention, but vital for encoder-decoder attention. A proper combination of deformable convolution with key content only saliency achieves the best accuracy-efficiency tradeoff in self-attention. Our results suggest that there exists much room for improvement in the design of attention mechanisms.

---

---

---

三 代码实现

四 问题思索