Self-Attention Generative Adversarial Networks笔记

中心思想

The self-attention module is complementary to convolutions and helps with modeling long range, multi-level dependencies across image regions. Armed with self-attention, the generator can draw images in which fine details at
every location are carefully coordinated with fine details in distant portions of the image. Moreover, the discriminator can also more accurately enforce complicated geometric constraints on the global image structure.

Self-Attention Generative Adversarial Networks

注意力的计算：

注意力层的输出：

类似残差结构，attention层的输出加上输入：

训练目标：
In SAGAN, the proposed attention module has been applied to both generator and discriminator, which are trained in an alternating fashion by minimizing the hinge version of the adversarial loss

测试生成图像质量的指标

Inception score

Fréchet Inception distance

稳定GAN训练的tricks

文章指出下面两个技巧是有效的。

Spectral normalization

这个技巧用在generator和discriminator，用于加速训练和稳定训练。

two-timescale update rule (TTUR)

用于加速regularized discriminator的训练。

文章中出现的一些名词

wall-clock time

对比wall-clock time和cpu time来理解。
Wall-clock time is the time that a clock on the wall (or a stopwatch in hand) would measure as having elapsed between the start of the process and ‘now’.
The user-cpu time and system-cpu time is the amount of time spent in user code and the amount of time spent in kernel code.
The wall-clock time is not the number of seconds that the process has spent on the CPU; it is the elapsed time, including time spent waiting for its turn on the CPU (while other processes get to run).