SaGAN:Generative Adversarial Network with Spatial Attention for Face Attribute Editing

#Problem

1. Traditional GAN methods directly operate on the whole image, and inevitably change the attribute-irrelevant regions
2. The performance of traditional regression methods heavily depends on the paired training data, which are however quite difficult to acquire

#Relative work

• ResGAN: learning the residual image avoids changing the attribute-irrelevant region by restraining most regions of the residual image as zero.

• Improvement:This work is quite insightful to enforce the manipulation mainly concentrate on local areas especially for those local attributes.

• Drawback: the location and the appearance of target attributes are modeled in single sparse residual image which is actually hard for a favorable optimization than modeling them separately

#Method

1. SaGAN: only alter the attribute specific region and keep the rest unchanged
2. The generator contains an attribute manipulation network (AMN) to edit the face image, and a spatial attention network (SAN) to localize the attribute-speciic region which restricts the alternation of AMN within this region.

#Contribution

1. The spatial attention is introduced to the GAN framework, forming an end-to-end generative model for face attribute editing (referred to as SaGAN),which can only alter those attribute-speciic region and keep the rest irrelevant region remain the same.
2. The proposed SaGAN adopts single generator with attribute as conditional signal rather than two dual ones for two inverse face attribute editing.
3. The proposed SaGAN achieves quite promising results especially for those local attributes with the attribute-irrelevant details well preserved. Besides, our approach also benefits the face recognition by data augmentation.

#Generative Adversarial Network with Spatial Attention

notation	meaning
$I$	input image
$\hat{I}$	output image
$I_a$	an edited face image output by AMN
$c$	attribute value
$c_g$	ground truth attribute label of the real image $I$
$D_{src}(I)$	probability of an image $I$ to be a real one
$D_{cls}(c\Vert I)$	probability of an image $I$ with the attribute $c$
$F_m$	an attribute manipulation network (AMN)
$F_a$	a spatial attention network(SAN)
$b$	a spatial attention mask, used to restrict the alternation of AMN within this region
${\lambda }_1$	balance parameters
${\lambda }_2$	balance parameters
${\lambda }_{gp}$	hyper-parameters control the gradient penalty, default = 10

• the goal of face attribute editing is to translate I into an
  new image $\hat{I}$, which should be realistic, with attribute c and look the same as the input image excluding the attribute-specific region

##Discriminator

• Two objectives, one to distinguish the generated images from the real ones, and another to classify the attributes of the generated and real images
• The two classifiers are both designed as a CNN with softmax function, denoted as $D_{src}$ and $D_{cls}$ respectively.
• The two networks can share the first few convolutional layers followed by distinct fully-connected layers for different classifications
  KaTeX parse error: Got function '\hat' with no arguments as subscript at position 61: …(I)]+\mathbb{E}_̲\hat{I}(log(1-D…
  $${\mathcal{L}}_{cls}^D={\mathbb{E}}_{I,c^g}[-log{D}_{cls}(c^g|I)]$$
  discriminator D:
  $$\underset{D_{src},D_{cls}}{\min }{\mathcal{L}}_D={\mathcal{L}}_{src}^D+{\mathcal{L}}_{cls}^D$$

##Generator

• G contains two modules, an attribute manipulation network(AMN) and a spatial attention network(SAN)
• AMN focuses on how to manipulate and SAN focuses on where to manipulate.
• The attribute manipulation network takes a face image $I$ and an attribute value $c$ as input, and outputs an edited face image $I_a$
  $$I_a=F_m(I,c)$$
• The spatial attention network takes the face image $I$ as input, and predict a spatial attention mask $b$, which is used to restrict the alternation of AMN within this region
• Ideally, the attribute-specific region of $b$ should be 1, and the rest regions should be 0.
• Regions with non-zeros attention values are all regarded as attribute-specific region, and the rest with zero attention values are regarded as attribute-irrelevant region
  $$b=F_a(I)$$
• the attribute-specfiic regions are manipulated towards the target attribute while the rest regions remain the same
  $$\hat{I}=G(I,c)=I_a\cdot b+I\cdot (1-b)$$

1. To make the edited face image $\hat{I}$ photo-realistic: an adversarial loss is designed to confuse the real/fake classifier
   $${\mathcal{L}}_{src}^G={\mathbb{E}}_{\hat{I}}[[-log{D}_{src}(\hat{I})]]\text{\hspace{1em}(2)}$$
2. To make $\hat{I}$ be correctly with target attribute $c$: an attribute classification loss is designed to enforce the attribute prediction of $\hat{I}$ from the attribute classifier approximates the target value $c$
   KaTeX parse error: Got function '\hat' with no arguments as subscript at position 33: …^G = \mathbb{E}_̲\hat{I}[-logD_{…
3. To keep the attribute-irrelevant region unchanged: a reconstruction loss is employed similar as CycleGAN and StarGAN
   $${\mathcal{L}}_{rec}^G={\lambda }_1{\mathbb{E}}_{I,c,c^g}[(||I-G(G(I,c),c^g)|{|}_1]+{\lambda }_2{\mathbb{E}}_{I,c^g}[(||I-G(I,c^g)|{|}_1]$$
4. generator G
   $$\underset{F_m,F_a}{\min }{\mathcal{L}}_G={\mathcal{L}}_{adv}^G+{\mathcal{L}}_{cls}^G+{\mathcal{L}}_{rec}^G$$

#Implementation
##Optimization
To optimize the adversarial real/fake classification more stably, in all experiments the objectives in Eq.(1) and Eq.(2) is optimized by using WGAN-GP
KaTeX parse error: Got function '\hat' with no arguments as subscript at position 59: …(I)]+\mathbb{E}_̲\hat{I}[D_{src}…

$\tilde{I}$ is sampled uniformly along a straight line between the edited images $\hat{I}$ and the real images $I$

##Network Architecture

• For the generator, the two networks of AMN and SAN share the same network architecture except slight difference in the input and output:

Network	Input	Output	Activation function
AMN	4-channel input, an input image and a attribute	3-channel RGB image	Tanh
SAN	3-channel input, an input image	1-channel attention mask image	Sigmoid

• For the discriminator, the same architecture as PatchGAN, is used considering its promising performance.