StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation

#Problem

• existing models are both inefficient and ineffective in such multi-domain image translation tasks
• incapable of jointly training domains from different datasets

New method

• Stargan, a novel and scalable approch that can perform image-to-image translations for multiple domains using only a single model
• A mask vector to domain label enables joint training between domains of different datasets

##Star Generative Adversarial Networks

###1. Multi-Domain Image-to-Image Translation

notation	meaning
$x$	input image
$y$	output image
$c$	target domain label
$c^{\prime }$	original domain label
Dsrc(x)	a probability distribution over sources given by D
Dcls(c’|x)	a probability distribution over domain labels computed by D
λcls	hyper-parameters that control the relative importance of domain classification and reconstruction losses
λrec	hyper-parameters control the relative importance of reconstruction losses
m	a mask vector
$[\cdot ]$	concatenation
$c_i$	a vector for the labels of the $i$-th dataset
$\hat{x}$	sampled uniformly along a straight line between a pair of a real and a generated images
${\lambda }_{gp}$	hyper-parameters control the gradient penalty

• Goals：To train a single generator G that learns mappings among multiple domains
• train G to translate an input image x into an output image y conditioned on the target domain label c, G(x, c) → y
• Discriminator produces probability distributions over both sources and domain labels, D : x → {D_src(x), D_cls(x)}， in order to allows a single discriminator to control multiple domains.
  ####Adversarial Loss
  $${\mathcal{L}}_{adv}={\mathbb{E}}_x[log{D}_{src}(x)]+{\mathbb{E}}_{x,c}[log(1-D_{src}(G(x,c))]\text{\hspace{1em}(1)}$$

D_src(x) as a probability distribution over sources given by D. The generator G tries to minimize this objective, while the discriminator D tries to maximize it
####Domain Classification Loss

• add an auxiliary classifier on top of D and impose the domain classification loss when optimizing both D and G
• decompose the objective into two terms: a domain classification loss of
  real images used to optimize D, and a domain classification loss of fake images used to optimize G
  $${\mathcal{L}}_{cls}^r={\mathbb{E}}_{x,c^{\prime }}[-log{D}_{cls}(c^{\prime }|x)]\text{\hspace{1em}(2)}$$
  $${\mathcal{L}}_{cls}^f={\mathbb{E}}_{x,c}[-log{D}_{cls}(c|G(x,c))]\text{\hspace{1em}(3)}$$
  ####Reconstruction Loss
• problem: minimizing the losses(Eqs. (1) and (3)) does not guarantee that translated images preserve the content of its input images while changing only the domain-related part of the inputs
• method: apply a cycle consistency loss to the generator
  $${\mathcal{L}}_{rec}={\mathbb{E}}_{x,c,c^{\prime }}[||x-G(G(x,c),c^{\prime })|{|}_1]$$
  G takes in the translated image G(x, c) and the original domain label c^’ as input and tries to reconstruct the original image x. We adopt the L1 norm as our reconstruction loss.
  Note that we use a single generator twice, first to translate an original image into an image in the target domain and then to reconstruct the original image from the translated image.
  ####Full Objective
  $${\mathcal{L}}_D=-{\mathcal{L}}_{adv}+{\lambda }_{cls}{\mathcal{L}}_{cls}^r$$
  $${\mathcal{L}}_G={\mathcal{L}}_{adv}+{\lambda }_{cls}{\mathcal{L}}_{cls}^f+{\lambda }_{rec}{\mathcal{L}}_{rec}$$

We use ${\lambda }_{cls}$ = 1 and ${\lambda }_{rec}$ = 10 in all of our experiments
###2. Training with Multiple Datasets

• Problem:the complete information on the label vector $c^{\prime }$ is required when reconstructing the input image $x$ from the translated image $G(x,c)$
  ####Mask Vector
• introduce a mask vector $m$ that allows StarGAN to ignore unspecified
  labels and focus on the explicitly known label provided by a particular dataset.
• use an n-dimensional one-hot vector to represent $m$, with $n$ being the number of datasets. In addition, we define a unified version of the label as a vector

$$\tilde{c}=[c_1,c_2...c_n,m]$$
For the remaining $n$-1 unknown labels we simply assign zero values

####Training Strategy

• use the domain label $\tilde{c}$ as input to the generator
• the generator learns to ignore the unspecified labels, which are zero vectors, and focus on the explicitly given label
• extend the auxiliary classifier of the discriminator to generate probability distributions over labels for all datasets
• train the model in a multi-task learning setting, where the discriminator tries to minimize only the classification error associated to the known label
• Under these settings, by alternating between CelebA and RaFD the discriminator learns all of the discriminative features for both datasets, and the generator learns to control all the labels in both datasets.

##Implementation
###Improved GAN Training

• replace Eq. (1) with Wasserstein GAN objective with gradient penalty defined as

KaTeX parse error: Got function '\hat' with no arguments as subscript at position 102: …_{gp}\mathbb{E}_̲\hat{x}[||\nabl…

where $\hat{x}$ is sampled uniformly along a straight line between a pair of a real and a generated images. We use ${\lambda }_{gp}$ = 10 for all experiments

###Network Architecture

• generator network composed of two convolutional layers with the stride size of two for downsampling, six residual blocks, and two transposed convolutional layers with the stride size of two for upsampling.
• use instance normalization for the generator but no normalization for
  the discriminator.
• leverage PatchGANs for the discriminator network, which classifies whether local image patches are real or fake.