StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation

Introduction

The task of image-to-image translation is to change a particular aspect of a given image to another, e.g., changing the facial expression of a person from smiling to frowning (see Fig. StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation). This task has experienced significant improvements following the introduction of generative adversarial networks (GANs), with results ranging from changing hair color , reconstructing photos from edge maps , and changing the seasons of scenery images .

Given training data from two different domains, these models learn to translate images from one domain to the other. We denote the terms attribute as a meaningful feature inherent in an image such as hair color, gender or age, and attribute value as a particular value of an attribute, e.g., black/blond/brown for hair color or male/female for gender. We further denote domain as a set of images sharing the same attribute value. For example, images of women can represent one domain while those of men represent another.

Several image datasets come with a number of labeled attributes. For instance, the CelebA dataset contains 40 labels related to facial attributes such as hair color, gender, and age, and the RaFD dataset has 8 labels for facial expressions such as ‘happy’, ‘angry’ and ‘sad’. These settings enable us to perform more interesting tasks, namely multi-domain image-to-image translation, where we change images according to attributes from multiple domains. The first five columns in Fig. StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation show how a CelebA image can be translated according to any of the four domains, ‘blond hair’, ‘gender’, ‘aged’, and ‘pale skin’. We can further extend to training multiple domains from different datasets, such as jointly training CelebA and RaFD images to change a CelebA image’s facial expression using features learned by training on RaFD, as in the rightmost columns of Fig. StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation.

As a solution to such problems we propose StarGAN, a novel and scalable approach capable of learning mappings among multiple domains. As demonstrated in Fig. 2 (b), our model takes in training data of multiple domains, and learns the mappings between all available domains using only a single generator. The idea is simple. Instead of learning a fixed translation (e.g., black-to-blond hair), our generator takes in as inputs both image and domain information, and learns to flexibly translate the image into the corresponding domain. We use a label (e.g., binary or one-hot vector) to represent domain information. During training, we randomly generate a target domain label and train the model to flexibly translate an input image into the target domain. By doing so, we can control the domain label and translate the image into any desired domain at testing phase.

We also introduce a simple but effective approach that enables joint training between domains of different datasets by adding a mask vector to the domain label. Our proposed method ensures that the model can ignore unknown labels and focus on the label provided by a particular dataset. In this manner, our model can perform well on tasks such as synthesizing facial expressions of CelebA images using features learned from RaFD, as shown in the rightmost columns of Fig. StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation. As far as our knowledge goes, our work is the first to successfully perform multi-domain image translation across different datasets.

Overall, our contributions are as follows:

We propose StarGAN, a novel generative adversarial network that learns the mappings among multiple domains using only a single generator and a discriminator, training effectively from images of all domains.

We demonstrate how we can successfully learn multi-domain image translation between multiple datasets by utilizing a mask vector method that enables StarGAN to control all available domain labels.

We provide both qualitative and quantitative results on facial attribute transfer and facial expression synthesis tasks using StarGAN, showing its superiority over baseline models.

Related Work

Generative Adversarial Networks. Generative adversarial networks (GANs) have shown remarkable results in various computer vision tasks such as image generation , image translation , super-resolution imaging , and face image synthesis . A typical GAN model consists of two modules: a discriminator and a generator. The discriminator learns to distinguish between real and fake samples, while the generator learns to generate fake samples that are indistinguishable from real samples. Our approach also leverages the adversarial loss to make the generated images as realistic as possible.

Conditional GANs. GAN-based conditional image generation has also been actively studied. Prior studies have provided both the discriminator and generator with class information in order to generate samples conditioned on the class . Other recent approaches focused on generating particular images highly relevant to a given text description . The idea of conditional image generation has also been successfully applied to domain transfer , super-resolution imaging, and photo editing . In this paper, we propose a scalable GAN framework that can flexibly steer the image translation to various target domains, by providing conditional domain information.

Image-to-Image Translation. Recent work have achieved impressive results in image-to-image translation . For instance, pix2pix learns this task in a supervised manner using cGANs. It combines an adversarial loss with a L1 loss, thus requires paired data samples. To alleviate the problem of obtaining data pairs, unpaired image-to-image translation frameworks have been proposed. UNIT combines variational autoencoders (VAEs) with CoGAN , a GAN framework where two generators share weights to learn the joint distribution of images in cross domains. CycleGAN and DiscoGAN preserve key attributes between the input and the translated image by utilizing a cycle consistency loss. However, all these frameworks are only capable of learning the relations between two different domains at a time. Their approaches have limited scalability in handling multiple domains since different models should be trained for each pair of domains. Unlike the aforementioned approaches, our framework can learn the relations among multiple domains using only a single model.

Star Generative Adversarial Networks

We first describe our proposed StarGAN, a framework to address multi-domain image-to-image translation within a single dataset. Then, we discuss how StarGAN incorporates multiple datasets containing different label sets to flexibly perform image translations using any of these labels.

Our goal is to train a single generator $G$ that learns mappings among multiple domains. To achieve this, we train $G$ to translate an input image $x$ into an output image $y$ conditioned on the target domain label $c$, $G(x,c)\rightarrow y$. We randomly generate the target domain label $c$ so that $G$ learns to flexibly translate the input image. We also introduce an auxiliary classifier that allows a single discriminator to control multiple domains. That is, our discriminator produces probability distributions over both sources and domain labels, $D:x\rightarrow\{{D}_{src}(x),{D}_{cls}(x)\}$. Fig. 3 illustrates the training process of our proposed approach.

Adversarial Loss. To make the generated images indistinguishable from real images, we adopt an adversarial loss

where $G$ generates an image $G(x,c)$ conditioned on both the input image $x$ and the target domain label $c$, while $D$ tries to distinguish between real and fake images. In this paper, we refer to the term ${D}_{src}(x)$ as a probability distribution over sources given by $D$. The generator $G$ tries to minimize this objective, while the discriminator $D$ tries to maximize it.

Domain Classification Loss. For a given input image $x$ and a target domain label $c$, our goal is to translate $x$ into an output image $y$, which is properly classified to the target domain $c$. To achieve this condition, we add an auxiliary classifier on top of $D$ and impose the domain classification loss when optimizing both $D$ and $G$. That is, we decompose the objective into two terms: a domain classification loss of real images used to optimize $D$, and a domain classification loss of fake images used to optimize $G$. In detail, the former is defined as

where the term ${D}_{cls}(c^{\prime}|x)$ represents a probability distribution over domain labels computed by $D$. By minimizing this objective, $D$ learns to classify a real image $x$ to its corresponding original domain $c^{\prime}$. We assume that the input image and domain label pair $(x,c^{\prime})$ is given by the training data. On the other hand, the loss function for the domain classification of fake images is defined as

In other words, $G$ tries to minimize this objective to generate images that can be classified as the target domain $c$.

Reconstruction Loss. By minimizing the adversarial and classification losses, $G$ is trained to generate images that are realistic and classified to its correct target domain. However, minimizing the losses (Eqs. (1) and (3)) does not guarantee that translated images preserve the content of its input images while changing only the domain-related part of the inputs. To alleviate this problem, we apply a cycle consistency loss to the generator, defined as

where $G$ takes in the translated image $G(x,c)$ and the original domain label $c^{\prime}$ as input and tries to reconstruct the original image $x$. We adopt the L1 norm as our reconstruction loss. Note that we use a single generator twice, first to translate an original image into an image in the target domain and then to reconstruct the original image from the translated image.

Full Objective. Finally, the objective functions to optimize $G$ and $D$ are written, respectively, as

where ${\lambda}_{cls}$ and ${\lambda}_{rec}$ are hyper-parameters that control the relative importance of domain classification and reconstruction losses, respectively, compared to the adversarial loss. We use ${\lambda}_{cls}=1$ and ${\lambda}_{rec}=10$ in all of our experiments.

2 Training with Multiple Datasets

An important advantage of StarGAN is that it simultaneously incorporates multiple datasets containing different types of labels, so that StarGAN can control all the labels at the test phase. An issue when learning from multiple datasets, however, is that the label information is only partially known to each dataset. In the case of CelebA and RaFD , while the former contains labels for attributes such as hair color and gender, it does not have any labels for facial expressions such as ‘happy’ and ‘angry’, and vice versa for the latter. This is problematic because the complete information on the label vector $c^{\prime}$ is required when reconstructing the input image $x$ from the translated image $G(x,c)$ (See Eq. (4)).

Mask Vector. To alleviate this problem, we introduce a mask vector $m$ that allows StarGAN to ignore unspecified labels and focus on the explicitly known label provided by a particular dataset. In StarGAN, we use an $n$-dimensional one-hot vector to represent $m$, with $n$ being the number of datasets. In addition, we define a unified version of the label as a vector

Implementation

Improved GAN Training. To stabilize the training process and generate higher quality images, we replace Eq. (1) with Wasserstein GAN objective with gradient penalty defined as

where $\hat{x}$ is sampled uniformly along a straight line between a pair of a real and a generated images. We use ${\lambda}_{gp}=10$ for all experiments.

Network Architecture. Adapted from CycleGAN , StarGAN has the generator network composed of two convolutional layers with the stride size of two for downsampling, six residual blocks , and two transposed convolutional layers with the stride size of two for upsampling. We use instance normalization for the generator but no normalization for the discriminator. We leverage PatchGANs for the discriminator network, which classifies whether local image patches are real or fake. See the appendix (Section 7.2) for more details about the network architecture.

Experiments

In this section, we first compare StarGAN against recent methods on facial attribute transfer by conducting user studies. Next, we perform a classification experiment on facial expression synthesis. Lastly, we demonstrate empirical results that StarGAN can learn image-to-image translation from multiple datasets. All our experiments were conducted by using the model output from unseen images during the training phase.

As our baseline models, we adopt DIAT and CycleGAN , both of which performs image-to-image translation between two different domains. For comparison, we trained these models multiple times for every pair of two different domains. We also adopt IcGAN as a baseline which can perform attribute transfer using a cGAN .

DIAT uses an adversarial loss to learn the mapping from $x\in X$ to $y\in Y$, where $x$ and $y$ are face images in two different domains $X$ and $Y$, respectively. This method has a regularization term on the mapping as ${||x-F(G(x))||}_{1}$ to preserve identity features of the source image, where $F$ is a feature extractor pretrained on a face recognition task.

CycleGAN also uses an adversarial loss to learn the mapping between two different domains $X$ and $Y$. This method regularizes the mapping via cycle consistency losses, ${||x-({G}_{YX}({G}_{XY}(x)))||}_{1}$ and ${||y-({G}_{XY}({G}_{YX}(y)))||}_{1}$. This method requires two generators and discriminators for each pair of two different domains.

IcGAN combines an encoder with a cGAN model. cGAN learns the mapping $G:\{z,c\}\rightarrow x$ that generates an image $x$ conditioned on both the latent vector $z$ and the conditional vector $c$. In addition, IcGAN introduces an encoder to learn the inverse mappings of cGAN, ${E}_{z}:x\rightarrow z$ and ${E}_{c}:x\rightarrow c$. This allows IcGAN to synthesis images by only changing the conditional vector and preserving the latent vector.

2 Datasets

CelebA. The CelebFaces Attributes (CelebA) dataset contains 202,599 face images of celebrities, each annotated with 40 binary attributes. We crop the initial $178\times 218$ size images to $178\times 178$, then resize them as $128\times 128$. We randomly select 2,000 images as test set and use all remaining images for training data. We construct seven domains using the following attributes: hair color (black, blond, brown), gender (male/female), and age (young/old).

RaFD. The Radboud Faces Database (RaFD) consists of 4,824 images collected from 67 participants. Each participant makes eight facial expressions in three different gaze directions, which are captured from three different angles. We crop the images to $256\times 256$, where the faces are centered, and then resize them to $128\times 128$.

3 Training

All models are trained using Adam with ${\beta}_{1}=0.5$ and ${\beta}_{2}=0.999$. For data augmentation we flip the images horizontally with a probability of 0.5. We perform one generator update after five discriminator updates as in . The batch size is set to 16 for all experiments. For experiments on CelebA, we train all models with a learning rate of 0.0001 for the first 10 epochs and linearly decay the learning rate to 0 over the next 10 epochs. To compensate for the lack of data, when training with RaFD we train all models for 100 epochs with a learning rate of 0.0001 and apply the same decaying strategy over the next 100 epochs. Training takes about one day on a single NVIDIA Tesla M40 GPU.

4 Experimental Results on CelebA

We first compare our proposed method to the baseline models on a single and multi-attribute transfer tasks. We train the cross-domain models such as DIAT and CycleGAN multiple times considering all possible attribute value pairs. In the case of DIAT and CycleGAN, we perform multi-step translations to synthesize multiple attributes (e.g. transferring a gender attribute after changing a hair color).

Qualitative evaluation. Fig. 4 shows the facial attribute transfer results on CelebA. We observed that our method provides a higher visual quality of translation results on test data compared to the cross-domain models. One possible reason is the regularization effect of StarGAN through a multi-task learning framework. In other words, rather than training a model to perform a fixed translation (e.g., brown-to-blond hair), which is prone to overfitting, we train our model to flexibly translate images according to the labels of the target domain. This allows our model to learn reliable features universally applicable to multiple domains of images with different facial attribute values.

Furthermore, compared to IcGAN, our model demonstrates an advantage in preserving the facial identity feature of an input. We conjecture that this is because our method maintains the spatial information by using activation maps from the convolutional layer as latent representation, rather than just a low-dimensional latent vector as in IcGAN.

Quantitative evaluation protocol. For quantitative evaluations, we performed two user studies in a survey format using Amazon Mechanical Turk (AMT) to assess single and multiple attribute transfer tasks. Given an input image, the Turkers were instructed to choose the best generated image based on perceptual realism, quality of transfer in attribute(s), and preservation of a figure’s original identity. The options were four randomly shuffled images generated from four different methods. The generated images in one study have a single attribute transfer in either hair color (black, blond, brown), gender, or age. In another study, the generated images involve a combination of attribute transfers. Each Turker was asked 30 to 40 questions with a few simple yet logical questions for validating human effort. The number of validated Turkers in each user study is 146 and 100 in single and multiple transfer tasks, respectively.

Quantitative results. Tables 1 and 2 show the results of our AMT experiment on single- and multi-attribute transfer tasks, respectively. StarGAN obtained the majority of votes for best transferring attributes in all cases. In the case of gender changes in Table 1, the voting difference between our model and other models was marginal, e.g., 39.1% for StarGAN vs. 31.4% for DIAT. However, in multi-attribute changes, e.g., the ‘G+A’ case in Table 2, the performance difference becomes significant, e.g., 49.8% for StarGAN vs. 20.3% for IcGAN), clearly showing the advantages of StarGAN in more complicated, multi-attribute transfer tasks. This is because unlike the other methods, StarGAN can handle image translation involving multiple attribute changes by randomly generating a target domain label in the training phase.

5 Experimental Results on RaFD

We next train our model on the RaFD dataset to learn the task of synthesizing facial expressions. To compare StarGAN and baseline models, we fix the input domain as the ‘neutral’ expression, but the target domain varies among the seven remaining expressions.

Qualitative evaluation. As seen in Fig. 5, StarGAN clearly generates the most natural-looking expressions while properly maintaining the personal identity and facial features of the input. While DIAT and CycleGAN mostly preserve the identity of the input, many of their results are shown blurry and do not maintain the degree of sharpness as seen in the input. IcGAN even fails to preserve the personal identity in the image by generating male images.

We believe that the superiority of StarGAN in the image quality is due to its implicit data augmentation effect from a multi-task learning setting. RaFD images contain a relatively small size of samples, e.g., 500 images per domain. When trained on two domains, DIAT and CycleGAN can only use 1,000 training images at a time, but StarGAN can use 4,000 images in total from all the available domains for its training. This allows StarGAN to properly learn how to maintain the quality and sharpness of the generated output.

Quantitative evaluation. For a quantitative evaluation, we compute the classification error of a facial expression on synthesized images. We trained a facial expression classifier on the RaFD dataset (90%/10% splitting for training and test sets) using a ResNet-18 architecture , resulting in a near-perfect accuracy of 99.55%. We then trained each of image translation models using the same training set and performed image translation on the same, unseen test set. Finally, we classified the expression of these translated images using the above-mentioned classifier. As can be seen in Table 3, our model achieves the lowest classification error, indicating that our model produces the most realistic facial expressions among all the methods compared.

Another important advantage of our model is the scalability in terms of the number of parameters required. The last column in Table 3 shows that the number of parameters required to learn all translations by StarGAN is seven times smaller than that of DIAT and fourteen times smaller than that of CycleGAN. This is because StarGAN requires only a single generator and discriminator pair, regardless of the number of domains, while in the case of cross-domain models such as CycleGAN, a completely different model should be trained for each source-target domain pair.

6 Experimental Results on CelebA+RaFD

Finally, we empirically demonstrate that our model can learn not only from multiple domains within a single dataset, but also from multiple datasets. We train our model jointly on the CelebA and RaFD datasets using the mask vector (see Section 3.2). To distinguish between the model trained only on RaFD and the model trained on both CelebA and RaFD, we denote the former as StarGAN-SNG (single) and the latter as StarGAN-JNT (joint).

Effects of joint training. Fig. 6 shows qualitative comparisons between StarGAN-SNG and StarGAN-JNT, where the task is to synthesize facial expressions of images in CelebA. StarGAN-JNT exhibits emotional expressions with high visual quality, while StarGAN-SNG generates reasonable but blurry images with gray backgrounds. This difference is due to the fact that StarGAN-JNT learns to translate CelebA images during training but not StarGAN-SNG. In other words, StarGAN-JNT can leverage both datasets to improve shared low-level tasks such facial keypoint detection and segmentation. By utilizing both CelebA and RaFD, StarGAN-JNT can improve these low-level tasks, which is beneficial to learning facial expression synthesis.

Learned role of mask vector. In this experiment, we gave a one-hot vector $c$ by setting the dimension of a particular facial expression (available from the second dataset, RaFD) to one. In this case, since the label associated with the second data set is explicitly given, the proper mask vector would be $$. Fig. 7 shows the case where this proper mask vector was given and the opposite case where a wrong mask vector of$$ was given. When the wrong mask vector was used, StarGAN-JNT fails to synthesize facial expressions, and it manipulates the age of the input image. This is because the model ignores the facial expression label as unknown and treats the facial attribute label as valid by the mask vector. Note that since one of the facial attributes is ‘young’, the model translates the image from young to old when it takes in a zero vector as input. From this behavior, we can confirm that StarGAN properly learned the intended role of a mask vector in image-to-image translations when involving all the labels from multiple datasets altogether.

Conclusion

In this paper, we proposed StarGAN, a scalable image-to-image translation model among multiple domains using a single generator and a discriminator. Besides the advantages in scalability, StarGAN generated images of higher visual quality compared to existing methods , owing to the generalization capability behind the multi-task learning setting. In addition, the use of the proposed simple mask vector enables StarGAN to utilize multiple datasets with different sets of domain labels, thus handling all available labels from them. We hope our work to enable users to develop interesting image translation applications across multiple domains.

Acknowledgements. This work was mainly done while the first author did a research internship at Clova AI Research, NAVER. We thank all the researchers at NAVER, especially Donghyun Kwak, for insightful discussions. This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF2016R1C1B2015924). Jaegul Choo is the corresponding author.

References

Appendix

Fig. 8 shows an overview of StarGAN when learning from both the CelebA and RaFD datasets. As can be seen at the top of the figure, the label for CelebA contains binary attributes (Black, Blond, Brown, Male, and Young), while the label for RaFD provides information on categorical attributes (Angry, Fearful, Happy, Sad, and Disgusted). The mask vector is a two-dimensional one-hot vector which indicates whether the CelebA or RaFD label is valid.

2 Network Architecture

The network architectures of StarGAN are shown in Table 4 and 5. For the generator network, we use instance normalization in all layers except the last output layer. For the discriminator network, we use Leaky ReLU with a negative slope of 0.01. There are some notations; ${n}_{d}$: the number of domain, ${n}_{c}$: the dimension of domain labels (${n}_{d}+2$ when training with both the CelebA and RaFD datasets, otherwise same as ${n}_{d}$), N: the number of output channels, K: kernel size, S: stride size, P: padding size, IN: instance normalization.

3 Additional Qualitative Results

Figs. 9, 10, 11, and 12 show additional images with $256\times 256$ resolutions generated by StarGAN. All images were generated by a single generator trained on both the CelebA and RaFD datasets. We trained StarGAN on a single NVIDIA Pascal M40 GPU for seven days.