Diverse Image-to-Image Translation via Disentangled Representations

Introduction

Image-to-Image (I2I) translation aims to learn the mapping between different visual domains. Many vision and graphics problems can be formulated as I2I translation problems, such as colorization (grayscale $\rightarrow$ color), super-resolution (low-resolution $\rightarrow$ high-resolution), and photorealistic image synthesis (label $\rightarrow$ image). Furthermore, I2I translation has recently shown promising results in facilitating domain adaptation .

Learning the mapping between two visual domains is challenging for two main reasons. First, aligned training image pairs are either difficult to collect (e.g., day scene $\leftrightarrow$ night scene) or do not exist (e.g., artwork $\leftrightarrow$ real photo). Second, many such mappings are inherently multimodal — a single input may correspond to multiple possible outputs. To handle multimodal translation, one possible approach is to inject a random noise vector to the generator for modeling the data distribution in the target domain. However, mode collapse may still occur easily since the generator often ignores the additional noise vectors.

Several recent efforts have been made to address these issues. Pix2pix applies conditional generative adversarial network to I2I translation problems. Nevertheless, the training process requires paired data. A number of recent work relaxes the dependency on paired training data for learning I2I translation. These methods, however, produce a single output conditioned on the given input image. As shown in , simply incorporating noise vectors as additional inputs to the generator does not lead the increased variations of the generated outputs due to the mode collapsing issue. The generators in these methods are inclined to overlook the added noise vectors. Very recently, BicycleGAN tackles the problem of generating diverse outputs in I2I problems by encouraging the one-to-one relationship between the output and the latent vector. Nevertheless, the training process of BicycleGAN requires paired images.

In this paper, we propose a disentangled representation framework for learning to generate diverse outputs with unpaired training data. Specifically, we propose to embed images onto two spaces: 1) a domain-invariant content space and 2) a domain-specific attribute space as shown in Figure 2. Our generator learns to perform I2I translation conditioned on content features and a latent attribute vector. The domain-specific attribute space aims to model the variations within a domain given the same content, while the domain-invariant content space captures information across domains. We achieve this representation disentanglement by applying a content adversarial loss to encourage the content features not to carry domain-specific cues, and a latent regression loss to encourage the invertible mapping between the latent attribute vectors and the corresponding outputs. To handle unpaired datasets, we propose a cross-cycle consistency loss using the disentangled representations. Given a pair of unaligned images, we first perform a cross-domain mapping to obtain intermediate results by swapping the attribute vectors from both images. We can then reconstruct the original input image pair by applying the cross-domain mapping one more time and use the proposed cross-cycle consistency loss to enforce the consistency between the original and the reconstructed images. At test time, we can use either 1) randomly sampled vectors from the attribute space to generate diverse outputs or 2) the transferred attribute vectors extracted from existing images for example-guided translation. Figure 1 shows examples of the two testing modes.

We evaluate the proposed model through extensive qualitative and quantitative evaluation. In a wide variety of I2I tasks, we show diverse translation results with randomly sampled attribute vectors and example-guided translation with transferred attribute vectors from existing images. We evaluate the realism of our results with a user study and the diversity using perceptual distance metrics . Furthermore, we demonstrate the potential application of unsupervised domain adaptation. On the tasks of adapting domains from MNIST to MNIST-M and Synthetic Cropped LineMod to Cropped LineMod , we show competitive performance against state-of-the-art domain adaptation methods.

1) We introduce a disentangled representation framework for image-to-image translation. We apply a content discriminator to facilitate the factorization of domain-invariant content space and domain-specific attribute space, and a cross-cycle consistency loss that allows us to train the model with unpaired data.

2) Extensive qualitative and quantitative experiments show that our model compares favorably against existing I2I models. Images generated by our model are both diverse and realistic.

3) We demonstrate the application of our model on unsupervised domain adaptation. We achieve competitive results on both the MNIST-M and the Cropped LineMod datasets.

Our code, data and more results are available at https://github.com/HsinYingLee/DRIT/.

Related Work

Generative adversarial networks. Recent years have witnessed rapid progress on generative adversarial networks (GANs) for image generation. The core idea of GANs lies in the adversarial loss that enforces the distribution of generated images to match that of the target domain. The generators in GANs can map from noise vectors to realistic images. Several recent efforts explore conditional GAN in various contexts including conditioned on text , low-resolution images , video frames , and image . Our work focuses on using GAN conditioned on an input image. In contrast to several existing conditional GAN frameworks that require paired training data, our model produces diverse outputs without paired data. This suggests that our method has wider applicability to problems where paired training datasets are scarce or not available.

Image-to-image translation. I2I translation aims to learn the mapping from a source image domain to a target image domain. Pix2pix applies a conditional GAN to model the mapping function. Although high-quality results have been shown, the model training requires paired training data. To train with unpaired data, CycleGAN , DiscoGAN , and UNIT leverage cycle consistency to regularize the training. However, these methods perform generation conditioned solely on an input image and thus produce one single output. Simply injecting a noise vector to a generator is usually not an effective solution to achieve multimodal generation due to the lack of regularization between the noise vectors and the target domain. On the other hand, BicycleGAN enforces the bijection mapping between the latent and target space to tackle the mode collapse problem. Nevertheless, the method is only applicable to problems with paired training data. Table 1 shows a feature-by-feature comparison among various I2I models. Unlike existing work, our method enables I2I translation with diverse outputs in the absence of paired training data.

Very recently, several concurrent works (all independently developed) also adopt a disentangled representation similar to our work for learning diverse I2I translation from unpaired training data. We encourage the readers to review these works for a complete picture.

Disentangled representations. The task of learning disentangled representation aims at modeling the factors of data variations. Previous work makes use of labeled data to factorize representations into class-related and class-independent components . Recently, the unsupervised setting has been explored . InfoGAN achieves disentanglement by maximizing the mutual information between latent variables and data variation. Similar to DrNet that separates time-independent and time-varying components with an adversarial loss, we apply a content adversarial loss to disentangle an image into domain-invariant and domain-specific representations to facilitate learning diverse cross-domain mappings.

Domain adaptation. Domain adaptation techniques focus on addressing the domain-shift problem between a source and a target domain. Domain Adversarial Neural Network (DANN) and its variants tackle domain adaptation through learning domain-invariant features. Sun et al. aims to map features in the source domain to those in the target domain. I2I translation has been recently applied to produce simulated images in the target domain by translating images from the source domain . Different from the aforementioned I2I based domain adaptation algorithms, our method does not utilize source domain annotations for I2I translation.

Disentangled Representation for I2I Translation

In this section, we first discuss the strategies used to disentangle the content and attribute representations in Section 3.1 and then introduce the proposed cross-cycle consistency loss that enables the training on unpaired data in Section 3.2. Finally, we detail the loss functions in Section 3.3.

Our approach embeds input images onto a shared content space $\mathcal{C}$, and domain-specific attribute spaces, $\mathcal{A}_{\mathcal{X}}$ and $\mathcal{A}_{\mathcal{Y}}$. Intuitively, the content encoders should encode the common information that is shared between domains onto $\mathcal{C}$, while the attribute encoders should map the remaining domain-specific information onto $\mathcal{A}_{\mathcal{X}}$ and $\mathcal{A}_{\mathcal{Y}}$.

To achieve representation disentanglement, we apply two strategies: weight-sharing and a content discriminator. First, similar to , based on the assumption that two domains share a common latent space, we share the weight between the last layer of $E^{c}_{\mathcal{X}}$ and $E^{c}_{\mathcal{Y}}$ and the first layer of $G_{\mathcal{X}}$ and $G_{\mathcal{Y}}$. Through weight sharing, we force the content representation to be mapped onto the same space. However, sharing the same high-level mapping functions cannot guarantee the same content representations encode the same information for both domains. Therefore, we propose a content discriminator $D^{c}$ which aims to distinguish the domain membership of the encoded content features $z_{x}^{c}$ and $z_{y}^{c}$. On the other hand, content encoders learn to produce encoded content representations whose domain membership cannot be distinguished by the content discriminator $D^{c}$. We express this content adversarial loss as:

2 Cross-cycle Consistency Loss

With the disentangled representation where the content space is shared among domains and the attribute space encodes intra-domain variations, we can perform I2I translation by combining a content representation from an arbitrary image and an attribute representation from an image of the target domain. We leverage this property and propose a cross-cycle consistency. In contrast to cycle consistency constraint in (i.e., $\mathcal{X}\to\mathcal{Y}\to\mathcal{X}$) which assumes one-to-one mapping between the two domains, the proposed cross-cycle constraint exploit the disentangled content and attribute representations for cyclic reconstruction.

Our cross-cycle constraint consists of two stages of I2I translation.

Forward translation. Given a non-corresponding pair of images $x$ and $y$, we encode them into $\{z_{x}^{c},z_{x}^{a}\}$ and $\{z_{y}^{c},z_{y}^{a}\}$. We then perform the first translation by swapping the attribute representation (i.e., $z_{x}^{a}$ and $z_{y}^{a}$) to generate $\{u,v\}$, where $u\in\mathcal{X},v\in\mathcal{Y}$.

Backward translation. After encoding $u$ and $v$ into $\{z_{u}^{c},z_{u}^{a}\}$ and $\{z_{v}^{c},z_{v}^{a}\}$, we perform the second translation by once again swapping the attribute representation (i.e., $z_{u}^{a}$ and $z_{v}^{a}$).

Here, after two I2I translation stages, the translation should reconstruct the original images $x$ and $y$ (as illustrated in Figure 3). To enforce this constraint, we formulate the cross-cycle consistency loss as:

where $u=G_{\mathcal{X}}(E_{\mathcal{Y}}^{c}(y)),E_{\mathcal{X}}^{a}(x))$ and $v=G_{\mathcal{Y}}(E_{\mathcal{X}}^{c}(x)),E_{\mathcal{Y}}^{a}(y))$.

3 Other Loss Functions

Other than the proposed content adversarial loss and cross-cycle consistency loss, we also use several other loss functions to facilitate network training. We illustrate these additional losses in Figure 4. Starting from the top-right, in the counter-clockwise order:

The full objective function of our network is:

where the hyper-parameters $\lambda$s control the importance of each term.

Experimental Results

Implementation details. We implement our model with PyTorch . We use the input image size of $216\times 216$ for all of our experiments except domain adaptation. For the content encoder $E^{c}$, we use an architecture consisting of three convolution layers followed by four residual blocks. For the attribute encoder $E^{a}$, we use a CNN architecture with four convolution layers followed by fully-connected layers. We set the size of the attribute vector to $z^{a}\in R^{8}$ for all experiments. For the generator $G$, we use an architecture containing four residual blocks followed by three fractionally strided convolution layers. For more details of architecture design, please refer to the supplementary material.

Datasets. We evaluate our model on several datasets include Yosemite (summer and winter scenes), artworks (Monet and van Gogh), edge-to-shoes and photo-to-portrait cropped from subsets of the WikiArt dataset https://www.wikiart.org/ and the CelebA dataset . We also perform domain adaptation on the classification task with MNIST to MNIST-M , and on the classification and pose estimation tasks with Synthetic Cropped LineMod to Cropped LineMod .

Compared methods. We perform the evaluation on the following algorithms:

DRIT: We refer to our proposed model, Disentangled Representation for Image-to-Image Translation, as DRIT.

DRIT w/o $D^{c}$: Our proposed model without the content discriminator.

Cycle/Bicycle: As there is no previous work addressing the problem of multimodal generation from unpaired training data, we construct a baseline using a combination of CylceGAN and BicycleGAN. Here, we first train CycleGAN on unpaired data to generate corresponding images as pseudo image pairs. We then use this pseudo paired data to train BicycleGAN.

Diversity. We first demonstrate the diversity of the generated images on several different tasks in Figure 5. In Figure 6, we compare the proposed model with other methods. Both our model without $D^{c}$ and Cycle/Bicycle can generate diverse results. However, the results contain clearly visible artifacts. Without the content discriminator, our model fails to capture domain-related details (e.g., the color of tree and sky). Therefore, the variations take place in global color difference. Cycle/Bicycle is trained on pseudo paired data generated by CycleGAN. The quality of the pseudo paired data is not uniformly ideal. As a result, the generated images are of ill-quality.

To have a better understanding of the learned domain-specific attribute space, we perform linear interpolation between two given attributes and generate the corresponding images as shown in Figure 7. The interpolation results verify the continuity in the attribute space and show that our model can generalize in the distribution, rather than memorize trivial visual information.

Attribute transfer. We demonstrate the results of the attribute transfer in Figure 8. Thanks to the representation disentanglement of content and attribute, we are able to perform attribute transfer from images of desired attributes, as illustrated in Figure 3(c). Moreover, since the content space is shared between two domains, we can generate images conditioned on content features encoded from either domain. Thus our model can achieve not only inter-domain but also intra-domain attribute transfer. Note that intra-domain attribute transfer is not explicitly involved in the training process.

2 Quantitative Evaluation

Realism vs. diversity. Here we have the quantitative evaluation on the realism and diversity of the generated images. We conduct the experiment using winter $\rightarrow$ summer translation with the Yosemite dataset. For realism, we conduct a user study using pairwise comparison. Given a pair of images sampled from real images and translated images generated from various methods, users need to answer the question “Which image is more realistic?” For diversity, similar to , we use the LPIPS metric to measure the similarity among images. We compute the distance between 1000 pairs of randomly sampled images translated from 100 real images.

Figure 9 and Table 3 show the results of realism and diversity, respectively. UNIT obtains low realism score, suggesting that their assumption might not be generally applicable. CycleGAN achieves the highest scores in realism, yet the diversity is limited. The diversity and the visual quality of Cycle/Bicycle are constrained by the data CycleGAN can generate. Our results also demonstrate the need for the content discriminator.

Reconstruction ability. In addition to diversity evaluation, we conduct an experiment on the edge-to-shoes dataset to measure the quality of the disentangled encoding. Our model was trained using unpaired data. At test time, given a paired data $\{x,y\}$, we can evaluate the quality of content-attribute disentanglement by measuring the reconstruction errors of $y$ with $\hat{y}=G_{\mathcal{Y}}(E_{\mathcal{X}}^{c}(x),E_{\mathcal{Y}}^{a}(y))$.

We compare our model with BicycleGAN, which requires paired data during training. Table 3 shows our model performs comparably with BicycleGAN despite training without paired data. Moreover, the result suggests that the content discriminator contributes greatly to the quality of disentangled representation.

3 Domain Adaptation

We demonstrate that the proposed image-to-image translation scheme can benefit unsupervised domain adaptation. Following PixelDA , we conduct experiments on the classification and pose estimation tasks using MNIST to MNIST-M , and Synthetic Cropped LineMod to Cropped LineMod . Several example images in these datasets are shown in Figure 10 (a) and (b). To evaluate our method, we first translate the labeled source images to the target domain. We then treat the generated labeled images as training data and train the classifiers of each task in the target domain. For a fair comparison, we use the classifiers with the same architecture as PixelDA. We compare the proposed method with CycleGAN, which generates the most realistic images in the target domain according to our previous experiment, and three state-of-the-art domain adaptation algorithms: PixelDA, DANN and DSN .

We present the quantitative comparisons in Table 4 and visual results from our method in Figure 10(c)(d). Since our model can generate diverse output, we generate one time, three times, and five times (denoted as $\times 1,\times 3,\times 5$) of target images using the same amount of source images. Our results validate that the proposed method can simulate diverse images in the target domain and improve the performance in target tasks. While our method does not outperform PixelDA, we note that unlike PixelDA, we do not leverage label information during training. Compared to CycleGAN, our method performs favorably even with the same amount of generated images (i.e., $\times 1$). We observe that CycleGAN suffers from the mode collapse problem and generates images with similar appearances, which degrade the performance of the adapted classifiers.

4 Limitations

Our method has the following limitations. First, due to the limited amount of training data, the attribute space is not fully exploited. Our I2I translation fails when the sampled attribute vectors locate in under-sampled space, see Figure 11(a). Second, it remains difficult when the domain characteristics differ significantly. For example, Figure 11(b) shows a failure case on the human figure due to the lack of human-related portraits in Monet collections.

Conclusions

In this paper, we present a novel disentangled representation framework for diverse image-to-image translation with unpaired data. we propose to disentangle the latent space to a content space that encodes common information between domains, and a domain-specific attribute space that can model the diverse variations given the same content. We apply a content discriminator to facilitate the representation disentanglement. We propose a cross-cycle consistency loss for cyclic reconstruction to train in the absence of paired data. Qualitative and quantitative results show that the proposed model produces realistic and diverse images. We also apply the proposed method to domain adaptation and achieve competitive performance compared to the state-of-the-art methods.

Acknowledgements

This work is supported in part by the NSF CAREER Grant #1149783, the NSF Grant #1755785, and gifts from Verisk, Adobe and Nvidia.

References