Unsupervised Image-to-Image Translation Networks

Introduction

Many computer visions problems can be posed as an image-to-image translation problem, mapping an image in one domain to a corresponding image in another domain. For example, super-resolution can be considered as a problem of mapping a low-resolution image to a corresponding high-resolution image; colorization can be considered as a problem of mapping a gray-scale image to a corresponding color image. The problem can be studied in supervised and unsupervised learning settings. In the supervised setting, paired of corresponding images in different domains are available . In the unsupervised setting, we only have two independent sets of images where one consists of images in one domain and the other consists of images in another domain—there exist no paired examples showing how an image could be translated to a corresponding image in another domain. Due to lack of corresponding images, the UNsupervised Image-to-image Translation (UNIT) problem is considered harder, but it is more applicable since training data collection is easier.

When analyzing the image translation problem from a probabilistic modeling perspective, the key challenge is to learn a joint distribution of images in different domains. In the unsupervised setting, the two sets consist of images from two marginal distributions in two different domains, and the task is to infer the joint distribution using these images. The coupling theory states there exist an infinite set of joint distributions that can arrive the given marginal distributions in general. Hence, inferring the joint distribution from the marginal distributions is a highly ill-posed problem. To address the ill-posed problem, we need additional assumptions on the structure of the joint distribution.

To this end we make a shared-latent space assumption, which assumes a pair of corresponding images in different domains can be mapped to a same latent representation in a shared-latent space. Based on the assumption, we propose a UNIT framework that are based on generative adversarial networks (GANs) and variational autoencoders (VAEs). We model each image domain using a VAE-GAN. The adversarial training objective interacts with a weight-sharing constraint, which enforces a shared-latent space, to generate corresponding images in two domains, while the variational autoencoders relate translated images with input images in the respective domains. We applied the proposed framework to various unsupervised image-to-image translation problems and achieved high quality image translation results. We also applied it to the domain adaptation problem and achieved state-of-the-art accuracies on benchmark datasets. The shared-latent space assumption was used in Coupled GAN for joint distribution learning. Here, we extend the Coupled GAN work for the UNIT problem. We also note that several contemporary works propose the cycle-consistency constraint assumption , which hypothesizes the existence of a cycle-consistency mapping so that an image in the source domain can be mapped to an image in the target domain and this translated image in the target domain can be mapped back to the original image in the source domain. In the paper, we show that the shared-latent space constraint implies the cycle-consistency constraint.

Assumptions

Let $\mathcal{X}_{1}$ and $\mathcal{X}_{2}$ be two image domains. In supervised image-to-image translation, we are given samples $(x_{1},x_{2})$ drawn from a joint distribution $P_{\mathcal{X}_{1},\mathcal{X}_{2}}(x_{1},x_{2})$. In unsupervised image-to-image translation, we are given samples drawn from the marginal distributions $P_{\mathcal{X}_{1}}(x_{1})$ and $P_{\mathcal{X}_{2}}(x_{2})$. Since an infinite set of possible joint distributions can yield the given marginal distributions, we could infer nothing about the joint distribution from the marginal samples without additional assumptions.

We make the shared-latent space assumption. As shown Figure 1, we assume for any given pair of images $x_{1}$ and $x_{2}$, there exists a shared latent code $z$ in a shared-latent space, such that we can recover both images from this code, and we can compute this code from each of the two images. That is, we postulate there exist functions $E_{1}^{*}$, $E_{2}^{*}$, $G_{1}^{*}$, and $G_{2}^{*}$ such that, given a pair of corresponding images $(x_{1},x_{2})$ from the joint distribution, we have $z=E_{1}^{*}(x_{1})=E_{2}^{*}(x_{2})$ and conversely $x_{1}=G_{1}^{*}(z)$ and $x_{2}=G_{2}^{*}(z)$. Within this model, the function $x_{2}=F_{1\rightarrow 2}^{*}(x_{1})$ that maps from $\mathcal{X}_{1}$ to $\mathcal{X}_{2}$ can be represented by the composition $F_{1\rightarrow 2}^{*}(x_{1})=G_{2}^{*}(E_{1}^{*}(x_{1}))$. Similarly, $x_{1}=F_{2\rightarrow 1}^{*}(x_{2})=G_{1}^{*}(E_{2}^{*}(x_{2}))$. The UNIT problem then becomes a problem of learning $F_{1\rightarrow 2}^{*}$ and $F_{2\rightarrow 1}^{*}$. We note that a necessary condition for $F_{1\rightarrow 2}^{*}$ and $F_{2\rightarrow 1}^{*}$ to exist is the cycle-consistency constraint : $x_{1}=F_{2\rightarrow 1}^{*}(F_{1\rightarrow 2}^{*}(x_{1}))$ and $x_{2}=F_{1\rightarrow 2}^{*}(F_{2\rightarrow 1}^{*}(x_{2}))$. We can reconstruct the input image from translating back the translated input image. In other words, the proposed shared-latent space assumption implies the cycle-consistency assumption (but not vice versa).

To implement the shared-latent space assumption, we further assume a shared intermediate representation $h$ such that the process of generating a pair of corresponding images admits a form of

Consequently, we have $G_{1}^{*}\equiv G_{L,1}^{*}\circ G_{H}^{*}$ and $G_{2}^{*}\equiv G_{L,2}^{*}\circ G_{H}^{*}$ where $G_{H}^{*}$ is a common high-level generation function that maps $z$ to $h$ and $G_{L,1}^{*}$ and $G_{L,2}^{*}$ are low-level generation functions that map $h$ to $x_{1}$ and $x_{2}$, respectively. In the case of multi-domain image translation (e.g., sunny and rainy image translation), $z$ can be regarded as the compact, high-level representation of a scene ("car in front, trees in back"), and $h$ can be considered a particular realization of $z$ through $G_{H}^{*}$ ("car/tree occupy the following pixels"), and $G_{L,1}^{*}$ and $G_{L,2}^{*}$ would be the actual image formation functions in each modality ("tree is lush green in the sunny domain, but dark green in the rainy domain"). Assuming $h$ also allow us to represent $E_{1}^{*}$ and $E_{2}^{*}$ by $E_{1}^{*}\equiv E_{H}^{*}\circ E_{L,1}^{*}$ and $E_{2}^{*}\equiv E^{*}_{H}\circ E_{L,2}^{*}$.

In the next section, we discuss how we realize the above ideas in the proposed UNIT framework.

Framework

Our framework, as illustrated in Figure 1, is based on variational autoencoders (VAEs) and generative adversarial networks (GANs) . It consists of 6 subnetworks: including two domain image encoders $E_{1}$ and $E_{2}$, two domain image generators $G_{1}$ and $G_{2}$, and two domain adversarial discriminators $D_{1}$ and $D_{2}$. Several ways exist to interpret the roles of the subnetworks, which we summarize in Table 1. Our framework learns translation in both directions in one shot.

Utilizing the reparameterization trick , the non-differentiable sampling operation can be reparameterized as a differentiable operation using auxiliary random variables. This reparameterization trick allows us to train VAEs using back-prop. Let $\eta$ be a random vector with a multi-variate Gaussian distribution: $\eta\sim\mathcal{N}(\eta|0,I)$. The sampling operations of $z_{1}\sim q_{1}(z_{1}|x_{1})$ and $z_{2}\sim q_{2}(z_{2}|x_{2})$ can be implemented via $z_{1}=E_{\mu,1}(x_{1})+\eta$ and $z_{2}=E_{\mu,2}(x_{2})+\eta$, respectively.

Weight-sharing. Based on the shared-latent space assumption discussed in Section 2, we enforce a weight-sharing constraint to relate the two VAEs. Specifically, we share the weights of the last few layers of $E_{1}$ and $E_{2}$ that are responsible for extracting high-level representations of the input images in the two domains. Similarly, we share the weights of the first few layers of $G_{1}$ and $G_{2}$ responsible for decoding high-level representations for reconstructing the input images.

Note that the weight-sharing constraint alone does not guarantee that corresponding images in two domains will have the same latent code. In the unsupervised setting, no pair of corresponding images in the two domains exists to train the network to output a same latent code. The extracted latent codes for a pair of corresponding images are different in general. Even if they are the same, the same latent component may have different semantic meanings in different domains. Hence, the same latent code could still be decoded to output two unrelated images. However, we will show that through adversarial training, a pair of corresponding images in the two domains can be mapped to a common latent code by $E_{1}$ and $E_{2}$, respectively, and a latent code will be mapped to a pair of corresponding images in the two domains by $G_{1}$ and $G_{2}$, respectively.

The shared-latent space assumption allows us to perform image-to-image translation. We can translate an image $x_{1}$ in $\mathcal{X}_{1}$ to an image in $\mathcal{X}_{2}$ through applying $G_{2}(z_{1}\sim q_{1}(z_{1}|x_{1}))$. We term such an information processing stream as the image translation stream. Two image translation streams exist in the proposed framework: $\mathcal{X}_{1}\rightarrow\mathcal{X}_{2}$ and $\mathcal{X}_{2}\rightarrow\mathcal{X}_{1}$. The two streams are trained jointly with the two image reconstruction streams from the VAEs. Once we could ensure that a pair of corresponding images are mapped to a same latent code and a same latent code is decoded to a pair of corresponding images, $(x_{1},G_{2}(z_{1}\sim q_{1}(z_{1}|x_{1})))$ would form a pair of corresponding images. In other words, the composition of $E_{1}$ and $G_{2}$ functions approximates $F^{*}_{1\rightarrow 2}$ for unsupervised image-to-image translation discussed in Section 2, and the composition of $E_{2}$ and $G_{1}$ function approximates $F^{*}_{2\rightarrow 1}$.

Cycle-consistency (CC). Since the shared-latent space assumption implies the cycle-consistency constraint (See Section 2), we could also enforce the cycle-consistency constraint in the proposed framework to further regularize the ill-posed unsupervised image-to-image translation problem. The resulting information processing stream is called the cycle-reconstruction stream.

Learning. We jointly solve the learning problems of the VAE1, VAE2, GAN1 and GAN2 for the image reconstruction streams, the image translation streams, and the cycle-reconstruction streams:

VAE training aims for minimizing a variational upper bound In (2), the VAE objects are

where the hyper-parameters $\lambda_{1}$ and $\lambda_{2}$ control the weights of the objective terms and the KL divergence terms penalize deviation of the distribution of the latent code from the prior distribution. The regularization allows an easy way to sample from the latent space . We model $p_{G_{1}}$ and $p_{G_{2}}$ using Laplacian distributions, respectively. Hence, minimizing the negative log-likelihood term is equivalent to minimizing the absolute distance between the image and the reconstructed image. The prior distribution is a zero mean Gaussian $p_{\eta}(z)=\mathcal{N}(z|0,I)$.

In (2), the GAN objective functions are given by

The objective functions in (5) and (6) are conditional GAN objective functions. They are used to ensure the translated images resembling images in the target domains, respectively. The hyper-parameter $\lambda_{0}$ controls the impact of the GAN objective functions.

We use a VAE-like objective function to model the cycle-consistency constraint, which is given by

where the negative log-likelihood objective term ensures a twice translated image resembles the input one and the KL terms penalize the latent codes deviating from the prior distribution in the cycle-reconstruction stream (Therefore, there are two KL terms). The hyper-parameters $\lambda_{3}$ and $\lambda_{4}$ control the weights of the two different objective terms.

Inheriting from GAN, training of the proposed framework results in solving a mini-max problem where the optimization aims to find a saddle point. It can be seen as a two player zero-sum game. The first player is a team consisting of the encoders and generators. The second player is a team consisting of the adversarial discriminators. In addition to defeating the second player, the first player has to minimize the VAE losses and the cycle-consistency losses. We apply an alternating gradient update scheme similar to the one described in to solve (2). Specifically, we first apply a gradient ascent step to update $D_{1}$ and $D_{2}$ with $E_{1}$, $E_{2}$, $G_{1}$, and $G_{2}$ fixed. We then apply a gradient descent step to update $E_{1}$, $E_{2}$, $G_{1}$, and $G_{2}$ with $D_{1}$ and $D_{2}$ fixed.

Translation: After learning, we obtain two image translation functions by assembling a subset of the subnetworks. We have $F_{1\rightarrow 2}(x_{1})=G_{2}(z_{1}\sim q_{1}(z_{1}|x_{1}))$ for translating images from $\mathcal{X}_{1}$ to $\mathcal{X}_{2}$ and $F_{2\rightarrow 1}(x_{2})=G_{1}(z_{2}\sim q_{2}(z_{2}|x_{2}))$ for translating images from $\mathcal{X}_{2}$ to $\mathcal{X}_{1}$.

Experiments

We first analyze various components of the proposed framework. We then present visual results on challenging translation tasks. Finally, we apply our framework to the domain adaptation tasks.

Performance Analysis. We used ADAM for training where the learning rate was set to 0.0001 and momentums were set to 0.5 and 0.999. Each mini-batch consisted of one image from the first domain and one image from the second domain. Our framework had several hyper-parameters. The default values were $\lambda_{0}=10$, $\lambda_{3}=\lambda_{1}=0.1$ and $\lambda_{4}=\lambda_{2}=100$. For the network architecture, our encoders consisted of 3 convolutional layers as the front-end and 4 basic residual blocks as the back-end. The generators consisted of 4 basic residual blocks as the front-end and 3 transposed convolutional layers as the back-end. The discriminators consisted of stacks of convolutional layers. We used LeakyReLU for nonlinearity. The details of the networks are given in Appendix A.

We used the map dataset (visualized in Figure 2), which contained corresponding pairs of images in two domains (satellite image and map) useful for quantitative evaluation. Here, the goal was to learn to translate between satellite images and maps. We operated in an unsupervised setting where we used the 1096 satellite images from the training set as the first domain and 1098 maps from the validation set as the second domain. We trained for 100K iterations and used the final model to translate 1098 satellite images in the test set. We then compared the difference between a translated satellite image (supposed to be maps) and the corresponding ground truth maps pixel-wisely. A pixel translation was counted correct if the color difference was within 16 of the ground truth color value. We used the average pixel accuracy over the images in the test set as the performance metric. We could use color difference for measuring translation accuracy since the target translation function was unimodal. We did not evaluate the translation from maps to images since the translation was multi-modal, which was difficult to construct a proper evaluation metric.

In one experiment, we varied the number of weight-sharing layers in the VAEs and paired each configuration with discriminator architectures of different depths during training. We changed the number of weight-sharing layers from 1 to 4. (Sharing 1 layer in VAEs means sharing 1 layer for $E_{1}$ and $E_{2}$ and, at the same time, sharing 1 layer for $G_{1}$ and $G_{2}$.) The results were reported in Figure 2(b). Each curve corresponded to a discriminator architecture of a different depth. The x-axis denoted the number of weigh-sharing layers in the VAEs. We found that the shallowest discriminator architecture led to the worst performance. We also found that the number of weight-sharing layer had little impact. This was due to the use of the residual blocks. As tying the weight of one layer, it effectively constrained the other layers since the residual blocks only updated the residual information. In the rest of the experiments, we used VAEs with 1 sharing layer and discriminators of 5 layers.

We analyzed impact of the hyper-parameter values to the translation accuracy. For different weight values on the negative log likelihood terms (i.e., $\lambda_{2}$, $\lambda_{4}$), we computed the achieved translation accuracy over different weight values on the KL terms (i.e., $\lambda_{1}$, $\lambda_{3}$). The results were reported in Figure 2(c). We found that, in general, a larger weight value on the negative log likelihood terms yielded a better translation accuracy. We also found setting the weights of the KL terms to 0.1 resulted in consistently good performance. We hence set $\lambda_{1}=\lambda_{3}=0.1$ and $\lambda_{2}=\lambda_{4}=100$.

We performed an ablation study measuring impact of the weight-sharing and cycle-consistency constraints to the translation performance and showed the results in Figure 2(d). We reported average accuracy over 5 trials (trained with different initialized weights.). We note that when we removed the weight-sharing constraint (as a consequence, we also removed the reconstruction streams in the framework), the framework was reduced to the CycleGAN architecture . We found the model achieved an average pixel accuracy of 0.569. When we removed the cycle-consistency constraint and only used the weight-sharing constraintWe used this architecture in an earlier version of the paper., it achieved 0.568 average pixel accuracy. But when we used the full model, it achieved the best performance of 0.600 average pixel accuracy. This echoed our point that for the ill-posed joint distribution recovery problem, more constraints are beneficial.

Qualitative results. Figure 6 to 6 showed results of the proposed framework on various UNIT tasks.

Street images. We applied the proposed framework to several unsupervised street scene image translation tasks including sunny to rainy, day to night, summery to snowy, and vice versa. For each task, we used a set of images extracted from driving videos recorded at different days and cities. The numbers of the images in the sunny/day, rainy, night, summery, and snowy sets are 86165, 28915, 36280, 6838, and 6044. We trained the network to translate street scene image of size 640$\times$480. In Figure 6, we showed several example translation results . We found that our method could generate realistic translated images. We also found that one translation was usually harder than the other. Specifically, the translation that required adding more details to the image was usually harder (e.g. night to day). Additional results are available in https://github.com/mingyuliutw/unit.

Synthetic to real. In Figure 6, we showed several example results achieved by applying the proposed framework to translate images between the synthetic images in the SYNTHIA dataset and the real images in the Cityscape dataset . For the real to synthetic translation, we found our method made the cityscape images cartoon like. For the synthetic to real translation, our method achieved better results in the building, sky, road, and car regions than in the human regions.

Dog breed conversion. We used the images of Husky, German Shepherd, Corgi, Samoyed, and Old English Sheep dogs in the ImageNet dataset to learn to translate dog images between different breeds. We only used the head regions, which were extracted by a template matching algorithm. Several example results were shown in Figure 6. We found our method translated a dog to a different breed.

Cat species conversion. We also used the images of house cat, tiger, lion, cougar, leopard, jaguar, and cheetah in the ImageNet dataset to learn to translate cat images between different species. We only used the head regions, which again were extracted by a template matching algorithm. Several example results were shown in Figure 6. We found our method translated a cat to a different specie.

Face attribute. We used the CelebA dataset for attribute-based face images translation. Each face image in the dataset had several attributes, including blond hair, smiling, goatee, and eyeglasses. The face images with an attribute constituted the 1st domain, while those without the attribute constituted the 2nd domain. In Figure 6, we visualized the results where we translated several images that do not have blond hair, eye glasses, goatee, and smiling to corresponding images with each of the individual attributes. We found that the translated face images were realistic.

Domain Adaptation. We applied the proposed framework to the problem for adapting a classifier trained using labeled samples in one domain (source domain) to classify samples in a new domain (target domain) where labeled samples in the new domain are unavailable during training. Early works have explored ideas from subspace learning to deep feature learning .

We performed multi-task learning where we trained the framework to 1) translate images between the source and target domains and 2) classify samples in the source domain using the features extracted by the discriminator in the source domain. Here, we tied the weights of the high-level layers of $D_{1}$ and $D_{2}$. This allows us to adapt a classifier trained in the source domain to the target domain. Also, for a pair of generated images in different domains, we minimized the L1 distance between the features extracted by the highest layer of the discriminators, which further encouraged $D_{1}$ and $D_{2}$ to interpret a pair of corresponding images in the same way. We applied the approach to several tasks including adapting from the Street View House Number (SVHN) dataset to the MNIST dataset and adapting between the MNIST and USPS datasets. Table 2 reported the achieved performance with comparison to the competing approaches. We found that our method achieved a 0.9053 accuracy for the SVHN$\rightarrow$MNIST task, which was much better than 0.8488 achieved by the previous state-of-the-art method . We also achieved better performance for the MNIST$\leftrightarrow$SVHN task than the Coupled GAN approach, which was the state-of-the-art. The digit images had a small resolution. Hence, we used a small network. We also found that the cycle-consistency constraint was not necessary for this task. More details about the experiments are available in Appendix B.

Related Work

Several deep generative models were recently proposed for image generation including GANs , VAEs , and PixelCNN . The proposed framework was based on GANs and VAEs but it was designed for the unsupervised image-to-image translation task, which could be considered as a conditional image generation model. In the following, we first review several recent GAN and VAE works and then discuss related image translation works.

GAN learning is via staging a zero-sum game between the generator and discriminator. The quality of GAN-generated images had improved dramatically since the introduction. LapGAN proposed a Laplacian pyramid implementation of GANs. DCGAN used a deeper convolutional network. Several GAN training tricks were proposed in . WGAN used the Wasserstein distance.

VAEs optimize a variational bound. By improving the variational approximation, better image generation results were achieved . In , a VAE-GAN architecture was proposed to improve image generation quality of VAEs. VAEs were applied to translate face image attribute in .

Conditional generative model is a popular approach for mapping an image from one domain to another. Most of the existing works were based on supervised learning . Our work differed to the previous works in that we do not need corresponding images. Recently, proposed the domain transformation network (DTN) and achieved promising results on translating small resolution face and digit images. In addition to faces and digits, we demonstrated that the proposed framework can translate large resolution natural images. It also achieved a better performance in the unsupervised domain adaptation task. In , a conditional generative adversarial network-based approach was proposed to translate a rendering images to a real image for gaze estimation. In order to ensure the generated real image was similar to the original rendering image, the L1 distance between the generated and original image was minimized. We note that two contemporary papers independently introduced the cycle-consistency constraint for the unsupervised image translation. We showed that that the cycle-consistency constraint is a natural consequence of the proposed shared-latent space assumption. From our experiment, we found that cycle-consistency and the weight-sharing (a realization of the shared-latent space assumption) constraints rendered comparable performance. When the two constraints were jointed used, the best performance was achieved.

Conclusion and Future Work

We presented a general framework for unsupervised image-to-image translation. We showed it learned to translate an image from one domain to another without any corresponding images in two domains in the training dataset. The current framework has two limitations. First, the translation model is unimodal due to the Gaussian latent space assumption. Second, training could be unstable due to the saddle point searching problem. We plan to address these issues in the future work.

References

Appendix A Network Architecture

The network architecture used for the unsupervised image-to-image translation experiments is given in Table 3. We use the following abbreviation for ease of presentation: N=Neurons, K=Kernel size, S=Stride size. The transposed convolutional layer is denoted by DCONV. The residual basic block is denoted as RESBLK.

Appendix B Domain Adaptation

MNIST$\leftrightarrow$USPS. For the MNIST and USPS adaptation experiments, we used the entire training sets in the two domains (60000 training images for MNIST and 7291 training images for USPS) for learning and reported the classification performance on the test sets (10000 test images for MNIST and 2007 test images for USPS). The MNIST and USPS images had different sizes, we resized them to 28$\times$28 for facilitating the experiments. We trained the Coupled GAN algorithm using the same setting for a fair comparison.

The encoder and generator architecture was given in Table 6. For the discriminators and classifiers, we used an architecture akin to the one used in the Coupled GAN paper, which is given in Table 6.

SVHN$\rightarrow$MNIST. For the SVHN and MNIST domain adaptation experiment, we used the extra training set (consisting of 531131 images) in the SVHN dataset for the source domain and the training set (consisting of 60000 images) in the MNIST dataset for the target domain as in the DTN work . The test set consists of 10000 images in the MNIST test dataset. The MNIST images were in gray-scale. We converted them to RGB images and performed a data augmentation where we also used the inversions of the original MNIST images for training. All the images were resized to 32$\times$32 for facilitating the experiment. We also found spatial context information was useful. For each input image, we created a 5-channel variant where the first three channels were the original RGB images and the last two channels were the normalized x and y coordinates.

The encoder and generator architecture was the same as the one used in the MNIST$\leftrightarrow$USPS domain adaptation. For the discriminators and the classifier, we used a network architecture akin to the one used in the DTN paper. The details are given in Table 6. We used dropout at every layer in the classifier for avoiding over-fitting.