Generating Images from Captions with Attention

Introduction

Statistical natural image modelling remains a fundamental problem in computer vision and image understanding. The challenging nature of this task has motivated recent approaches to exploit the inference and generative capabilities of deep neural networks. Previously studied deep generative models of images often defined distributions that were restricted to being either unconditioned or conditioned on classification labels. In real world applications, however, images rarely appear in isolation as they are often accompanied by unstructured textual descriptions, such as on web pages and in books. The additional information from these descriptions could be used to simplify the image modelling task. Moreover, learning generative models conditioned on text also allows a better understanding of the generalization performance of the model, as we can create textual descriptions of completely new scenes not seen at training time.

There are numerous ways to learn a generative model over both image and text modalities. One approach is to learn a generative model of text conditioned on images, known as caption generation (Kiros et al., 2014a; Karpathy & Li, 2015; Vinyals et al., 2015; Xu et al., 2015). These models take an image descriptor and generate unstructured texts using a recurrent decoder. In contrast, in this paper we explore models that condition in the opposite direction, i.e. taking textual descriptions as input and using them to generate relevant images. Generating high dimensional realistic images from their descriptions combines the two challenging components of language modelling and image generation, and can be considered to be more difficult than caption generation.

In this paper, we illustrate how sequential deep learning techniques can be used to build a conditional probabilistic model over natural image space effectively. By extending the Deep Recurrent Attention Writer (DRAW) (Gregor et al., 2015), our model iteratively draws patches on a canvas, while attending to the relevant words in the description. Overall, the main contributions of this work are the following: we introduce a conditional alignDRAW model, a generative model of images from captions using a soft attention mechanism. The images generated by our alignDRAW model are refined in a post-processing step by a deterministic Laplacian pyramid adversarial network (Denton et al., 2015). We further illustrate how our method, learned on Microsoft COCO (Lin et al., 2014), generalizes to captions describing novel scenes that are not seen in the dataset, such as “A stop sign is flying in blue skies” (see Fig. 1).

Related Work

Deep Neural Networks have achieved significant success in various tasks such as image recognition (Krizhevsky et al., 2012), speech transcription (Graves et al., 2013), and machine translation (Bahdanau et al., 2015). While most of the recent success has been achieved by discriminative models, generative models have not yet enjoyed the same level of success. Most of the previous work in generative models has focused on variants of Boltzmann Machines (Smolensky, 1986; Salakhutdinov & Hinton, 2009) and Deep Belief Networks (Hinton et al., 2006). While these models are very powerful, each iteration of training requires a computationally costly step of MCMC to approximate derivatives of an intractable partition function (normalization constant), making it difficult to scale them to large datasets.

Kingma & Welling (2014), Rezende et al. (2014) have introduced the Variational Auto-Encoder (VAE) which can be seen as a neural network with continuous latent variables. The encoder is used to approximate a posterior distribution and the decoder is used to stochastically reconstruct the data from latent variables. Gregor et al. (2015) further introduced the Deep Recurrent Attention Writer (DRAW), extending the VAE approach by incorporating a novel differentiable attention mechanism.

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are another type of generative models that use noise-contrastive estimation (Gutmann & Hyvärinen, 2010) to avoid calculating an intractable partition function. The model consists of a generator that generates samples using a uniform distribution and a discriminator that discriminates between real and generated images. Recently, Denton et al. (2015) have scaled those models by training conditional GANs at each level of a Laplacian pyramid of images.

While many of the previous approaches have focused on unconditional models or models conditioned on labels, in this paper we develop a generative model of images conditioned on captions.

Model

Our proposed model defines a generative process of images conditioned on captions. In particular, captions are represented as a sequence of consecutive words and images are represented as a sequence of patches drawn on a canvas $c_{t}$ over time $t=1,...,T$. The model can be viewed as a part of the sequence-to-sequence framework (Sutskever et al., 2014; Cho et al., 2014; Srivastava et al., 2015).

Let ${\bf{y}}$ be the input caption, represented as a sequence of 1-of-K encoded words ${\bf y}=(y_{1},y_{2},...,y_{N})$, where $K$ is the size of the vocabulary and $N$ is the length of the sequence. We obtain the caption sentence representation by first transforming each word $y_{i}$ to an $m$-dimensional vector representation $h^{lang}_{i}$, $i=1,..,N$ using the Bidirectional RNN. In a Bidirectional RNN, the two LSTMs (Hochreiter & Schmidhuber, 1997) with forget gates (Gers et al., 2000) process the input sequence from both forward and backward directions. The Forward LSTM computes the sequence of forward hidden states $[\overrightarrow{h}^{lang}_{1},\overrightarrow{h}^{lang}_{2},...,\overrightarrow{h}^{lang}_{N}]$ , whereas the Backward LSTM computes the sequence of backward hidden states $[\overleftarrow{h}^{lang}_{1},\overleftarrow{h}^{lang}_{2},...,\overleftarrow{h}^{lang}_{N}]$. These hidden states are then concatenated together into the sequence $h^{lang}=[h^{lang}_{1},h^{lang}_{2},...,h^{lang}_{N}]$, with $h^{lang}_{i}=[\overrightarrow{h}^{lang}_{i},\overleftarrow{h}^{lang}_{i}],1\leq i\leq N$.

2 Image Model: the Conditional DRAW Network

Unlike the original DRAW network where latent variables are independent spherical Gaussians $\mathcal{N}(0,I)$, the latent variables in the proposed alignDRAW model have their mean and variance depend on the previous hidden states of the generative LSTM $h^{gen}_{t-1}$, except for $P(Z_{1})=\mathcal{N}(0,I)$. Namely, the mean and variance of the prior distribution over $Z_{t}$ are parameterized by:

Formally, an image is generated by iteratively computing the following set of equations for $t=1,...,T$ (see Fig. 2), with $h^{gen}_{0}$ and $c_{0}$ initialized to learned biases:

The $align$ function is used to compute the alignment between the input caption and intermediate image generative steps (Bahdanau et al., 2015). Given the caption representation from the language model, $h^{lang}=[h^{lang}_{1},h^{lang}_{2},...,h^{lang}_{N}]$, the $align$ operator outputs a dynamic sentence representation $s_{t}$ at each step through a weighted sum using alignment probabilities $\alpha_{1...N}^{t}$:

The corresponding alignment probability $\alpha_{k}^{t}$ for the $k^{th}$ word in the caption is obtained using the caption representation $h^{lang}$ and the current hidden state of the generative model $h^{gen}_{t-1}$:

The $\mathit{LSTM}^{gen}$ function of Eq. 3 is defined by the LSTM network with forget gates (Gers et al., 2000) at a single time-step. To generate the next hidden state $h^{gen}_{t}$, the $\mathit{LSTM}^{gen}$ takes the previous hidden state $h^{gen}_{t-1}$ and combines it with the input from both the latent sample $z_{t}$ and the sentence representation $s_{t}$.

Finally, each entry $c_{T,i}$ from the final canvas matrix $c_{T}$ is transformed using a sigmoid function $\boldsymbol{\sigma}$ to produce a conditional Bernoulli distribution with mean vector $\boldsymbol{\sigma}(c_{T})$ over the $h\times w$ image pixels ${\bf{x}}$ given the latent variables $Z_{1:T}$ and the input caption ${\bf{y}}$We also experimented with a conditional Gaussian observation model, but it worked worse compared to the Bernoulli model.. In practice, when generating an image ${\bf{x}}$, instead of sampling from the conditional Bernoulli distribution, we simply use the conditional mean ${\bf{x}}=\boldsymbol{\sigma}(c_{T})$.

3 Learning

The model is trained to maximize a variational lower bound $\mathcal{L}$ on the marginal likelihood of the correct image ${\bf{x}}$ given the input caption ${\bf{y}}$:

Similar to the DRAW model, the inference recurrent network produces an approximate posterior $Q(Z_{1:T}\,|\,{\bf{x}},{\bf{y}})$ via a $\mathit{read}$ operator, which reads a patch from an input image ${\bf{x}}$ using two arrays of 1D Gaussian filters (inverse of $\mathit{write}$ from section 3.2) at each time-step $t$. Specifically,

The terms in the variational lower bound Eq. 9 can be rearranged using the law of total expectation. Therefore, the variational bound $\mathcal{L}$ is calculated as follows:

The model can be trained using stochastic gradient descent. In all of our experiments, we used only a single sample from $Q(Z_{1:T}\,|\,{\bf{y}},{\bf{x}})$ for parameter learning. Training details, hyperparameter settings, and the overall model architecture are specified in Appendix B. The code is available at https://github.com/emansim/text2image.

4 Generating Images from Captions

During the image generation step, we discard the inference network and instead sample from the prior distribution. Due to the blurriness of samples generated by the DRAW model, we perform an additional post processing step where we use an adversarial network trained on residuals of a Laplacian pyramid conditioned on the skipthought representation (Kiros et al., 2015) of the captions to sharpen the generated images, similar to (Denton et al., 2015). By fixing the prior of the adversarial generator to its mean, it gets treated as a deterministic neural network that allows us to define the conditional data term in Eq. 3.3 on the sharpened images and estimate the variational lower bound accordingly.

Experiments

Microsoft COCO (Lin et al., 2014) is a large dataset containing 82,783 images, each annotated with at least 5 captions. The rich collection of images with a wide variety of styles, backgrounds and objects makes the task of learning a good generative model very challenging. For consistency with related work on caption generation, we used only the first five captions when training and evaluating our model. The images were resized to $32\times 32$ pixels for consistency with other tiny image datasets (Krizhevsky, 2009). In the following subsections, we analyzed both the qualitative and quantitative aspects of our model as well as compared its performance with that of other, related generative models.To see more generated images, visit http://www.cs.toronto.edu/~emansim/cap2im.html Appendix A further reports some additional experiments using the MNIST dataset.

The main goal of this work is to learn a model that can understand the semantic meaning expressed in the textual descriptions of images, such as the properties of objects, the relationships between them, and then use that knowledge to generate relevant images. To examine the understanding of our model, we wrote a set of captions inspired by the COCO dataset and changed some words in the captions to see whether the model made the relevant changes in the generated samples.

First, we explored whether the model understood one of the most basic properties of any object, the color. In Fig. 3, we generated images of school buses with four different colors: yellow, red, green and blue. Although, there are images of buses with different colors in the training set, all mentioned school buses are specifically colored yellow. Despite that, the model managed to generate images of an object that is visually reminiscent of a school bus that is painted with the specified color.

Apart from changing the colors of objects, we next examined whether changing the background of the scene described in a caption would result in the appropriate changes in the generated samples. The task of changing the background of an image is somewhat harder than just changing the color of an object because the model will have to make alterations over a wider visual area. Nevertheless, as shown in Fig. 4 changing the skies from blue to rainy in a caption as well as changing the grass type from dry to green in another caption resulted in the appropriate changes in the generated image.

Despite a large number of ways of changing colors and backgrounds in descriptions, in general we found that the model made appropriate changes as long as some similar pattern was present in the training set. However, the model struggled when the visual difference between objects was very small, such as when the objects have the same general shape and color. In Fig. 3, we demonstrate that when we swap two objects that are both visually similar, for example cats and dogs, it is difficult to discriminate solely from the generated samples whether it is an image of a cat or dog, even though we might notice an animal-like shape. This highlights a limitation of the model in that it has difficulty modelling the fine-grained details of objects.

As a test of model generalization, we tried generating images corresponding to captions that describe scenarios that are highly unlikely to occur in real life. These captions describe a common object doing unusual things or set in a strange location, for example “A toilet seat sits open in the grass field”. Even though some of these scenarios may never occur in real life, it is very easy for humans to imagine the corresponding scene. Nevertheless, as you can see in Fig. 1, the model managed to generate reasonable images.

1.2 Analysis of Attention

After flipping sets of words in the captions, we further explored which words the model attended to when generating images. It turned out that during the generation step, the model mostly focused on the specific words (or nearby words) that carried the main semantic meaning expressed in the sentences. The attention values of words in sentences helped us interpret the reasons why the model made the changes it did when we flipped certain words. For example, in Fig. 5, top row, we can see that when we flipped the word “desert” to “forest”, the attention over words in the sentence did not change drastically. This suggests that, in their respective sentences, the model looked at “desert” and “forest” with relatively equal probability, and thus made the correct changes. In contrast, when we swap words “beach” and “sun”, we can see a drastic change between sentences in the probability distribution over words. By noting that the model completely ignores the word “sun” in the second sentence, we can therefore gain a more thorough understanding of why we see no visual differences between the images generated by each caption.

We also tried to analyze the way the model generated images. Unfortunately, we found that there was no significant connection between the patches drawn on canvas and the most attended words at particular time-steps.

1.3 Comparison With Other Models

Quantitatively evaluating generative models remains a challenging task in itself as each method of evaluation suffers from its own specific drawbacks. Compared to reporting classification accuracies in discriminative models, the measures defining generative models are intractable most of the times and might not correctly define the real quality of the model. To get a better comparison between performances of different generative models, we report results on two different metrics as well as a qualitative comparison of different generative models.

We compared the performance of the proposed model to the DRAW model conditioned on captions without the $align$ function (noalignDRAW) as well as the DRAW model conditioned on the skipthought vectors of (Kiros et al., 2015) (skipthoughtDRAW). All of the conditional DRAW models were trained with a binary cross-entropy cost function, i.e. they had Bernoulli conditional likelihoods. We also compared our model with Fully-Connected (Fully-Conn) and Convolutional-Deconvolutional (Conv-Deconv) Variational Autoencoders which were trained with the least squares cost function. The LAPGAN model of (Denton et al., 2015) was trained on a two level Laplacian Pyramid with a GAN as a top layer generator and all stages were conditioned on the same skipthought vector.

In Fig. 5, bottom row, we generated several samples from the prior of each of the current state-of-the-art generative models, conditioned on the caption “A group of people walk on a beach with surf boards”. While all of the samples look sharp, the images generated by LAPGAN look more noisy and it is harder to make out definite objects, whereas the images generated by variational models trained with least squares cost function have a watercolor effect on the images. We found that the quality of generated samples was similar among different variants of conditional DRAW models.

As for the quantitative comparison of different models, we first compare the performances of the model trained with variational methods. We rank the images in the test set conditioned on the captions based on the variational lower bound of the log-probabilities and then report the Precision-Recall metric as an evaluation of the quality of the generative model (see Table 1.). Perhaps unsurprisingly, generative models did not perform well on the image retrieval task. To deal with the large computational complexity involved in looping through each test image, we create a shortlist of one hundred images including the correct one, based on the images having the closest Euclidean distance in the last fully-connected feature space of a VGG-like model (Simonyan & Zisserman, 2015) trained on the CIFAR datasetThe architecture of the model is described here http://torch.ch/blog/2015/07/30/cifar.html. The shortlist of test images used for evaluation can be downloaded from http://www.cs.toronto.edu/~emansim/cap2im/test-nns.pkl. (Krizhevsky, 2009). Since there are “easy” images for which the model assigns high log-probabilities independent of the query caption, we instead look at the ratio of the likelihood of the image conditioned on the sentence to the likelihood of the image conditioned on the mean sentence representation in the training set, following the retrieval protocol of (Kiros et al., 2014b). We also found that the lower bound on the test log-probabilities decreased for sharpened images, and that sharpening considerably hurt the retrieval results. Since sharpening changes the statistics of images, the estimated log-probabilities of image pixels is not necessarily a good metric. Some examples of generated images before and after sharpening are shown in Appendix C.

Instead of calculating error per pixel, we turn to a smarter metric, the Structural Similarity Index (SSI) (Wang et al., 2004), which incorporates luminance and contrast masking into the error calculation. Strong inter-dependencies of closer pixels are also taken into account and the metric is calculated on small windows of the images. Due to independence property of test captions, we sampled fifty images from the prior of each generative model for every caption in the test set in order to calculate SSI. As you can see on Table 1, SSI scores achieved by variational models were higher compared to SSI score achieved by LAPGAN.

Discussion

In this paper, we demonstrated that the alignDRAW model, a combination of a recurrent variational autoencoder with an alignment model over words, succeeded in generating images that correspond to a given input caption. By extensively using attentional mechanisms, our model gained several advantages. Namely, the use of the visual attention mechanism allowed us to decompose the problem of image generation into a series of steps instead of a single forward pass, while the attention over words provided us an insight whenever our model failed to generate a relevant image. Additionally, our model generated images corresponding to captions which generalized beyond the training set, such as sentences describing novel scenarios which are highly unlikely to occur in real life.

Because the alignDRAW model tends to output slightly blurry samples, we augmented the model with a sharpening post-processing step in which GAN generated edges which were added to the alignDRAW samples. Unfortunately, this is not an ideal solution due to the fact that the whole model was not trained in an end-to-end fashion. Therefore a direction of future work would be to find methods that can bypass the separate post-processing step and output sharp images directly in an end-to-end manner.

Acknowledgments: This work was supported by Samsung and IARPA, Raytheon BBN Contract No. D11PC20071. We would like to thank developers of Theano (Bastien et al., 2012), the authors of (Denton et al., 2015) for open sourcing their code, and Ryan Kiros and Nitish Srivastava for helpful discussions.

References

Appendix A: MNIST With Captions

As an additional experiment, we trained our model on the MNIST dataset with artificial captions. Either one or two digits from the MNIST training dataset were placed on a $60\times 60$ blank image. One digit was placed in one of the four (top-left, top-right, bottom-left or bottom-right) corners of the image. Two digits were either placed horizontally or vertically in non-overlapping fashion. The corresponding artificial captions specified the identity of each digit along with their relative positions, e.g. “The digit three is at the top of the digit one”, or “The digit seven is at the bottom left of the image”.

The generated images together with the attention alignments are displayed in Figure 6. The model correctly displayed the specified digits at the described positions and even managed to generalize reasonably well to the configurations that were never present during training. In the case of generating two digits, the model would dynamically attend to the digit in the caption it was drawing at that particular time-step. Similarly, in the setting where the caption specified only a single digit, the model would correctly attend to the digit in the caption during the whole generation process. In both cases, the model placed small attention values on the words describing the position of digits in the images.

Appendix B: Training Details

Each parameter in alignDRAW was initialized by sampling from a Gaussian distribution with mean and standard deviation $0.01$. The model was trained using RMSprop with an initial learning rate of $0.001$. For the Microsoft COCO task, we trained our model for $18$ epochs. The learning rate was reduced to $0.0001$ after $11$ epochs. For the MNIST with Captions task, the model was trained for 150 epochs and the learning rate was reduced to $0.0001$ after 110 epochs. During each epoch, randomly created $10,000$ training samples were used for learning. The norm of the gradients was clipped at $10$ during training to avoid the exploding gradients problem.

The GAN model used for sharpening had the same configuration as the $28\times 28$ model trained by Denton et al. (2015) on the edge residuals of the CIFAR dataset. The configuration can be found at https://gist.github.com/soumith/e3f722173ea16c1ea0d9. The model was trained for $6$ epochs.

Evaluation

Table 3 shows the estimated variational lower bounds on the average train/validation/test log-probabilities. Note that the alignDRAW model does not suffer much from overfitting. The results substantially worsen after sharpening test images.

Appendix C: Effect of Sharpening Images.

Some examples of generated images before (top row) and after (bottom row) sharpening images using an adversarial network trained on residuals of a Laplacian pyramid conditioned on the skipthought vectors of the captions.