Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering

Introduction

Visual Question Answering (VQA) is an emerging interdisciplinary research problem at the intersection of computer vision, natural language processing and artificial intelligence. It has many real-life applications, such as automatic querying of surveillance video or assisting the visually impaired . Compared to the recently popular image captioning task , VQA requires a deeper understanding of the image, but is considerably easier to evaluate. It also puts more focus on artificial intelligence, namely the inference process needed to produce the answer to the visual question.

In one of the early works , VQA is seen as a Turing test proxy. The authors propose an approach based on handcrafted features using a semantic parse of the question and scene analysis of the image combined in a latent-world Bayesian framework. More recently, several end-to-end deep neural networks that learn features directly from data have been applied to this problem . Most of these are directly adapted from captioning models , and utilize a recurrent LSTM network, which takes the question and Convolutional Neural Net (CNN) image features as input, and outputs the answer. Though the deep learning methods in have shown great improvement compared to the handcrafted feature method , they have their own drawbacks. These models based on the LSTM reading in both the question and the image features do not show a clear improvement compared to an LSTM reading in the question only . Furthermore, the rather complicated LSTM models obtain similar or worse accuracy to a baseline model which concatenates CNN features and a bag-of-words question embedding to predict the answer, see the IMG+BOW model in and the iBOWIMG model in .

A major drawback of existing models is that they do not have any explicit notion of object position, and do not support the computation of intermediate results based on spatial attention. Our intuition is that answering visual questions often involves looking at different spatial regions and comparing their contents and/or locations. For example, to answer the questions in Fig. 1, we need to look at a portion of the image, such as the child or the phone booth. Similarly, to answer the question “Is there a cat in the basket?” in Fig. 2, we can first find the basket and the cat objects, and then compare their locations.

We propose a new deep learning approach to VQA that incorporates explicit spatial attention, which we call the Spatial Memory Network VQA (SMem-VQA). Our approach is based on memory networks, which have recently been proposed for text Question Answering (QA) . Memory networks combine learned text embeddings with an attention mechanism and multi-step inference. The text QA memory network stores textual knowledge in its “memory” in the form of sentences, and selects relevant sentences to infer the answer. However, in VQA, the knowledge is in the form of an image, thus the memory and the question come from different modalities. We adapt the end-to-end memory network to solve visual question answering by storing the convolutional network outputs obtained from different receptive fields into the memory, which explicitly allows spatial attention over the image. We also propose to repeat the process of gathering evidence from attended regions, enabling the model to update the answer based on several attention steps, or “hops”. The entire model is trained end-to-end and the evidence for the computed answer can be visualized using the attention weights.

To summarize our contributions, in this paper we

propose a novel multi-hop memory network with spatial attention for the VQA task which allows one to visualize the spatial inference process used by the deep network (a CAFFE implementation will be made available),

design an attention architecture in the first hop which uses each word embedding to capture fine-grained alignment between the image and question,

create a series of synthetic questions that explicitly require spatial inference to analyze the working principles of the network, and show that it learns logical inference rules by visualizing the attention weights,

provide an extensive evaluation of several existing models and our own model on the same publicly available datasets.

Sec. 2 introduces relevant work on memory networks and attention models. Sec. 3 describes our design of the multi-hop memory network architecture for visual question answering (SMem-VQA). Sec. 4 visualizes the inference rules learned by the network for synthetic spatial questions and shows the experimental results on DAQUAR and VQA datasets. Sec. 5 concludes the paper.

Related work

Before the popularity of visual question answering (VQA), text question answering (QA) had already been established as a mature research problem in the area of natural language processing. Previous QA methods include searching for the key words of the question in a search engine ; parsing the question as a knowledge base (KB) query ; or embedding the question and using a similarity measurement to find evidence for the answer . Recently, memory networks were proposed for solving the QA problem. first introduces the memory network as a general model that consists of a memory and four components: input feature map, generalization, output feature map and response. The model is investigated in the context of question answering, where the long-term memory acts as a dynamic knowledge base and the output is a textual response. proposes a competitive memory network model that uses less supervision, called end-to-end memory network, which has a recurrent attention model over a large external memory. The Neural Turing Machine (NTM) couples a neural network to external memory and interacts with it by attentional processes to infer simple algorithms such as copying, sorting, and associative recall from input and output examples. In this paper, we solve the VQA problem using a multimodal memory network architecture that applies a spatial attention mechanism over an input image guided by an input text question.

The neural attention mechanism has been widely used in different areas of computer vision and natural language processing, see for example the attention models in image captioning , video description generation , machine translation and machine reading systems . Most methods use the soft attention mechanism first proposed in , which adds a layer to the network that predicts soft weights and uses them to compute a weighted combination of the items in memory. The two main types of soft attention mechanisms differ in the function that aligns the input feature vector and the candidate feature vectors in order to compute the soft attention weights. The first type uses an alignment function based on “concatenation” of the input and each candidate (we use the term “concatenation” as described ), and the second type uses an alignment function based on the dot product of the input and each candidate. The “concatenation” alignment function adds one input vector (e.g. hidden state vector of the LSTM) to each candidate feature vector, embeds the resulting vectors into scalar values, and then applies the softmax function to generate the attention weight for each candidate. use the “concatenation” alignment function in their soft attention models and gives a literature review of such models applied to different tasks. On the other hand, the dot product alignment function first projects both inputs to a common vector embedding space, then takes the dot product of the two input vectors, and applies a softmax function to the resulting scalar value to produce the attention weight for each candidate. The end-to-end memory network uses the dot product alignment function. In , the authors compare these two alignment functions in an attention model for the neural machine translation task, and find that their implementation of the “concatenation” alignment function does not yield good performance on their task. Motivated by this, in this paper we use the dot product alignment function in our Spatial Memory Network.

VQA is related to image captioning. Several early papers about VQA directly adapt the image captioning models to solve the VQA problem by generating the answer using a recurrent LSTM network conditioned on the CNN output. But these models’ performance is still limited . proposes a new dataset and uses a similar attention model to that in image captioning , but does not give results on the more common VQA benchmark , and our own implementation of this model is less accurate on than other baseline models. summarizes several recent papers reporting results on the VQA dataset on arxiv.org and gives a simple but strong baseline model (iBOWIMG) on this dataset. This simple baseline concatenates the image features with the bag of word embedding question representation and feeds them into a softmax classifier to predict the answer. The iBOWIMG model beats most VQA models considered in the paper. Here, we compare our proposed model to the VQA models (namely, the ACK model and the DPPnet model ) which have comparable or better results than the iBOWIMG model. The ACK model in is essentially the same as the LSTM model in , except that it uses image attribute features, the generated image caption and relevant external knowledge from a knowledge base as the input to the LSTM’s first time step. The DPPnet model in tackles VQA by learning a convolutional neural network (CNN) with some parameters predicted from a separate parameter prediction network. Their parameter prediction network uses a Gate Recurrent Unit (GRU) to generate a question representation, and based on this question input, maps the predicted weights to CNN via hashing. Neither of these models contain a spatial attention mechanism, and they both use external data in addition to the VQA dataset , e.g. the knowledge base in and the large-scale text corpus used to pre-train the GRU question representation . In this paper, we explore a complementary approach of spatial attention to both improve performance and visualize the network’s inference process, and obtain improved results without using external data compared to the iBOWIMG model as well as the ACK model and the DPPnet model which use external data.

Spatial Memory Network for VQA

We first give an overview of the proposed SMem-VQA network, illustrated in Fig. 2 (a). Sec. 3.1 details the word-guided spatial attention process of the first hop shown in Fig. 2 (b), and Sec. 3.2 describes adding a second hop into SMem-VQA network.

The convolutional image feature vectors at each location are embedded into a common semantic space with the word vectors. Two different embeddings are used: the “attention” embedding $W_{A}$ and the “evidence” embedding $W_{E}$. The attention embedding projects each visual feature vector such that its combination with the embedded question words generates the attention weight at that location. The evidence embedding detects the presence of semantic concepts or objects, and the embedding results are multiplied with attention weights and summed over all locations to generate the visual evidence vector $S_{att}$.

Finally, the visual evidence vector is combined with the question representation and used to predict the answer for the given image and question. In the next section, we describe the one-hop Spatial Memory network model and the specific attention mechanism it uses in more detail.

The spatial attention weights $W_{att}$ are calculated by taking maximum over the word dimension $T$ for the correlation matrix $C$, selecting the highest correlation value for each spatial location, and then applying the softmax function

Finally, the sum of this evidence vector $S_{att}$ and the question embedding $Q$ is used to predict the answer for the given image and question. For the question representation $Q$, we choose the bag-of-words (BOW). Other question representations, such as an LSTM, can also be used, however, BOW has fewer parameters yet has shown good performance. As noted in , the simple BOW model performs roughly as well if not better than the sequence-based LSTM for the VQA task. Specifically, we compute

2 Spatial Attention in Two-Hop Model

We can repeat hops to promote deeper inference, gathering additional evidence at each hop. Recall that the visual evidence vector $S_{att}$ is added to the question representation $Q$ in the first hop to produce an updated question vector,

The attention weights in the second hop $W_{att2}$ are obtained by applying the softmax function to the correlation vector $C_{hop2}$.

The final answer $P$ is predicted by combining the whole question representation $Q$, the local visual evidence $S_{att}$ from each word vector in the first hop and the global visual evidence $S_{att2}$ from the whole question in the second hop,

The entire network is differentiable and is trained using stochastic gradient descent via standard backpropagation, allowing image feature extraction, image embedding, word embedding and answer prediction to be jointly optimized on the training image/question/answer triples.

Experiments

In this section, we conduct a series of experiments to evaluate our model. To explore whether the model learns to perform the spatial inference necessary for answering visual questions that explicitly require spatial reasoning, we design a set of experiments using synthetic visual question/answer data in Sec. 4.1. The experimental results of our model in standard datasets (DAQUAR and VQA datasets) are reported in Sec. 4.2.

The questions in the public VQA datasets are quite varied and difficult and often require common sense knowledge to answer (e.g., “Does this man have 20/20 vision?” about a person wearing glasses). Furthermore, past work showed that the question text alone (no image) is a very strong predictor of the answer. Therefore, before evaluating on standard datasets, we would first like to understand how the proposed model uses spatial attention to answer simple visual questions where the answer cannot be predicted from question alone. Our visualization demonstrates that the attention mechanism does learn to attend to objects and gather evidence via certain inference rules.

We investigate whether the model has the ability to recognize the absolute location of the object in the image. We explore this by designing a simple task where an object (a red square) appears in some region of a white-background image, and the question is “Is there a red square on the [top$\mid$bottom$\mid$left$\mid$right]?” For each image, we randomly place the square in one of the four regions, and generate the four questions above, together with three “no” answers and one “yes” answer. The generated data is split into training and testing sets.

Due to the simplicity of this synthetic dataset, the SMem-VQA one-hop model achieves 100% test accuracy. However, the baseline model (iBOWIMG) cannot infer the answer and only obtains accuracy of around 75%, which is the prior probability of the answer “no” in the training set. The SMem-VQA one-hop model is equivalent to the iBOWIMG model if the attention weights in our one-hop model are set equally for each location, since the iBOWIMG model uses the mean pool of the convolutional feature ($inception\_5b/output$) in GoogLeNet that we use in SMem-VQA model. We check the visualization of the attention weights and find that the relationship between the high attention position and the answer can be expressed by logical expressions. We show the attention weights of several typical examples in Fig. 3 which reflect two logic rules: 1) Look at the position specified in question (top$\mid$bottom$\mid$right$\mid$left), if it contains a square, then answer “yes”; if it does not contain a square, then answer “no”. 2) Look at the region where there is a square, then answer “yes” for the question about that position and “no” for the questions about the other three positions.

In the iBOWIMG model, the mean-pooled GoogLeNet visual features lose spatial information and thus cannot distinguish images with a square in different positions. On the contrary, our SMem-VQA model can pay high attention to different regions according to the question, and generate an answer based on the selected region, using some learned inference rules. This experiment demonstrates that the attention mechanism in our model is able to make absolute spatial location inference based on the spatial attention.

1.2 Relative Position Recognition

In order to check whether the model has the ability to infer the position of one object relative to another object, we collect all the cat images from the MS COCO Detection dataset , and add a red square on the [top$\mid$bottom$\mid$left$\mid$right] of the bounding box of the cat in the images. For each generated image, we create four questions, “Is there a red square on the [top$\mid$bottom$\mid$left$\mid$right] of the cat?” together with three “no” answers and one “yes” answer. We select 2639 training cat images and 1395 testing cat images from MS COCO Detection dataset.

Our SMem-VQA one-hop model achieves 96% test accuracy on this synthetic task, while the baseline model (iBOWIMG) accuracy is around 75%. We also check that another simple baseline that predicts the answer based on the absolute position of the square in the image gets around 70% accuracy. We visualize the evidence embedding $W_{E}$ features and the attention weights $W_{att}$ of several typical examples in Fig. 4. The evidence embedding $W_{E}$ has high activations on the cat and the red square, while the attention weights pay high attention to certain locations around the cat. We can analyze the attention in the correctly predicted examples using the same rules as in absolute position recognition experiment. These rules still work, but the position is relative to the cat object: 1) Check the specified position relative to the cat, if it finds the square, then answer “yes”, otherwise “no”; 2) Find the square, then answer “yes” for the specified position, and answer “no” for the other positions around the cat. We also check the images where our model makes mistakes, and find that the mistakes mainly occur in images with more than one cats. The red square appears near only one of the cats in the image, but our model might make mistakes by focusing on the other cats. We conclude that our SMem-VQA model can infer the relative spatial position based on the spatial attention around the specified object, which can also be represented by some logical inference rules.

2 Experiments on Standard Datasets

The DAQUAR dataset is a relatively small dataset which builds on the NYU Depth Dataset V2 . We use the reduced DAQUAR dataset . The evaluation metric for this dataset is 0-1 accuracy. The embedding dimension is 512 for our models running on the DAQUAR dataset. We use several reported models on DAQUAR as baselines, which are listed below: $\bullet$ Multi-World : an approach based on handcrafted features using a semantic parse of the question and scene analysis of the image combined in a latent-world Bayesian framework. $\bullet$ Neural-Image-QA : uses an LSTM to encode the question and then decode the hidden information into the answer. The image CNN feature vector is shown at each time step of the encoding phase. $\bullet$ Question LSTM : only shows the question to the LSTM to predict the answer without any image information. $\bullet$ VIS+LSTM : similar to Neural-Image-QA, but only shows the image features to the LSTM at the first time step, and the question in the remaining time steps to predict the answer. $\bullet$ Question BOW : only uses the BOW question representation and a single hidden layer neural network to predict the answer, without any image features. $\bullet$ IMG+BOW : concatenates the BOW question representation with image features, and then uses a single hidden layer neural network to predict the answer. This model is similar to the iBOWIMG baseline model in .

Results of our SMem-VQA model on the DAQUAR dataset and the baseline model results reported in previous work are shown in Tab. 1. From the DAQUAR result in Tab. 1, we see that models based on deep features significantly outperform the Multi-World approach based on hand-crafted features. Modeling the question only with either the LSTM model or Question BOW model does equally well in comparison, indicating the the question text contains important prior information for predicting the answer. Also, on this dataset, the VIS+LSTM model achieves better accuracy than Neural-Image-QA model; the former shows the image only at the first timestep of the LSTM, while the latter does so at each timestep. In comparison, both our One-Hop model and Two-Hop spatial attention models outperform the IMG+BOW, as well as the other baseline models. A major advantage of our model is the ability to visualize the inference process in the deep network. To illustrate this, two attention weights visualization examples in SMem-VQA One-Hop and Two-Hop models on DAQUAR dataset are shown in Fig. 5 (bottom row).

2.2 Results on VQA

The VQA dataset is a recent large dataset based on MS COCO . We use the full release (V1.0) open-ended dataset, which contains a train set and a val set. Following standard practice, we choose the top 1000 answers in train and val sets as possible prediction answers, and only keep the examples whose answers belong to these 1000 answers as training data. The question vocabulary size is 7477 with the word frequency of at least three. Because of the larger training size, the embedding dimension is 1000 on the VQA dataset. We report the test-dev and test-standard results from the VQA evaluation server. The server evaluation uses the evaluation metric introduced by , which gives partial credit to certain synonym answers: $Acc({ans})=\min\left\{(\text{\# humans that said }{ans})/3,1\right\}$.

For the attention models, we do not mirror the input image when using the CNN to extract convolutional features, since this might cause confusion about the spatial locations of objects in the input image. The optimization algorithm used is stochastic gradient descent (SGD) with a minibatch of size 50 and momentum of 0.9. The base learning rate is set to be 0.01 which is halved every six epoches. Regularization, dropout and L2 norm are cross-validated and used.

For the VQA dataset, we use the simple iBOWIMG model in as one baseline model, which beats most existing VQA models currently on arxiv.org. We also compare to two models in which have comparable or better results to the iBOWIMG model. These three baseline models as well the best model in VQA dataset paper are listed in the following: $\bullet$ LSTM Q+I : uses the element-wise multiplication of the LSTM encoding of the question and the image feature vector to predict the answer. This is the best model in the VQA dataset paper. $\bullet$ ACK : shows the image attribute features, the generated image caption and relevant external knowledge from knowledge base to the LSTM at the first time step, and the question in the remaining time steps to predict the answer. $\bullet$ DPPnet : uses the Gated Recurrent Unit (GRU) representation of question to predict certain parameters for a CNN classification network. They pre-train the GRU for question representation on a large-scale text corpus to improve the GRU generalization performance. $\bullet$ iBOWIMG : concatenates the BOW question representation with image feature (GoogLeNet), and uses a softmax classification to predict the answer.

The overall accuracy and per-answer category accuracy for our SMem-VQA models and the four baseline models on VQA dataset are shown in Tab. 2. From the table, we can see that the SMem-VQA One-Hop model obtains slightly better results compared to the iBOWIMG model. However, the SMem-VQA Two-Hop model achieves an improvement of 2.27% on test-dev and 2.35% on test-standard compared to the iBOWIMG model, demonstrating the value of spatial attention. The SMem-VQA Two-Hop model also shows best performance in the per-answer category accuracy. The SMem-VQA Two-Hop model has slightly better result than the DPPnet model. The DPPnet model uses a large-scale text corpus to pre-train the Gated Recurrent Unit (GRU) network for question representation. Similar pre-training work on extra data to improve model accuracy has been done in . Considering the fact that our model does not use extra data to pre-train the word embeddings, its results are very competitive. We also experiment with adding a third hop into our model on the VQA dataset, but the result does not improve further.

The attention weights visualization examples for the SMem-VQA One-Hop and Two-Hop models on the VQA dataset are shown in Fig. 5 (top row). From the visualization, we can see that the two-hop model collects supplementary evidence for inferring the answer, which may be necessary to achieve an improvement on these complicated real-world datasets. We also visualize the fine-grained alignment in the first hop of our SMem-VQA Two-Hop model in Fig. 6. The correlation vector values (blue bars) measure the correlation between image regions and each word vector in the question. Higher values indicate stronger correlation of that particular word with the specific location’s image features. We observe that the fine-grained visual evidence collected using each local word vector, together with the global visual evidence from the whole question, complement each other to infer the correct answer for the given image and question, as shown in Fig. 1.

Conclusion

In this paper, we proposed the Spatial Memory Network for VQA, a memory network architecture with a spatial attention mechanism adapted to the visual question answering task. We proposed a set of synthetic spatial questions and demonstrated that our model learns inference rules based on spatial attention through attention weight visualization. Evaluation on the challenging DAQUAR and VQA datasets showed improved results over previously published models. Our model can be used to visualize the inference steps learned by the deep network, giving some insight into its processing. Future work may include further exploring the inference ability of our SMem-VQA model and exploring other VQA attention models.

References