Contrastive Learning for Image Captioning

Introduction

Image captioning, a task to generate natural descriptions of images, has been an active research topic in computer vision and machine learning. Thanks to the advances in deep neural networks, especially the wide adoption of RNN and LSTM, there has been substantial progress on this topic in recent years . However, studies have shown that even the captions generated by state-of-the-art models still leave a lot to be desired. Compared to human descriptions, machine-generated captions are often quite rigid and tend to favor a “safe” (i.e. matching parts of the training captions in a word-by-word manner) but restrictive way. As a consequence, captions generated for different images, especially those that contain objects of the same categories, are sometimes very similar , despite their differences in other aspects.

We argue that distinctiveness, a property often overlooked in previous work, is significant in natural language descriptions. To be more specific, when people describe an image, they often mention or even emphasize the distinctive aspects of an image that distinguish it from others. With a distinctive description, someone can easily identify the image it is referring to, among a number of similar images. In this work, we performed a self-retrieval study (see Section 4.1), which reveals the lack of distinctiveness affects the quality of descriptions.

From a technical standpoint, the lack of distinctiveness is partly related to the way that the captioning model was learned. A majority of image captioning models are learned by Maximum Likelihood Estimation (MLE), where the probabilities of training captions conditioned on corresponding images are maximized. While well grounded in statistics, this approach does not explicitly promote distinctiveness. Specifically, the differences among the captions of different images are not explicitly taken into account. We found empirically that the resultant captions highly resemble the training set in a word-by-word manner, but are not distinctive.

In this paper, we propose Contrastive Learning (CL), a new learning method for image captioning, which explicitly encourages distinctiveness, while maintaining the overall quality of the generated captions. Specifically, it employs a baseline, e.g. a state-of-the-art model, as a reference. During learning, in addition to true image-caption pairs, denoted as $(I,c)$, this method also takes as input mismatched pairs, denoted as $(I,c_{/})$, where $c_{/}$ is a caption describing another image. Then, the target model is learned to meet two goals, namely (1) giving higher probabilities $p(c|I)$ to positive pairs, and (2) lower probabilities $p(c_{/}|I)$ to negative pairs, compared to the reference model. The former ensures that the overall performance of the target model is not inferior to the reference; while the latter encourages distinctiveness.

It is noteworthy that the proposed learning method (CL) is generic. While in this paper, we focused on models based on recurrent neural networks , the proposed method can also generalize well to models based on other formulations, e.g. probabilistic graphical models . Also, by choosing the state-of-the-art model as the reference model in CL, one can build on top of the latest advancement in image captioning to obtain improved performances.

Related Work

The history of image captioning can date back to decades ago. Early attempts are mostly based on detections, which first detect visual concepts (e.g. objects and their attributes) followed by template filling or nearest neighbor retrieving for caption generation . With the development of neural networks, a more powerful paradigm, encoder-and-decoder, was proposed by , which then becomes the core of most state-of-the-art image captioning models. It uses a CNN to represent the input image with a feature vector, and applies a LSTM net upon the feature to generate words one by one.

Based on the encoder-and-decoder, many variants are proposed, where attention mechanism appears to be the most effective add-on. Specifically, attention mechanism replaces the feature vector with a set of feature vectors, such as the features from different regions , and those under different conditions . It also uses the LSTM net to generate words one by one, where the difference is that at each step, a mixed guiding feature over the whole feature set, will be dynamically computed. In recent years, there are also approaches combining attention mechanism and detection. Instead of doing attention on features, they consider the attention on a set of detected visual concepts, such as attributes and objects .

Despite of the specific structure of any image captioning model, it is able to give $p(c|I)$, the probability of a caption conditioned on an image. Therefore, all image captioning models can be used as the target or the reference in CL method.

Learning Methods for Image Captioning

Many state-of-the-art image captioning models adopt Maximum Likelihood Estimation (MLE) as their learning method, which maximizes the conditional log-likelihood of the training samples, as:

where $\boldsymbol{\theta}$ is the parameter vector, $I_{i}$ and $c_{i}=(w^{(1)}_{i},w^{(2)}_{i},...,w^{(T_{i})}_{i})$ are a training image and its caption. Although effective, some issues, including high resemblance in model-gerenated captions, are observed on models learned by MLE.

Facing these issues, alternative learning methods are proposed in recent years. Techniques of reinforcement learning (RL) have been introduced in image captioning by and . RL sees the procedure of caption generation as a procedure of sequentially sampling actions (words) in a policy space (vocabulary). The rewards in RL are defined to be evaluation scores of sampled captions. Note that distinctiveness has not been considered in both approaches, RL and MLE.

Prior to this work, some relevant ideas have been explored . Specifically, proposed an introspective learning (IL) approach that learns the target model by comparing its outputs on $(I,c)$ and $(I_{/},c)$. Note that IL uses the target model itself as a reference. On the contrary, the reference model in CL provides more independent and stable indications about distinctiveness. In addition, $(I_{/},c)$ in IL is pre-defined and fixed across the learning procedure, while the negative sample in CL, i.e. $(I,c_{/})$, is dynamically sampled, making it more diverse and random. Recently, Generative Adversarial Networks (GAN) was also adopted for image captioning , which involves an evaluator that may help promote the distinctiveness. However, this evaluator is learned to directly measure the distinctiveness as a parameterized approximation, and the approximation accuracy is not ensured in GAN. In CL, the fixed reference provides stable bounds about the distinctiveness, and the bounds are supported by the model’s performance on image captioning. Besides that, is specifically designed for models that generate captions word-by-word, while CL is more generic.

Background

Our formulation is partly inspired by Noise Contrastive Estimation (NCE) . NCE is originally introduced for estimating probability distributions, where the partition functions can be difficult or even infeasible to compute. To estimate a parametric distribution $p_{m}(.;\boldsymbol{\theta})$, which we refer to as the target distribution, NCE employs not only the observed samples $X=(\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{T_{m}})$, but also the samples drawn from a reference distribution $p_{n}$, denoted as $Y=(\mathbf{y}_{1},\mathbf{y}_{2},...,\mathbf{y}_{T_{n}})$. Instead of estimating $p_{m}(.;\boldsymbol{\theta})$ directly, NCE estimates the density ratio $p_{m}/p_{n}$ by training a classifier based on logistic regression.

Specifically, let $U=(\mathbf{u}_{1},...,\mathbf{u}_{T_{m}+T_{n}})$ be the union of $X$ and $Y$. A binary class label $C_{t}$ is assigned to each $u_{t}$, where $C_{t}=1$ if $u_{t}\in X$ and $C_{t}=0$ if $u_{t}\in Y$. The posterior probabilities for the class labels are therefore

where $\nu=T_{n}/T_{m}$. Let $G(\mathbf{u};\boldsymbol{\theta})=\ln p_{m}(\mathbf{u};\boldsymbol{\theta})-\ln p_{n}(\mathbf{u})$ and $h(\mathbf{u},\boldsymbol{\theta})=P(C=1|\mathbf{u},\boldsymbol{\theta})$, then we can write

The objective function of NCE is the joint conditional log-probabilities of $C_{t}$ given the samples $U$, which can be written as

Maximizing this objective with respect to $\boldsymbol{\theta}$ leads to an estimation of $G(\cdot;\boldsymbol{\theta})$, the logarithm of the density ratio $p_{m}/p_{n}$. As $p_{n}$ is a known distribution, $p_{m}(:|\boldsymbol{\theta})$ can be readily derived.

Contrastive Learning for Image Captioning

Learning a model by characterizing desired properties relative to a strong baseline is a convenient and often quite effective way in situations where it is hard to describe these properties directly. Specifically, in image captioning, it is difficult to characterize the distinctiveness of natural image descriptions via a set of rules, without running into the risk that some subtle but significant points are missed. Our idea in this work is to introduce a baseline model as a reference, and try to enhance the distinctiveness on top, while maintaining the overall quality of the generated captions.

In the following we will first present an empirical study on the correlation between distinctiveness of its generated captions and the overall performance of a captioning model. Subsequently, we introduce the main framework of Contrastive Learning in detail.

In most of the existing learning methods of image captioning, models are asked to generate a caption that best describes the semantics of a given image. In the meantime, distinctiveness of the caption, which, on the other hand, requires the image to be the best matching among all images for the caption, has not been explored. However, distinctiveness is crucial for high-quality captions. A study by Jas showed that specificity is common in human descriptions, which implies that image descriptions often involve distinctive aspects. Intuitively, a caption satisfying this property is very likely to contain key and unique content of the image, so that the original image could easily be retrieved when the caption is presented.

To verify this intuition, we conducted an empirical study which we refer to as self retrieval. In this experiment, we try to retrieve the original image given its model-generated caption and investigate top-$k$ recalls, as illustrated in Figure 1. Specifically, we randomly sampled $5,000$ images $(I_{1},I_{2},...,I_{5000})$ from standard MSCOCO test set as the experiment benchmark. For an image captioning model $p_{m}(:,\boldsymbol{\theta})$, we first ran it on the benchmark to get corresponding captions $(c_{1},c_{2},...,c_{5000})$ for the images. After that, using each caption $c_{t}$ as a query, we computed the conditional probabilities $(p_{m}(c_{t}|I_{1}),p_{m}(c_{t}|I_{2}),...,p_{m}(c_{t}|I_{5000}))$, which were used to get a ranked list of images, denoted by $\mathbf{r}_{t}$. Based on all ranked lists, we can compute top-$k$ recalls, which is the fraction of images within top-$k$ positions of their corresponding ranked lists. The top-$k$ recalls are good indicators of how well a model captures the distinctiveness of descriptions.

In this experiment, we compared three different models, including Neuraltalk2 and AdaptiveAttention that are learned by MLE, as well as AdaptiveAttention learned by our method. The top-$k$ recalls are listed in Table 1, along with overall performances of these models in terms of Rouge and Cider . These results clearly show that the recalls of self retrieval are positively correlated to the performances of image captioning models in classical captioning metrics. Although most of the models are not explicitly learned to promote distinctiveness, the one with better recalls of self retrieval, which means the generated-captions are more distinctive, performs better in the image captioning evaluation. Such positive correlation clearly demonstrates the significance of distinctiveness to captioning performance.

2 Contrastive Learning

In Contrastive Learning (CL), we learn a target image captioning model $p_{m}(:;\boldsymbol{\theta})$ with parameter $\boldsymbol{\theta}$ by constraining its behaviors relative to a reference model $p_{n}(:;\boldsymbol{\phi})$ with parameter $\boldsymbol{\phi}$. The learning procedure requires two sets of data: (1) the observed data $X$, which is a set of ground-truth image-caption pairs $((c_{1},I_{1}),(c_{2},I_{2}),...,(c_{T_{m}},I_{T_{m}}))$, and is readily available in any image captioning dataset, (2) the noise set $Y$, which contains mismatched pairs $((c_{/1},I_{1}),(c_{/2},I_{2}),...,(c_{/T_{n}},I_{T_{n}}))$, and can be generated by randomly sampling $c_{/t}\in\mathcal{C}_{/I_{t}}$ for each image $I_{t}$, where $\mathcal{C}_{/I_{t}}$ is the set of all ground-truth captions except captions of image $I_{t}$. We refer to $X$ as positive pairs while $Y$ as negative pairs.

For any pair $(c,I)$, the target model and the reference model will respectively give their estimated conditional probabilities $p_{m}(c|I,\boldsymbol{\theta})$ and $p_{n}(c|I,\boldsymbol{\phi})$. We wish that $p_{m}(c_{t}|I_{t},\boldsymbol{\theta})$ is greater than $p_{n}(c_{t}|I_{t},\boldsymbol{\phi})$ for any positive pair $(c_{t},I_{t})$, and vice versa for any negative pair $(c_{/t},I_{t})$. Following this intuition, our initial attempt was to define $D((c,I);\boldsymbol{\theta},\boldsymbol{\phi})$, the difference between $p_{m}(c|I,\boldsymbol{\theta})$ and $p_{n}(c|I,\boldsymbol{\phi})$, as

In practice, this formulation would meet with several difficulties. First, $p_{m}(c|I,\boldsymbol{\theta})$ and $p_{n}(c|I,\boldsymbol{\phi})$ are very small ($\sim 1e$-$8$), which may result in numerical problems. Second, Eq (6) treats easy samples, hard samples, and mistaken samples equally. This, however, is not the most effective way. For example, when $D((c_{t},I_{t});\boldsymbol{\theta},\boldsymbol{\phi})\gg 0$ for some positive pair, further increasing $D((c_{t},I_{t});\boldsymbol{\theta},\boldsymbol{\phi})$ is probably not as effective as updating $D((c_{t^{\prime}},I_{t^{\prime}});\boldsymbol{\theta},\boldsymbol{\phi})$ for another positive pair, for which $D((c_{t^{\prime}},I_{t^{\prime}});\boldsymbol{\theta},\boldsymbol{\phi})$ is much smaller.

To resolve these issues, we adopted an alternative formulation inspired by NCE (sec 3), where we replace the difference function $D((c,I);\boldsymbol{\theta},\boldsymbol{\phi})$ with a log-ratio function $G((c,I);\boldsymbol{\theta},\boldsymbol{\phi})$:

and further use a logistic function $r_{\nu}$ (Eq(3)) after $G((c,I);\boldsymbol{\theta},\boldsymbol{\phi})$ to saturate the influence of easy samples. Following the notations in NCE, we let $\nu=T_{n}/T_{m}$, and turn $D((c,I);\boldsymbol{\theta},\boldsymbol{\phi})$ into:

Note that $h((c,I);\boldsymbol{\theta},\boldsymbol{\phi})\in(0,1)$. Then, we define our updated loss function as:

For the setting of $\nu=T_{n}/T_{m}$, we choose $\nu=1$, i.e. $T_{n}=T_{m}$, to ensure balanced influences from both positive and negative pairs. This setting consistently yields good performance in our experiments. Furthermore, we copy $X$ for $K$ times and sample $K$ different $Y$s, in order to involve more diverse negative pairs without overfitted to them. In practice we found $K=5$ is sufficient to make the learning stable. Finally, our objective function is defined to be

Note that $J(\boldsymbol{\theta})$ attains its upper bound if positive and negative pairs can be perfectly distinguished, namely, for all $t$, $h((c_{t},I_{t});\boldsymbol{\theta},\boldsymbol{\phi})=1$ and $h((c_{/t},I_{t});\boldsymbol{\theta},\boldsymbol{\phi})=0$. In this case, $G((c_{t},I_{t});\boldsymbol{\theta},\boldsymbol{\phi})\to\infty$ and $G((c_{/t},I_{t});\boldsymbol{\theta},\boldsymbol{\phi})\to-\infty$, which indicates the target model will give higher probability $p(c_{t}|I_{t})$ and lower probability $p(c_{/t}|I_{t})$, compared to the reference model. Towards this goal, the learning process would encourage distinctiveness by suppressing negative pairs, while maintaining the overall performance by maximizing the probability values on positive pairs.

3 Discussion

Maximum Likelihood Estimation (MLE) is a popular learning method in the area of image captioning . The objective of MLE is to maximize only the probabilities of ground-truth image-caption pairs, which may lead to some issues , including high resemblance in generated captions. While in CL, the probabilities of ground-truth pairs are indirectly ensured by the positive constraint (the first term in Eq(9)), and the negative constraint (the second term in Eq(9)) suppresses the probabilities of mismatched pairs, forcing the target model to also learn from distinctiveness.

Generative Adversarial Network (GAN) is a similar learning method that involves an auxiliary model. However, in GAN the auxiliary model and the target model follow two opposite goals, while in CL the auxiliary model and the target model are models in the same track. Moreover, in CL the auxiliary model is stable across the learning procedure, while itself needs careful learning in GAN.

It’s worth noting that although our CL method bears certain level of resemblance with Noise Contrastive Estimation (NCE) . The motivation and the actual technical formulation of CL and NCE are essentially different. For example, in NCE the logistic function is a result of computing posterior probabilities, while in CL it is explicitly introduced to saturate the influence of easy samples.

As CL requires only $p_{m}(c|I)$ and $p_{n}(c|I)$, the choices of the target model and the reference model can range from models based on LSTMs to models in other formats, such as MRFs and memory-networks . On the other hand, although in CL, the reference model is usually fixed across the learning procedure, one can replace the reference model with the latest target model periodically. The reasons are (1) $\nabla J(\boldsymbol{\theta})\neq\mathbf{0}$ when the target model and the reference model are identical, (2) latest target model is usually stronger than the reference model, (3) and a stronger reference model can provide stronger bounds and lead to a stronger target model.

Experiment

We use two large scale datasets to test our contrastive learning method. The first dataset is MSCOCO , which contains $122,585$ images for training and validation. Each image in MSCOCO has $5$ human annotated captions. Following splits in , we reserved $2,000$ images for validation. A more challenging dataset, InstaPIC-1.1M , is used as the second dataset, which contains $648,761$ images for training, and $5,000$ images for testing. The images and their ground-truth captions are acquired from Instagram, where people post images with related descriptions. Each image in InstaPIC-1.1M is paired with $1$ caption. This dataset is challenging, as its captions are natural posts with varying formats. In practice, we reserved $2,000$ images from the training set for validation.

On both datasets, non-alphabet characters except emojis are removed, and alphabet characters are converted to lowercases. Words and emojis that appeared less than $5$ times are replaced with UNK. And all captions are truncated to have at most $18$ words and emojis. As a result, we obtained a vocabulary of size $9,567$ on MSCOCO, and a vocabulary of size $22,886$ on InstaPIC-1.1M.

2 Settings

To study the generalization ability of proposed CL method, we tested it on two different image captioning models, namely Neuraltalk2 and AdaptiveAttention . Both models are based on encoder-and-decoder , where no attention mechanism is used in the former, and an adaptive attention component is used in the latter.

For both models, we have pretrained them by MLE, and use the pretrain checkpoints as initializations. In all experiments except for the experiment on model choices, we choose the same model and use the same initialization for target model and reference model. In all our experiments, we fixed the learning rate to be $1e$-$6$ for all components, and used Adam optimizer. Seven evaluation metrics have been selected to compare the performances of different models, including Bleu-1,2,3,4 , Meteor , Rouge and Cider . All experiments for ablation studies are conducted on the validation set of MSCOCO.

3 Results

We compared our best model (AdaptiveAttention learned by CL) with state-of-the-art models on two datasets. On MSCOCO, we submitted the results to the online COCO testing server. The results along with other published results are listed in Table 2. Compared to MLE-learned AdaptiveAttention, CL improves the performace of it by significant margins across all metrics. While most of state-of-the-art results are achieved by ensembling multiple models, our improved AdaptiveAttention gains competitive results as a single model. Specifically, on Cider, CL improves AdaptiveAttention from $1.003$ to $1.029$, which is the best single-model result on C40 among all published ones. In terms of Cider, if we use MLE, we need to combine $5$ models to get $4.5\%$ boost on C40 for AdaptiveAttention. Using CL, we improve the performance by $2.5\%$ with just a single model. On InstaPIC-1.1M, CL improves the performance of AdaptiveAttention by $14\%$ in terms of Cider, which is the state-of-the-art. Some qualitative results are shown in Figure 2. It’s worth noting that the proposed learning method can be used with stronger base models to obtain better results without any modification.

Compare Learning Methods

Using AdaptiveAttention learned by MLE as base model and initialization, we compared our CL with similar learning methods, including CL(P) and CL(N) that respectively contains only the positive constraint and the negative constraint in CL. We also compared with IL , and GAN . The results on MSCOCO are listed in Table 4, where (1) among IL, CL and GAN, CL improves performance of the base model, while both IL and GAN decrease the results. This indicates the trade-off between learning distinctiveness and maintaining overall performance is not well settled in IL and GAN. (2) comparing models learned by CL(P), CL(N) and CL, we found using the positive constraint or the negative constraint alone is not sufficient, as only one source of guidance is provided. While CL(P) gives the base model lower improvement than full CL, CL(N) downgrades the base model, indicating overfits on distinctiveness. Combining CL(P) and CL(N), CL is able to encourage distinctiveness while also emphasizing on overall performance, resulting in largest improvements on all metrics.

Compare Model Choices

To study the generalization ability of CL, AdaptiveAttention and Neuraltalk2 are respectively chosen as both the target and the reference in CL. In addition, AdaptiveAttention learned by MLE, as a better model, is chosen to be the reference, for Neuraltalk2. The results are listed in Table 5, where compared to models learned by MLE, both AdaptiveAttention and Neuraltalk2 are improved after learning using CL. For example, on Cider, AdaptiveAttention improves from $1.042$ to $1.142$, and Neuraltalk2 improves from $0.882$ to $0.905$. Moreover, by using a stronger model, AdaptiveAttention, as the reference, Neuraltalk2 improves further from $0.905$ to $0.956$, which indicates stronger references empirically provide tighter bounds on both the positive constraint and the negative constraint.

Reference Replacement

As discussed in sec 4.3, one can periodically replace the reference with latest best target model, to further improve the performance. In our study, using AdaptiveAttention learned by MLE as a start, each run we fix the reference model util the target saturates its performance on the validation set, then we replace the reference with latest best target model and rerun the learning. As listed in Table 6, in second run, the relative improvements of the target model is incremental, compared to its improvement in the first run. Therefore, when learning a model using CL, with a sufficiently strong reference, the improvement is usually saturated in the first run, and there is no need, in terms of overall performance, to replace the reference multiple times.

Conclusion

In this paper, we propose Contrastive Learning, a new learning method for image captioning. By employing a state-of-the-art model as a reference, the proposed method is able to maintain the optimality of the target model, while encouraging it to learn from distinctiveness, which is an important property of high quality captions. On two challenging datasets, namely MSCOCO and InstaPIC-1.1M, the proposed method improves the target model by significant margins, and gains state-of-the-art results across multiple metrics. On comparative studies, the proposed method extends well to models with different structures, which clearly shows its generalization ability.

This work is partially supported by the Big Data Collaboration Research grant from SenseTime Group (CUHK Agreement No.TS1610626), the General Research Fund (GRF) of Hong Kong (No.14236516) and the Early Career Scheme (ECS) of Hong Kong (No.24204215).

References