Self-Supervised Learning of Pretext-Invariant Representations

Introduction

Modern image-recognition systems learn image representations from large collections of images and corresponding semantic annotations. These annotations can be provided in the form of class labels , hashtags , bounding boxes , etc. Pre-defined semantic annotations scale poorly to the long tail of visual concepts , which hampers further improvements in image recognition.

Self-supervised learning tries to address these limitations by learning image representations from the pixels themselves without relying on pre-defined semantic annotations. Often, this is done via a pretext task that applies a transformation to the input image and requires the learner to predict properties of the transformation from the transformed image (see Figure 1). Examples of image transformations used include rotations , affine transformations , and jigsaw transformations . As the pretext task involves predicting a property of the image transformation, it encourages the construction og image representations that are covariant to the transformations. Although such covariance is beneficial for tasks such as predicting 3D correspondences , it is undesirable for most semantic recognition tasks. Representations ought to be invariant under image transformations to be useful for image recognition because the transformations do not alter visual semantics.

Motivated by this observation, we propose a method that learns invariant representations rather than covariant ones. Instead of predicting properties of the image transformation, Pretext-Invariant Representation Learning (PIRL) constructs image representations that are similar to the representation of transformed versions of the same image and different from the representations of other images. We adapt the “Jigsaw” pretext task to work with PIRL and find that the resulting invariant representations perform better than their covariant counterparts across a range of vision tasks. PIRL substantially outperforms all prior art in self-supervised learning from ImageNet (Figure 2) and from uncurated image data (Table 4). Interestingly, PIRL even outperforms supervised pre-training in learning image representations suitable for object detection (Tables 1 & 6).

PIRL: Pretext-Invariant Representation Learning

Our work focuses on pretext tasks for self-supervised learning in which a known image transformation is applied to the input image. For example, the “Jigsaw” task divides the image into nine patches and perturbs the image by randomly permuting the patches . Prior work used Jigsaw as a pretext task by predicting the permutation from the perturbed input image. This requires the learner to construct a representation that is covariant to the perturbation. The same is true for a range of other pretext tasks that have recently been studied . In this work, we adopt the existing Jigsaw pretext task in a way that encourages the image representations to be invariant to the image patch perturbation. While we focus on the Jigsaw pretext task in this paper, our approach is applicable to any pretext task that involves image transformations (see Section 4.3).

where $p(\mathcal{T})$ is some distribution over the transformations in $\mathcal{T}$, and $\mathbf{I}^{t}$ denotes image $\mathbf{I}$ after application of transformation $t$, that is, $\mathbf{I}^{t}=t(\mathbf{I})$. The function $L(\cdot,\cdot)$ is a loss function that measures the similarity between two image representations. Minimization of this loss encourages the network $\phi_{\theta}(\cdot)$ to produce the same representation for image $\mathbf{I}$ as for its transformed counterpart $\mathbf{I}^{t}$, i.e., to make representation invariant under transformation $t$.

We contrast our loss function to losses that aim to learn image representations $\mathbf{v}_{\mathbf{I}}=\phi_{\theta}(\mathbf{I})$ that are covariant to image transformations $t\in\mathcal{T}$ by minimizing:

where $z$ is a function that measures some properties of transformation $t$. Such losses encourage network $\phi_{\theta}(\cdot)$ to learn image representations that contain information on transformation $t$, thereby encouraging it to maintain information that is not semantically relevant.

Herein, $\mathcal{D}_{N}\subseteq\mathcal{D}\setminus\{\mathbf{I}\}$ is a set of $N$ negative samples that are drawn uniformly at random from dataset $\mathcal{D}$ excluding image $\mathbf{I}$, $\tau$ is a temperature parameter, and $s(\cdot,\cdot)$ is the cosine similarity between the representations.

In practice, we do not use the convolutional features $\mathbf{v}$ directly but apply to different “heads” to the features before computing the score $s(\cdot,\cdot)$. Specifically, we apply head $f(\cdot)$ on features ($\mathbf{v}_{\mathbf{I}}$) of $\mathbf{I}$ and head $g(\cdot)$ on features ($\mathbf{v}_{\mathbf{I}^{t}}$) of $\mathbf{I}^{t}$; see Figure 3 and Section 2.3. NCE then amounts to minimizing the following loss:

This loss encourages the representation of image $\mathbf{I}$ to be similar to that of its transformed counterpart $\mathbf{I}^{t}$, whilst also encouraging the representation of $\mathbf{I}^{t}$ to be dissimilar to that of other images $\mathbf{I}^{\prime}$.

2 Using a Memory Bank of Negative Samples

Prior work has found that it is important to use a large number of negatives in the NCE loss of Equation 4 . In a mini-batch SGD optimizer, it is difficult to obtain a large number of negatives without increasing the batch to an infeasibly large size. To address this problem, we follow and use a memory bank of “cached” features. Concurrent work used a similar memory-bank approach .

The memory bank, $\mathcal{M}$, contains a feature representation $\mathbf{m}_{\mathbf{I}}$ for each image $\mathbf{I}$ in dataset $\mathcal{D}$. The representation $\mathbf{m}_{\mathbf{I}}$ is an exponential moving average of feature representations $f(\mathbf{v}_{\mathbf{I}})$ that were computed in prior epochs. This allows us to replace negative samples, $f(\mathbf{v}_{\mathbf{I}}^{\prime})$, by their memory bank representations, $\mathbf{m}_{\mathbf{I}^{\prime}}$, in Equation 4 without having to increase the training batch size. We emphasize that the representations that are stored in the memory bank are all computed on the original images, $\mathbf{I}$, without the transformation $t$.

Herein, the first term is simply the loss of Equation 4 but uses memory representations $\mathbf{m}_{\mathbf{I}}$ and $\mathbf{m}_{\mathbf{I}^{\prime}}$ instead of $f(\mathbf{v}_{\mathbf{I}})$ and $f(\mathbf{v}_{\mathbf{I}}^{\prime})$, respectively. The second term does two things: (1) it encourages the representation $f(\mathbf{v}_{\mathbf{I}})$ to be similar to its memory representation $\mathbf{m}_{\mathbf{I}}$, thereby dampening the parameter updates; and (2) it encourages the representations $f(\mathbf{v}_{\mathbf{I}})$ and $f(\mathbf{v}_{\mathbf{I}}^{\prime})$ to be dissimilar. We note that both the first and the second term use $\mathbf{m}_{\mathbf{I}^{\prime}}$ instead of $f(\mathbf{v}_{\mathbf{I}}^{\prime})$ in Equation 4. Setting $\lambda=0$ in Equation 2.2 leads to the loss used in . We study the effect of $\lambda$ on the learned representations in Section 4.

3 Implementation Details

Although PIRL can be used with any pretext task that involves image transformations, we focus on the Jigsaw pretext task in this paper. To demonstrate that PIRL is more generally applicable, we also experiment with the Rotation pretext task and with a combination of both tasks in Section 4.3. Below, we describe the implementation details of PIRL with the Jigsaw pretext task.

Convolutional network. We use a ResNet-50 (R-50) network architecture in our experiments . The network is used to compute image representations for both $\mathbf{I}$ and $\mathbf{I}^{t}$. These representations are obtained by applying function $f(\cdot)$ or $g(\cdot)$ on features extracted from the the network.

Specifically, we compute the representation of $\mathbf{I}$, $f(\mathbf{v}_{\mathbf{I}})$, by extracting res5 features, average pooling, and a linear projection to obtain a $128$-dimensional representation.

To compute the representation $g(\mathbf{v}_{\mathbf{I}^{t}})$ of a transformed image $\mathbf{I}^{t}$, we closely follow . We: (1) extract nine patches from image $\mathbf{I}$, (2) compute an image representation for each patch separately by extracting activations from the res5 layer of the ResNet-50 and average pool the activations, (3) apply a linear projection to obtain a $128$-dimensional patch representations, and (4) concatenate the patch representations in random order and apply a second linear projection on the result to obtain the final $128$-dimensional image representation, $g(\mathbf{v}_{\mathbf{I}^{t}})$. Our motivation for this design of $g(\mathbf{v}_{\mathbf{I}^{t}})$ is the desire to remain as close as possible to the covariant pretext task of . This allows apples-to-apples comparisons between the covariant approach and our invariant approach.

Hyperparameters. We implement the memory bank as described in and use the same hyperparameters for the memory bank. Specifically, we set the temperature in Equation 3 to $\tau\!=\!0.07$, and use a weight of $0.5$ to compute the exponential moving averages in the memory bank. Unless stated otherwise, we use $\lambda\!=0.5$ in Equation 2.2.

Experiments

Following common practice in self-supervised learning , we evaluate the performance of PIRL in transfer-learning experiments. We perform experiments on a variety of datasets, focusing on object detection and image classification tasks. Our empirical evaluations cover: (1) a learning setting in which the parameters of the convolutional network are finetuned during transfer, thus evaluating the network “initialization” obtained using self-supervised learning and (2) a learning setting in which the parameters of the network are fixed during transfer learning, thus using the network as a feature extractor. Code reproducing the results of our experiments will be published online.

Baselines. Our most important baseline is the Jigsaw ResNet-50 model of . This baseline implements the covariant counterpart of our PIRL approach with the Jigsaw pretext task.

We also compare PIRL to a range of other self-supervised methods. An important comparison is to NPID . NPID is a special case of PIRL: setting $\lambda=0$ in Equation 2.2 leads to the loss function of NPID. We found it is possible to improve the original implementation of NPID by using more negative samples and training for more epochs (see Section 4). We refer to our improved version of NPID as NPID++. Comparisons between PIRL and NPID++ allow us to study the effect of the pretext-invariance that PIRL aims to achieve, i.e., the effect of using $\lambda>0$ in Equation 2.2.

Pre-training data. To facilitate comparisons with prior work, we use the $1.28$M images from the ImageNet train split (without labels) to pre-train our models.

Training details. We train our models using mini-batch SGD using the cosine learning rate decay scheme with an initial learning rate of $1.2\times 10^{-1}$ and a final learning rate of $1.2\times 10^{-4}$. We train the models for $800$ epochs using a batch size of $1,024$ images and using $N=32,000$ negative samples in Equation 3. We do not use data-augmentation approaches such as Fast AutoAugment because they are the result of supervised-learning approaches. We provide a full overview of all hyperparameter settings that were used in the supplemental material.

Transfer learning. Prior work suggests that the hyperparameters used in transfer learning can play an important role in the evaluation pre-trained representations . To facilitate fair comparisons with prior work, we closely follow the transfer-learning setup described in .

Following prior work , we perform object-detection experiments on the the Pascal VOC dataset using the VOC07+12 train split. We use the Faster R-CNN C4 object-detection model implemented in Detectron2 with a ResNet-50 (R-50) backbone. We pre-train the ResNet-50 using PIRL to initialize the detection model before finetuning it on the VOC training data. We use the same training schedule as for all models finetuned on VOC and follow to keep the BatchNorm parameters fixed during finetuning. We evaluate object-detection performance in terms of APall, AP50, and AP75 .

The results of our detection experiments are presented in Table 1. The results demonstrate the strong performance of PIRL: it outperforms all alternative self-supervised learnings in terms of all three AP measures. Compared to pre-training on the Jigsaw pretext task, PIRL achieves AP improvements of $\mathbf{5}$ points. These results underscore the importance of learning invariant (rather than covariant) image representations. PIRL also outperforms NPID++, which demonstrates the benefits of learning pretext invariance.

Interestingly, PIRL even outperforms the supervised ImageNet-pretrained model in terms of the more conservative APall and AP75 metrics. Similar to concurrent work , we find that a self-supervised learner can outperform supervised pre-training for object detection. We emphasize that PIRL achieves this result using the same backbone model, the same number of finetuning epochs, and the exact same pre-training data (but without the labels). This result is a substantial improvement over prior self-supervised approaches that obtain slightly worse performance than fully supervised baselines despite using orders of magnitude more curated training data or much larger backbone models . In Table 6, we show that PIRL also outperforms supervised pretraining when finetuning is done on the much smaller VOC07 train+val set. This suggests that PIRL learns image representations that are amenable to sample-efficient supervised learning.

2 Image Classification with Linear Models

Next, we assess the quality of image representations by training linear classifiers on fixed image representations. We follow the evaluation setup from and measure the performance of such classifiers on four image-classification datasets: ImageNet , VOC07 , Places205 , and iNaturalist2018 . These datasets involve diverse tasks such as object classification, scene recognition and fine-grained recognition. Following , we evaluate representations extracted from all intermediate layers of the pre-trained network, and report the image-classification results for the best-performing layer in Table 2.

ImageNet results. The results on ImageNet highlight the benefits of learning invariant features: PIRL improves recognition accuracies by over $15\%$ compared to its covariant counterpart, Jigsaw. PIRL achieves the highest single-crop top-1 accuracy of all self-supervised learners that use a single ResNet-50 model.

The benefits of pretext invariance are further highlighted by comparing PIRL with NPID. Our re-implementation of NPID (called NPID++) substantially outperforms the results reported in . Specifically, NPID++ achieves a single-crop top-1 accuracy of $59\%$, which is higher or on par with existing work that uses a single ResNet-50. Yet, PIRL substantially outperforms NPID++. We note that PIRL also outperforms concurrent work in this setting.

Akin to prior approaches, the performance of PIRL improves with network size. For example, CMC uses a combination of two ResNet-50 models and trains the linear classifier for longer to obtain $64.1\%$ accuracy. We performed an experiment in which we did the same for PIRL, and obtained a top-1 accuracy of $65.7\%$; see “PIRL-ens.” in Figure 2. To compare PIRL with larger models, we also performed experiments in which we followed by doubling the number of channels in ResNet-50; see “PIRL-c2x” in Figure 2. PIRL-c2x achieves a top-1 accuracy of $67.4\%$, which is close to the accuracy obtained by AMDIM with a model that has $6\times$ more parameters.

Altogether, the results in Figure 2 demonstrate that PIRL outperforms all prior self-supervised learners on ImageNet in terms of the trade-off between model accuracy and size. Indeed, PIRL even outperforms most self-supervised learners that use much larger models .

Results on other datasets. The results on the other image-classification datasets in Table 2 are in line with the results on ImageNet: PIRL substantially outperforms its covariant counterpart (Jigsaw). The performance of PIRL is within 2% of fully supervised representations on Places205, and improves the previous best results of on VOC07 by more than 16 AP points. On the challenging iNaturalist dataset, which has over $8,000$ classes, we obtain a gain of 11% in top-1 accuracy over the prior best result . We observe that the NPID++ baseline performs well on these three datasets but is consistently outperformed by PIRL. Indeed, PIRL sets a new state-of-the-art for self-supervised representations in this learning setting on the VOC07, Places205, and iNaturalist datasets.

3 Semi-Supervised Image Classification

We perform semi-supervised image classification experiments on ImageNet following the experimental setup of . Specifically, we randomly select $1\%$ and $10\%$ of the ImageNet training data (with labels). We finetune our models on these training-data subsets following the procedure of . Table 3 reports the top-5 accuracy of the resulting models on the ImageNet validation set.

The results further highlight the quality of the image representations learned by PIRL: finetuning the models on just $1\%$ ($\sim$13,000) labeled images leads to a top-5 accuracy of $57\%$. PIRL performs at least as well as S4L and better than VAT , which are both methods specifically designed for semi-supervised learning. In line with earlier results, PIRL also outperforms Jigsaw and NPID++.

4 Pre-Training on Uncurated Image Data

Most representation learning methods are sensitive to the data distribution used during pre-training . To study how much changes in the data distribution impact PIRL, we pre-train models on uncurated images from the unlabeled YFCC dataset . Following , we randomly select a subset of 1 million images (YFCC-1M) from the 100 million images in YFCC. We pre-train PIRL ResNet-50 networks on YFCC-1M using the same procedure that was used for ImageNet pre-training. We evaluate using the setup in Section 3.2 by training linear classifiers on fixed image representations.

Table 4 reports the top-1 accuracy of the resulting classifiers. In line with prior results, PIRL outperforms competing self-supervised learners. In fact, PIRL even outperforms Jigsaw and DeeperCluster models that were trained on $100\times$ more data from the same distribution. Comparing pre-training on ImageNet (Table 2) with pre-training YFCC-1M (Table 4) leads to a mixed set of observations. On ImageNet classification, pre-training (without labels) on ImageNet works substantially better than pre-training on YFCC-1M. In line with prior work , however, pre-training on YFCC-1M leads to better representations for image classification on the Places205 dataset.

Analysis

We performed a set of experiments aimed at better understanding the properties of PIRL. To make it feasible to train the larger number of models needed for these experiments, we train the models we study in this section for fewer epochs ($400$) and with fewer negatives ($N=4,096$) than in Section 3. As a result, we obtain lower absolute performances. Apart from that, we did not change the experimental setup or any of the other hyperparameters. Throughout the section, we use the evaluation setup from Section 3.2 that trains linear classifiers on fixed image representations to measure the quality of image representations.

PIRL was designed to learn representations that are invariant to image transformation $t\in\mathcal{T}$. We analyzed whether the learned representations actually have the desired invariance properties. Specifically, we normalize the representations to have unit norm and compute $l_{2}$ distances between the (normalized) representation of image, $f(\mathbf{v}_{\mathbf{I}})$, and the (normalized) representation its transformed version, $g(\mathbf{v}_{\mathbf{I}^{t}})$. We repeat this for all transforms $t\in\mathcal{T}$ and for a large set of images. We plot histograms of the distances thus obtained in Figure 4. The figure shows that, for PIRL, an image representation and the representation of a transformed version of that image are generally similar. This suggests that PIRL has learned representations that are invariant to the transformations. By contrast, the distances between Jigsaw representations have a much larger mean and variance, which suggests that Jigsaw representations covary with the image transformations that were applied.

Which layer produces the best representations?

All prior experiments used PIRL representations that were extracted from the res5 layer and Jigsaw representations that were extracted from the res4 layer (which work better for Jigsaw). Figure 5 studies the quality of representations in earlier layers of the convolutional networks. The figure reveals that the quality of Jigsaw representations improves from the conv1 to the res4 layer but that their quality sharply decreases in the res5 layer. We surmise this happens because the res5 representations in the last layer of the network covary with the image transformation $t$ and are not encouraged to contain semantic information. By contrast, PIRL representations are invariant to image transformations, which allows them to focus on modeling semantic information. As a result, the best image representations are extracted from the res5 layer of PIRL-trained networks.

2 Analyzing the PIRL Loss Function

The PIRL loss function in Equation 2.2 contains a hyperparameter $\lambda$ that trades off between two NCE losses. All prior experiments were performed with $\lambda\!=\!0.5$. NPID(++) is a special case of PIRL in which $\lambda\!=\!0$, effectively removing the pretext-invariance term from the loss. At $\lambda\!=\!1$, the network does not compare untransformed images at training time and updates to the memory bank $\mathbf{m}_{\mathbf{I}}$ are not dampened.

We study the effect of $\lambda$ on the quality of PIRL representations. As before, we measure representation quality by the top-1 accuracy of linear classifiers operating on fixed ImageNet representations. Figure 6 shows the results of these experiments. The results show that the performance of PIRL is quite sensitive to the setting of $\lambda$, and that the best performance if obtained by setting $\lambda\!=\!0.5$.

What is the effect of the number of image transforms?

Both in PIRL and Jigsaw, it is possible to vary the complexity of the task by varying the number of permutations of the nine image patches that are included in the set of image transformations, $\mathcal{T}$. Prior work on Jigsaw suggests that increasing the number of possible patch permutations leads to better performance . However, the largest value $|\mathcal{T}|$ can take is restricted because the number of learnable parameters in the output layer grows linearly with the number of patch permutations in models trained to solve the Jigsaw task. This problem does not apply to PIRL because it never outputs the patch permutations, and thus has a fixed number of model parameters. As a result, PIRL can use all $9!\approx 3.6$ million permutations in $\mathcal{T}$.

We study the quality of PIRL and Jigsaw as a function of the number of patch permutations included in $\mathcal{T}$. To facilitate comparison with , we measure quality in terms of performance of linear models on image classification using the VOC07 dataset, following the same setup as in Section 3.2. The results of these experiments are presented in Figure 7. The results show that PIRL outperforms Jigsaw for all cardinalities of $\mathcal{T}$ but that PIRL particularly benefits from being able to use very large numbers of image transformations (i.e., large $|\mathcal{T}|$) during training.

What is the effect of the number of negative samples?

We study the effect of the number of negative samples, $N$, on the quality of the learned image representations. We measure the accuracy of linear ImageNet classifiers on fixed representations produced by PIRL as a function of the value of $N$ used in pre-training. The results of these experiments are presented in Figure 8. They suggest that increasing the number of negatives tends to have a positive influence on the quality of the image representations constructed by PIRL.

3 Generalizing PIRL to Other Pretext Tasks

Although we studied PIRL in the context of Jigsaw in this paper, PIRL can be used with any set of image transformations, $\mathcal{T}$. We performed an experiment evaluating the performance of PIRL using the Rotation pretext task . We define $\mathcal{T}$ to contain image rotations by $\{0^{\circ},90^{\circ},180^{\circ},270^{\circ}\}$, and measure representation quality in terms of image-classification accuracy of linear models (see the supplemental material for details).

The results of these experiments are presented in Table 5 (top). In line with earlier results, models trained using PIRL (Rotation) outperform those trained using the Rotation pretext task of . The performance gains obtained from learning a rotation-invariant representation are substantial, e.g. $+11\%$ top-1 accuracy on ImageNet. We also note that PIRL (Rotation) outperforms NPID++ (see Table 2). In a second set of experiments, we combined the pretext image transforms from both the Jigsaw and Rotation tasks in the set of image transformations, $\mathcal{T}$. Specifically, we obtain $\mathbf{I}^{t}$ by first applying a rotation and then performing a Jigsaw transformation. The results of these experiments are shown in Table 5 (bottom). The results demonstrate that combining image transforms from multiple pretext tasks can further improve image representations.

Related Work

Our study is related to prior work that tries to learn characteristics of the image distribution without considering a corresponding (image-conditional) label distribution. A variety of work has studied reconstructing images from a small, intermediate representation, e.g., using sparse coding , adversarial training , autoencoders , or probabilistic versions thereof .

More recently, interest has shifted to specifying pretext tasks that require modeling a more limited set of properties of the data distribution. For video data, these pretext tasks learn representations by ordering video frames , tracking , or using cross-modal signals like audio .

Our work focuses on image-based pretext tasks. Prior pretext tasks include image colorization , orientation prediction , affine transform prediction , predicting contextual image patches , re-ordering image patches , counting visual primitives , or their combinations . In contrast, our works learns image representations that are invariant to the image transformations rather than covariant.

PIRL is related to approaches that learn invariant image representations via contrastive learning , clustering , or maximizing mutual information . PIRL is most similar to methods that learn representations that are invariant under standard data augmentation . PIRL learns representations that are invariant to both the data augmentation and to the pretext image transformations.

Finally, PIRL is also related to approaches that use a contrastive loss in predictive learning . These prior approaches predict missing parts of the data, e.g., future frames in videos , or operate on multiple views . In contrast to those approaches, PIRL learns invariances rather than predicting missing data.

Discussion and Conclusion

We studied Pretext-Invariant Representation Learning (PIRL) for learning representations that are invariant to image transformations applied in self-supervised pretext tasks. The rationale behind PIRL is that invariance to image transformations maintains semantic information in the representation. We obtain state-of-the-art results on multiple benchmarks for self-supervised learning in image classification and object detection. PIRL even outperforms supervised ImageNet pre-training on object detection.

In this paper, we used PIRL with the Jigsaw and Rotation image transformations. In future work, we aim to extend to richer sets of transformations. We also plan to investigate combinations of PIRL with clustering-based approaches . Like PIRL, those approaches use inter-image statistics but they do so in a different way. A combination of the two approaches may lead to even better image representations.

We are grateful to Rob Fergus, and Andrea Vedaldi for encouragement and their feedback on early versions of the manuscript; Yann LeCun for helpful discussions; Aaron Adcock, Naman Goyal, Priya Goyal, and Myle Ott for their help with the code development for this research; and Rohit Girdhar, and Ross Girshick for feedback on the manuscript. We thank Yuxin Wu, and Kaiming He for help with Detectron2.

References

Supplemental Material

Appendix A Training architecture and Hyperparameters

PIRL models in Section 3, 4 and 5 in the main paper are standard ResNet-50 models with 25.6 million parameters. Figure 2 and Section 3.2 also use a larger model ‘PIRL-c2x’ which has double the number of channels in each ‘stage’ of the ResNet, e.g., 128 channels in the first convolutional layer, 4096 channels in the final res5 layer etc., and a total of 98 million parameters. The heads $f(\cdot)$ and $g(\cdot)$ are linear layers that are used for the pre-training stage (Figure 3). When evaluating the model using transfer learning (Section 4 and 5 of the main paper), the heads $f$, $g$ are removed.

Training Hyperparameters for Section 3

PIRL models in Section 3 are trained for $800$ epochs using the ImageNet training set (1.28 million images). We use a batchsize of 32 per GPU and use a total of 32 GPUs to train each model. We optimize the models using mini-batch SGD with the cosine learning rate decay scheme with an initial learning rate of $1.2\times 10^{-1}$ and a final learning rate of $1.2\times 10^{-4}$, momentum of $0.9$ and a weight decay of $10^{-4}$. We use a total of $32,000$ negatives to compute the NCE loss (Equation 2.2) and the negatives are sampled randomly for each data point in the batch.

Training Hyperparameters for Section 4

The hyperparameters used are exactly the same as Section 3, except that the models are trained with $4096$ negatives and for $400$ epochs only. This results in a lower absolute performance, but the main focus of Section 4 is to analyze PIRL.

Common Data Pre-processing

We use data augmentations as implemented in PyTorch and the Python Imaging Library. These data augmentations are used for all methods including PIRL and NPID++. We do not use supervised policies like Fast AutoAugment. Our ‘geometric’ data pre-processing involves extracting a randomly resized crop from the image, and horizontally flipping it. We follow this by ‘photometric’ pre-processing that alter the RGB color values by using random values of color jitter, contrast, brightness, saturation, sharpness, equalize etc. transforms to the image.

A.1 Details for PIRL with Jigsaw

We base our Jigsaw implementation on . To construct $\mathbf{I}^{t}$, we extract a random resized crop that occupies at least 60% of the image. We resize this crop to $255\times 255$, and then divide into a $3\times 3$ grid. We extract a patch of $64\times 64$ from each of these grids and get $9$ patches. We apply photometric data augmentation (random color jitter, hue, contrast, saturation) to each patch independently and finally obtain the $9$ patches that constitute $\mathbf{I}^{t}$.

The image $\mathbf{I}$ is a $224\times 224$ image obtained by applying standard data augmentation (flip, random resized crop, color jitter, hue, contrast, saturation) to the image from the dataset.

Architecture

The 9 patches of $\mathbf{I}^{t}$ are individually input to the network to get their res5 features which are average pooled to get a single $2048$ dimensional vector for each patch. We then apply a linear projection to these features to get a $128$ dimensional vector for each patch. These patch features are concatenated to get a $1152$ dimensional vector which is then input to another single linear layer to get the final $128$ dimensional feature $\mathbf{v}_{\mathbf{I}^{t}}$.

We feed forward the image $\mathbf{I}$ and average pool its res5 feature to get a $2048$ dimensional vector for the image. We then apply a single linear projection to get a $128$ dimensional feature $\mathbf{v}_{\mathbf{I}}$.

A.2 Details for PIRL with Rotation

We base our Rotation implementation on . To construct $\mathbf{I}^{t}$ we use the standard data augmentation described earlier (geometric + photometric) to get a $224\times 224$ image, followed by a random rotation from $\{0^{\circ},90^{\circ},180^{\circ},270^{\circ}\}$.

The image $\mathbf{I}$ is a $224\times 224$ image obtained by applying standard data augmentation (flip, random resized crop, color jitter, hue, contrast, saturation) to the image from the dataset.

Architecture

We feed forward the image $\mathbf{I}^{t}$, average pool the res5 feature, and apply a single linear layer to get the final $128$ dimensional feature for $\mathbf{v}_{\mathbf{I}^{t}}$.

We feed forward the image $\mathbf{I}$ and average pool its res5 feature to get a $2048$ dimensional vector for the image. We then apply a single linear projection to get a $128$ dimensional feature $\mathbf{v}_{\mathbf{I}}$.

Appendix B Object Detection

In Section 3.1 of the main paper, we presented object detection results using the VOC07+12 training set which has 16K images. In this section, we use the smaller VOC07 train+val set (5K images) for finetuning the Faster R-CNN C4 detection models. All models are finetuned using the Detectron2 codebase and hyperparameters from . We report the detection AP in Table 6. We see that the PIRL model outperforms the ImageNet supervised pretrained models on the stricter AP75 and APall metrics without extra pretraining data or changes to the network architecture.

Detection hyperparameters: We use a batchsize of 2 images per GPU, a total of 8 GPUs and finetune models for $12.5$K iterations with the learning rate dropped by $0.1$ at $9.5$K iterations. The supervised and Jigsaw baseline models use an initial learning rate of $0.02$ with a linear warmup for $100$ iterations with a slope of $1/3$, while the NPID++ and PIRL models use an initial learning rate of $0.003$ with a linear warmup for $1000$ iterations and a slope of $1/3$. We keep the BatchNorm parameters of all models fixed during the detection finetuning stage.

B.2 VOC07+12 train set for detection

In Table 1, we use the VOC07+12 training split and the VOC07 test set as used in prior work .

Detection hyperparamters: We use a batchsize of 2 images per GPU, a total of 8 GPUs and finetune models for $25$K iterations with the learning rate dropped by $0.1$ at $17$K iterations. The supervised and Jigsaw baseline models use an initial learning rate of $0.02$ with a linear warmup for $100$ iterations with a slope of $1/3$, while the NPID++ and PIRL models use an initial learning rate of $0.003$ with a linear warmup for $1000$ iterations and a slope of $1/3$. We keep the BatchNorm parameters of all models fixed during the detection finetuning stage.

Appendix C Linear Models for Transfer

We train linear models on representations from the intermediate layers of a ResNet-50 model following the procedure outlined in . We briefly outline the hyperparameters from their work. The features from each of the layers are average pooled such that they are of about $9,000$ dimensions each. The linear model is trained with mini-batch SGD using a learning rate of $0.01$ decayed by $0.1$ at two equally spaced intervals, momentum of $0.9$ and weight decay of $5\times 10^{-4}$. We train the linear models for $28$ epochs on ImageNet (1.28M training images), $14$ epochs on Places205 (2.4M training images) and for $84$ epochs on iNaturalist-2018 (437K training images). Thus, the number of parameter updates for training the linear models are roughly constant across all these datasets. We report the center crop top-1 accuracy for the ImageNet, Places205 and iNaturalist-2018 datasets in Table 2.

For training linear models on VOC07, we train linear SVMs following the procedure in and report mean average Precision (mAP).

The results of all these models are in Table 7.