Detecting Twenty-thousand Classes using Image-level Supervision

Introduction

Object detection consists of two sub-problems - finding the object (localization) and naming it (classification). Traditional methods tightly couple these two sub-problems and thus rely on box labels for all classes. Despite many data collection efforts, detection datasets are much smaller in overall size and vocabularies than classification datasets . For example, the recent LVIS detection dataset has 1000+ classes with 120K images; OpenImages has 500 classes in 1.8M images. Moreover, not all classes contain sufficient annotations to train a robust detector (see Figure 1 Top). In classification, even the ten-year-old ImageNet has 21K classes and 14M images (Figure 1 Bottom).

In this paper, we propose Detector with image classes (Detic) that uses image-level supervision in addition to detection supervision. We observe that the localization and classification sub-problems can be decoupled. Modern region proposal networks already localize many ‘new’ objects using existing detection supervision. Thus, we focus on the classification sub-problem and use image-level labels to train the classifier and broaden the vocabulary of the detector. We propose a simple classification loss that applies the image-level supervision to the proposal with the largest size, and do not supervise other outputs for image-labeled data. This is easy to implement and massively expands the vocabulary.

Most existing weakly-supervised detection techniques use the weakly labeled data to supervise both the localization and classification sub-problems of detection. Since image-classification data has no box labels, these methods develop various label-to-box assignment techniques based on model predictions to obtain supervision. For example, YOLO9000 and DLWL assign the image label to proposals that have high prediction scores on the labeled class. Unfortunately, this prediction-based assignment requires good initial detections which leads to a chicken-and-egg problem—we need a good detector for good label assignment, but we need many boxes to train a good detector. Our method completely side-steps the prediction-based label assignment process by supervising the classification sub-problem alone when using classification data. This also enables our method to learn detectors for new classes which would have been impossible to predict and assign.

Experiments on the open-vocabulary LVIS and the open-vocabulary COCO benchmarks show that our method can significantly improve over a strong box-supervised baseline, on both novel and base classes. With image-level supervision from ImageNet-21K , our model trained without novel class detection annotations improves the baseline by $8.3$ point and matches the performance of using full class annotations in training. With the standard LVIS annotations, our model reaches $41.7$ mAP and $41.7$ mAP${}_{\text{rare}}$, closing the gap between rare classes and all classes. On open-vocabulary COCO, our method outperforms the previous state-of-the-art OVR-CNN by $5$ point with the same detector and data. Finally, we train a detector using the full ImageNet-21K with more than twenty-thousand classes. Our detector generalizes much better to new datasets with disjoint label spaces, reaching $21.5$ mAP on Objects365 and $55.2$ mAP50 on OpenImages, without seeing any images from the corresponding training sets. Our contributions are summarized below:

We identify issues and propose a simpler alternative to existing weakly-supervised detection techniques in the open-vocabulary setting.

Our proposed family of losses significantly improves detection performance on novel classes, closely matching the supervised performance upper bound.

Our detector transfers to new datasets and vocabularies without finetuning.

We release our code (in supplement). It is ready-to-use for open-vocabulary detection in the real world. See examples in supplement.

Related Work

Weakly-supervised object detection (WSOD) trains object detector using image-level labels. Many works use only image-level labels without any box supervision . WSDDN and OIRC use a subnetwork to predict per-proposal weighting and sum up proposal scores into a single image scores. PCL first clusters proposals and then assign image labels at the cluster level. CASD further introduces feature-level attention and self-distillation. As no bounding box supervision is used in training, these methods rely on low-level region proposal techniques , which leads to reduced localization quality.

Another line of WSOD work uses bounding box supervision together with image labels, known as semi-supervised WSOD . YOLO9000 mixes detection data and classification data in the same mini-batch, and assigns classification labels to anchors with the highest predicted scores. DLWL combines self-training and clustering-based WSOD , and again assigns image labels to max-scored proposals. MosaicOS handles domain differences between detection and image datasets by mosaic augmentation and proposed a three-stage self-training and finetuning framework. In segmentation, Pinheiro et al. use a log-sum-exponential function to aggregate pixels scores into a global classification. Our work belongs to semi-supervised WSOD. Unlike prior work, we use a simple image-supervised loss. Besides image labels, researchers have also studied complementary methods for weak localization supervision like points or scribles .

Open-vocabulary object detection, or also named zero-shot object detection, aims to detect objects outside of the training vocabulary. The basic solution is to replace the last classification layer with language embeddings (e.g., GloVe ) of the class names. Rahman et al. and Li et al. improve the classifier embedding using external text information. OVR-CNN pretrains the detector on image-text pairs. ViLD , OpenSeg and langSeg upgrade the language embedding to CLIP . ViLD further distills region features from CLIP image features. We use CLIP classifier as well, but do not use distillation. Instead, we use additional image-labeled data for co-training.

Large-vocabulary object detection requires detecting 1000+ classes. Many existing works focus on handling the long-tail problem . Equalization losses and SeeSaw loss reweights the per-class loss by balancing the gradients or number of samples . Federated Loss subsamples classes per-iteration to mimic the federated annotation . Yang et al. detects 11K classes with a label hierarchy. Our method builds on these advances, and we tackle the problem from a different aspect: using additional image-labeled data.

Proposal Network Generalization. ViLD reports that region proposal networks have certain generalization abilities for new classes by default. Dave et al. shows segmentation and localization generalizes across classes. Kim et al. further improves proposal generalization with a localization quality estimator. In our experiments, we found proposals to generalize well enough (see Appendix 0.A), as also observed in ViLD . Further improvements to RPNs can hopefully lead to better results.

Preliminaries

We train object detectors using both object detection and image classification datasets. We propose a simple way to leverage image supervision to learn object detectors, including for classes without box labels. We first describe the object detection problem and then detail our approach.

Detic: Detector with Image Classes

Non-prediction-based losses. We now describe a variety of simple ways to use image labels and evaluate them empirically in Table 1. Our first idea is to use the whole image as a new “proposal” box. We call this loss image-box. We ignore all proposals from the RPN, and instead use an injected box of the whole image $\mathbf{b}^{\prime}=(0,0,w,h)$. We then apply the classification loss to its RoI features $\mathbf{f}^{\prime}$ for all classes $c\in\{c_{k}\}_{k=1}^{K}$:

where $BCE(s,c)=-log\sigma(s_{c})-\sum_{k\neq c}log(1-\sigma(s_{k}))$ is the binary cross-entropy loss, and $\sigma$ is the sigmoid activation. Thus, our loss uses the features from the same ‘proposal’ for solving the classification problem for all the classes $\{c_{k}\}$.

In practice, the image-box can be replaced by smaller boxes. We introduce two alternatives: the proposal with the max object score or the proposal with the max size:

We show that all these three losses can effectively leverage the image-level supervision, while the max-size loss performs the best. We thus use the max-size loss by default for image-supervised data. We also note that the classification parameters $\mathbf{W}$ are shared across both detection and classification data, which greatly improves detection performance. The overall training objective is

where $L_{\text{rpn}}$, $L_{\text{reg}}$, $L_{\text{cls}}$ are standard losses in a two-stage detector, and $\lambda=0.1$ is the weight of our loss.

Relation to prediction-based assignments. In traditional weakly-supervised detection , a popular idea is to assign the image to the proposals based on model prediction. Let $\mathbf{F}=(\mathbf{f}_{1},\dots,\mathbf{f}_{N})$ be the stacked feature of all object proposals and $\mathbf{S}=\mathbf{W}\mathbf{F}$ be their classification scores. For each $c\in\{c_{k}\}_{k=1}^{K}$, $L=BCE(\mathbf{S}_{j},c),j=\mathcal{F}(\mathbf{S},c)$, where $\mathcal{F}$ is the label-to-box assignment process. In most methods, $\mathcal{F}$ is a function of the prediction $\mathbf{S}$. For example, $\mathcal{F}$ selects the proposal with max score on $c$. Our key insight is that $\mathcal{F}$ should not depend on the prediction $\mathbf{S}$. In large-vocabulary detection, the initial recognition ability of rare or novel classes is low, making the label assignment process inaccurate. Our method side-steps this prediction-and-assignment process entirely and relies on a fixed supervision criteria.

Experiments

We evaluate Detic on the large-vocabulary object detection dataset LVIS . We mainly use the open-vocabulary setting proposed by Gu et al. , and also report results on the standard LVIS setting. We describe our experiment setup below.

LVIS. The LVIS dataset has object detection and instance segmentation labels for $1203$ classes with 100K images. The classes are divided into three groups - frequent, common, rare based on the number of training images. We refer to this standard LVIS training set as LVIS-all. Following ViLD , we remove the labels of $337$ rare-class from training and consider them as novel classes in testing. We refer to this partial training set with only frequent and common classes as LVIS-base. We report mask mAP which is the official metric for LVIS. While our model is developed for box detection, we use a standard class-agnostic mask head to produce segmentation masks for boxes. We train the mask head only on detection data.

Image-supervised data. We use two sources of image-supervised data: ImageNet-21K and Conceptual Captions . ImageNet-21K (IN-21K) contains 14M images for 21K classes. For ease of training and evaluation, most of our experiments use the 997 classes that overlap with the LVIS vocabulary and denote this subset as IN-L. Conceptual Captions (CC) is an image captioning dataset containing 3M images. We extract image labels from the captions using exact text-matching and keep images whose captions mention at least one LVIS class. See Appendix 0.B for results of directly using captions. The resulting dataset contains 1.5M images with 992 LVIS classes. We summarize the datasets used below.

Box-Supervised: a strong LVIS baseline. We first establish a strong baseline on LVIS to demonstrate that our improvements are orthogonal to recent advances in object detection. The baseline only uses the supervised bounding box labels. We use the CenterNet2 detector with ResNet50 backbone. We use Federated Loss and repeat factor sampling . We use large scale jittering with input resolution $640\!\times\!640$ and train for a $4\times$ ($\sim\!\!48$ LVIS epochs) schedule. To show our method is compatible with better pretraining, we use ImageNet-21k pretrained backbone weights . As described in § 3, we use the CLIP embedding as the classifier. Our baseline is $9.1$ mAP higher than the detectron2 baseline ($31.5$ vs. $22.4$ mAP${}^{\text{mask}}$) and trains in a similar time (17 vs. 12 hours on 8 V100 GPUs). See Appendix 0.C for more details.

Resolution change for image-labeled images. ImageNet images are inherently smaller and more object-focused than LVIS images . In practice, we observe it is important to use smaller image resolution for ImageNet images. Using smaller resolution in addition allows us to increase the batch-size with the same computation. In our implementation, we use $320\!\!\times\!\!320$ for ImageNet and CC and ablate this in Appendix 0.D.

Multi-dataset training. We sample detection and classification mini-batches in a $1:1$ ratio, regardless of the original dataset size. We group images from the same dataset on the same GPU to improve training efficiency .

Training schedules. To shorten the experimental cycle and have a good initialization for prediction-based WSOD losses , we always first train a converged base-class-only model ($4\times$ schedule) and finetune on it with additional image-labeled data for another $4\times$ schedule. We confirm finetuning the model using only box supervision does not improve the performance. The $4\times$ schedule for our joint training consists of $\sim\!\!24$ LVIS epochs plus $\sim\!\!4.8$ ImageNet epochs or $\sim\!\!3.8$ CC epochs. Training our ResNet50 model takes $\sim\!22$ hours on 8 V100 GPUs. The large 21K Swin-B model trains in $\sim\!24$ hours on 32 GPUs.

2 Prediction-based vs non-prediction-based methods

Table 1 shows the results of the box-supervised baseline, existing prediction-based methods, and our proposed non-prediction-based methods. The baseline (Box-Supervised) is trained without access to novel class bounding box labels. It uses the CLIP classifier and has open-vocabulary capabilities with 16.3 mAP${}_{\text{novel}}$. In order to leverage additional image-labeled data like ImageNet or CC, we use prior prediction-based methods or our non-prediction-based method.

We compare a few prediction-based methods that assign image labels to proposals based on predictions. Self-training assigns predictions of Box-Supervised as pseudo-labels offline with a fixed score threshold ($0.5$). The other prediction-based methods use different losses to assign predictions to image labels online. See Appendix 0.E for implementation details. For DLWL , we implement a simplified version that does not include bootstrapping and refer to it as DLWL*.

Table 1 (third block) shows the results of our non-prediction-based methods in § 4. All variants of our proposed simpler method outperform the complex prediction-based counterparts, with both image-supervised datasets. On the novel classes, Detic provides a significant gain of $\sim 4.2$ points with ImageNet over the best prediction-based methods.

Using non-object centric images from Conceptual Captions. ImageNet images typically have a single large object . Thus, our non-prediction-based methods, for example image-box which considers the entire image as a bounding box, are well suited for ImageNet. To test whether our losses work with different image distributions with multiple objects, we test it with the Conceptual Captions (CC) dataset. Even on this challenging dataset with multiple objects/labels per image, Detic provides a gain of $\sim 2.6$ points on novel class detection over the best prediction-based methods. This suggests that our simpler Detic method can generalize to different types of image-labeled data. Overall, the results from Table 1 suggest that complex prediction-based methods that overly rely on model prediction scores do not perform well for open-vocabulary detection. Amongst our non-prediction-based variants, the max-size loss consistently performs the best, and is the default for Detic in our following experiments.

Why does max-size work? Intuitively, our simpler non-prediction methods outperform the complex prediction-based method by side-stepping a hard assignment problem. Prediction-based methods rely on strong initial detections to assign image-level labels to predicted boxes. When the initial predictions are reliable, prediction-based methods are ideal. However, in open-vocabulary scenarios, such strong initial predictions are absent, which explains the limited performance of prediction-based methods. Detic’s simpler assignment does not rely on strong predictions and is more robust under the challenges of open-vocabulary setting.

We now study two additional advantages of the Detic max-size variant over prediction-based methods that may contribute to improved performance: 1) the selected max-size proposal can safely cover the target object; 2) the selected max-size proposal is consistent during different training iterations.

Figure 4 provides typical qualitative examples of the assigned region for the prediction-based method and our max-size variant. On an annotated subset of IN-L, Detic max-size covers $92.8\%$ target objects, vs. $69.0\%$ for the prediction-based method. Overall, unlike prediction-based methods, Detic’s simpler assignment yields boxes that are more likely to contain the object. Indeed, Detic may miss certain objects (especially small objects) or supervise to a loose region. However, in order for Detic to yield a good detector, the selected box need not be perfect, it just needs to 1) provide meaningful training signal (cover the objects and be consistent during training); 2) be ‘more correct’ than the box selected by the prediction-based method. We provide details about our metrics, more quantitative evaluation, and more discussions in Appendix 0.F.

3 Comparison with a fully-supervised detector

In Table 1, compared with the strong baseline Box-Supervised, Detic improves the detection performance by $2.4$ mAP and $8.3$ mAP${}_{\text{novel}}$. Thus, Detic with image-level labels leads to strong open-vocabulary detection performance and can provide orthogonal gains to existing open-vocabulary detectors . To further understand the open-vocabulary capabilities of Detic, we also report the top-line results trained with box labels for all classes (Table 1 last row). Despite not using box labels for the novel classes, Detic with ImageNet performs favorably compared to the fully-supervised detector. This result also suggests that bounding box annotations may not be required for new classes. Detic combined with large image classification datasets is a simple and effective alternative for increasing detector vocabulary.

4 Comparison with the state-of-the-art

We compare Detic’s open-vocabulary object detectors with state-of-the-art methods on the open-vocabulary LVIS and the open-vocabulary COCO benchmarks. In each case, we strictly follow the architecture and setup from prior work to ensure fair comparisons.

Open-vocabulary LVIS. We compare to ViLD , which first uses CLIP embeddings for open-vocabulary detection. We strictly follow their training setup and model architecture (Appendix 0.G) and report results in Table 2. Here ViLD-text is exactly our Box-Supervised baseline. Detic provides a gain of $7.7$ points on mAP${}_{\text{novel}}$. Compared to ViLD-text, ViLD, which uses knowledge distillation from the CLIP visual backbone, improves mAP${}_{\text{novel}}$ at the cost of hurting overall mAP. Ensembling the two models, ViLD-ens provides improvements for both metrics. On the other hand, Detic uses a single model which improves both novel and overall mAP, and outperforms the ViLD ensemble.

Open-vocabulary COCO. Next, we compare with prior works on the popular open-vocabulary COCO benchmark (see benchmark and implementation details in Appendix 0.H). We strictly follow OVR-CNN to use Faster R-CNN with ResNet50-C4 backbone and do not use any improvements from § 5.1. Following , we use COCO captions as the image-supervised data. We extract nouns from the captions and use both the image labels and captions as supervision.

Table 3 summarizes our results. As the training set contains only 48 base classes, the base-class only model (second row) yields low mAP on novel classes. Detic improves the baseline and outperforms OVR-CNN by a large margin, using exactly the same model, training recipe, and data.

Additionally, similar to Table 1, we compare to prior prediction-based methods on the open-vocabulary COCO benchmark in Appendix 0.H. In this setting too, Detic improves over prior work providing significant gains on novel class detection and overall detection performance.

5 Detecting 21K classes across datasets without finetuning

Next, we train a detector with the full 21K classes of ImageNet. We use our strong recipe with Swin-B backbone. In practice, training a classification layer of 21K classes is computationally involved.This is more pronounced in detection than classification, as the “batch-size” for the classification layer is $512\times$ image-batch-size, where $512$ is #RoIs per image. We adopt a modified Federated Loss that uniformly samples $50$ classes from the vocabulary at every iteration. We only compute classification scores and back-propagate on the sampled classes.

As there are no direct benchmark to evaluate detectors with such large vocabulary, we evaluate our detectors on new datasets without finetuning. We evaluate on two large-scale object detection datasets: Objects365v2 and OpenImages , both with around $1.8$M training images. We follow LVIS to split $\frac{1}{3}$ of classes with the fewest training images as rare classes. Table 4 shows the results. On both datasets, Detic improves the Box-Supervised baseline by a large margin, especially on classes with fewer annotations. Using all the 21k classes further improves performance owing to the large vocabulary. Our single model significantly reduces the gap towards the dataset-specific oracles and reaches $70\%$-$80\%$ of their performance without using the corresponding $1.8$M detection annotations. See Figure 5 for qualitative results.

6 Ablation studies

We now ablate our key components under the open-vocabulary LVIS setting with IN-L as the image-classification data. We use our strong training recipe as described in § 5.1 for all these experiments.

Classifier weights. We study the effect of different classifier weights $\mathbf{W}$. While our main open-vocabulary experiments use CLIP , we show the gain of Detic is independent of CLIP. We train Box-Supervised and Detic with different classifiers, including a standard random initialized and trained classifier, and other fixed language models The results are shown in Table 5. By default, a trained classifier cannot recognize novel classes. However, Detic enables novel class recognition ability even in this setting ($17.4$ mAP${}_{\text{novel}}$ for classes without detection labels). Using language models such as FastText or an open-source version of CLIP leads to better novel class performance. CLIP performs the best among them.

Effect of Pretraining. Many existing methods use additional data only for pretraining , while we use image-labeled data for co-training. We present results of Detic with different types of pretraining in Table 6. Detic provides similar gains across different types of pretraining, suggesting that our gains are orthogonal to advances in pretraining. We believe that this is because pretraining improves the overall features, while Detic uses co-training which improves both the features and the classifier.

7 The standard LVIS benchmark

Finally, we evaluate Detic on the standard LVIS benchmark . In this setting, the baseline (Box-Supervised) is trained with box and mask labels for all classes while Detic uses additional image-level labels from IN-L. We train Detic with the same recipe in § 5.1 and use a strong Swin-B backbone and $896\times 896$ input size. We report the mask mAP across all classes and also split into rare, common, and frequent classes. Notably, Detic achieves $41.7$ mAP and $41.7$ mAP${}_{\text{r}}$, closing the gap between the overall mAP and the rare mAP. This suggests Detic effectively uses image-level labels to improve the performance of classes with very few boxes labels. Appendix 0.I provides more comparisons to prior work on LVIS. Appendix 0.J shows Detic generalizes to DETR-based detectors.

Limitations and Conclusions

We present Detic which is a simple way to use image supervision in large-vocabulary object detection. While Detic is simpler than prior assignment-based weakly-supervised detection methods, it supervises all image labels to the same region and does not consider overall dataset statistics. We leave incorporating such information for future work. Moreover, open vocabulary generalization has no guarantees on extreme domains. Our experiments show Detic improves large-vocabulary detection with various weak data sources, classifiers, detector architectures, and training recipes.

Acknowledgments. We thank Bowen Cheng and Ross Girshick for helpful discussions and feedback. This material is in part based upon work supported by the National Science Foundation under Grant No. IIS-1845485 and IIS-2006820. Xingyi is supported by a Facebook PhD Fellowship.

References

Appendix 0.A Region proposal quality

In this section, we show the region proposal network trained on LVIS is satisfactory and generalizes well to new classes by default. We experiment under our strong baseline in § 5.1. Table 8(a) shows the proposal recalls with or without rare classes in training. First, we observe the recall gaps between the two models on rare classes are small (79.7 vs. 78.5); second, the gaps between rare classes and all classes are small (79.7 vs. 80.9); third, the absolute recall is relatively high ($\sim\!80\%$, note recall at IoU threshold 0.5 can be translated into oracle mAP-pool given perfect classifier and regressor). All observations indicate the proposals have good generalization abilities to new classes even though they are supervised to background during training. We consider the proposal generalization is currently not the performance bottleneck in open-vocabulary detection. This especially the case as modern detectors use an over-sufficient number of proposals in testing (1K proposals for $<$ 20 objects per image). Our observations are consistent with ViLD .

We in addition evaluate a more strict setting, where we uniformly split LVIS classes into two halves. I.e., we use classes ID $1,3,5,\cdots$ as the first half, and the rest as the second half. These two subsets have completely different definitions of “objects”. We then train a proposal network on each of them, and evaluate on both subsets. As shown in Table 8(b), the proposal networks give non-trivial recalls at the complementary other half ($69.6\%$ over $82.2\%$ percent of the full generalizability). This again supports proposal networks trained on a diverse vocabulary learned a general concept of objects.

Appendix 0.B Direct captions supervision

We do not back-propagate into the language encoder.

We evaluate the effectiveness of the caption loss in Table 9 on both open-vocabulary LVIS and COCO (see dataset details in Appendix 0.H). We compare individually applying the max-size loss for image labels and the caption loss, and applying both of them. Both image labels and captions can improve both overall mAP and novel class mAP. Combining both losses gives a more significant improvement. Our open-vocabulary COCO results in Table 3 uses both the max-size loss and the caption loss.

Appendix 0.C LVIS baseline details

We first describe the standard LVIS baseline from the detectron2 model zoohttps://github.com/facebookresearch/detectron2/blob/main/configs/LVISv1-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml. This baseline uses ResNet-50 FPN backbone and a $2\times$ training schedule ($180k$ iterations with batch-size $16$)We are aware different projects use different notations of a $1\times$ schedule. In this paper we always refer $1\times$ schedule to $16\times 90k$ images. Data augmentation includes horizontal flip and random resize short side [$640$, $800$], long side $<1333$. The baseline uses SGD optimizer with a learning rate 0.02 (dropped by $10\!\times$ at $120k$ and $160k$ iteration). The bounding box regression head and the mask head are class-specific.

Table 10 shows the roadmap from the detectron2 baseline to our baseline (§ 5.1). First, we prepare the model for new classes by making the box and mask heads class-agnostic. This slightly hurts performance. We then use Federated loss and upgrade the detector to CenterNet2 (i.e., replacing RPN with CenterNet and multiplying proposal score to classification score). Both modifications improve mAP and mAP${}_{\text{r}}$significantly, and CenterNet2 slightly increases the training time.

Next, we use the EfficientDet style large-scale jittering and train a longer schedule ($4\times$). To balance the training time, we also reduce the training image size to $640\times 640$ (the testing size is unchanged at $800\times 1333$) and increase batch-size to $64$ (with the learning rate scaled up to $0.08$). The resulting augmentation and schedule is slightly better than the default multi-scale training, with $30\%$ more training time. A longer schedule is beneficial when using more data, and can be improved by larger resolution.

Next, we switch in the CLIP classifier . We follow ViLD to L2 normalize the embedding and RoI feature before dot-product. Note CenterNet2 uses a cascade classifier . We use CLIP for all of them. Using CLIP classifier improves rare class mAP.

Finally, we use an ImageNet-21k pretrained ResNet-50 model from Ridnik et al. . We remark the ImageNet-21k pretrained model requires using Adam optimizer (with learning rate $2e$-$4$). Combing all the improvements results in $35.3$ mAP${}^{\text{box}}$ and $31.5$ mAP${}^{\text{mask}}$, and trains in a favorable time ($17$h on 8 V100 GPUs). We use this model as our baseline in the main paper.

Increasing the training resolution or using a larger backbone can further increase performance significantly, at a cost of longer training time. We use the large models only when compared to the state-of-the-art models.

Appendix 0.D Resolution change for classification data

Table 11 ablates the resolution change in § 5.1. Using a smaller input resolution improves $\sim\!1$ point for both mAP and mAP${}_{\text{novel}}$ with ImageNet, but does not impact much with CC. Using more batches for the weak datasets is slightly better than a $1:1$ ratio.

Appendix 0.E Prediction-based losses implementation details

Following the notations in § 4, we implement the prediction-based weakly-supervised detection losses as below:

WSDDN learns a soft weight on the proposals to weight-sum the proposal classification scores into a single image classification score:

where $\mathbf{W}^{\prime}$ is a learnable network parameter.

Predicted selects the proposal with the max predicted score on class $c$:

DLWL* first runs a clustering algorithm with IoU threshold 0.5. Let $\mathcal{J}$ be the set of peaks of each cluster (i.e., the proposal within the cluster and has the max predicted score on class $c$), We then select the top $N_{c}=3$ peaks with the highest prediction scores on class $c$.

The original DLWL in addition upgrades $\mathbf{S}$ using an IoU-based assignment matrix from self-training and bootstrapping (See their Section 3.2). In our implementation, we did not include this part, as our goal is to only compare the training losses.

Appendix 0.F More comparison between prediction-based and non-prediction-based methods

Our non-prediction-based losses perform significantly better than prediction-based losses as is shown in Table 1. In this section, we take the max-size loss and the predicted-loss as the representitives and conduct more detailed comparisons between them. A straightforward reason is that the predicted loss requires a good initial prediction to guide the pseudo-label-based training. However in the open-vocabulary detection setting the initial predictions are inherently flawed. To verify this, in Table 12(a), we show both improving the backbone and including rare classes in training can narrow the gap. However in the current performance regime, our max-size loss performs better.

We highlight two additional advantages of the max-size loss that may contribute to the good performance: (1) the max-size loss is a safe approximation of object regions; (2) the max-size loss is consistent during training. Figure 4 provides qualitative examples of the assigned region for the predicted loss and the max-size loss. First, we observe that while being coarse at the boundary, the max-size loss can cover the target object in most cases. Second, the assigned regions of the predicted loss are usually different across training iterations, especially in the early phase where the model predictions are unstable. On the contrary, max-size loss supervises consistent regions across training iterations.

Table 12(b) quantitatively evaluates these two properties. We use the ground truth box annotation in the full COCO detection dataset and a subset of ImageNet with bounding box annotation https://image-net.org/download-bboxes.php. 213K of the 1.2M IN-L images have bounding box annotations. to evaluate the cover rate. We define cover rate as the ratio of image labels whose ground-truth box has $>0.5$ intersection-over-area with the assigned region. We define the consistency metric as the average assigned-region IoU of the same image between the $1/2$ schedule and the final schedule. Table 12(b) shows max-size loss is more favorable than predicted loss on these two metrics. However we highlight that these two metrics alone do not always correlate to the final performance, as the image-box loss is perfect on both metrics but underperforms max-size loss.

Appendix 0.G ViLD baseline details

The baseline in ViLD is very different from detectron2. They use MaskRCNN detector with Res50-FPN backbone, but trains the network from scratch without ImageNet pretraining. They use large-scale jittering with input resolution $1024\times 1024$ and train a $32\times$ schedule. The optimizer is SGD with batch size $256$ and learning rate $0.32$. We first reproduce their baselines (both the oracle detector and ViLD-text) under the same setting. We observe half of their schedule ($16\times$) is sufficient to closely match their numbers. The half training schedule takes $4$ days on $4$ nodes (each with 8 V100 GPUs). We then finetune another $16\times$ schedule using ImageNet data with our max-size loss.

Appendix 0.H Open-vocabulary COCO benchmark details

Open-vocabulary COCO is proposed by Bansal et al. . They manually select 48 classes from the 80 COCO classes as base classes, and 17 classes as novel classes. The training set is the same as the full COCO, but only images containing at least one base class are used. During testing, we report results under the “generalized zero-shot detection” setting , where all COCO validation images are used.

We strictly follow the literatures to use FasterRCNN with ResNet50-C4 backbone and the $1\times$ training schedule ($90k$ iterations). We use horizontal flip as the only data augmentation in training and keep the input resolution fixed to $800\times 1333$ in both training and testing. We use SGD optimizer with a learning rate $0.02$ (dropped by $10\times$ at $60k$ and $80k$ iteration) and batch size $16$. The evaluation metric on open-vocabulary COCO is box mAP at IoU threshold 0.5. Our reproduced baseline matches OVR-CNN . Our model is finetuned on the baseline model with another $1\times$ schedule. We sample detection data and image-supervised data in a $1:1$ ratio.

Table 13 repeats the experiments in Table 1 on open-vocabulary COCO. The observations are consistent: our proposed non-prediction-based methods outperform existing prediction-based counterparts, and the max-size loss performs the best among our variants.

Appendix 0.I Compare to MosaicOS [73]

MosaicOS first uses image-level annotations to improve LVIS detectors. We compare to MosaicOS by strictly following their baseline setup (without any improvements in § 5.1). The detailed hyper-parameters follow the detectron2 baseline as described in Appendix 0.C. We finetune on the Box-supervised model with an additional $2\times$ schedule with Adam optimizer. Table 14 shows our re-trained baseline exactly matches their reported results from the paper. Our method is developed based on the CLIP classifier, and we also report our baseline with CLIP. The baseline has slightly lower mAP and higher mAP${}_{\text{r}}$. MosaicOS uses IN-L and additional web-search images as image-supervised data. Detic outperforms MosaicOS in mAP and mAP${}_{\text{r}}$, without using their multi-stage training and mosaic augmentation. Our relative improvements over the baseline are slightly higher than MosiacOS . We highlight our training framework is simpler and we use less additional training data (Google-searched images).

Appendix 0.J Generalization to Deformable-DETR.

We apply Detic to the recent Transformer based Deformable-DETR to study its generalization. We use their default training recipe, Federated Loss and train for a $4\times$ schedule ($\sim\!48$ LVIS epochs). We apply the image supervision to the query from the encoder with the max predicted size. Table 15 shows that Detic improves over the baseline (+$0.8$ mAP and $+4.8$ mAP${}_{\text{r}}$) and generalizes to Transformer based detectors.

Appendix 0.K Improvements breakdown to classes

Table 16 shows mAP breakdown into classes with and without image labels for both the Box-Supervised baseline and Detic. As expected, most of the improvements are from classes with image-level labels. On ImageNet, Detic also improves classes without image labels thanks to the CLIP classifier which leverages inter-class relations.

Appendix 0.L mAPFixedFixed{}^{\text{Fixed}} evaluation

Table 17 compares our improvements under the new mAP${}^{\text{fix}}$ proposed in Dave et al. . Our improvements are consistent under the new metric.

Appendix 0.M Image Attributions

License for the images from OpenImages in Figure 5:

“Oyster”: Photo by The Local People Photo Archive (CC BY 2.0)

“Cheetah”: Photo by Michael Gil (CC BY 2.0)

“Harbor seal”: Photo by Alden Chadwick (CC BY 2.0)

“Dinosaur”: Photo by Paxson Woelber (CC BY 2.0)