CenterNet: Keypoint Triplets for Object Detection

Introduction

Object detection has been significantly improved and advanced with the help of deep learning, especially convolutional neural networks (CNNs). In the current era, one of the most popular flowcharts is anchor-based , which placed a set of rectangles with pre-defined sizes, and regressed them to the desired place with the help of ground-truth objects. These approaches often need a large number of anchors to ensure a sufficiently high IoU (intersection over union) rate with the ground-truth objects, and the size and aspect ratio of each anchor box need to be manually designed. In addition, anchors are usually not aligned with the ground-truth boxes, which is not conducive to the bounding box classification task.

To overcome the drawbacks of anchor-based approaches, a keypoint-based object detection pipeline named CornerNet was proposed. It represented each object by a pair of corner keypoints, which bypassed the need of anchor boxes and achieved the state-of-the-art one-stage object detection accuracy. Nevertheless, the performance of CornerNet is still restricted by its relatively weak ability of referring to the global information of an object. That is to say, since each object is constructed by a pair of corners, the algorithm is sensitive to detect the boundary of objects, meanwhile not being aware of which pairs of keypoints should be grouped into objects. Consequently, as shown in Figure 1, it often generates some incorrect bounding boxes, most of which could be easily filtered out with complementary information, e.g., the aspect ratio.

To address this issue, we equip CornerNet with an ability of perceiving the visual patterns within each proposed region, so that it can identify the correctness of each bounding box by itself. In this paper, we present a low-cost yet effective solution named CenterNet, which explores the central part of a proposal, i.e., the region that is close to the geometric center, with one extra keypoint. Our intuition is that, if a predicted bounding box has a high IoU with the ground-truth box, then the probability that the center keypoint in its central region is predicted as the same class is high, and vice versa. Thus, during inference, after a proposal is generated as a pair of corner keypoints, we determine if the proposal is indeed an object by checking if there is a center keypoint of the same class falling within its central region. The idea, as shown in Figure 1, is to use a triplet, instead of a pair, of keypoints to represent each object.

Accordingly, for better detecting center keypoints and corners, we propose two strategies to enrich center and corner information, respectively. The first strategy is named center pooling, which is used in the branch for predicting center keypoints. Center pooling helps the center keypoints obtain more recognizable visual patterns within objects, which makes it easier to perceive the central part of a proposal. We achieve this by getting out the max summed response in both horizontal and vertical directions of the center keypoint on a feature map for predicting center keypoints. The second strategy is named cascade corner pooling, which equips the original corner pooling module with the ability of perceiving internal information. We achieve this by getting out the max summed response in both boundary and internal directions of objects on a feature map for predicting corners. Empirically, we verify that such a two-directional pooling method is more stable, i.e., being more robust to feature-level noises, which contributes to the improvement of both precision and recall.

The remainder of this paper is organized as follows. Section 2 briefly reviews related work, and Section 3 details the proposed CenterNet. Experimental results are given in Section 4, followed by the conclusion in Section 5.

Related Work

Object detection involves locating and classifying the objects. In the deep learning era, powered by deep convolutional neural networks, object detection approaches can be roughly categorized into two main types of pipelines, namely, two-stage approaches and one-stage approaches.

Two-stage approaches divide the object detection task into two stages: extract RoIs, then classify and regress the RoIs.

R-CNN uses a selective search method to locate RoIs in the input images and uses a DCN-based regionwise classifier to classify the RoIs independently. SPP-Net and Fast-RCNN improve R-CNN by extracting the RoIs from the feature maps. Faster-RCNN is allowed to be trained end to end by introducing RPN (region proposal network). RPN can generate RoIs by regressing the anchor boxes. Later, the anchor boxes are widely used in the object detection task. Mask-RCNN adds a mask prediction branch on the Faster-RCNN, which can detect objects and predict their masks at the same time. R-FCN replaces the fully connected layers with the position-sensitive score maps for better detecting objects. Cascade R-CNN addresses the problem of overfitting at training and quality mismatch at inference by training a sequence of detectors with increasing IoU thresholds. The keypoint-based object detection approaches are proposed to avoid the disadvantages of using anchor boxes and bounding boxes regression. Other meaningful works are proposed for different problems in object detection, e.g., focus on the architecture design, focus on the contextual relationship, focus on the multi-scale unification.

One-stage approaches remove the RoI extraction process and directly classify and regress the candidate anchor boxes.

Our Approach

This paper uses CornerNet as the baseline. For detecting corners, CornerNet produces two heatmaps: a heatmap of top-left corners and a heatmap of bottom-right corners. The heatmaps represent the locations of keypoints of different categories and assigns a confidence score for each keypoint. Besides, it also predicts an embedding and a group of offsets for each corner. The embeddings are used to identify if two corners are from the same object. The offsets learn to remap the corners from the heatmaps to the input image. For generating object bounding boxes, top-$k$ left-top corners and bottom-right corners are selected from the heatmaps according to their scores, respectively. Then, the distance of the embedding vectors of a pair of corners is calculated to determine if the paired corners belong to the same object. An object bounding box is generated if the distance is less than a threshold. The bounding box is assigned a confidence score, which equals to the average scores of the corner pair.

In this paper, we propose a highly efficient alternative called CenterNet to explore the visual patterns within each bounding box. For detecting an object, our approach uses a triplet, rather than a pair, of keypoints. By doing so, our approach is still a one-stage detector, but partially inherits the functionality of RoI pooling. Our approach only pays attention to the center information, the cost of our approach is minimal. Meanwhile, we further introduce the visual patterns within objects into the keypoint detection process by using center pooling and cascade corner pooling.

2 Object Detection as Keypoint Triplets

The overall network architecture is shown in Figure 2. We represent each object by a center keypoint and a pair of corners. Specifically, we embed a heatmap for the center keypoints on the basis of CornerNet and predict the offsets of the center keypoints. Then, we use the method proposed in CornerNet to generate top-$k$ bounding boxes. However, to effectively filter out the incorrect bounding boxes, we leverage the detected center keypoints and resort to the following procedure: (1) select top-$k$ center keypoints according to their scores; (2) use the corresponding offsets to remap these center keypoints to the input image; (3) define a central region for each bounding box and check if the central region contains center keypoints. Note that the class labels of the checked center keypoints should be same as that of the bounding box; (4) if a center keypoint is detected in the central region, we will preserve the bounding box. The score of the bounding box will be replaced by the average scores of the three points, i.e., the top-left corner, the bottom-right corner and the center keypoint. If there are no center keypoints detected in its central region, the bounding box will be removed.

where $n$ is odd that determines the scale of the central region $j$. In this paper, $n$ is set to be $3$ and $5$ for the scales of bounding boxes less and greater than $150$, respectively. Figure 3 shows two central regions when $n=3$ and $n=5$, respectively. According to Equation (1), we can determine a scale-aware central region, then we check if the central region contains center keypoints.

3 Enriching Center and Corner Information

Center pooling. The geometric centers of objects do not necessarily convey very recognizable visual patterns (e.g., the human head contains strong visual patterns, but the center keypoint is often in the middle of the human body). To address this issue, we propose center pooling to capture richer and more recognizable visual patterns. Figure 4 shows the principle of center pooling. The detailed process of center pooling is as follows: the backbone outputs a feature map, and to determine if a pixel in the feature map is a center keypoint, we need to find the maximum value in its both horizontal and vertical directions and add them together. By doing this, center pooling helps the better detection of center keypoints.

Cascade corner pooling. Corners are often outside the objects, which lacks local appearance features. CornerNet uses corner pooling to address this issue. The principle of corner pooling is shown in Figure 4. Corner pooling aims to find the maximum values on the boundary directions so as to determine corners. However, it makes corners sensitive to the edges. To address this problem, we need to let corners “see” the visual patterns of objects. The principle of cascade corner pooling is presented in Figure 4. It first looks along a boundary to find a boundary maximum value, then looks inside along the location of the boundary maximum valueFor the topmost, leftmost, bottommost and rightmost boundary, look vertically towards the bottom, horizontally towards the right, vertically towards the top and horizontally towards the left, respectively. to find an internal maximum value, and finally, add the two maximum values together. By doing this, the corners obtain both the the boundary information and the visual patterns of objects.

Both the center pooling and the cascade corner pooling can be easily achieved by combining the corner pooling at different directions. Figure 5(a) shows the structure of the center pooling module. To take a maximum value in a direction, e.g., the horizontal direction, we only need to connect the left pooling and the right pooling in series. Figure 5(b) shows the structure of a cascade top corner pooling module. Compared with the top corner pooling in CornerNet , we add a left corner pooling before the top corner pooling.

4 Training and Inference

Training. Our method is implemented in Pytorch and the network is trained from scratch. The resolution of the input image is $511\times 511$, leading to heatmaps of size $128\times 128$. We use the data augmentation strategy presented in to train a robust model. Adam is used to optimize the training loss:

Inference. Following , for the single-scale testing, we input both the original and horizontally flipped images with the original resolutions into the network. While for the multi-scale testing, we input both the original and horizontally flipped images with the resolutions of $0.6,1,1.2,1.5$ and $1.8$. We select top $70$ center keypoints, top $70$ top-left corners and top $70$ bottom-right corners from the heatmaps to detect the bounding boxes. We flip the bounding boxes detected in the horizontally flipped images and mix them into the original bounding boxes. Soft-nms is used to remove the redundant bounding boxes. We finally select top $100$ bounding boxes according to their scores as the final detection results.

Experiments

Our direct baseline is CornerNet . Following it, we use the stacked hourglass network with $52$ and $104$ layers as the backbone – the latter has two hourglass modules while the former has only one. All modifications on the hourglass architecture, made by , are preserved.

2 Comparisons with State-of-the-art Detectors

Table 2 shows the comparison with the state-of-the-art detectors on the MS-COCO test-dev set.

Compared with the baseline CornerNet , the proposed CenterNet achieves a remarkable improvement. For example, CenterNet511-52 (means that the resolution of input images is $511\times 511$ and the backbone is Hourglass-52) reports a single-scale testing AP of $41.6\%$, an improvement of $3.8\%$ over $37.8\%$, and a multi-scale testing AP of $43.5\%$, an improvement of $4.1\%$ over $39.4\%$, achieved by CornerNet under the same setting. When using the deeper backbone (i.e., Hourglass-104), the AP improvement over CornerNet are $4.4\%$ (from $40.5\%$ to $44.9\%$) and $4.9\%$ (from $42.1\%$ to $47.0\%$) under the single-scale and multi-scale testing, respectively. These results firmly demonstrate the effectiveness of CenterNet.

Meanwhile, it can be seen that the most contribution comes from the small objects. For instance, CenterNet511-52 improves the AP for small objects by $5.5\%$ (single-scale) and by $6.4\%$ (multi-scale). As for the backbone Hourglass-104, the improvements are $6.2\%$ (single-scale) and by $8.1\%$ (multi-scale), respectively. The benefit stems from the center information modeled by the center keypoints: the smaller the scale of an incorrect bounding box is, the lower probability a center keypoint can be detected in its central region. Figure 6 and Figure 6 show some qualitative comparisons, which demonstrate the effectiveness of CenterNet in reducing small incorrect bounding boxes.

CenterNet also leads to a large improvement for reducing medium and large incorrect bounding boxes. As Table 2 shows, CenterNet511-104 improves the single-scale testing AP by $4.7\%$ (from $42.7\%$ to $47.4\%$) and $3.5\%$ (from $53.9\%$ to $57.4\%$), respectively. Figure 6 and Figure 6 show some qualitative comparisons for reducing medium and large incorrect bounding boxes. It is worth noting that the AR is also significantly improved, with the best performance achieved with multi-scale testing. This is because our approach removes lots of incorrect bounding boxes, which is equivalent to improving the confidence of those bounding boxes with accurate locations but lower scores.

When comparing other one-stage approaches, CenterNet511-52 reports $41.6\%$ single-scale testing AP. This achievement is already better than those using deeper models (e.g., RetinaNet800 and RefineDet ). The best performance of CenterNet is AP 47.0%, dramatically surpassing all the published one-stage approaches to our best knowledge.

At last, one can observe that the performance of CenterNet is also competitive with the two-stage approaches, e.g., the single-scale testing AP of CenterNet511-52 is comparable to the two-stage approach Fitness R-CNN ($41.6\%$ vs. $41.8\%$) and that of CenterNet511-104 is comparable to D-RFCN + SNIP ($44.9\%$ vs. $45.7\%$), respectively. Nevertheless, it should be mentioned that two-stage approaches usually use larger resolution input images (e.g., $\sim 1000\times 600$), which significantly improves the detection accuracy especially for small objects. The multi-scale testing AP 47.0% achieved by CenterNet511-104 closely matches the state-of-the-art AP $47.4\%$, achieved by the two-stage detector PANet . We present some qualitative detection results in Figure 7.

3 Incorrect Bounding Box Reduction

4 Inference Speed

5 Ablation Study

Our work has contributed three components, including central region exploration, center pooling and cascade corner pooling. To analyze the contribution of each individual component, an ablation study is given here. The baseline is CornerNet511-52 . We add the three components to the baseline one by one and follow the default parameter setting detailed in Section 4.1. The results are given in Table 4.

Central region exploration. To understand the importance of the central region exploration (see CRE in the table), we add a center heatmap branch to the baseline and use a triplet of keypoints to detect bounding boxes. For the center keypoint detection, we only use conventional convolutions. As presented in the third row in Table 4, we improve the AP by $2.3\%$ (from $37.6\%$ to $39.9\%$). However, we find that the improvement for the small objects (that is $4.6$%) is more significant than that for other object scales. The improvement for large objects is almost negligible (from $52.2\%$ to $52.3\%$). This is not surprising because, from a probabilistic point of view, the center keypoint for a small object is easier to be located than that of a large object.

Center pooling. To demonstrate the effectiveness of proposed center pooling, we then add the center pooling module to the network (see CTP in the table). The fourth row in Table 4 shows that center pooling improves the AP by $0.9\%$ (from $39.9\%$ to $40.8\%$). Notably, with the help of center pooling, we improve the AP for large objects by $1.4\%$ (from $52.2\%$ to $53.6\%$), which is much higher than the improvement using conventional convolutions (i.e., $1.4\%$ vs. $0.1\%$). It demonstrates that our center pooling is effective in detecting center keypoints of objects, especially for large objects. Our explanation is that center pooling can extract richer internal visual patterns, and larger objects contain more accessible internal visual patterns. Figure 6 shows the results of detecting center keypoints without/with center pooling. We can see the conventional convolution fails to locate the center keypoint for the cow, but with center pooling, it successfully locates the center keypoint.

Cascade corner pooling. We replace corner pooling with cascade corner pooling to detect corners (see CCP in the table). The second row in Table 4 shows the results that we test on the basis of CornerNet511-52. We find that cascade corner pooling improves the AP by $0.7\%$ (from $37.6\%$ to $38.3\%$). The last row shows the results that we test on the basis of CenterNet511-52, which improves the AP by $0.5\%$ (from $40.8\%$ to $41.3\%$). The results of the second row show there is almost no change in the AP for large objects (i.e., $52.2\%$ vs. $52.2\%$), but the AR is improved by $1.8\%$ (from $74.0\%$ to $75.8\%$). This suggests that cascade corner pooling can “see” more objects due to the rich internal visual patterns in large objects, but too rich visual patterns may interfere with its perception for the boundary information, leading to many inaccurate bounding boxes. After equipping with our CenterNet, the inaccurate bounding boxes are effectively suppressed, which improves the AP for large objects by $2.2\%$ (from $53.6\%$ to $55.8\%$). Figure 6 shows the result of detecting corners with corner pooling or cascade corner pooling. We can see that cascade corner pooling can successfully locate a pair of corners for the cat on the left while corner pooling cannot.

6 Error Analysis

The exploration of visual patterns within each bounding box depends on the center keypoints. In other words, once a center keypoint is missed, the proposed CenterNet would miss the visual patterns within the bounding box. To understand the importance of center keypoints, we replace the predicted center keypoints with the ground-truth values and evaluate performance on the MS-COCO validation dataset. Table 5 shows that using the ground-truth center keypoints improves the AP from $41.3\%$ to $56.5\%$ for CenterNet511-52 and from $44.8\%$ to $58.1\%$ for CenterNet511-104, respectively. APs for small, medium and large objects are improved by $15.5\%$, $16.5\%$, and $14.5\%$ for CenterNet511-52 and $14.5\%$, $14.1\%$, and $13.3\%$ for CenterNet511-104, respectively. This demonstrates that the detection of center keypoints is far from the bottleneck.

Conclusions

In this paper, we propose CenterNet, which detects objects using a triplet, including one center keypoint and two corners. Our approach addresses the problem that CornerNet lacks an additional look into the cropped regions by exploring the visual patterns within each proposed region with minimal costs. In fact, this is a common defect for all one-stage approaches. As one-stage approaches remove the RoI extraction process, they cannot pay attention to internal information within cropped regions.

An intuitive explanation of our contribution lies in that we equip a one-stage detector with the ability of two-stage approaches, with an efficient discriminator being added. We believe that our idea of adding an extra branch for the center keypoint can be potentially generalized to other existing one-stage approaches (e.g., SSD ). Meanwhile, some advanced training strategies can be used for better performance. We leave as our future work.

References