3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation

Introduction

With the availability of commodity RGB-D sensors such as Kinect or Intel RealSense, the computer vision and graphics communities have achieved impressive results on 3D reconstruction methods that can now even achieve global pose tracking in real time . In addition to the reconstruction of the geometry, semantic scene understanding is critical to many real-world computer vision applications, including robotics, upcoming applications on mobile devices, or AR/VR headsets. In order to understand reconstructed 3D environments, researchers have already made significant progress with 3D deep learning methods that operate on volumetric grids , point clouds , meshes or multi-view hybrids . While early 3D learning approaches focus mostly on semantic segmentation, we have recently seen many works on 3D semantic instance segmentation and 3D object detection , both of which we believe are critical for real-world 3D perception.

One of the fundamental challenges in 3D object detection lies in how to predict and process object proposals: On one side, top-down methods first predict a large number of rough object bounding box proposals (e.g., anchor mechanisms in Faster R-CNN ), followed by a second stage refinement step. Here, results can be generated in a single forward pass, but there is little outlier tolerance to wrongly detected box anchors. On the other side, bottom-up approaches utilize metric-learning methods with the goal of learning a per-point feature embedding space which is subsequently clustered into object instances . This strategy can effectively handle outliers, but it heavily depends on manually tuning cluster parameters and is inherently expensive to compute at inference time due to $O(N^{2})$ pairwise relationships.

In this work, we propose 3D-MPA which follows a hybrid approach that takes advantage of the benefits of both top-down and bottom-up techniques: from an input point cloud representing a 3D scan, we generate votes from each point for object centers and group those into object proposals; then – instead of rejecting proposals using non-maximum-suppression – we learn higher-level features for each proposal, which we use to cluster the proposals into final object detections. The key idea behind this strategy is that the number of generated proposals is orders of magnitude smaller than the number of raw input points in a 3D scan, which makes grouping computationally very efficient. At the same time, each object can receive multiple proposals, which simplifies proposal generation since objects of all sizes are handled in the same fashion, and we can easily tolerate outlier proposals further down the pipeline.

To this end, our method first generates object-centric proposals using a per-point voting scheme from a sparse volumetric feature backbone. We then interpret the proposals as nodes of a proposal graph which we feed into a graph convolutional neural network in order to enable higher-order interactions between neighboring proposal features. In addition to proposal losses, the network is trained with a proxy loss between proposals similar to affinity scores in metric learning; however, due to the relatively small number of proposals, we can efficiently train the network and cluster proposals. In the end, each node predicts a semantic class, an object foreground mask, an objectness score, and additional features that are used to group nodes together.

In summary, our contributions are the following:

A new method for 3D instance segmentation based on dense object center prediction leveraging learned semantic features from a sparse volumetric backbone.

To obtain the final object detections and semantic instances from the object proposals, we replace the commonly used NMS with our multi proposal aggregation strategy based on jointly learned proposal features and report significantly improved scores over NMS.

We employ a graph convolutional network that explicitly models higher-order interactions between neighboring proposal features in addition to the lower-level point features.

Related Work

In the 2D domain, object detection and instance segmentation have most notably been influenced by Faster R-CNN from Ren et al. , which introduced the anchor mechanism to predict proposals with associated objectness scores and regions of interest that enable the regression of semantic bounding boxes. This approach was extended in Mask-RCNN to predict per-pixel object instance masks. Hou et al. apply the 2D proposal ideas onto the 3D domain by means of dense 3D convolutional networks. As an alternative, proposal-free methods were proposed in which rely on metric learning. In the 2D domain, Fathi et al. estimate how likely pixels are to belong to the same object. De Brabandere et al. define a discriminative loss, which moves feature points of the same object towards their mean while pushing means of different objects apart. This discriminative loss is adopted by Lahoud et al. to perform instance segmentation in 3D space. Final instances are obtained via clustering of the learned feature space. Yang et al. directly predict object bounding boxes from a learned global feature vector and obtain instance masks by segmenting points inside a bounding box. The recent VoteNet highlights the challenge of directly predicting bounding box centers in sparse 3D data as most surface points are far away from object centers. Instead, they predict bounding boxes by grouping points from the same object based on their votes for object centers. We adopt the object-centric approach, extend it with a branch for instance mask prediction and replace NMS with a grouping mechanism of jointly-learned proposal features.

D Deep Learning.

PointNets have pioneered the use of deep learning methods for point cloud processing. Since then, we have seen impressive progress in numerous different fields, including 3D semantic segmentation , 3D instance segmentation , object detection and relocalization , flow estimation , scene-graph reconstruction and scene over-segmentation . Point-based architectures, such as PointNet and PointNet++ operate directly on unstructured sets of points, while voxel based approaches, such as 3DMV or SparseConvNets transform the continuous 3D space into a discrete grid representation and define convolutional operators on the volumetric grid, analogously to image convolutions in the 2D domain. Graph-based approaches define convolutional operators over graph-structured data such as 3D meshes , citation networks , or molecules . Here, we leverage the voxel-based approach of Graham et al. as point feature backbone and use the graph neural network of Wang et al. to enable higher-level interactions between proposals.

Method

The overall architecture of 3D-MPA is depicted in Fig. 2. The model consists of three parts: the first one takes as input a 3D point cloud and learns object proposals from sampled and grouped point features that voted for the same object center (Sec. 3.1). The next part consolidates the proposal features using a graph convolutional network enabling higher-level interactions between proposals which results in refined proposal features (Sec. 3.2). Last, the object generator consumes the object proposals and generates the final object detections, i.e. semantic instances. We parameterize an object as a set of points associated with that object and a semantic class. (Sec. 3.3).

where $||\cdot||_{H}$ is the Huber-loss (or smooth L1-loss) and $\mathds{1}(\cdot)$ is a binary function indicating whether a point $x_{i}$ belongs to an object. $M$ is a normalization factor equal to the total number of points on objects. All in all, the feature backbone has two heads (Fig. 2, $\blacksquare$$\square$): a semantic head (which performs semantic classification of points) and a center head (which regresses object centers for each point). They are jointly supervised using the combined loss $\mathcal{L}_{\text{point}}$ where $\lambda$ is a weighting factor set to 0.1:

2 Proposal Consolidation

So far, proposal features encode local information of their associated objects. During proposal consolidation, proposals become aware of their global neighborhood by explicitly modeling higher-order interactions between neighboring proposals. To this end, we define a graph convolutional network (GCN) over the proposals. While the initial point-feature backbone operates at the level of points, the GCN operates at the level of proposals. In particular, the nodes of the graph are defined by the proposal positions $y_{i}$ with associated proposal features $g_{i}$. An edge between two nodes exists if the Euclidean distance $d$ between two 3D proposal positions $y_{\{i,j\}}$ is below 2 m. We adopt the convolutional operator from DGCNN to define edge-features $e_{ij}$ between two neighboring proposals as:

3 Object Generation

At this stage, we have $K$ proposals $\{(y_{i},h_{i},s_{i})\}_{i=1}^{K}$ with positions $y_{i}$, refined features $h_{i}$ and sets of points $s_{i}$. The goal is to obtain the final semantic instances (or object detections) from these proposals. To this end, we predict for every proposal a semantic class, an aggregation feature vector, an objectness score and a binary foreground-background mask over the points $s_{i}$ associated with the proposal. Specifically, the proposal features $h_{i}$ are input to an MLP with output sizes $(128,128,D_{out})$ where $D_{out}=S+E+2$ with $S$ semantic classes, $E$-dimensional aggregation feature and a 2D (positive, negative) objectness score (Fig. 2, $\blacksquare$$\square$).

The objectness score classifies proposals into positive or negative examples. It is supervised via a cross-entropy loss $\mathcal{L}_{obj.}$. Proposals near a ground truth center ($<0.3$ m) are classified as positive. They are classified as negative, if they are far away ($>0.6$ m) from any ground truth center, or if they are equally far away from two ground truth centers since then the correct ground truth object is ambiguous. This is the case when $d_{1}>0.6\cdot d_{2}$ where $d_{i}$ is the distance to the $i^{th}$ closest ground truth center.

Positive proposals are further supervised to predict a semantic class, aggregation features, and a binary mask. Negative ones are ignored. We use a cross-entropy loss $\mathcal{L}_{\text{sem.}}$ to predict the semantic label of the closest ground truth object.

Previous methods such as VoteNet or 3D-BoNet rely on non-maximum-suppression (NMS) to obtain the final objects. NMS iteratively selects proposals with the highest objectness score and removes all others that overlap with a certain IoU. However, this is sensitive to the quality of the objectness scores and can discard correct predictions. Instead of rejecting potentially useful information, we combine multiple proposals. To this end, we learn aggregation features for each proposal which are then clustered using DBScan .

All proposals whose aggregation features end up in the same cluster are aggregated together, yielding the final object detections. The points of a final object are the union over the foreground masks of combined proposals. As the number of proposals is relatively small ($K$ $\approx$ $500$) compared to the full point cloud ($N$ $\approx$ $10^{6}$), this step is very fast ($\sim 8$ ms). This is a significant advantage over clustering full point clouds , which can be prohibitively slow.

In our experiments, we set $\gamma=0.001$ and $\delta_{v}=\delta_{d}=0.1$. $C$ is the total number of ground truth objects and $N_{C}$ the number of proposals belonging to one object. $\mathcal{L}_{\text{var.}}$ pulls features that belong to the same instance towards their mean, $\mathcal{L}_{\text{dist.}}$ pushes clusters with different instance labels apart, and $\mathcal{L}_{\text{reg.}}$ is a regularization term pulling the means towards the origin. Further details and intuitions are available in the original work by DeBrabandere et al. . In Sec. 4, we will show that geometric features outperform embedding features.

Mask Prediction.

Each positive proposal predicts a class-agnostic binary segmentation mask over the points $s_{i}$ associated with that proposal, where the number of points per proposal $i$ is $|s_{i}|=n_{i}$ (Fig. 2, $\blacksquare$$\square$). Prior approaches obtain masks by segmenting 2D regions of interest (RoI) (Mask-RCNN ) or 3D bounding boxes (3D-BoNet ). Since we adopt an object-centric approach, mask segmentation can directly be performed on the points $s_{i}$ associated with a proposal. In particular, for each proposal, we select the per-point features $f_{i}$ of points that voted for a center within a distance $r$ of the proposal position $y_{i}$. Formally, the set of selected per-point features is defined as $M_{f}=\{f_{i}|\,\lVert(x_{i}+\Delta x_{i})-y_{i}\rVert_{2}<r\}$ with $r=0.3$ m. The selected features $M_{f}$ are passed to a PointNet for binary segmentation, i.e., we apply a shared MLP on each per-point feature, compute max-pooling over all feature channels, and concatenate the result to each feature before passing it through another MLP with feature sizes (256, 128, 64, 32, 2). Points that have the same ground truth instance label as the closest ground truth object instance label are supervised as foreground, while all others are background. Similar to , the mask loss $\mathcal{L}_{\text{mask}}$ is implemented as FocalLoss instead of a cross-entropy loss to cope with the foreground-background class imbalance.

4 Training Details

The model is trained end-to-end from scratch using the multi-task loss $\mathcal{L}=\mathcal{L}_{\text{point}}+\mathcal{L}_{\text{obj.}}+0.1\cdot\mathcal{L}_{\text{sem.}}+\mathcal{L}_{\text{mask}}+\mathcal{L}_{\text{agg.}}$. The batch size is 4 and the initial learning rate 0.1 which is reduced by half every 2$\cdot 10^{4}$ iterations and trained for 15$\cdot 10^{4}$ iterations in total. Our model is implemented in TensorFlow and runs on an Nvidia TitanXp GPU (12GB).

Our network is trained on 3 m$\times$3 m point cloud crops of $N$ points sampled from the surface of a 3D mesh. During test time, we evaluate on full scenes. Input features are the 3D position, color and normal assigned to each point. Data augmentation is performed by randomly rotating the scene by $\text{Uniform}[-180^{\circ},180^{\circ}]$ around the upright axis and $\text{Uniform}[-10^{\circ},10^{\circ}]$ around the other axis. The scenes are randomly flipped in both horizontal directions and randomly scaled by $\text{Uniform}[0.9,1.1]$.

Experiments

We compare our approach to previous state-of-the-art methods on two large-scale 3D indoor datasets (Sec. 4.1). Our ablation study analyzes the contribution of each component of our model and shows in particular the improvement of aggregating proposals over NMS (Sec. 4.2).

The ScanNetV2 benchmark dataset consists of richly-annotated 3D reconstructions of indoor scenes. It comprises 1201 training scenes, 312 validation scenes and 100 hidden test scenes. The benchmark is evaluated on 20 semantic classes which include 18 different object classes.

The S3DIS dataset is a collection of six large-scale indoor areas annotated with 13 semantic classes and object instance labels. We follow the standard evaluation protocol and report scores on Area 5, as well as 6-fold cross validation results over all six areas.

Object detection scores

are shown in Tab. 1. Object detections are obtained by fitting a tight axis-aligned bounding box around the predicted object point-masks. We compare 3D-MPA to recent approaches including VoteNet on the ScanNetV2 dataset. Scores are obtained by using the evaluation methodology provided by . Our method outperforms all previous methods by at least +5.8 mAP@25% and +15.7 mAP@50%.

Instance segmentation scores

on S3DIS are shown in Tab. 2. Per-class instance segmentation results are shown in Tab. 7. We report mean average precision (mAP) and mean average recall (mAR) scores. Our scores are computed using the evaluation scripts provided by Yang et al. . Our approach outperforms all previous methods. In particular, we report an increased recall of +17.8 mAR@50% on Area5 and +16.5 mAR@50% on 6-fold cross validation, which means we detect significantly more objects, while simultaneously achieving higher precision.

We show results on ScanNetV2 validation and hidden test set in Tab. 3 and per-class scores with mAP@25% in Tab. 4 and mAP@50% in Tab. 5. We improve over previous methods by at least +18.1 mAP@50% and +17.0 mAP@25%. In particular, our 3D-MPA outperforms all other methods in every object class on mAP@50 (Tab. 5). On mAP@25, we outperform on all classes except chair and sofa. Qualitative results on ScanNetV2 are visualized in Fig. 3 and failure cases in Fig. 4.

2 Ablation study

In Tab. 6, we show the result of our ablation study analyzing the design choices of each component of our model. The evaluation metric is mean average precision (mAP) on the task of instance segmentation, evaluated on the ScanNetV2 validation set.

The main result of this work is that grouping multiple proposals is superior to non-maximum-suppression (NMS). We demonstrate this experimentally by comparing two baseline variants of our model: In experiment \footnotesize{1}⃝ (Tab. 6), we apply the traditional approach of predicting a number of proposals and applying NMS to obtain the final predictions. The model corresponds to the one depicted in Fig. 2 without proposal consolidation and with the aggregation replaced by NMS. NMS chooses the most confident prediction and suppresses all other predictions with an IoU larger than a specified threshold, in our case 25%. For experiment \footnotesize{2}⃝, we perform a naive grouping of proposals by clustering the proposal positions $y_{i}$. The final object instance masks are obtained as the union over all proposal masks in one cluster. We observe a significant increase of +4.9 mAP by replacing NMS with aggregation.

How important are good aggregation features?

In experiment \footnotesize{2}⃝, we group proposals based on their position $y_{i}$. These are still relatively simple features. In experiments \footnotesize{3}⃝ and \footnotesize{4}⃝, proposals are grouped based on learned embedding features and learned geometric features, respectively. These features are described in Sec. 3.3. Again, we observe a notable improvement of +5.4 mAP compared to experiment \footnotesize{2}⃝ and even +10.3 mAP over \footnotesize{1}⃝. In our experiments, the geometric features performed better than the embedding features (+1.1 mAP). One possible explanation could be that the geometric features have an explicit meaning and are therefore easier to train than the 5-dimensional embedding space used in experiment \footnotesize{3}⃝. Therefore, for the next experiment in the ablation study and our final model, we make use of the geometric features. In summary, the quality of the aggregation features has a significant impact.

Does the graph convolutional network help?

The graph convolutional network (GCN) defined on top of proposals enables higher-order interaction between proposals. Experiment \footnotesize{5}⃝ corresponds to the model depicted in Fig. 2 with a 10 layer GCN. Experiment \footnotesize{4}⃝ differs from experiment \footnotesize{5}⃝ in that it does not include the GCN for proposal consolidation. Adding the GCN results in another improvement of +1.3 mAP. In total, by incorporating the GCN and replacing NMS with multi-proposal aggregation, we observe an improvement of +11.6 mAP over the same network architecture without those changes.

Conclusion

In this work, we introduced 3D-MPA, a new method for 3D semantic instance segmentation. Our core idea is to combine the benefits of both top-down and bottom-up object detection strategies. That is, we first produce a number of proposals using an object-centric voting scheme based on a sparse volumetric backbone. Each object may receive multiple proposals, which makes our method robust to potential outliers in the object proposal stage. However, at the same time we obtain only a handful of proposals such that clustering them is computationally inexpensive. To address this, we first allow higher-order feature interactions between proposals via a graph convolutional network. We then aggregate proposals based on graph relationship results and proposal feature similarities. We show that graph convolutions help to achieve high evaluation scores, although, the largest improvement originates from our multi proposal aggregation strategy. Our combined approach achieves state-of-the-art instance segmentation and object detection results on the popular ScanNetV2 and S3DIS datasets, thus validating our algorithm design.

Overall, we believe that multi proposal aggregation is a promising direction for object detection, in particular in the 3D domain. However, there still remain many interesting future avenues, for instance, how to combine detection with tracking in semi-dynamic sequences. We see a variety of interesting ideas, where proposals could be distributed in 4D space and accumulated along the time-space axis.

We would like to thank Theodora Kontogianni, Jonas Schult, Jonathon Luiten, Mats Steinweg, Ali Athar, Dan Jia and Sabarinath Mahadevan for helpful feedback as well as Angela Dai for help with the video. This work was funded by the ERC Consolidator Grant DeeViSe (ERC-2017-COG-773161) and the ERC Starting Grant Scan2CAD (804724).

References