Occupancy Networks: Learning 3D Reconstruction in Function Space

Introduction

Recently, learning-based approaches for 3D reconstruction have gained popularity . In contrast to traditional multi-view stereo algorithms, learned models are able to encode rich prior information about the space of 3D shapes which helps to resolve ambiguities in the input.

While generative models have recently achieved remarkable successes in generating realistic high resolution images , this success has not yet been replicated in the 3D domain. In contrast to the 2D domain, the community has not yet agreed on a 3D output representation that is both memory efficient and can be efficiently inferred from data. Existing representations can be broadly categorized into three categories: voxel-based representations , point-based representations and mesh representations , see Fig. 1.

Voxel representations are a straightforward generalization of pixels to the 3D case. Unfortunately, however, the memory footprint of voxel representations grows cubically with resolution, hence limiting naïve implementations to $32^{3}$ or $64^{3}$ voxels. While it is possible to reduce the memory footprint by using data adaptive representations such as octrees , this approach leads to complex implementations and existing data-adaptive algorithms are still limited to relatively small $256^{3}$ voxel grids. Point clouds and meshes have been introduced as alternative representations for deep learning, using appropriate loss functions. However, point clouds lack the connectivity structure of the underlying mesh and hence require additional post-processing steps to extract 3D geometry from the model. Existing mesh representations are typically based on deforming a template mesh and hence do not allow arbitrary topologies. Moreover, both approaches are limited in the number of points/vertices which can be reliably predicted using a standard feed-forward network.

In this paperAlso see for concurrent work that proposes similar ideas., we propose a novel approach to 3D-reconstruction based on directly learning the continuous 3D occupancy function (Fig. 1(d)). Instead of predicting a voxelized representation at a fixed resolution, we predict the complete occupancy function with a neural network $f_{\theta}$ which can be evaluated at arbitrary resolution. This drastically reduces the memory footprint during training. At inference time, we extract the mesh from the learned model using a simple multi-resolution isosurface extraction algorithm which trivially parallelizes over 3D locations.

In summary, our contributions are as follows:

We introduce a new representation for 3D geometry based on learning a continuous 3D mapping.

We show how this representation can be used for reconstructing 3D geometry from various input types.

We experimentally validate that our approach is able to generate high-quality meshes and demonstrate that it compares favorably to the state-of-the-art.

Related Work

Existing work on learning-based 3D reconstruction can be broadly categorized by the output representation they produce as either voxel-based, point-based or mesh-based.

Voxel Representations: Due to their simplicity, voxels are the most commonly used representation for discriminative and generative 3D tasks.

Early works have considered the problem of reconstructing 3D geometry from a single image using 3D convolutional neural networks which operate on voxel grids . Due to memory requirements, however, these approaches were limited to relatively small $32^{3}$ voxel grids. While recent works have applied 3D convolutional neural networks to resolutions up to $128^{3}$, this is only possible with shallow architectures and small batch sizes, which leads to slow training.

The problem of reconstructing 3D geometry from multiple input views has been considered in . Ji et al. and Kar et al. encode the camera parameters together with the input images in a 3D voxel representation and apply 3D convolutions to reconstruct 3D scenes from multiple views. Paschalidou et al. introduced an architecture that predicts voxel occupancies from multiple images, exploiting multi-view geometry constraints .

Other works applied voxel representations to learn generative models of 3D shapes. Most of these methods are either based on variational auto-encoders or generative adversarial networks . These two approaches were pursued in and , respectively.

Due to the high memory requirements of voxel representations, recent works have proposed to reconstruct 3D objects in a multi-resolution fashion . However, the resulting methods are often complicated to implement and require multiple passes over the input to generate the final 3D model. Furthermore, they are still limited to comparably small $256^{3}$ voxel grids. For achieving sub-voxel precision, several works have proposed to predict truncated signed distance fields (TSDF) where each point in a 3D grid stores the truncated signed distance to the closest 3D surface point. However, this representation is usually much harder to learn compared to occupancy representations as the network must reason about distance functions in 3D space instead of merely classifying a voxel as occupied or not. Moreover, this representation is still limited by the resolution of the underlying 3D grid.

Point Representations: An interesting alternative representation of 3D geometry is given by 3D point clouds which are widely used both in the robotics and in the computer graphics communities. Qi et al. pioneered point clouds as a representation for discriminative deep learning tasks. They achieved permutation invariance by applying a fully connected neural network to each point independently followed by a global pooling operation. Fan et al. introduced point clouds as an output representation for 3D reconstruction. However, unlike other representations, this approach requires additional non-trivial post-processing steps to generate the final 3D mesh.

Mesh Representations: Meshes have first been considered for discriminative 3D classification or segmentation tasks by applying convolutions on the graph spanned by the mesh’s vertices and edges .

More recently, meshes have also been considered as output representation for 3D reconstruction . Unfortunately, most of these approaches are prone to generating self-intersecting meshes. Moreover, they are only able to generate meshes with simple topology , require a reference template from the same object class or cannot guarantee closed surfaces . Liao et al. proposed an end-to-end learnable version of the marching cubes algorithm . However, their approach is still limited by the memory requirements of the underlying 3D grid and hence also restricted to $32^{3}$ voxel resolution.

In contrast to the aforementioned approaches, our approach leads to high resolution closed surfaces without self-intersections and does not require template meshes from the same object class as input. This idea is related to classical level set approaches to multi-view 3D reconstruction . However, instead of solving a differential equation, our approach uses deep learning to obtain a more expressive representation which can be naturally integrated into an end-to-end learning pipeline.

Method

In this section, we first introduce Occupancy Networks as a representation of 3D geometry. We then describe how we can learn a model that infers this representation from various forms of input such as point clouds, single images and low-resolution voxel representations. Lastly, we describe a technique for extracting high-quality 3D meshes from our model at test time.

We call this network the Occupancy Network.

2 Training

Here, $x_{i}$ is the $i$’th observation of batch $\mathcal{B}$, $o_{ij}\equiv o(p_{ij})$ denotes the true occupancy at point $p_{ij}$, and $\mathcal{L}(\cdot,\cdot)$ is a cross-entropy classification loss.

The performance of our method depends on the sampling scheme that we employ for drawing the locations $p_{ij}$ that are used for training. In Section 4.6 we perform a detailed ablation study comparing different sampling schemes. In practice, we found that sampling uniformly inside the bounding box of the object with an additional small padding yields the best results.

where $\operatorname{KL}$ denotes the KL-divergence, $p_{0}(z)$ is a prior distribution on the latent variable $z_{i}$ (typically Gaussian) and $z_{i}$ is sampled according to $q_{\psi}(z_{i}|(p_{ij},o_{ij})_{j=1:K})$.

3 Inference

For extracting the isosurface corresponding to a new observation given a trained occupancy network, we introduce Multiresolution IsoSurface Extraction (MISE), a hierarchical isosurface extraction algorithm (Fig. 2). By incrementally building an octree , MISE enables us to extract high resolution meshes from the occupancy network without densely evaluating all points of a high-dimensional occupancy grid.

We first discretize the volumetric space at an initial resolution and evaluate the occupancy network $f_{\theta}(p,x)$ for all $p$ in this grid. We mark all grid points $p$ as occupied for which $f_{\theta}(p,x)$ is bigger or equal to some thresholdThe threshold $\tau$ is the only hyperparameter of our occupancy network. It determines the “thickness” of the extracted 3D surface. In our experiments we cross-validate this threshold on a validation set. $\tau$. Next, we mark all voxels as active for which at least two adjacent grid points have differing occupancy predictions. These are the voxels which would intersect the mesh if we applied the marching cubes algorithm at the current resolution. We subdivide all active voxels into 8 subvoxels and evaluate all new grid points which are introduced to the occupancy grid through this subdivision. We repeat these steps until the desired final resolution is reached. At this final resolution, we apply the Marching Cubes algorithm to extract an approximate isosurface

Our algorithm converges to the correct mesh if the occupancy grid at the initial resolution contains points from every connected component of both the interior and the exterior of the mesh. It is hence important to take an initial resolution which is high enough to satisfy this condition. In practice, we found that an initial resolution of $32^{3}$ was sufficient in almost all cases.

The initial mesh extracted by the Marching Cubes algorithm can be further refined. In a first step, we simplify the mesh using the Fast-Quadric-Mesh-Simplification algorithmhttps://github.com/sp4cerat/Fast-Quadric-Mesh-Simplification . Finally, we refine the output mesh using first and second order (i.e., gradient) information. Towards this goal, we sample random points $p_{k}$ from each face of the output mesh and minimize the loss

where $n(p_{k})$ denotes the normal vector of the mesh at $p_{k}$. In practice, we set $\lambda=0.01$. Minimization of the second term in (6) uses second order gradient information and can be efficiently implemented using Double-Backpropagation .

Note that this last step removes the discretization artifacts of the Marching Cubes approximation and would not be possible if we had directly predicted a voxel-based representation. In addition, our approach also allows to efficiently extract normals for all vertices of our output mesh by simply backpropagating through the occupancy network. In total, our inference algorithm requires 3s per mesh.

4 Implementation Details

We implemented our occupancy network using a fully-connected neural network with 5 ResNet blocks and condition it on the input using conditional batch normalization . We exploit different encoder architectures depending on the type of input. For single view 3D reconstruction, we use a ResNet18 architecture . For point clouds we use the PointNet encoder . For voxelized inputs, we use a 3D convolutional neural network . For unconditional mesh generation, we use a PointNet for the encoder network $g_{\psi}$. More details are provided in the supplementary material.

Experiments

We conduct three types of experiments to validate the proposed occupancy networks. First, we analyze the representation power of occupancy networks by examining how well the network can reconstruct complex 3D shapes from a learned latent embedding. This gives us an upper bound on the results we can achieve when conditioning our representation on additional input. Second, we condition our occupancy networks on images, noisy point clouds and low resolution voxel representations, and compare the performance of our method to several state-of-the-art baselines. Finally, we examine the generative capabilities of occupancy networks by adding an encoder to our model and generating unconditional samples from this model.The code to reproduce our experiments is available under https://github.com/LMescheder/Occupancy-Networks.

Baselines: For the single image 3D reconstruction task, we compare our approach against several state-of-the-art baselines which leverage various 3D representations: we evaluate against 3D-R2N2 as a voxel-based method, Point Set Generating Networks (PSGN) as a point-based technique and Pixel2Mesh as well as AtlasNet as mesh-based approaches. For point cloud inputs, we adapted 3D-R2N2 and PSGN by changing the encoder. As mesh-based baseline, we use Deep Marching Cubes (DMC) which has recently reported state-of-the-art results on this task. For the voxel super-resolution task we assess the improvements wrt. the input.

Dataset: For all of our experiments we use the ShapeNet subset of Choy et al. . We also use the same voxelization, image renderings and train/test split as Choy et al. Moreover, we subdivide the training set into a training and a validation set on which we track the loss of our method and the baselines to determine when to stop training.

In order to generate watertight meshes and to determine if a point lies in the interior of a mesh (e.g., for measuring IoU) we use the code provided by Stutz et al. . For a fair comparison, we sample points from the surface of the watertight mesh instead of the original model as ground truth for PSGN , Pixel2Mesh and DMC . All of our evaluations are conducted wrt. these watertight meshes.

Metrics: For evaluation we use the volumetric IoU, the Chamfer-$L_{1}$ distance and a normal consistency score.

Volumetric IoU is defined as the quotient of the volume of the two meshes’ union and the volume of their intersection. We obtain unbiased estimates of the volume of the intersection and the union by randomly sampling 100k points from the bounding volume and determining if the points lie inside our outside the ground truth / predicted mesh.

The Chamfer-$L_{1}$ distance is defined as the mean of an accuracy and and a completeness metric. The accuracy metric is defined as the mean distance of points on the output mesh to their nearest neighbors on the ground truth mesh. The completeness metric is defined similarly, but in opposite direction. We estimate both distances efficiently by randomly sampling 100k points from both meshes and using a KD-tree to estimate the corresponding distances. Like Fan et al. we use $1/10$ times the maximal edge length of the current object’s bounding box as unit $1$.

Finally, to measure how well the methods can capture higher order information, we define a normal consistency score as the mean absolute dot product of the normals in one mesh and the normals at the corresponding nearest neighbors in the other mesh.

In our first experiment, we investigate how well occupancy networks represent 3D geometry, independent of the inaccuracies of the input encoding. The question we try to answer in this experiment is whether our network can learn a memory efficient representation of 3D shapes while at the same time preserving as many details as possible. This gives us an estimate of the representational capacity of our model and an upper bound on the performance we may expect when conditioning our model on additional input. Similarly to , we embed each training sample in a 512 dimensional latent space and train our neural network to reconstruct the 3D shape from this embedding.

We apply our method to the training split of the “chair” category of the ShapeNet dataset. This subset is challenging to represent as it is highly varied and many models contain high-frequency details. Since we are only interested in reconstructing the training data, we do not use separate validation and test sets for this experiment.

For evaluation, we measure the volumetric IoU to the ground truth mesh. Quantitative results and a comparison to voxel representations at various resolutions are shown in Fig. 4. We see that the Occupancy Network (ONet) is able to faithfully represent the entire dataset with a high mean IoU of 0.89 while a low-resolution voxel representation is not able to represent the meshes accurately. At the same time, the occupancy network is able to encode all 4746 training samples with as little as 6M parameters, independently of the resolution. In contrast, the memory requirements of a voxel representation grow cubically with resolution. Qualitative results are shown in Fig. 3. We observe that the occupancy network enables us to represent details of the 3D geometry which are lost in a low-resolution voxelization.

2 Single Image 3D Reconstruction

In our second experiment, we condition the occupancy network on an additional view of the object from a random camera location. The goal of this experiment is to evaluate how well occupancy functions can be inferred from complex input. While we train and test our method on the ShapeNet dataset, we also present qualitative results for the KITTI and the Online Products dataset .

ShapeNet: In this experiment, we use a ResNet-18 image encoder, which was pretrained on the ImageNet dataset. For a fair comparison, we use the same image encoder for both 3D-R2N2 and PSGNSee supplementary for a comparison to the original architectures.. For PSGN we use a fully connected decoder with 4 layers and 512 hidden units in each layer. The last layer projects the hidden representation to a 3072 dimensional vector which we reshape into 1024 3D points. As we use only a single input view, we remove the recurrent network in 3D-R2N2. We reimplemented the method of in PyTorch, closely following the Tensorflow implementation provided by the authors. For the method of , we use the code and pretrained model from the authorshttps://github.com/ThibaultGROUEIX/AtlasNet.

For all methods, we track the loss and other metrics on the validation set and stop training as soon as the target metric reaches its optimum. For 3D-R2N2 and our method we use the IoU to the ground truth mesh as target metric, for PSGN and Pixel2Mesh we use the Chamfer distance to the ground truth mesh as target metric. To extract the final mesh, we use a threshold of $0.4$ for 3D-R2N2 as suggested in the original publication . To choose the threshold parameter $\tau$ for our method, we perform grid search on the validation set (see supplementary) and found that $\tau=0.2$ yields a good trade-off between accuracy and completeness.

Qualitative results from our model and the baselines are shown in Fig. 5. We observe that all methods are able to capture the 3D geometry of the input image. However, 3D-R2N2 produces a very coarse representation and hence lacks details. In contrast, PSGN produces a high-fidelity output, but lacks connectivity. As a result, PSGN requires additional lossy post-processing steps to produce a final meshSee supplementary material for meshing results.. Pixel2Mesh is able to create compelling meshes, but often misses holes in the presence of more complicated topologies. Such topologies are frequent, for example, for the “chairs“ category in the ShapeNet dataset. Similarly, AtlasNet captures the geometry well, but produces artifacts in form of self-intersections and overlapping patches.

In contrast, our method is able to capture complex topologies, produces closed meshes and preserves most of the details. Please see the supplementary material for additional high resolution results and failure cases.

Quantitative results are shown in Table 1. We observe that our method achieves the highest IoU and normal consistency to the ground truth mesh. Surprisingly, while not trained wrt. Chamfer distance as PSGN, Pixel2Mesh or AtlasNet, our method also achieves good results for this metric. Note that it is not possible to evaluate the IoU for PSGN or AtlasNet, as they do not yield watertight meshes.

Real Data: To test how well our model generalizes to real data, we apply our network to the KITTI and Online Products datasets . To capture the variety in viewpoints of KITTI and Online Products, we rerendered all ShapeNet objects with random camera locations and retrained our network for this task.

For the KITTI dataset, we additionally use the instance masks provided in to mask and crop car regions. We then feed these images into our neural network to predict the occupancy function. Some selected qualitative results are shown in Fig. 6(a). Despite only trained on synthetic data, we observe that our method is also able to generate realistic reconstructions in this challenging setting.

For the Online Products dataset, we apply the same pretrained model. Several qualitative results are shown in Fig. 6(b). Again, we observe that our method generalizes reasonably well to real images despite being trained solely on synthetic data. An additional quantitative evaluation on the Pix3D dataset can be found in the supplementary.

3 Point Cloud Completion

As a second conditional task, we apply our method to the problem of reconstructing the mesh from noisy point clouds. Towards this goal, we subsample 300 points from the surface of each of the (watertight) ShapeNet models and apply noise using a Gaussian distribution with zero mean and standard deviation $0.05$ to the point clouds.

Again, we measure both the IoU and Chamfer-$L_{1}$ distance wrt. the ground truth mesh. The results are shown in Table 2. We observe that our method achieves the highest IoU and normal consistency as well as the lowest Chamfer-$L_{1}$ distance. Note that all numbers are significantly better than for the single image 3D reconstruction task. This can be explained by the fact that this task is much easier for the recognition model, as there is less ambiguity and the model only has to fill in the gaps.

4 Voxel Super-Resolution

As a final conditional task, we apply occupancy networks to 3D super-resolution . Here, the task is to reconstruct a high-resolution mesh from a coarse $32^{3}$ voxelization of this mesh.

The results are shown in Table 3. We observe that our model considerably improves IoU, Chamfer-$L_{1}$ distance and normal consistency compared to the coarse input mesh. Please see the supplementary for qualitative results.

5 Unconditional Mesh Generation

Finally, we apply our occupancy network to unconditional mesh generation, training it separately on four categories of the ShapeNet dataset in an unsupervised fashion. Our goal is to explore how well our model can represent the latent space of 3D models. Some samples are shown in Figure 7. Indeed, we find that our model can generate compelling new models. In the supplementary material we show interpolations in latent space for our model.

6 Ablation Study

In this section, we test how the various components of our model affect its performance on the single-image 3D-reconstruction task.

First, we examine how the sampling strategy affects the performance of our final model. We try three different sampling strategies: (i) sampling 2048 points uniformly in the bounding volume of the ground truth mesh (uniform sampling), (ii) sampling 1024 points inside and 1024 points outside mesh (equal sampling) and (iii) sampling 1024 points uniformly and 1024 points on the surface of the mesh plus some Gaussian noise with standard deviation $0.1$ (surface sampling). We also examine the effect of the number of sampling points by decreasing this number from 2048 to 64.

The results are shown in Table 8(a). To our surprise, we find that uniform, the simplest sampling strategy, works best. We explain this by the fact that other sampling strategies introduce bias to the model: for example, when sampling an equal number of points inside and outside the mesh, we implicitly tell the model that every object has a volume of $0.5$. Indeed, when using this sampling strategy, we observe thickening artifacts in the model’s output. Moreover, we find that reducing the number of sampling points from 2048 to 64 still leads to good performance, although the model does not perform as well as a model trained with 2048 sampling points.

Effect of architecture

To test the effect of the various components of our architecture, we test two variations: (i) we remove the conditional batch normalization and replace it with a linear layer in the beginning of the network that projects the encoding of the input to the required hidden dimension and (ii) we remove all ResNet blocks in the decoder and replace them with linear blocks. The results are presented in Table 8(b). We find that both components are helpful to achieve good performance.

Conclusion

In this paper, we introduced occupancy networks, a new representation for 3D geometry. In contrast to existing representations, occupancy networks are not constrained by the discretization of the 3D space and can hence be used to represent realistic high-resolution meshes.

Our experiments demonstrate that occupancy networks are very expressive and can be used effectively both for supervised and unsupervised learning. We hence believe that occupancy networks are a useful tool which can be applied to a wide variety of 3D tasks.

Acknowledgements

This work was supported by the Intel Network on Intelligent Systems and by Microsoft Research through its PhD Scholarship Programme.

References