Mending Neural Implicit Modeling for 3D Vehicle Reconstruction in the Wild

@article{Duggal2022MendingNI,
  title={Mending Neural Implicit Modeling for 3D Vehicle Reconstruction in the Wild},
  author={Shivam Duggal and Zihao Wang and Wei-Chiu Ma and Sivabalan Manivasagam and Justin Liang and Shenlong Wang and Raquel Urtasun},
  journal={2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2022},
  pages={277-286}
}
Reconstructing high-quality 3D objects from sparse, partial observations from a single view is of crucial importance for various applications in computer vision, robotics, and graphics. While recent neural implicit modeling methods show promising results on synthetic or dense data, they perform poorly on sparse and noisy real-world data. We discover that the limitations of a popular neural implicit model are due to lack of robust shape priors and lack of proper regularization. In this work, we… 
Learning Smooth Neural Functions via Lipschitz Regularization
TLDR
A novel regularization designed to encourage smooth latent spaces in neural fields by penalizing the upper bound on the field’s Lipschitz constant is introduced, showing both qualitative and quantitative improvements over existing state-of-the-art and non-regularized baselines.
Topologically-Aware Deformation Fields for Single-View 3D Reconstruction
TLDR
A topologically-aware deformation that maps 3D points in object space to a higher-dimensional canonical space and achieves state-of-the-art reconstruction and correspondence on several datasets: ShapeNet, Pascal3D+, CUB, and Pix3D chairs.
Online Adaptation for Implicit Object Tracking and Shape Reconstruction in the Wild
TLDR
A novel and unified framework which utilizes a neural implicit function to simultaneously track and reconstruct 3D objects in the wild and experiment with both Waymo and KITTI datasets and shows significant improvements over state-of-the-art methods for both tracking and shape reconstruction tasks.

References

SHOWING 1-10 OF 69 REFERENCES
Deep Optimized Priors for 3D Shape Modeling and Reconstruction
TLDR
This work introduces a new learning framework for 3D modeling and reconstruction that greatly improves the generalization ability of a deep generator and proposes to further optimize the learned prior and latent code according to the input physical measurements after the training.
Learning 3D Shape Completion Under Weak Supervision
TLDR
This work proposes a weakly-supervised learning-based approach to 3D shape completion which neither requires slow optimization nor direct supervision and outperforms the data-driven approach of Engelmann et al.
Occupancy Networks: Learning 3D Reconstruction in Function Space
TLDR
This paper proposes Occupancy Networks, a new representation for learning-based 3D reconstruction methods that encodes a description of the 3D output at infinite resolution without excessive memory footprint, and validate that the representation can efficiently encode 3D structure and can be inferred from various kinds of input.
Autolabeling 3D Objects With Differentiable Rendering of SDF Shape Priors
TLDR
An automatic annotation pipeline to recover 9D cuboids and 3D shapes from pre-trained off-the-shelf 2D detectors and sparse LIDAR data is presented and a curriculum learning strategy is proposed, iteratively retraining on samples of increasing difficulty in subsequent self-improving annotation rounds.
DirectShape: Direct Photometric Alignment of Shape Priors for Visual Vehicle Pose and Shape Estimation
TLDR
This paper proposes a novel approach to jointly infer the 3D rigid-body poses and shapes of vehicles from a stereo image pair using shape priors and shows superior performance of the method on 3D pose and shape estimation over the previous geometric approach.
Convolutional Occupancy Networks
TLDR
Convolutional Occupancy Networks is proposed, a more flexible implicit representation for detailed reconstruction of objects and 3D scenes that enables the fine-grained implicit 3D reconstruction of single objects, scales to large indoor scenes, and generalizes well from synthetic to real data.
3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare
TLDR
A differentiable Render-and-Compare loss is proposed that allows 3D shape and pose to be learned with 2D supervision and produces a compact 3D representation of the scene, which can be readily used for applications like autonomous driving.
Weakly-supervised 3D Shape Completion in the Wild
TLDR
Experiments show that it is feasible and promising to learn 3D shape completion through large-scale data without shape and pose supervision, and jointly optimizes canonical shapes and poses with multi-view geometry constraints during training.
Fostering Generalization in Single-view 3D Reconstruction by Learning a Hierarchy of Local and Global Shape Priors
TLDR
This work argues that exploiting local priors allows this method to efficiently use input observations, thus improving generalization in visible areas of novel shapes, and shows that the hierarchical approach generalizes much better than the global approach.
Learning Category-Specific Mesh Reconstruction from Image Collections
TLDR
A learning framework for recovering the 3D shape, camera, and texture of an object from a single image by incorporating texture inference as prediction of an image in a canonical appearance space and shows that semantic keypoints can be easily associated with the predicted shapes.
...
...