Summary
This work presents a novel approach for zero-shot 3D occupancy prediction for autonomous driving applications. The key idea consists of three parts:
- a 2D-3D encoder to create a 3D voxel feature grid, based on TPVFormer
- An MLP-based voxel occupancy predictor that is class agnostic
- an MLP-based feature encoder, called a "3D language head" that takes the TPV-former features that have positive occupancy, and learns to predict the corresponding features of an off-the-shelf MaskCLIP+ model. In effect this is distilling the 2D MaskCLIP+ features into 3D.
At inference time LiDAR information is not necessary because occupancy is predicted directly from RGB images. Further, the model can be used zero-shot at inference time by computing the similarty of the output of the 3D language head features with language features from a CLIP text encoder. During training, only direct supervision is needed derived from class-agnostic occupancy from the LiDAR data, the distillation loss only needs an off-the-shelf vision language model. The proposed approach obtains ~78% of the fully supervised TPVFormer, while needing no class label supervision.
Strengths
Originality
- While there are a variety of works trying to leverage vision-language models, this is the first work to show promising results on open vocabulary 3D occupancy prediction. The proposed model is simple but highly effective taking advantage of existing LIDAR datasets, the TPVFormer architecture, and off the shelf vision language models. The simplicity of this technique opens up the potential for removing the dependence of expensive supervision in this problem domain and instead focusing on extracting information from existing vision-language models.
Quality
- The main claim of the paper is that the proposed system can perform zero-shot semantic occupancy prediction without using any 3D semantic labels at training time. This paper provides sufficient evidence for this claim by evaluating on nuScenes and obtaining ~78% of the performance of the fully supervised model. This highlights the power of vision-language models.
- The related work is detailed and positions the proposed work well relative to previous techniques semantic 3D occupancy prediction, multi-modal learning and open vocabulary segmentation.
- Section 4.3 presents empirical evidence for the choice of hyperparameters, and the effect of image resolutions.
Clarity
- The paper writing is excellent, it easy to follow and understand. Figure 2 is particularly helpful for following the technical description of the model in Section 3.
Weaknesses
Quality - small scale evaluation
- The evaluation is done only on one dataset. This is understandable given the computation costs for training and evaluating models, but having evaluation on more than one dataset would significantly improve the paper. A potential candidate dataset is https://pandaset.org/ which contains LIDAR and 5 wide angle cameras.
- Evaluation is done only against one open-vocabulary baseline and one supervised basline. While TPVFormer appears to be the only fully supervised reasonable choice, there are baselines like ODISE (https://github.com/NVlabs/ODISE) or OVSeg (https://jeff-liangf.github.io/projects/ovseg/) for open vocabulary segmentation, whose predictions could get back-projected into the class-agnostic occupancy predictions, to obtain 3D semantic labels. These are relevant baselines in light of the comparison with MaskCLIP.
- Open-vocabulary recognition is powerful because it allows tackling difficult long-tailed scenarios. Currently such a benchmark doesn't exist for occupancy prediction, limiting the evidence to qualitative examples, if we want to evaluate performance beyond the classes defined in nuScenes. It would be a great improvement to the paper if there were some annotated samples to set quantitative benchmark.
Questions
- Why is evaluation limited to only one dataset when there are other datasets that can be used for evaluation?
- Why aren't existing open-vocabulary segmentation models, backprojected in 3D, used as baselines in addition to MaskCLIP? This is particularly important given the small scale experiments.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors have adequately addressed the limitations and the potential negative societal impact.