Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection

Summary

This paper presents a new task named object-centric occupancy completion as a fine-grained object representation to supplement the coarse-grained 3D bounding boxes. To accomplish this task, a new dataset, which annotates instance-level high-resolution occupancy, is created in an automated pipeline. This paper also introduces an implicit shape decoder to fuse multi-frame information, predict instance occupancy and refine 3D bounding boxes. Experiments on Waymo datasets above several baselines demonstrate the effectiveness of the proposed method on both occupancy prediction and 3D detection.

Strengths

1. This paper is well-written and organized. 2. A novel task, occupancy augments 3D object detection, and a corresponding new instance-level occupancy datasets is proposed. 3. A implicit shape decoder is proposed and achieves great improvements both in occupancy and 3D detection.

Weaknesses

1. The motivation of this paper does not seem to be very reasonable. The authors claim that a. high-resolution scene-level occupancy is constrained by computational cost and foreground objects is more import, and b. 3D detection is too coarse to capture the object geometry information. So why not just predict foreground instance-level occupancy in the whole scene, instead of pursuing higher detection accuracy by using the occupancy results? 2. Time and memory cost bought by the proposed shape decoder are not provided. The paper is trying to make a trade-off between occupancy and detection in fine-/coarse-grained level and computational cost level. But the authors only report the occupancy and detection accuracy. 3. Some methods, like VoxelNeXt, FSDv2, HEDNet are missing and are not compared in Table 1. 4. Typos/mis-leading descriptions. For example, ‘Tab. 5.4’ on line 351 -> ‘Tab. 3’.

Questions

1. Why not just predict foreground instance-level occupancy in the whole scene, instead of pursuing higher detection accuracy by using the occupancy results? (the same as weakness 1) 2. Could you provide the computational cost of your method or the proposed module?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors adequately addressed the limitations.

Summary

In this work, the authors propose a novel task called object-centric occupancy. It extends the 3D detected bounding box representation to provide a more detailed description of the internal object shape. The method provides higher voxel resolution in large scenes by focusing on foreground objects only. It not only achieves state-of-the-art performance on shape completion but can also help refine the object detection tasks on the Waymo Open Dataset (WOD).

Strengths

- The motion of the proposed task is clear, and the task itself shows good potential in scene understanding. It can enhance 3D detection results even at a far distance. - The extensive ablation studies validate each contribution. Various detector results with different settings help prove the robustness of the proposed methods. Using implicit representation from a 3D reconstruction task to complete shapes is neat and interesting. It will be interesting to see how this work can be applied to popular 3D Gaussian representation.

Weaknesses

- The experimental results are only obtained on the Waymo Open Dataset. It will be nicer to conduct the experiments on nuScenes or Argoverse 2 to validate its robustness for different datasets. - Although the authors say it is a new task, so there are no learning baselines for shape completion, it will be interesting to compare the results with other scene occupancy methods. So that we can see the flaws of using coarse resolution quantitatively.

Questions

- The extrapolated results of shape completion are interesting, showing that it can achieve a performance similar to that of using GT boxes. Will it also help with 3D Object Detection results?

Rating

6

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes

Summary

The manuscript introduces the idea of representing the shape of objects at higher fidelity (and independent of) the rest of the scene. This is explored in the context of autonomous vehicles research on 3d car detection and representation. The proposed model regresses a shape code and an updated 3d bounding box from a 3D bounding box tracklet (derived from any other algorithm) and the points included in it. The shape code can be queried for occupancy to produce a full shape representation during inference. The proposed approach is able to infer complete shapes from partial inputs and the updated 3D BBs improve the input 3D BBs.

Strengths

The proposed approach is relatively straightforward, effective and well motivated. This makes it reusable for other works and the paper more reproducable. The manuscript is well written and the illustrations help convey the message and improve understanding of the written parts. The evaluation is comprehensive and the sensitivity studies are well chosen and help motivate architecture and training choices. In particular it is great to see that the addition of the high resolution shape code and updated 3D BB does lead to substantial performance improvements on the 3D BB detection task (especially for far away OBBs). And that the shape code (if given the GT OBB) does produce a high IoU occupancy grid even if the input 3D BBs are subpar (table 1).

Weaknesses

The manuscript's related work section misses out on an existing related field of 3D CAD model retrieval (which also produces complete shapes) and shape regression from RGB (and depth data) in indoor scenes. Relevant related works include: - Scan2CAD https://openaccess.thecvf.com/content_CVPR_2019/papers/Avetisyan_Scan2CAD_Learning_CAD_Model_Alignment_in_RGB-D_Scans_CVPR_2019_paper.pdf - SLAM++ https://www.doc.ic.ac.uk/~ajd/Publications/salas-moreno_etal_cvpr2013.pdf - FroDO https://openaccess.thecvf.com/content_CVPR_2020/papers/Runz_FroDO_From_Detections_to_3D_Objects_CVPR_2020_paper.pdf I would have wanted to see a few renderings of the shape codes; This would support the claim that the model learns to complete shapes. The appendix has a few but the visualizations are hard to understand without a better renderer. Some kind of shading or edges for the 3D voxels are essential to see any kind of depth and thus shape (Fig 6 and 7). Extracting a mesh using marching cubes at the 0.5 isolevel might also work.

Questions

The model takes in a series of 3D BBs and outputs one updated 3D BB - at which timestamp is this 3D BB output? The latest?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation of rigid objects only is addressed in the appendix. This means humans are not supported for example.

Summary

This paper addresses the limitations of 3D object bounding box representations in autonomous driving by introducing object-centric occupancy. It uses an implicit shape decoder to manage dynamic-size occupancy generation. The method demonstrates robust performance under noisy conditions, significantly enhancing detection results in the Waymo Open Dataset.

Strengths

1. The presentation is well-executed, with figures and charts effectively aiding reader comprehension. 2. The overall performance is impressive, demonstrating significant improvements across multiple baselines.

Weaknesses

1. Creating detailed occupancy for each object seems unnecessary. In most downstream tasks in autonomous driving, using bounding boxes (bboxes) is sufficient. 2. The performance improvement primarily stems from temporal feature fusion, which lacks significant technical innovation. 3. It is unclear whether the loss on occ heads in Fig. 4 enhances detection performance. The authors should compare detection performance with and without occ heads after obtaining the Shape Emb. Z to determine if occ heads contribute to learning useful features, such as yaw estimation.

Questions

See weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors discuss the limitation concerning non-rigid objects, which is indeed a constraint. They could start by exploring whether reconstructing the noisy occupancy of non-rigid objects improves detection performance.

Post rebuttal

Thanks for the response. My concerns regarding the motivation and computation cost have been well-addressed, therefore I raise my rating to weak accept.