OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queries

Summary

This paper proposes a new occupancy predictor that utilizes the octree structure to reduce the total number of occupancy query. The octree tree is initialized with semantic information and recursively rectified, so that it can accurately represent the structure of the driving scenes.

Strengths

1. This work gives a new way to represent occupancy instead of voxel feature or dense query, which is enlightening for later works. 2. The prediction precision of OctreeOcc exceeds FB-Occ and PanoOcc, indicating a new SOTA. The visualization results also demonstrate its ability to predict fine structures.

Weaknesses

1. OctreeOcc requires extra segmentation model to provide the reference for the initialization of octree queries, which reduces the efficiency gain. More details of the segmentation model should be published. 2. The ablation study shows that the accuracy is largely improved after using the extra semantic information. A fair comparison between OctreeOcc without extra semantic information and other methods is missing.

Questions

1. It is not clear how the high-confidence and low-confidence regions are determined in Iterative Structure Rectification. Is the confidence related to the split probability? 2. How does OctreeOcc handle the conflict between the octree queries initialized by current semantic information and the octree queries aligned from the previous frame?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have described the limitation of this work, and there is no potential negative societal impact.

Summary

This paper introduces OctreeOcc, aiming to tackle the heavy computational demands of the dense and regular grid representations employed by the previous methods. Instead of randomly initializing the octree structure, OctreeOcc incorporates the semantic priors of images as guidance. The octree structure are further updated iteratively to correct the potential errors.

Strengths

1. Octree representation is a good solution to the heavy computation burden of the dense grid representation. OctreeOcc presents the details of the network components clearly. 2. OctreeOcc provides detailed ablation experiments. 3. OctreeOcc achieves good balance between the accuracy and the efficiency.

Weaknesses

1. The image resolution and visible mask should be marked for fairly comparing with other methods. To my understanding, using the visible mask or not results in significant performance differences. 2. This paper mentioned temporal fusion in line 131. Is the temporal information obtained similar to the bevformer? Besides, how many frames are employed? (e.g., 4 frames in the PanoOcc) 3. The metrics of the Symponies in the tables should be updated. The performance differences are acceptable because Symponies uses stereo information (If OctreeOcc were to rely solely on monocular input). 4. Dose the Memory presented in the tables refer to training memory? To my understanding, the inference memory for most of the methods are highly lower than this. 5. In Table 6, query form of octree attend to the image features in a hierarchical manner, what is the number of the attention layers of the 3D voxel?

Questions

Please see above weaknesses.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations.

Summary

This paper introduces OctreeOcc, an innovative 3D occupancy prediction framework that leverages the octree representation to adaptively capture valuable information in 3D, offering variable granularity to accommodate object shapes and semantic regions of varying sizes and complexities. The authors incorporate image semantic information to improve the accuracy of initial octree structures and design an effective rectification mechanism to refine the octree structure iteratively. Extensive evaluations show that OctreeOcc not only surpasses state-of-the-art methods in occupancy prediction, but also achieves a 15% − 24% reduction in computational overhead compared to dense-grid-based methods.

Strengths

- OctreeOcc utilises octree to represent 3D space, which is novel and no one has done it before as far as my knowledge. - Owing to the design of octree queries, OctreeOcc improves the accuracy of occupancy prediction with the latency also decreasing. - Extensive experiments on Occ3D and SemanticKitti demonstrate the effectiveness of OctreeOcc.

Weaknesses

- The pipeline is quite complex and heavily relies on manual operations, such as the selection strategy in Iterative Structure Rectification Module. My concern is that the complicated design may reduce the generalization. - The updating processing of octree queries is not very clear. In my understanding, if the octree mask changes, the newly generated octree queries cannot directly match the original octree queries. - Different octree queries may have different granularity, but, as shown in Eql (3), all octree queries seem to interact with image features only referring to their centers, which ignoring the influence of granularity.

Questions

please refer to weaknesses.

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

NaN

Summary

The authors aim to tackle the problem of high memory usage in dense occupancy prediction for 3D scenes. They introduce OctreeOcc, a method that uses octree structures to make predictions more efficiently. Experimental results show that OctreeOcc reduces computational load and achieves competitive performance.

Strengths

S1: The authors address the significant issue of high memory consumption in dense occupancy prediction, highlighting the importance of finding more efficient solutions. S2: The use of octree representation for occupancy prediction is a novel and effective approach, offering adaptive granularity for various object shapes and regions. S3: Experimental results show some reduction in memory usage and latency, demonstrating the efficiency of the proposed method.

Weaknesses

W1: The proposed method seems difficult to optimize and relies on segmentation and historical information. W2: The results on the SemanticKITTI dataset do not achieve state-of-the-art performance, and the reduction in memory usage is not very significant. W3: The training process is complex and costly, requiring three days on 8 A100 GPUs, which seems less efficient compared to previous methods.

Questions

Q1: The proposed method seems to be quite time-consuming to train. Can the authors provide a comparison of training times with other methods? This is especially relevant given the emphasis on efficiency throughout the paper. Q2: There is some confusion regarding the reported memory consumption in the experiments. For instance, the original VoxFormer [1] paper reports memory usage of less than 16GB, but the authors here report around 23GB. What accounts for this significant discrepancy? Are there differences in the settings or configurations used? Q3: The implementation details provided focus on the nuScenes dataset. Could the authors include the specifics of the implementation for the SemanticKITTI dataset as well? Q4: The paper lacks results or discussions on other commonly used datasets, such as SSCBench-KITTI360 [2]. Including these would provide a more comprehensive evaluation of the proposed method. [1] VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion [2] SSCBench: Monocular 3D Semantic Scene Completion Benchmark in Street Views

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have identified a significant limitation in their work: the dependency on the quality of the occupancy ground truth (GT). This reliance can lead to suboptimal performance if the GT data, derived from sparse LiDAR point clouds and surface reconstruction, is not accurate or complete.