Context and Geometry Aware Voxel Transformer for Semantic Scene Completion

Summary

This paper proposes CGFormer for semantic scene completion. It generates distinct voxel queries for different input images, instead of simply predefining a set of trainable parameters. The deformable cross attention is extend to 3D pixel space, avoiding sampling the same features for different projecting points. This method further enhances the 3D volume from both local and global perspectives. This paper incorporates stereo depth to improve the accuracy of depth probability by a depth refinement strategy.

Strengths

1. The code is submitted in the supplement material, whose guideline is detailed and easy to follow. 2. The proposed blocks bring notable performance gain, with elaborate experiments. 3. The CGFormer surpasses previous methods on both the SemanticKITTI and SSC-Bench-KITTI360 benchmarks. 4. The paper is well-organized the motivation is clear.

Weaknesses

1. EfficientNetB7 contains much more parameters than the ResNet50 employed in the previous coarse-to-fine methods (e.g., VoxFormer, Symphonize). Comparing the performance under the same setting will be fairer. 2. Text error in Figure 1. 3. Although the paper presents both superior qualitative and visualization results, it would be better to provide several failure cases.

Questions

Please see the weaknesses above.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

In this paper, the limitations have been well discussed. The accuracy on most of the categories is unsatisfactory. Furthermore, there is a need to explore designing depth estimation networks under multi-view scenarios to extend the geometry-aware view transformation to these scenes. The method is worth further exploration.

Summary

The authors present Context and Geometry Aware Voxel Transformer (CGFormer) for the semantic scene completion task. Their method extends the baseline VoxFormer with a Context-Aware Query Generator (CAQG), 3D deformable attention layers, a depth refinement block, and a dynamic fusion of voxel and TPV features. Experiments on two datasets demonstrated that CGFormer can achieve state-of-the-art mIoU performance.

Strengths

1. solid experiments. Two datasets are experimented with and results tables are presented clearly along with many baseline methods. 2. detailed ablation study for every designed module. 3. The figures are well-organized and easy to follow, which is greatly appreciated.

Weaknesses

1. The biggest concern to the reviewer is that the empirical results are not convincing enough to support the main claims of the proposed method. Specifically, a) The authors claim that context-aware queries are a major novelty that helps the model perform better in its region of interest. However, the empirical results show that the performance is superior mainly in categories with larger areas such as the roads and sidewalks, while not as good in categories with finer details such as trucks, persons, and bicycles. The fact that CGFormer is comparable or worse compared to the baselines with context-independent queries on 50% of the categories makes the claims less compelling. b) Besides a), the qualitative difference in Figure 3 is not illustrative. According to the first two columns, the context-dependent query should sample more at locations with details such as cars and bicycles, therefore, in the 3rd column, shouldn’t the context-aware queries be focused more on the white car? It is not clear why the points in (b) are irrelevant. c) The model is named CGFormer, while how context-aware plays a part has been discussed in the paper, how the performance reflects geometry-aware is barely touched on. 2. Some minor issues with the writing. The proposed CAQG seems to be a key component in the model. However, its details are glossed over in Section 3.2 lines 137--146. Please consider elaborating on this part to emphasize the novelty.

Questions

1. details of the CAQG should be put in Section 3 since it is a key component of the proposed method. 2. Table 3 shows that directly adding TPV slightly increases mIoU while hurting the IoU. This is interesting and can the authors provide their thoughts and analysis? 2. In Table 3, LB-only, DF, and adding each of the TPV planes are ablated. Is it possible to also ablate the TPV branch only? Conceptually this is the only missing setting.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have discussed the limitations in their appendix.

Summary

This manuscript studies the problem of road scene semantic scene completion from RGB images. The architecture and benchmarking frameworks follow a widely accepted literature. The innovation proposed here is better queries that are informed of the geometry and semantics of the input scene, a cross-attention variant that leverages rich information from different depth planes in the cost volume, and better information fusion from different represetnations. The whole architecture achieves state-of-the-art performance on public benchmarks.

Strengths

(1+) The paper aims to improve the query quality and solve the deep ambiguity and context-independence problems, which are important and motivated for the SSC task. (2+) The paper proposes a novel method, the CGFormer, which introduce a context aware query generator to capture context -dependent queries and a novel Depth Net utilizing stereo depth and monocular depth for effective refinement. (3+) Experiments on SemanticKITTI (table 1.) and SSCBench-KITTI360 (table 2.) show CGFormer outperforms prior methods. The motivation and the proposed components are validated by detailed ablation studies. (Table 3.,4.,5.) (4+) The paper is basically clearly written and model architecture is easy to follow.

Weaknesses

(1-) The paper notes that context-dependent query tends to aggregate information from the points within the region of interest, Ablation on the the number of the cross-attention and self-attention layers should be provided. (2-) In alignment with previous methods, it would be better to demonstrate the performance when using only a monocular image as input (See Table 3. in VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion). (3-) The visualization (figure 4. and figure 5.) shows some common scenes (It would be even better if the corresponding image views could be added). Could you show more performance improvements for objects like the bicycle in Figure 3 (due to capturing the regions of interest)?

Questions

My questions are incorporated into the weakness section.

Rating

6

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The manuscript is limited in scholarship and can be improved by incorporating semantic scene completion references like [A] and [B]. [A] Efficient semantic scene completion network with spatial group convolution, ECCV 2018 [B] Lode: Locally conditioned eikonal implicit scene completion from sparse lidar, ICRA 2023

Summary

This paper proposes a state-of-the-art Semantic Scene Completion method called CGFormer. It introduces a Context and Geometry Aware Voxel Transformer that dynamically generates queries tailored to individual input images, addressing depth ambiguity through a 3D deformable cross-attention mechanism. The network leverages a multi-scale 3D representation by combining voxel and tri-perspective view to enhance both local semantic and global geometric information. State-of-the-art results are achieved in key benchmark tests.

Strengths

1. This paper focuses on limitations of existing methods that utilizes a combination of voxel and tri-perspective view representations to capture both local details and global structures. 2. The idea that different input images have unique contextual features is very interesting. 3. The experiments are adequate with complete qualitative analysis, ablation experiments and visualization results. 4. In the SemanticKITTI test, the proposed method improves most of the metrics.

Weaknesses

1. In the SSCBench-KITTI360 test set, CGFormer does not improve much compared to Symphonie. Symphonie has almost half of the metrics higher than CGFormer, and Symphonie seems has fewer parameters. 2. Context and Geometry Aware Voxel Transformer seems to be a redesign of Symphonie and adds Deformable Self-Attention after Deformable Cross-Attention, which should be added with more explanation as to why it's done. 3. Context net is not described in the main text or in the supplementary material. 4. The table font is too small to read.

Questions

Refer to the weaknesses.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

The limitations of this paper are primarily in the accuracy of certain categories (e.g., pedestrians and bicyclists), which suggests that there is room for improvement in these areas, as outlined by the authors in Section 5.

Thanks

I'd like to thank the authors for the thoughtful rebuttal. I am glad to see the performance using only monocular image, which demonstrates that CGformer still performs best with a monocular image as input. It dispelled my doubts that there might be an unfair comparison with other methods. Futhermore, Fig. R1 shows two visualization examples or objects with finer details that are convincing. Based on the authors' responses and the comments of other reviewers I am willing to change my rating to weak accept.

Official Comment by Authors

Dear reviewers, We sincerely appreciate the dedication and constructive suggestions you’ve provided for our manuscript. We have made efforts to address each of your comments, including conducting additional experiments and providing further explanations of our proposed method. Given the importance of the discussion between authors and reviewers, we kindly ask for your timely review of our response. We genuinely appreciate your feedback and look forward to any further discussions or suggestions you may have. Best, Authors of Paper ID 3532