Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention

Summary

This paper improves upon the previous work, M3DRef-CLIP, through three key modifications: First, the authors incorporate an additional proposal probability prediction branch followed by a NMS operator to filter out low-confidence and redundant object proposals. Second, they learn camera pose residuals to dynamically render the object proposals from multiple views and extract multi-view CLIP features as the 2D feature. Third, they incorporate spatial relations weighted by learnable weights from visual and text features. The proposed method outperforms M3DRef-CLIP in several settings and datasets.

Strengths

• The proposed method demonstrates state-of-the-art performance in both multi-object and single-object 3D grounding tasks. • The ablation study clearly shows the performance improvement brought by each modification.

Weaknesses

1. Despite the performance improvement, the time and memory complexity of the proposed methods seem substantial, particularly the multi-view rendering step. It would be beneficial to include a complexity comparison with baseline methods. 2. In M3DRef-CLIP, both F1@0.25 and F1@0.5 are reported. Including the same evaluation metrics would be advantageous, as this paper is an improvement over M3DRef-CLIP. 3. The paper would benefit from more detailed motivations and explanations. For example, what motivates the design of camera pose offset prediction (Eq. 5) in this manner? Why is the designed LISA module better than the language-conditioned spatial self-attention in ViL3DRel [9]? 4. What would the performance be if the camera pose offset were removed while keeping the average of multi-view CLIP features? 5. Is there any supervision on the predicted proposal probability (Eq. 1)? How does the threshold in Eq. 2 affect the performance?

Questions

Please refer to the questions in the weaknesses section. I would consider increasing the rating if the authors can address my questions and concerns properly.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors have discussed some limitations and indicated that there are no potential societal impacts of the proposed method.

Summary

This paper introduces a novel two-stage approach for multi-object 3D grounding from a point cloud based on a given query phrase. The first stage of D-LISA uses a dynamic proposal module that selects a variable number of box proposals instead of a fixed maximum, addressing the issue of determining the optimal number of proposals in the scene. D-LISA incorporates a dynamic multi-view renderer module that optimizes viewing angles for each proposal based on the specific scene, moving away from the fixed camera poses used in prior work. The second stage introduces a module that reasons over the spatial relationships among objects, guided by the textual description, improving the contextual understanding of the model. Experiments conducted on the Multi3DRefer benchmark demonstrate that D-LISA outperforms the state-of-the-art methods by a significant 12.8% absolute increase in multi-object 3D grounding performance. It also shows competitive results in single-object 3D grounding tasks.

Strengths

1. The method shows a substantial improvement over existing baselines, indicating effective handling of complex 3D scenes with multiple objects. 2. The dynamic module can reduce the proposals effectively.

Weaknesses

1. The novelty of the dynamic vision module is limited. In fact, I think the statement of dynamic vision module is kind of exaggeration. From my perspective, the authors just calculate the probability of each box candidate to remove the low-probability boxes and use NMS to filter overlapping boxes. Both the two operations has nothing to do with a novel dynamic vision module. 2. Also, the novelty of the LISA module is also doubtful. The authors use spatial scores to modulate the self-attention and spatial distance matrix. Although this operation is innovative to some extent, it is not enough for acceptance. It is more like a trick rather than a novelty.

Questions

Could the authors re-organize the novelty according to the weakness to highlight the key points more clearly?

Rating

3

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are adequately addressed.

Summary

This paper proposes D-LISA, a two-stage framework for multi-object 3D grounding. D-LISA consists of three novel components that make the method effective, namely a dynamic box proposal module, a dynamic multi-view renderer and a language informed spatial fusion module. Comprehensive Experiments are done on Multi3DRefer, ScanRefer and Nr3D datasets to prove the superiority of D-LISA over previous methods. Experimental results show that D-LISA not only outperforms previous methods on multi-object grounding, but also achieves comparable results on single-object grounding.

Strengths

1. The paper is well-written and easy to understand and the figures help with the illustration of the overall idea. 2. Comprehensive experiments and ablation studies are conducted. 3. Implementation and evaluation details are clearly provided, making this work easy to follow.

Weaknesses

1. In Table A2, the proposed method falls behind state-of-the-art method by a large margin, which somehow is not competitive. 2. In the ablation study (Table 4), I'm curious what would happen if putting two modules together, e.g. abandon LIS and use DBP and DMR, since the improvements of the full model is not significant compared to the model with one component. 3. The analysis of dynamic pose distribution is lacking. Readers can't tell from Figure 4 (b) if the dynamic pose rendered results are better. Fixed pose results seem to be looking at the object from a more informative view. 4. How much effect NMS has on the proposed method seems to be unclear, the paper only conducted experiments by adding NMS to a baseline method, resulting in a much better performance.

Questions

Please see the weakness part.

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

The paper introduces D-LISA, a two-stage approach for multi-object 3D grounding that incorporates three innovative modules. First, a dynamic vision module generates variable and learnable box proposals. Second, a dynamic multi-view renderer extracts features from optimized viewing angles. Third, a language-informed spatial attention module reasons over the proposals to output final predictions. Empirically, D-LISA outperforms state-of-the-art methods by 12.8% in multi-object 3D grounding and is competitive in single-object 3D grounding.

Strengths

1. This paper is generally well-written and clearly stated. 2. The key idea lies in enhancing visual understanding and human-computer interaction by improving the ability to locate objects in 3D scenes based on natural language descriptions. 3. Experiments demonstrate that D-LISA outperforms the existing state-of-the-art, indicating the effectiveness of the proposed innovations.

Weaknesses

1. It is not clear what core issue this paper is targeting in the task of dynamic multi-object 3D grounding. I suggest the authors include this in the abstract and introduction section. To me, the three improvements over M3DRef-CLIP are very engineering. 2. The paper could benefit from a more detailed ablation study that isolates the impact of each dynamic component (the dynamic proposal module, the dynamic multi-view renderer, and the LISA module) on different types of scenes and queries to better understand their individual contributions. 3. The paper does not provide details on the computational cost of the dynamic components, such as the multi-view rendering and the language-informed spatial attention module.

Questions

See weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Response to rebuttal

Thank you to the authors for providing the rebuttal. However, most of my concerns remain inadequately addressed. Here are some additional comments: 1. I am puzzled by the complexity comparison between the baseline renderer and the dynamic multi-view renderer. How is it that the multi-view renderer and CLIP feature extraction have almost no impact on memory and time? 2. The LISA method also computes the pairwise L2 distance between box centers, which is a feature in Vil3DRef. Shouldn't this distance calculation also be affected by noisy detected proposals? 3. Rows 1 and 2 of Table 4 do not show the performance when removing the camera pose offset while keeping the average of multi-view CLIP features. 4. The results in Table R7 do not match the performance reported in Table 4, which is quite confusing. 5. I agree with Reviewer su8L and WVQA that the three improvements over M3DRef-CLIP proposed in this paper are more engineering-focused and show limited novelty. Hence, I would like to maintain my initial rating.