OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling

Summary

The authors propose a unified one-tower referring framework that introduces the MRefM paradigm to capture the referential relationships between vision and text. They demonstrate the effectiveness and generality of MRefM across three settings on REC, PG, and RES tasks, consistently achieving good results.

Strengths

The proposed method is structurally simpler than previous methods and achieves higher grounding and RES performance. The ablation experiments are thorough, validating the effectiveness of the two Ref mask modeling strategies.

Weaknesses

Adapting BEiT, a robust general-purpose multimodal model, to downstream grounding tasks and achieving performance improvement does not seem to be a particularly interesting finding. Additionally, it is puzzling why introducing two types of relation scores in MIM and MLM would enhance the model's referring performance. Especially regarding the four types of masks in the Visual target-relation score, their effective mechanism is not intuitive and lacks explanation. Why did the authors decide to introduce referring information by predicting these four scores? Are there any reference works? In Table 3, comparisons with some related works, such as UNINEXT [1] and HIPIE [2], are missing. [1] Universal instance perception as object discovery and retrieval. CVPR 2023. [2] Hierarchical Open-vocabulary Universal Image Segmentation. NeurIPS 2023.

Questions

See weaknesses.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper proposes a Mask Referring Modeling (MRefM) paragram and a unified and extremely concise grounding and referring segmentation framework named UniRef that no longer requires the fusion or interaction of the Transformer structure and the special grounding tokens. A masked referring modeling (MRefM) is proposed to model the referential relationship, which encompasses referring-aware mask image modeling and referring-aware mask language modeling.

Strengths

A mask-referring modeling paragram is proposed to model the referential relation between visual and language effectively. They also propose a one-tower framework for grounding and referring segmentation in a unified modality-shared feature space. Experiments demonstrate the effectiveness of this method and its components.

Weaknesses

1. Lack of discussion of related work [a]. I suggest discussing the difference between this paper's referring-aware mask language modeling and masked contrastive learning in [a]. 2. Lack of computation costs analysis with other methods, including parameters/FLOPs/speed. [a] VLT: Vision-Language Transformer and Query Generation for Referring Segmentation. In TPAMI 2023.

Questions

See weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Summary

This manuscript proposes UniRef, a framework aimed at unifying visual and linguistic feature spaces for referring expression comprehension and segmentation. The key presented is the Masked Referring Modeling (MRefM) paradigm, which includes referring-aware MIM and MLM. This approach seeks to streamline the architecture by eliminating the need for separate modality-specific encoders and complex interaction modules, achieving sota performance on several datasets.

Strengths

1. The introduction of the MRefM paradigm effectively captures the referential relationship between visual and linguistic features, contributing to the robustness and accuracy of the model. 2. The integration of visual and linguistic feature spaces into a unified framework simplifies the model architecture and potentially improves computational efficiency. 3. The authors provide extensive experimental results across five datasets, demonstrating the effectiveness of the proposed approach in outperforming existing methods.

Weaknesses

1. The technical contribution of the proposed method appears to be insufficient, as it primarily adapts traditional Masked Autoencoders into a 'Referring MAE', despite the rich experimentation presented. 2. Section 3.2 is intricate and challenging to follow. I recommend a thorough proofreading and revision of this section to enhance its clarity and readability. 3. Could you explain the rationale behind designing the system to encompass four masks: x-, y-, w-, and h-masks? 4. The manuscript asserts that the approach is lightweight, raising questions about its real-time applicability. It would be helpful if the authors detailed the computational demands and processing speed of the method in practical scenarios. 5. (Minor) While the manuscript has discussed the limitations of the proposed approach, I am interested in understanding how MrefM performs in more general contexts. For instance, evaluating the backbone linear probing (or its integration with LLMs) could demonstratebroader applicability. This should certainly be considered for future work.

Questions

See weakness.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

None.