Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehension

Summary

Aiming for the WRIS task, the work proposes a novel framework to leverage target-related textual cues from the description for progressively localizing the target object. The authors use a Large Language Model (LLM) to decompose the input text description into short phrases which are taken as target-related cues and fed into a Conditional Referring module. Besides, Region-aware Shrinking (RaS) loss and Instance-aware Disambiguation (IaD) loss are proposed to facilitate fine-grained cross-modal alignment.

Strengths

1. The motivation is clear and easy to understand. The idea of leveraging target-related textual cues decomposed by LLM for progressive localization makes sense. 2. This work develops multiple consecutive Conditional Referring Modules. RaS and IaD two loss objectives are seamlessly integrated into the framework. 3. It is novel to implement RaS and IaD two loss objectives by harnessing the capabilities of segmentation foundation models. The SAM[1] does not include strong semantic prior knowledge like GroundingDINO and Grounded-SAM[2], and it is suitable for the weakly-supervised setting. 4. The work shows outstanding object localization ability and outperforms counterparts on three common benchmarks. Especially after refined by the SAM, the results are promising. [1] Segment anything ICCV2023 [2] Grounding dino: Marrying dino with grounded pre-training for open-set object detection

Weaknesses

1. In table 1, only the results refined by SAM[1] are reported. Considering that FreeSOLO [2] is an unsupervised segmentation model, I am curious about the results refined by FreeSOLO. 2. This work introduces multiple stages for progressive localization. It may increase the training and inference time of the model. The time comparison of model training and inference may be needed. [1] Segment anything ICCV2023 [2] Freesolo: Learning to segment objects without annotations CVPR2022

Questions

1. The mask proposals by SAM are often part of the whole instance. It will affect the refined results. Does this work do anything special to alleviate the problem? The author can give more details about this issue. 2. Table 1 shows the results of the pseudo labels obtained at the wealy-supervised training stage. Can the authors give the results after pseudo labels based on supervised training?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

NaN

Summary

This paper proposes the Progressive Comprehension Network (PCNet) for the WRIS task. this model achieves visual localization by progressively incorporating target-related textual cues for visual-linguistic alignment. Although experimental results have demonstrated the effectiveness of this paper, several aspects of the paper lack clarity.

Strengths

1. The corresponding experiments validate the effectiveness of the proposed method on three popular benchmarks. the proposed method outperforms existing methods. 2. The two loss functions introduced in this paper effectively enhance the accuracy of the response maps.

Weaknesses

1. The Conditional Referring Module (CRM) is implemented through multiple cross-attentions, which is a relatively common approach, lacking novelty. 2. What is the general idea of classification loss mentioned in the paper? Please explain the fundamental research insights of this work. 3. In the Region-aware Shrinking (RaS) loss, what does the ⊙ symbol represent in Equation 5? Please clarify its meaning. 4. This article only conducted ablation experiments on the RefCOCOg dataset, so the ablation experiments are insufficient. It is recommended to validate on multiple datasets (such as RefCOCO+, RefCOCO) to comprehensively evaluate the effectiveness of our method. 5. The order of the tables is somewhat confusing, with Table 9 preceding Table 6. It is recommended to reorder the tables to make them more in line with logical sequence or reading habits. 6. Which datasets were the ablation experiments in Tables 7-9 conducted on? Please specify.

Questions

None

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

See the weakness

Summary

Inspired by human's step-by-step cognitive process for localizing a target object in an image, the paper proposes Progressive Comprehension Network (PCNet) for the task of weakly-supervised referring image segmentation (WRIS) where text is the only supervision signal. PCNet first decomposes a long, complex text description into multiple target-related cues using a large language model (LLM), and the specialized designed module, Conditional Referring Module (CRM), progressively refines the text-to-image response map using the generated text cues in multiple stages. Region-aware Shrinking (RaS) is introduced for constraining the latter stage to have a more compact, target-related response map, and Instance-aware Disambiguation (IaD) loss is proposed for differentiating between the target object and the other objects in the input image. The experiment results show the superiority of the proposed method.

Strengths

- The paper provides extensive experiment results proving the validity of the proposed method, PCNet, and also ablation studies showing the contributions of main components of the model. - Figures help readers understand the proposed method.

Weaknesses

- The proposed method uses the contrastive loss of TRIS, but not generates independent response maps from a positive cue and L negative cues. In Equation (3) and (4), it integrates all the text cue features and generates a single response map R_n that contains information from both positive and negative cues. This does not make sense. How did the authors differentiate between positive and negative cues when computing the classification loss L_Cls? (This is why I rated "poor" for presentation). - IaD, the loss to differentiate between the target object and other objects in the same image is similar to the calibration loss used in TRIS; the idea is the same while the calibration loss uses CLIP image-text similarity for differentiation. The authors should have discussed about this loss in the paper, and also done an ablation study comparing the two losses. - The PCNet uses the mask proposals generated by a pre-defined segmentation mask generator such as FreeSOLO and SAM. We cannot ignore the possibility of knowledge being distilled from the mask generator into PCNet, especially through RaS and IaD losses, main contributions of the paper. More specifically, we cannot be sure that the performance gains from using RaS and IaD do not come from the mask generator's knowledge.

Questions

Please rebut the above weaknesses. I also have minor questions: - In Figure 2, the arrow comes from Q_{n+1} to Cls. How is Q_{n+t} used for computing the classification loss? - How did the authors sample L negative textual cues? one cue from each of the examples in the same mini-batch? - I am curious about the rationale behind Equations (2) and (3). Why did the authors use residual connection only after the mlp layer, not also after the self-attention layer like in Transformer? - Does the order in which text cues are used affect the performance of the model? E.g., is there a performance difference between feeding into the CRM module in the order of q_0, q_1, q_2 and in the order of q_0, q_2, q_1?

Rating

5

Confidence

3

Soundness

2

Presentation

1

Contribution

2

Limitations

The paper addressed the limitations of the work.

Summary

This paper proposes a Progressive Comprehension Network for weakly-supervised referring image segmentation, which mimics the human process of progressive understanding by breaking down sentences into segments and gradually narrowing down the target range. The main contributions include: A multi-stage Conditional Referring Module to progressively comprehend text cues. A Region-aware Shrinking Loss to constrain the target region to gradually shrink. An Instance-aware Disambiguation Loss to eliminate overlap between different instances.

Strengths

1. Presentation: The majority of the paper is well-written and easy to follow, providing clear explanations of the key concepts. 2. Motivation: The motivation is reasonable and convincing. By mimicking the human process of understanding concepts from coarse to fine, the method progressively refines the final mask. Directly requiring the mask to continuously shrink could lead to trivial solutions, and two losses are introduced to further support this pipeline. The proposed method aligns well with the motivation. 3. Experiments: Comprehensive ablation experiments demonstrate the effectiveness of the proposed modules

Weaknesses

1. The concept of mimicking the human process of understanding concepts from coarse to fine-grained, along with multi-stage refinement for different parts of sentences, has been explored in prior REC research [1]. This approach involves parsing sentences into multiple parts and conducting multi-stage refinement. Applying REC's ideas [1] to WRIS can also be considered a contribution, but it should be thoroughly discussed in comparison to [1], highlighting any differences, and cited appropriately. 2. There is a lack of discussion and citation of recent WRIS work, such as [2], in related work and the main text. It is necessary to discuss and cite the latest WRIS work in both the related work section and the main table. 3. Section 3.4 is not entirely understandable. I suggest the authors re-describe this section to make it clearer and easier to understand. 4. The top of Fig.2 is not clear enough, especially the subscripts a and d, I suggest the author re-design this part. [1] Yang S, Li G, Yu Y. Dynamic graph attention for referring expression comprehension[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 4644-4653. [2] Dai Q, Yang S. Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13711-13722.

Questions

1. Why is the range of Equation 6 set to [0,1]? Can the combination of other proposals (denoted as 𝑚_𝑏 ) be simply approximated by the complement of the foreground, since the proposals segmented by SAM nearly cover the entire image? In this case, if IoU(𝑅_𝑛,𝑚_𝑓) < IoU(𝑅_𝑛,𝑚_𝑏), wouldn't Equation 6 be greater than 1? 2. Does 𝑚_𝑓 change at different stages in Equation 7, for instance, when different proposals are selected at different stages? What issues could this cause? In Equation 7, are 𝑛 and 𝑛+1 reversed? According to Section 3.3, the ambiguity score should be lower at later stages. 3. Section 3.1 mentions that there are five phrases for each sentence, and Section 3.2 states that each stage uses one text cue, so there should be five stages. Why do the implementation details mention only three stages? 4. In Section 3.4, how do you sample extra 𝑁_𝑑 texts? Are they randomly sampled from multiple text descriptions corresponding to the same image? How do you ensure that the sampled text descriptions correspond to different target referents? Because in refcoco series dataset, one referent may have multiple text expressions. 5. In the bottom part of Tab.1, are the numbers all obtained by using a single peak point as the prompt for SAM? Did you try using the response map as the prompt for SAM?

Rating

7

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes.