CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation

Summary

This paper proposes a prompt-based deep model for organ and tumor segmentation. The authors leverage two types of prompts—cropped target volumes and textual descriptions—to perform the segmentation. The proposed model demonstrates good performance across three datasets for the segmentation task.

Strengths

1. Point out several critical challenges in medical image segmentation, such as the long-tailed nature, variations in shape, size, and density distribution, and blurring boundaries. 2. Utilize both textual and visual prompts for medical image segmentation, achieving good performance.

Weaknesses

1. The motivation is not convincing. Although the authors highlight several critical issues, it remains unclear how the two types of prompts address or refine them. For instance, textual descriptions lack detailed and quantitative measurements. How can they denote invading target boundaries? How can the effectiveness of terms like 'greater' or 'deeper' be evaluated? 2. Several experimental settings are not standard. For example, in the Anatomical Prompt Encoder, the input sizes are the same. How does this account for small or large targets? 3. Figure 2 is unclear. What is the output? During inference, do both prompt inputs need to be used? 4. For comparisons, the authors trained the model on 10 public datasets. However, only ZePT, nnUNet, and Swin UNETR were trained in the same setting. Therefore, the conclusions drawn are not reasonable. Besides, in Table 1, CT-SAM3D achieves the best performance. 5. The overall organization is unsatisfactory. First, as mentioned, the motivations could be improved to better align with the technical designs. Second, despite the better performance, no new insights are provided.

Questions

See the above weaknesses.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have analyzed the limitations of this paper.

Summary

The paper "CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation" introduces a novel dual-prompt schema that leverages both anatomical and textual prompts for medical image segmentation. The proposed CAT model coordinates 3D anatomical prompts with enriched textual prompts to improve segmentation accuracy for various organs and tumors. Key contributions include the development of the ShareRefiner and PromptRefer modules to refine and integrate these prompts, resulting in superior performance on multiple public CT datasets and an in-house dataset. The approach demonstrates enhanced generalization capabilities and robustness in complex medical imaging scenarios.

Strengths

1. The paper introduces a novel approach by combining anatomical and textual prompts, leveraging the strengths of both to enhance medical image segmentation. 2. The development of the ShareRefiner and PromptRefer modules demonstrates a sophisticated method for refining and integrating multimodal prompts, leading to improved segmentation accuracy. 3. The paper provides a thorough experimental analysis, including ablation studies and qualitative comparisons, to substantiate the effectiveness of the proposed methods. 4. The work focuses on both organ and tumor segmentation, which is commendable and proves the model's performance overall.

Weaknesses

1. The motivation for using textual descriptions was to provide the model with specific knowledge about each image, including details such as cancer stages and density. However, the authors use only fixed generic textual information generated by a language model in this work. This approach does not fully capture the intended motivation. If the textual information included specific details for each sample, demonstrating how changes in staging affect segmentation performance, it would better support their claim about the utility of textual information. 2. I am not convinced of CAT's superiority in the comparative results. For example, CT-SAM3D seems to perform better in organ segmentation in most cases, and where CAT excels, the improvement is minimal compared to other text-based segmentation models. Additionally, why aren’t SOTA segmentation models like UNet, nn-UNet, and Swin UNETR, etc included for organ segmentation, and vice versa for tumor segmentation? Only comparing with SAM-based models for organ segmentation is insufficient, especially since SAM is known to have poor performance in medical image segmentation. 3. The authors claim that CAT performs better on the in-house dataset, especially for tumors at different stages. However, the textual prompts used by CAT are very generic and do not include the cancer stage information. Thus, the reasons for CAT's superior performance are unclear. 4. The paper lacks sufficient novelty, as it combines already existing methods, such as integrating text and visual prompts, where text is intended to provide rich semantic information. However, it does not adequately explain how the text contributes to performance improvement. This work appears to be an incremental extension of ZePT, with only the creation of visual prompts distinguishing it from similar efforts. 5. Exactly how contrastive alignment works in PromptRefer isn't clear. This should be further enhanced. 6. Many of the techniques are described in terms of their usage, but the underlying motivation for their utilization is not clearly articulated. For instance, the rationale for choosing hard assignments in ShareRefiner is not explained. If the intention is to follow ZePT, this choice is questionable because ZePT uses hard assignments to distinguish between healthy organs and tumors in feature space. The motivation behind these decisions is lacking. 7. Figure 2 (c) should have more explanations. It is not clear right now. Overall, the authors should place greater emphasis on the motivations behind the chosen techniques. Currently, it appears they are following these methods simply because they work, which is not a robust standard. Clear justification for each technique would strengthen the paper.

Questions

1. In equation 3, why is a Linear function used to convert EAE_AEA​ to QAQ_AQA​? There is no explanation provided for this choice. I assume it is to reduce the dimensions of the embedding space, but clarification is needed. 2. There should be more explanations about learnable segmentation queries. For example, how are they initialized? What is their purpose? Beyond their use in architecture, a high-level explanation of how they help predict masks would aid readers in understanding and relating to the motivation behind these techniques. 3. Why do some models present HD95 values while others do not? The current reasoning isn't sufficient. 4. Figure 1, some modules have the "snowflake" icons, which the authors don't explain why. Assuming these show that these are frozen models that have not been trained, do they mean any modules that don't have these are all trained?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Please refer to the Weaknesses section.

Summary

This paper proposed CAT, a promptable segmentation model that utilizes the strengths of both visual and textual prompts without human interaction, aiming at a fully automatic model for medical professionals. Extensive experiments demonstrate the benefits of coordinating anatomical prompts and textual within one model. CAT achieves state-of-the-art performance on multiple segmentation tasks and has generalization capability to diverse tumor types.

Strengths

1. The idea is OK. CAT combines text and visual prompts, which could be needed in clinical scenarios. 2. The experiment CAT proves that combining visual and textual prompts is essential. 3. The experiment shows that CAT can deal with challenging small region segmentation and tumor segmentation. 4. CAT applied domain knowledge generated from GPT4, which is innovative. And one board-certified physician is recruited to check the text prompts.

Weaknesses

1. It is not clear how those comparison methods were trained. Were they also trained on the same 10 datasets as CAT? For example, how was nnUNet trained and tested (since the number output channel of nnUNet is fixed)? 2. The writing is a bit confusing. For example, what is the backbone of ShareRefiner and PromptRefer? The similarity matrices need some mathematical illustration. 3. It is confusing that Table 3 has the same setting in the last two rows. Is that a typo? 4. The paper did not mention what pre-trained model is used.

Questions

1. What do the textual prompt and visual prompt look like? It is suggested that an example be given in the Supplementary. If CAM is only trained on one dataset, will it surpass nnUNet and other comparisons? 2. How large are CAT and those comparison models? It is suggested to report the number of parameters. 3. It is suggested to give a concrete example that how visual and text prompt improve the segmentation.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper proposed in the introduction that medical data is challenging because of the long-trailed problem but did not illustrate how CAT helps solve this problem.

Thank you for the reply

Thanks for the response and it solves most of my concerns. Using both text and visual prompts is a kind of combination and it is not surprising it could achieve better performance. Even though the technical novelty is not strong, the analysis of clinical usage is reasonable and I will raise my score to borderline acceptance. This is the final comment.

Nice rebuttal

Thanks for the excellent rebuttal. My confusion is solved.