Thank you very much for acknowledging our rebuttal and paper.
Below, we provide answers to your newly raised comments:
## Q1
Thanks for your question. Below we summarize the challenges lying in surgical images:
- In surgical images, it is common that instruments may exhibit significant similarity, necessitating the model that can distinguish their subtle differences for accurate segmentation.
- As surgeries progress, surgical instruments might cut, suture, or otherwise manipulate tissues, altering their shapes. This can lead to tissues obscuring the instruments and potentially making some parts of the tissues even resemble the instruments.
- In surgical settings, the use of diminutive endoscopic devices with small lenses inherently constrains the imaging quality due to hardware limitations.
- The areas into which the endoscope is inserted, like the gastrointestinal tract, are continuously moving, which can add complexities to the segmentation of surgical instruments.
Given above, on one hand, when comparing with those objects in natural scenes in RGB images which often have distinct and rigid boundaries, the continuous morphological changes of instruments and tissues during surgeries make instrument segmentation easily affected by tissue occlusions and variations in illumination, etc.
On the other hand, when dealing with radiological images, different segmentation challenges emerge: the presence of various imaging modalities, intrinsic noise, limited contrast, and the potential for artefacts collectively introduce complexity to the precise segmentation in radiological images.
Overall, we firmly believe that research on surgical instrument segmentation is both challenging and meaningful.
Our methods have achieved significant improvements over text-promptable methods (e.g., CRIS, CLIPSeg) developed on natural images.
We follow a problem-driven approach, in response to the aforementioned challenges in surgical images, we introduce the mixture of prompts module to address instrument similarity by integrating detailed text descriptions; moreover, the hard instrument area reinforcement module further amplifies the model's precise segmentation performance, especially in tricky regions where distinctions between instruments and tissues become blurred.
Finally, although our method is designed for surgical instrument segmentation, it should also have the merits in natural image segmentation, especially when encountering challenges similar to those in surgical contexts.
For instance, when undertaking fine-grained segmentation or in the presence of heavy occlusions. Therefore, our method holds potential for broader applications in computer vision tasks, and NeurIPS would be an excellent venue to showcase our work.
## Q2
Thanks for this question.
First, we emphasize that our method is not merely a synergy of a few components.
For the text-promptable pipeline itself, there are inherent novelties instead of a simple adaptation from existing approaches. For instance, we devise a multi-scale fusion scheme for image encoder and the mask decoder integrated both attention-based and convolution-based prompting schemes to facilitate text features in guiding visual features for segmentation prediction (see Tab. 3 & 4 in our paper for their improvements).
Building upon the text-promptable method, we introduce the mixture of prompts, substantially enhancing model performance, especially in the prediction of novel instruments (see Tab. 5 in our paper).
Additionally, to overcome classification inaccuracies during segmentation, we incorporate the hard instrument area reinforcement module (see Tab. 6 in our paper).
All these modules, as mentioned in the above answer, follow a problem-driven paradigm in the surgical instrument segmentation domain.
Next, although our modules have been primarily validated in surgical instrument segmentation, we believe they can offer insights or potential benefits for segmentation tasks in other domains (e.g., RGB and radiological images), especially when facing challenges akin to those found in our surgical images.
For example, in tasks such as fine-grained segmentation of natural images, our mixture of prompts module could be effectively employed and for scenarios with heavy occlusions, our hard instrument area reinforcement module could be particularly suited.
Our approach exhibits strong adaptability, as demonstrated on the CholecSeg8k dataset where it not only segments instruments but also various tissues.
Experimentally, our method surpasses the current SOTA in performance (see results in the global response).
In summary, our method was borne out of a problem-driven necessity.
To address these challenges, we introduce various modules which, when synergized, significantly enhance performance.
We believe our approach harbors immense potential for broader applications within the computer vision domain, especially for methods centered on text prompts.