Summary
The paper describes the utilization of SegVol, a deep learning model, to segment any organ/tumor/lesion on 3D CT data.
**objectives**
Create a universal segmentation model that can segment with :
- any type of labeling
- good performances on complex tasks
- low computational cost (i.e., no sliding window)
**contributions**
The authors contribute to the state of the art in 3 ways :
- first model for segmenting over 200 anatomical categories
- support spatial (points/bbox) and semantic prompt (text describing each category)
- introduction of a zoom-out-zoom-in mechanism to improve the model's efficiency and accuracy
**datasets**
- 25 open-source segmentation CT datasets + background filtering
- generation of pseudo-labels with Felzenswalb-Huttenlocher to remove spurious correlation between the datasets
**architecture**
The whole architectures is divided into four blocks:
- image encoder: 3D ViT pretrained using SimMIM on 96K unlabeled CTs + fine-tuning on 6K labeled CTs
- text encoder: the CLIP model is used to generate a text embedding from the text prompt. The text prompt only consists in the name of the anatomical region to be segmented.
- prompt encoder: text embedding and spatial prompt (bbox+points) are given to generate the prompt embedding.
- mask decoder: from the prompt embedding, the segmentation is generated.
**training algorithm**
- the encoder is pretrained using SimMIM
- the model is fine-tuned using an algorithm described in appendix B. For each epoch, an iteration of training is made using the ground truth, and a second one using the pseudo masks.
**results**
Experimental setup is given, as well as a description of the three external datasets used for the evaluation.
- comparison with SAM-like methods : Table 2 shows the superiority of the method with Dice score > 0.7 on each task. Figure 2 helps to visualize the dice score distribution for each model, on each task. An additional comparison to standard fully supervised segmentation models such as nnUNet, SwinUNetR is provided. Superiority of SegVol is also shown compared to this model on Figure 8 and Table 6.
- ablation studies
- Zoom-out-zoom-in mechanism : Table 3 shows the interest of the method, improving performances and reducing duration compared to sliding window
- Scaling up training data : Fig 3.(a) shows that the model performances improve as the number of training data increases.
- Prompt impact : Fig 3 (a and b) shows that point prompt is inferior to text prompt, and best prompt combination is bbox+text
- Case studies
- Disambiguation via semantic-prompt: Figure 4 shows that when the spatial prompt is not sufficient to describe the segmenting task, the text prompt adds the complementary information to locate the good anatomical region.
- spatial-prompt impact: Figure 5 shows how the points prompt are related to the results. When choosing two points, the category becomes clear, whereas with one point can identify two categories.
**discussion**
- Scalability : the authors believe that the model have a strong model scalability.
- Generalizability : Generalization of SegVol to MRI is discussed, with an example given in appendix C showing a strong dice score of 80%.
- Limitations : only limitation is that text prompt only consider a category, and not a full sentence with logical reasoning.
- Broader impact : the authors believe that their method is universal and doesn't have any negative impact.
Strengths
- **originality** : the concept of SegVol is not original itself. However, the addition of the zoom-in-zoom-out principle and the combination of bbox + text + points prompts is new in this kind of models.
- **quality** : the work is done rigorously, respecting a scientific reasoning all along the article.
- **clarity** : the paper lacks clarity as there are two parts difficult to understand.
- 1) The description of the training algorithm is not made in the text, only the steps are written in appendix B.
- 2) The hyperparameters regarding the model's dimension (number of layer, ViT dimension) is not explicited.
- **significance** : The results are impactful for the community of medical image segmentation, and could have plenty of applications. The property of segmenting multiple organs, from multiple entirely different datasets is remarkable.
Weaknesses
The only weakness regards the clarity of the training algorithm, which is hard to understand even with appendix B and training steps.
This point could be improved by describing the steps in the appendix with a paragraph, or maybe with another diagram that follows the exact same steps.
Questions
**datasets**
l. 83 : Why is Felzenswalb-Huttenlocher (FH) algorithm to generate pseudo masks ?
l. 81 : How is it supposed to remove spurious correlation between datasets ?
**architecture**
l. 96 : The architecture principle is described, but some details are lacking such as the number of layers of each network, the number of heads in the ViT, etc.
l. 133 : why is the computational cost reduced using the zoom-in-zoom-out principle compared to sliding window? clarify this point.
l. 140 : how are prompts generated from the coarse segmentation masks? clarify if the prompts are generated thanks to the prompt encoder as a bbox, or points.
**results**
l. 169 : Could you indicate the training time (SimMIM and finetuning) of the model?
l. 169 : Why did you choose these 3 specific datasets as external test sets?
**discussion**
l. 257 : Why is data salability a good thing ? Why is this part in the discussion? The data scalability of the model is not discussed or compared with other models. If not discussed, move to the conclusion.
Limitations
Only one limitation is addressed, considering the text prompt which cannot be a logical sentence at the moment.
However, the amount of data required to train the model is not discussed. Would a few-shot finetuning method work on small datasets (10 samples)?