Annotation by Clicks: A Point-Supervised Contrastive Variance Method for Medical Semantic Segmentation

Thank you for your careful and valuable advice. **Q1**. Contributions of the method. **Answer**: Firstly, we introduce an experiment aiming to segment multiple organs in medical images under only point annotations—an unexplored endeavour. Secondly, we propose a novel contrastive Mumford-Shah loss function, distinct from existing methods. Our proposed loss function is intentionally crafted rather than directly adapted from existing approaches. Calculated over the pixel-level variance distribution map rather than features, it compels each prediction into statistically plausible spatial regions—a crucial element for addressing point-supervised medical image segmentation tasks. In contrast to the previous Mumford-Shah loss, which primarily enforces similarity in pixel values within regions, our approach enhances the discriminative capacity for separating different categories. The proposed loss amalgamates the strengths of both Mumford-Shah and contrastive paradigms in a previously unexplored manner. Hence, our proposed loss function is both innovative and effective. **Q2**. Annotated point for each organ. **Answer**: We conducted randomly selected annotation experiments on the Synapse dataset (CT) because it presents more challenging categories than the ACDC dataset (MRI), making it a more representative test of the impact of randomness on our proposed PSCV. In response to your suggestion, we also performed random selections on the ACDC dataset, yielding results of 0.841, 0.838, 0.842, and 0.837. These experimental outcomes demonstrate the robustness of our method in producing consistent results based on different random points for both CT and MRI. For the random selection attempts, we adhered to the setting of point annotation in natural scene images, aligning with the operational practices of medical professionals during annotation. As per your suggestion, we will incorporate limit point attempts in the modified version, such as considering maximum and minimum intensities. **Q3**. Split of Synapse dataset. **Answer**: We use the same split as the fully supervised method (e.g. TransUNet or Swin-Unet). We use the training dataset for cross-validation during training and use the resulting parameters to apply to the test dataset. **Q4**. Post-processing step. **Answer**: We employed a post-processing operation commonly used in video object segmentation and image saliency detection. This operation, known for its simplicity, is a standard approach in various applications. The rationale behind applying this post-processing step is our observation of noise in the segmentation results on the Synapse dataset. Conversely, this method had negligible effects when applied to the ACDC dataset, given its simpler nature and the absence of prevalent background noise. We applied this post-processing step to all comparison methods to maintain fairness in our comparisons. Notably, even without this operation, our method achieves a DSC of 0.699 on the Synapse dataset, surpassing the performance of the comparison methods. **Q5**. Overall computational complexity **Answer**: We employ UNet as our backbone, consistent with our comparison methods (except for WSL4MIS, which uses one more decoder than we do). The primary computational load arises from the computation of positive/negative pairs. With a batch size 12 and utilizing a 2080Ti GPU, we calculated the computational overhead to be 3.456 milliseconds. **Q6**. More performances in different scenarios using ACDC MRI data in the future. **Answer**: Thank you for your valuable suggestions. In future studies, we will incorporate additional experiments covering various scenarios on the ACDC dataset.

Thanks for your valuable advice. **Q1**. More analyses of the loss. **Answer**: Our PSCV method is inspired by level set methods, treating image segmentation as a series of energy minimization steps. At its core is creating an implicit function in a higher-dimensional space, defining contours through the zero-level set. The Level Set Function evolves guided by a partial differential equation derived from the Lagrangian formulation of the active contour model. In the Mumford-Shah level set model context, segmenting an image $I$ entails identifying a parametric contour that partitions the image plane $\Omega \subset \mathbb{R}^2$ into $N$ class-specific, piecewise constant regions. However, existing methods approximate the image using smooth functions within regions, overlooking information discrepancies across various categories. To be specific, for a given input image $I^n$, the prediction $\hat{Y}^n$ is obtained using $f_{encoder}$ and $f_{decoder}$ (Eq.1). Each position in $\hat{Y}^n$ corresponds to a probability vector of K classes. Using Eqs. 5 and 4, we calculate the mean pixel value and derive the variance distribution map $z_k^{n}$ for the k-th organ category, serving as an anchor. Positive samples, represented by another image $I^{m_n}$ with category k in the same batch, have their variance distribution map $z_k^{m_n}$ considered a positive sample, expected to be similar to the anchor ($z_k^{n}$). Conversely, negative samples, involving all batch images except $I^n$, assume dissimilarity to the anchor's variance distribution map ($z_k^{n}$), acting as effective negative samples. **Q2**. Different locations’ pixels of point annotations. **Answer**: Our proposed PSCV demonstrates robustness to point annotations at various random locations. As illustrated in Figure 2 in the Supplementary Material, we individually randomly selected five different point annotations on the Synapse dataset. The experimental results indicate that the DSC value change is insignificant. This outcome serves as empirical evidence for the robustness of our proposed loss function. **Q3**. Comparison methods. **Answer**: All compared methods employ the same point annotation and backbone, ensuring a fair experimental comparison. In Tables 1 and 2 of the main text, GatedCRF is specifically designed for point and scribble annotations, while USTM and WSL4MIS are tailored for scribble annotations. EntMini is designed for semi-supervised segmentation. Additionally, the methods listed in Table 2 of the supplementary material are designed for point annotations. Currently, no existing methods are designed for segmenting multiple organs in medical images under only single-point annotation. Consequently, no directly comparable methods are available to evaluate these two datasets. By contrasting methods designed for point, scribble, and semi-supervised scenarios, our proposed PSCV consistently yields the best results. **Q4**. Failure results. **Answer**: Thanks for your suggestion. We will add failure examples and corresponding discussion in the revised version.

Thank you for your constructive advice. **Q1**. About the Comparisons. **Answer**: Firstly, we compare relevant medical image segmentation methods based on point annotation, as in Table 2 in the Supplementary Material. The experimental results demonstrate the superior performance of our proposed PSCV over all the methods considered, specifically on the Synapse dataset. Secondly, no existing methods are currently designed for segmenting multiple organs in medical images under only single-point annotation. Consequently, no directly comparable methods are available for assessing these two datasets. Thirdly, the scribble-supervised problem is viewed as an extended and simplified version of the point-supervised problem. Therefore, relevant methods developed for point supervision can be directly applied to this study. Lastly, it's worth noting that the reported results in the original paper of WSL4MIS appear higher due to their use of scribble annotations. In contrast, our experiments are conducted based on more challenging point annotations, making our tasks more demanding than their original settings. **Q2**. Sensitivity to hyperparameters. **Answer**: We present detailed analyses of parameter sensitivities in Section 4.5 of the main text and Tables 4 to 7 in the Supplementary Material, specifically focusing on DSC and HD95 values for each category in the Synapse datasets. The experimental findings indicate optimal performance when $\lambda_{cv}=0.3$ and $\tau=0.07$, with a more pronounced decrease in results for larger settings of these parameters. These discussions will be included in the revised version. The results depicted in Figure 3 are derived from the ACDC dataset, characterized by its simplicity with only three categories and a large organ area exhibiting a relatively uniform shape. Notably, the ACDC dataset achieved a DSC value of 0.722 using only pCE. Nevertheless, our proposed method consistently outperforms vanilla MS on ACDC. In contrast, the Synapse dataset, being more complex with eight categories and substantial shape variation among organs, exhibits less variation in organ appearance. Despite fluctuations in our results based on different parameters, our method consistently surpasses vanilla MS on this more intricate Synapse dataset. **Q3**. The novelty of this paper. **Answer**: Our PSCV diverges significantly from existing methods. Our proposed $L_{CV}$ exploits the spatial distribution properties of organs and variance distribution map representations within a contrastive learning framework. Unlike traditional methods that operate on features, $L_{CV}$ is calculated over the pixel-level variance distribution map, ensuring each prediction adheres to statistically plausible spatial regions. This distinction is crucial for addressing point-supervised medical image segmentation tasks. Our novel loss function amalgamates the merits of the Mumford-Shah Loss Function and contrastive paradigms in a unique manner not previously explored. Our PSCV method draws technical inspiration from level set methods, treating image segmentation as a series of energy minimization steps. At its core is creating an implicit function in a higher-dimensional space, characterizing contours through the zero-level set. This function, the Level Set Function, evolves guided by a partial differential equation derived from the Lagrangian formulation of the active contour model. **Q4**. The statements of pixel variations caused by different collection equipment or conditions. **Answer**: Thank you for pointing that out. We wanted to state that “one can effectively mitigate the irrelevant inter-image pixel variations caused by different collection equipment or conditions”. We will revise this sentence in the revised version so that it is more accurately expressed.

Thank you for your suggestions. **Q1**. The reason for multiplying the prediction function output in Eq .4. **Answer**: The prediction function output represents the probability of each pixel belonging to the $k$-th class. A value close to 1 in the prediction function output indicates a higher probability of classifying location $r$ into class $k$, and vice versa. Multiplying the prediction function output in Eq. 4 yields the variance corresponding to the $k$-th class region in the prediction result. This process enables $z_{k}^{n}$ to capture the variance information of a specific class region. Such information is instrumental in constructing contrastive relations based on the distinct class regions predicted in the subsequent contrastive learning paradigm. **Q2**. Different similarity methods. **Answer**: Upon your suggestion, we experimented with directly using the dot product instead of the cosine similarity method and achieved a Dice value result of 68.7% on the Synapse dataset. We will incorporate additional experiments exploring various similarity measures in the revised version. **Q3**. Compared methods. **Answer**: We compare relevant medical image segmentation methods based on point annotation in Table 2 in the Supplementary Material. Experimental results show that the proposed PSCV outperforms all the methods on the Synapse dataset. [1] employs multiple points (2, 5, and 10 points per category) and scribbles as supervisory information, whereas our approach utilizes only 1 point per category for supervision, making direct comparisons of results challenging. [2] focuses on nuclei and Histopathology segmentation, a cell segmentation task, with datasets featuring only two categories (foreground and background). In contrast, our experimental scenarios contain several different categories. [2] involves over-segmentation and dynamic label propagation for histopathology segmentation, addressing small target areas with similar appearance and shape but distinct edges. Unlike [1,2], our proposed $L_{CV}$ leverages spatial distribution properties and variance distribution maps in a contrastive learning paradigm. Computed over pixel-level variance distribution maps rather than features, it enforces predictions into statistically plausible spatial regions, a crucial element for solving point-supervised medical image segmentation tasks. Our loss function integrates the advantages of the Mumford-Shah (MS) and contrastive paradigms in a novel manner. Besides, [1,2] lacks publicly available code, precluding a direct comparison with our method. **Q4**. Experimental datasets **Answer**: We employed the same datasets as those used in fully supervised and scribble-supervised methods to assess the effectiveness of PSCV. Our evaluation datasets, sourced from authentic clinical exams, span diverse CT and MRI formats, encompassing vital abdominal organs and cardiac classifications. This diversity presents challenges for point-supervised setups, and our paper addresses these challenges through thorough data analysis and validation. Our evaluation's comprehensive diversity and robustness affirm the potential hurdles in PSCV applications. Your insights have inspired us to incorporate additional modalities and organ categories in our future work. **Q5**. About the assumptions. **Answer**: Medical images exhibit unique characteristics distinguishing them from natural landscape images: organs of the same category maintain consistent relative positions across different patients with minimal variation. This observation underpins our methodology, setting us apart from existing approaches. Consequently, our method specifically addresses medical organ segmentation challenges with only single-point annotations.