Response to Reviewer cXkh
### **Q1. Advantages of using diffusion for segmentation**
The advantages of diffusion model are mainly manifested in the following aspects:
* Firstly, the diffusion model exhibits a strong perceptual capability for the texture and edge details of images. This conclusion has been demonstrated in numerous diffusion-based image generation works since diffusion models can generate samples with extremely realistic details. The reason for this might lie in the fact that, as a latent variable generative model, diffusion's latent variables are full-sized. Similar to the findings in StyleGAN \[A\], larger-sized latent variables tend to be more inclined towards controlling details and textures. This might explain the perceptual sensitivity of diffusion towards details. Thus, such a detail-sensitive model is particularly suitable for pixel-level segmentation task (especially for high-accuracy segmentation which emphasizes capturing very fine details).
* Secondly, as mentioned in the previous response, addressing all errors in a single iteration can be exceptionally challenging. In contrast, the iterative strategy of diffusion model reduces the difficulty of segmentation refinement tasks by making the model focus on only some "most obvious errors" (subset of all errors) during each step/iteration. This iterative procedure gradually refines the segmentation mask, leading to improved accuracy. In image generation, this iterative strategy imparts the diffusion model with the ability to generate realistic images of challenging complex scenes. When applied to segmentation refinement task, it empowers our SegRefiner to handle more challenging examples and consistently rectify errors and produce precise predictions.
\[A\] Karras, Tero, Samuli Laine, and Timo Aila. "A style-based generator architecture for generative adversarial networks." in CVPR 2019.
### **Q2. Why is the addition of Gaussian noise unsuitable**
* Firstly, the reason for adding Gaussian noise in image generation fundamentally stems from the assumption that natural images, as target data, can be considered as high-dimensional Gaussian variables. Thus, the forward and backward processes are interpreted as noising and denoising and the core idea is to fit a Gaussian distribution. However, in this perspective, our target data is binary masks, which obviously do not conform to the Gaussian distribution. Therefore, we should not regard the process of obtaining segmentation masks as fitting a Gaussian variable. On the contrary, representing it with a discrete random states-transition process is very natural.
* Secondly, as a segmentation refinement method, our core idea is to treat the coarse prediction as a noisy ground truth and recover the high-quality segmentation by eliminating this noise. Since the coarse segmentation result entails misclassifications during foreground/background prediction, it is more akin to discrete states transitions occurring in certain pixels of the ground truth, rather than a binary ground truth overlayed with a continuous Gaussian noise.
### **Q3. Generazability of SegRefiner**
* The generazability largely comes from the model-agnostic and class-agnostic manners of SegRefiner. Its mechanism is first obtaining prompts from the coarse mask to determine the objects needed to be segmented and then providing accurate masks. Such a mechanism allows SegRefiner to have consistent objectives across different segmentation tasks, enhancing the generazability of SegRefiner.
* The difference lies in the fact that for DIS, the dataset resolution is exceptionally high, while the model's output size is only $256 \times 256$. This resolution is inadequate for such high-resolution datasets and results in the loss of numerous fine details. Therefore, we employed a combination of both global and local refinements strategy (as elaborated in line 217 of the paper). However, for instance segmentation on the LVIS dataset with relatively smaller resolution, we only employed instance-level refinement without local refinement.
* The alpha matting task requires predicting floating-point values for unknown region. Although our method is designed for binary masks using a discrete process, it can also handle image matting by converting the intermediate predicted fine mask into a floating-point format. However, since this approach would require the aforementioned modifications and training on matting datasets independently, resulting in substantial deviations from the overall framework of this work, we did not conduct the matting experiment within the scope of this work.
### **Q4. "whether the authors have uploaded the revised version of the paper (*.pdf) to the system."**
According to the author responses policy of NeurIPS 2023, *"Authors may not submit revisions of their paper or supplemental material"*, we have not uploaded the revised vision. We will update the paper according to the reviewers' comments. Thank you very much for your valuable comments!