DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut

Summary

This paper proposes a model for zero-shot semantic image segmentation based on clustering feeatures from an off-the-shelf text-to-image diffusion model. Features are extracted from the U-Net used in the diffusion model. The features are then clustered using a recursive normalized cut algorithm. The clustering is scaled up to the original image resolution by semantic guidance from the extracted features. The proposed model consistently outperforms previous methods on standard benchmarks for image segmentation.

Strengths

- The paper makes a very targeted contribution to the field. An off-the-shelf feature extractor is combined with an established clustering algorithm and a novel upscaling method. The method is evaluated on standard benchmarks which allows for a clear comparison to prior work. - The model is compared to a range of state-of-the-art methods and outperforms them consistently, in most cases by a substantial margin. - Several ablation studies are conducted to show the impact of the contributions in comparison to other diffusion model based segmentation methods.

Weaknesses

- The main contribution explitly states the improvements over the TokenCut and MaskCut models, but does not compare to these models in the experiments. - In my view, the comparison of the respective strengths and weaknesses of different base models is not detailed enough. The experiment on semantic coherence targets this direction. However, beyond this analysis it would have been interesting to directly compare the segmentation results after clustering that are obtained from the different base models, both in terms of quantiative metrics and example cases and further analyses that show the differences.

Questions

- The paper should state more clearly that it target *semantic segmention* to avoid confusion with other types of image segmentation. In my view this would have a major impact on the clarity of the paper. - How large is the computational improvement over previous methods? The differences are mentioned in terms of parameter count, but a comparison in terms of runtime would be interesting as well. - In comparison to standard semantic segmentation, the method proposed here predicts unlabelled classes. Are the features consistent enough to allow for an easy (e.g. linear) mapping to the labelled classes across images?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

I don't fully agree with the statement that the method does not have potential societal impacts. For example, as briefly mentioned by the authors elsewhere, resusing pre-trained models in a zero-shot fashion can save computational and labeling resources. In terms of energy consumption and human labour, this line of works might have a societal impact. One limitation in my view stems from the fact that the diffusion model used as a basis probably does not work well on all types of images. While unsupervised, the approach followed in the paper is dependent on the capabilities of the foundation model. This is not mentioned in the paper.

Summary

This work proposes a new strategy for the task of unsupervised zero-shot segmentation using diffusion model features. The semantic maps are extracted from the U-net features of a diffusion model by applying a recursive algorithm that allows various levels of granularities of the segmentation maps. The proposed method beats competing methods on unsupervised and open-vocabular semantic segmentation on six datasets.

Strengths

In general, the work addresses a problem of unsupervised zero-shot segmentation that has not been addressed a lot by the community, proposing new ways to extract knowledge from the rich U-Net features of a diffusion model. While the method is not overly complex and mainly builds on previously published works, it nicely connects more conventional non-DL research from the past with the opportunities arising in foundational models, and it adds a few more technical contributions that are demonstrated to improve performance. In general, the method seems to clearly outperform previous methods on various benchmarks, including the open-vocabulary setting. The paper is clearly written, and the structure allows to easily follow the work’s main components and contributions.

Weaknesses

My main concern is that the experimental comparison seems rather unfair. While the proposed method uses powerful distilled SD-XL model the baselines only use SD1.4, which might explain most of the performance improvements. In my opinion, the ablation with respect to SD1.4 is not comprehensive enough: 1) The proposed method is not applied with SD1.4, which I think would be mandatory. 2) The DiffSeg results in Tab. 2 seem to be worse than in Tab. 1. I would appreciate a clarification of the effect of using SD-XL features. The proposed method should also be evaluated using the less powerful SD1.4 features to allow a direct comparison with the original DiffSeg method. Otherwise, the comparisons cannot be really fair, in my opinion. Tab. 4: I assume, DiffSeg would again underperform DiffCut. However, having DiffSeg in this table would be a more complete comparison, not only relaying on the Hungarian algorithm-based evaluation strategy in Tab. 1. When considering Fig. 9, the masks do get more fine-granular for higher tau. However, the masks do not always align with the object boundaries (e.g. the surfer, the roofs, or the sign). I am wondering whether this is because of using the low-dimensional resolution and only bi-linearly upsampling? While the previous method DiffSeg also pursued this strategy, I am unsure if the application of the Hungarian matching algorithm to assign predicted masks to a ground truth mask is fair when comparing with methods that do not apply this strategy for the evaluation. It appears not directly comparable. I would appreciate a discussion of this fact and an argumentation why the presented evaluation strategy is reasonable and comparable to the other methods (that do not apply the Hungarian algorithm). I like the additional baseline AutoSC, which is also seems to be very effective. While the recursive and adaptive formulation of the baseline seem so to be reasonable, the gap to DiffCut is, however, not very clear. The results are almost the same. I would appreciate a more complete ablation of this. The set of chosen exponents seems to be arbitrary (l.302). Did you study whether AutoSC outperforms DiffCut if more possible alpha values were chosen? I would appreciate such an analysis and discussion of it in the rebuttal. Also, I am wondering – which model is faster, AutoSC or DiffCut?

Questions

ReCO is first listed training-free, in the open-vocabulary setting with extra-training, but this is not explained in the text? Regarding terminology: The section 4.4) model analysis is nicely to read. I would maybe consider replacing the term robustness with hyperparameter sensitivity (also in G). But I have no strong opinion on this. Tab. 3: Did you also study the results if only 64x64 resolution were used? l.339: Could you please discuss this strategy more comprehensively in the final manuscript (or refer to a related work if existing)? It is not very clear.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The method seems to be profound. However, some important evaluations are missing, as explained above.

Summary

This paper addresses harnessing semantic localization information in pretrained diffusion UNet models. With the proposed recursive normalized cut on the final self-attention features of the diffusion UNet encoder, this paper achieves better performance on unsupervised zero-shot segmentation comparing to previous methods utilizing other fundation vision encoders, with regulation to the granularity of details. Experiments on several benchmarks and ablation studies show the efficacy of the proposed method.

Strengths

This paper addresses an interesting and important problem of analyzing semantic information in pretrained generative diffusion models comparing to other fundation models. Conducting semantic clusterting on the latents of UNet encoder to reduce computation cost and then applying concept assignment while upsampling seems reasonable. The experiments are overall comprehensive and informative. The emperical results significantly outperform pervious baseline method.

Weaknesses

My main concerns consists of several aspects: 1. The writing seems a bit too vague, maybe with more visual illustraition of the observations? For example, can you demonstrate the "patch-level alignment" with visual examples? It's hard to imagine when reading this in Line 168. 2. Are there any visual verification and description for what kind of concepts you get in Section 3.3? 3. The experiments with the proposed DiffCut seems to mainly excute with SSD-1B. What about using other SD models? Regarding computation cost and performance benefit.

Questions

1. In Line 223, what do you mean by "perform a many-to-one matching"? 2. Are the hyperparameters the same for DiffCut on different datasets? Since you seem to use the same hyperparameters for DiffSeg?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

This author did not discuss the limitations in the paper, which potentially could include: 1. How is the computation efficiency of the proposed method comparing to other methods? Considering the recursive partitioning and concept assigning process. 2. The semantic clustering knowledge in pretrained diffusion models might not be able to transfer to data domains like biomedical images. 3. If LD, AX, or UA is given, would the proposed method still outperform the corresponding SOTAs? Or can it further improve its performance?

Summary

This paper introduces an innovative unsupervised, zero-shot image segmentation method called DiffCut. This method leverages the encoder features of a pre-trained diffusion model within a recursive graph partitioning algorithm to create finely detailed segmentation maps without requiring labels from downstream segmentation datasets. DiffCut exploits features from the last self-attention block of a diffusion UNet encoder to perform image segmentation. This approach does not rely on paired image and text data, making it suitable for unsupervised and zero-shot learning tasks. The core algorithmic innovation in DiffCut is the use of a recursive Normalized Cut (NCut) that allows the model to regulate the granularity of detected objects and consequently adapt the number of segments to the visual content of each image. Compared to existing methods like DiffSeg and other graph-based object localization techniques, DiffCut significantly outperforms in terms of the quality of segmentation maps and alignment with semantic visual concepts. The effectiveness of DiffCut was validated across multiple standard benchmarks with a focus on mIoU scores, where it consistently outperformed the state-of-the-art unsupervised semantic segmentation methods.

Strengths

1. The paper is clearly written and logically structured, making it easy to understand. 2. The authors conducted thorough experiments to demonstrate the effectiveness of DiffCut, which achieves state-of-the-art performance in many downstream tasks. 3. In contrast to earlier works that employ graph-based clustering methods for unsupervised segmentation, such as TokenCut and MaskCut, this method introduces a soft thresholding technique for constructing the affinity matrix. This approach effectively maintains high affinity between highly similar patches while reducing the weights between dissimilar patches to near zero—an improvement that addresses a limitation often neglected in previous studies.

Weaknesses

1. **[Unsupervised Instance or Panoptic Segmentation Performance]** While the paper positions itself as effective in segmenting more objects within an image, it primarily focuses on semantic segmentation, potentially overlooking the model's capability for instance discrimination. I am interested in how the model performs in unsupervised instance or panoptic segmentation tasks, as these require differentiating individual objects or integrating both semantic and instance segmentation. 2. **[Technical Contribution and Comparison with MaskCut]** MaskCut is actually capable of segmenting multiple objects per image, with the ability to set a large number of masks per image. This raises questions about the novelty and technical contributions of this paper, particularly in the absence of quantitative comparisons with established methods like TokenCut [30] or MaskCut [32]. 3. **[Lack of Comparisons with Previous Works]** The paper does not compare its results with some previous state-of-the-art works in unsupervised panoptic and semantic segmentation, such as U2Seg [N1] and EAGLE [N2]. Notably, [N2] also explores the use of prototypes derived from eigenvectors of feature maps. 4. **[Adequacy of Low-Resolution Features]** The use of low-resolution feature maps to produce concept-embeddings may result in missing small-scale objects in the images, which could lead to the exclusion of these small objects during the "High-Resolution Concept Assignment" phase. This could limit the model’s effectiveness in capturing medium or small-sized objects, which are crucial for detailed image segmentation. 5. **[Recursive Normalized Cuts or Multi-class Spectral Clustering]** I am curious if the authors have explored to use spectral clustering for multi-entity/object segmentation. The granularity of the segmentation masks can be controlled by using a different cluster numbers. [N1] Niu, Dantong, Xudong Wang, Xinyang Han, Long Lian, Roei Herzig, and Trevor Darrell. "Unsupervised universal image segmentation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22744-22754. 2024. [N2] Kim, Chanyoung, Woojung Han, Dayun Ju, and Seong Jae Hwang. "EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3523-3533. 2024.

Questions

Please check the weakness section.

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors adequately addressed the limitations.