Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features

Summary

The paper addresses the critical task of enhancing feature selection in diffusion models to improve performance in discriminative tasks such as semantic correspondence, semantic segmentation, and label-scarce segmentation. Previous methods often overlooked many potential activations within diffusion models, leading to suboptimal performance and limitations due to ignoring high-resolution activations and not effectively handling diffusion noises. The authors propose a comprehensive feature selection solution that leverages distinctive properties of diffusion U-Nets, including diffusion noises, in-resolution granularity changes, and locality without positional embeddings, to filter and select the most relevant features. Experimental results demonstrate that their method outperforms state-of-the-art techniques, achieving significant improvements in various metrics across multiple datasets.

Strengths

- This paper introduces a novel approach to feature selection in diffusion models, focusing on qualitative analysis to filter out suboptimal activations before performing quantitative comparisons. This methodology shifts from the traditional full-scale quantitative comparison, making the process more efficient and potentially more accurate. - The authors identify three distinct properties of diffusion U-Nets that are leveraged for feature selection: 1.  **Asymmetric Diffusion Noises**: The diffusion process introduces unique noises affecting both low- and high-frequency signals. 2.  **In-Resolution Granularity Changes**: Modern diffusion U-Nets exhibit significant granularity changes within a single resolution due to fewer but larger resolutions. 3. **Locality without Positional Embeddings**: Self-attention modules in diffusion U-Nets show a new type of locality that enhances activation quality without traditional positional embeddings. - The discovered properties means that the findings can be generalized to various diffusion models, providing valuable insights and a solid foundation for future research. This can guide the development of more effective and efficient discriminative models and applications in other fields. - The proposed method achieves SOTA performance across multiple discriminative tasks, including semantic correspondence, semantic segmentation, and label-scarce segmentation.

Weaknesses

- Some more comparisons are recommended. While the authors claim to find unique properties in diffusion U-Nets, it lacks a detailed comparison with traditional U-Nets. A more comprehensive analysis of how these properties differ would provide better context. - Some visualizations, such as Figure 3(c) showing positional information in self-attention activations, could be clearer and better explained to enhance understanding of the findings.

Questions

See the weakness.

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper highlights the importance of considering a broader range of activations within diffusion models. The authors propose three universal properties of diffusion U-Nets that aid in qualitatively filtering out activations that are clearly sub-optimal. On top of this, the authors can improve the efficiency of feature selection. By leveraging these properties, the proposed method demonstrates superior performance across multiple discriminative tasks, such as semantic correspondence and segmentations. This comprehensive approach to activation selection addresses a fundamental issue in diffusion models, enhancing their applicability and performance in various tasks.

Strengths

1) The authors propose an efficient method for selecting high-quality activations. With the help of qualitatively filtering out sub-optimal activations before quantitative comparison, the computational costs can be significantly reduced, compared with the prior arts. 2) The authors present their observations in a clear and organized manner, making it easy for readers to understand the complex concepts and follow the research process. Besides, the main content and the appendix include numerous visual samples and activation visualizations, which help illustrate the properties and effectiveness of the proposed activation selection method. 3) Extensive experimental results across multiple discriminative tasks validate the superiority of the proposed method over the competitors.

Weaknesses

W1: This paper could benefit from introducing more notations to clearly define and differentiate between various activations and components within the diffusion models, which would further enhance the clarity of the descriptions. W2: While the empirical results are robust, the paper could include more theoretical analysis to provide deeper insights. W3: Some more latest related work are commended to be cited: [1] Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features, CVPR 2024. [2] DiffSeg: Towards Detecting Diffusion-Based Inpainting Attacks Using Multi-Feature Segmentation, CVPR 2024.

Questions

I have no more questions.

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes

Summary

Diffusion models have achieved significant success in image generation and show great potential for various discriminative tasks. The authors rethink the foundational problem of feature selection within these models. To this end, they analyze the properties of diffusion models, including asymmetric diffusion noises, in-resolution granularity changes, and locality without positional embeddings, to filter out suboptimal activations. The results demonstrate that their proposed feature selection method outperforms state-of-the-art techniques across multiple discriminative tasks, achieving superior performances.

Strengths

- This paper points out three unique properties of diffusion models—asymmetric diffusion noises, in-resolution granularity changes, and locality without positional embeddings—that provide valuable insights into improving discriminative tasks. - The authors propose an off-the-shelf solution for feature selection. Besides, the findings and methodologies presented are not limited to the specific models studied (SDv1.5 and SDXL). - The paper conducts extensive experiments across multiple discriminative tasks, including semantic correspondence, semantic segmentation, and label-scarce segmentation. The results, as well as the visual illustration, validate the effectiveness of the proposed feature selection method, demonstrating significant improvements over state-of-the-art techniques.

Weaknesses

- The authors acknowledge the uncertainty about whether the findings can generalize to newer models like DiT (Diffusion Transformer) due to architectural differences. - The qualitative filtering approach proposed is novel, but its scalability and efficiency in very large-scale settings are not fully demonstrated. How do the authors alleviate this problem? - The paper focuses on empirical results and qualitative analysis. There is a lack of rigorous theoretical foundation or mathematical proofs to support the proposed feature selection methodology and the identified properties of diffusion U-Nets. A more robust theoretical framework would strengthen the claims made.

Questions

Besides those in weakness, I wonder can we combine multiple features during the qualitative analysis to further improve the performance? And how many features do you select for each image?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

4

Limitations

Yes, as stated in Sec.7.

Summary

This paper proposes a new feature selection method for diffusion models by evaluating a broader range of activations, particularly those in embedded Vision Transformers (ViTs). The authors identify the limitations of current approaches that consider only a narrow range of activations and introduce a qualitative analysis to filter out low-quality activations in diffusion U-Nets. They develop specific feature selection solutions for popular diffusion models SDv1.5 and SDXL. Experiments demonstrate that this method outperforms state-of-the-art techniques in tasks like semantic correspondence, semantic segmentation, and label-scarce segmentation, showcasing its effectiveness and generalizability.

Strengths

- This paper extends the evaluation to a wider range of activations within diffusion models, especially those in embedded ViT modules, which were previously overlooked. Since feature selection is a basic problem for diffusion features, this paper can boost future work in this direction. - The authors introduce a qualitative analysis to effectively filter out low-quality activations, simplifying the subsequent quantitative comparison and improving feature selection efficiency. What’s more, the observed properties are not limited to the discussed models, which can also inspire future work. - Extensive experiments demonstrate that the proposed feature selection method significantly outperforms state-of-the-art techniques in various discriminative tasks, validating its effectiveness and generalizability.

Weaknesses

- As pointed out by the authors, the observations and methods proposed do not necessarily generalize well to recently developed DiT models since their architecture is markedly different from diffusion U-Nets . - Some details can be more clarified. For example, in Appenidx C.3, the authors state that “this is also the setting for the quantitative comparison”. In other words, quantitative comparison is conducted on Label-Scarce Segmentation, and the experiments on the other tasks follow this setting. Is this understanding correct? If so, I wonder why the authors select this setting to conduct quantitative comparison. - Besides, there are some typos. For example, in the introduction, “a fundamental problem to select” should be “a fundamental problem in selecting”. In the caption of Figure 3, “existing knowledge of other models” should be “existing knowledge about other models”.

Questions

Please refer to the weakness.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors have clarified the limitations in Section 7.