A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence

Summary

This paper exploits Stable Diffusion (SD) features for semantic and dense correspondence tasks. The authors first conduct evaluations of SD features and found that SD features provide high-quality spatial information but sometimes inaccurate semantic matches. This paper further shows that such SD features are complementary to DINOv2 features which provide sparse but accurate matches. By fusing SD and DINOv2 features with a simple weighted concatenation strategy, this paper achieves significant performance gains over state-of-the-art methods on benchmark datasets, e.g., SPair-71k, PF-Pascal, and TSS.

Strengths

- **Extensive evaluation and analysis.** This paper conducts lots of experiments and visualizations to analyze the behavior of Stable Diffusion and DINOv2 features. The experiments are also well-organized, with a smooth logic flow, which makes it easy for readers to follow. - **Strong zero-shot performance.** By simply combining the Stable Diffusion and DINOv2 features, this paper shows strong zero-shot performance of the semantic correspondence task, outperforming previous task-specific methods by a large margin. - **The message shown in this paper could be interesting to the community.** Unlike previous methods that mostly focus on studying single image diffusion models, this paper studies the properties of diffusion model features across different images in the context of semantic correspondence. This paper shows that the publicly available Stable Diffusion and DINOv2 models contain rich information to solve the semantic correspondence task, even outperforming task-specific models in the zero-shot setting. This would be a strong message and might stimulate future work along this direction.

Weaknesses

It's unclear how the proposed method handles outliers in the correspondences. Since the correspondences are obtained with nearest neighbor search, which might be less robust for occlusion or out-of-frame points.

Questions

- I am wondering what might lead to the different behaviors of Stable Diffusion features and DINOv2 features? A more in-depth discussion might make this paper stronger. Could the authors further comment on this? - As also discussed in the limitation section, the spatial resolutions of Stable Diffusion features and DINOv2 features are both relative low, which might hurt the performance for fine-grained details. Besides Stable Diffusion, there are also other diffusion models that directly work on the original pixel space instead of the downsampled latent space, would these kind of diffusion models potentially alleviate this issue? I am not asking for such experiments, but just wondering the authors' opinion on this.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, the limitations are discussed in the paper.

Summary

This paper, for the first time, proposes a novel method to fuse Stable Diffusion features and DINOv2 features to obtain robust feature representation that readily surpasses the SOTA semantic correspondence work without further training, rather simply adopting Winner Takes All yields SOTA correspondence performance. Interestingly, with such correspondences, high-quality object swapping is also enabled without task-specific fine-tuning.

Strengths

1. Novel idea, approach and thorough analysis. 2. Motivation is really good and the paper was easy to read and understand. 3. Impressive performance. 4. Good application (swapping)

Weaknesses

1. This is more like a question rather than a weakness. In figure 1, the visualization shows that a cat given as a source image, the proposed approach finds dense "correspondence" to dog, horse and motorcycle. Is this really a correspondence? To me, it feels like this is more like foreground segmentation. From the colors of the features, iI could tell that the facial part of cat is matched to the frontal part of motorcycle and facial part of other animals. This is very interesting. Why do you think this visualization is obtained even though strictly speaking, there should not be correspondences? 2. Why is it that on SPair-71k, the proposed method yields leading results, but not for PF-PASCAL and TSS? As the authors state, PF-PASCAL and TSS are less challenging datasets, but results seem to show the opposite. 3. Missing baselines : for supervised ones, CATs++ (extension of CATs) and IFCAT from the same author achieve better performance. Also, PWarpC is a work that adopts unsupervised learning for semantic correspondence. I would suggest including this as well. It is better to compare with them as well. I don't think it would be a weakness even if the proposed method does not attain SOTA, since the contribution lies more on the fusion of features and their analysis. 4. In implementation detail, it says the input resolution is 960 x 960. What was the evaluation keypoint annotation resolution? The input resolution, evaluation resolution and many other factors related to resolutions have high influence on performance in this task. This is addressed by PwarpC (CVPR'22) and CATs++ (TPAMI). So if the evaluation is performed at higher resolution than other works, the comparison may not be fair at all. This needs to be clarified.

Questions

See the weaknesses above. If they are adequately addressed in the rebuttal, I think this paper will be a very strong paper. My rating is solely based on the performance part, which if clarified, then I am happy to increase my rating to accept this paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are adequately discussed.

Summary

The paper explores the use of Stable Diffusion (SD) features for dense correspondence. The authors investigate the potential of SD and DINOv2 features and show some complementarity. SD features provide high-quality spatial information but sometimes inaccurate semantic matches while DINOv2 features offer sparse but accurate matches. The authors show that averaging the features from the two models might achieve strong performances for dense correspondence. The fused features are evaluated using a zero-shot approach, where no specialized training is performed for the correspondence task, and nearest neighbors are used for evaluation. Surprisingly, the fused features outperform state-of-the-art methods on benchmark datasets like SPair-71k, PF-Pascal, and TSS. The authors also demonstrate that these correspondences enable interesting applications such as instance swapping between images while preserving object identities.

Strengths

**In-depth Qualitative Analysis** The paper conducts a detailed qualitative analysis of the Stable Diffusion (SD) features and DINOv2 features, shedding light on their respective strengths and weaknesses. This analysis provides valuable insights into the semantic and texture relevance of these features, highlighting their distinct properties. **Extensive Experiments** The paper presents extensive experimental results on benchmark datasets, including SPair-71k, PF-Pascal, and TSS. The authors report significant gains compared to SOTA methods. **Application of Instance Swapping** The paper well illustrate the complementarity of the feature on this task.

Weaknesses

**Lack of Strong Quantitative Assessments** While the paper provides an in-depth qualitative analysis of the complementarity between SD and DINOv2 features, it lacks strong quantitative assessments to support these claims. The true added value of the paper should have been a robust quantitative evaluation showcasing in particular the non-redundancy of the SD and DINOv2 features. This absence limits the overall impact and credibility of the proposed approach. **Incomplete Figure 2 ?** There appears to be an issue with Figure 2, as the bottom illustration is missing despite being mentioned in the legend. **Lack of Clarity in PCA Aggregation** The authors mention the use of PCA for aggregating SD features, but the explanation provided is not clear enough. The authors mention computing PCA across the pair of images for each layer of the feature map, followed by upsampling features from different layers to the same resolution to form the ensembled feature. I however have trouble understanding clearly what is the workflow here.

Questions

To provide a fair evaluation, it would be interesting to have the performances of other fused features based on DINO varaitions such as DINOv1 + SD or EsViT[1] + SD. [1] Chunyuan Li et al. “Efficient Self-supervised Vision Transformers for Representation Learning”, ICLR22

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

1 poor

Presentation

2 fair

Contribution

2 fair

Limitations

The authors underline the technical limitation of their work.

Summary

This paper proposes to study the effectiveness of features extracted from Stable Diffusion for dense correspondences. The extracted features are compared to that of DINOv2 and shown to be complementary. A very simple fusion scheme is then proposed and evaluation on datasets for sparse and dense correspondences as well as on instance swapping with convincing results. --- The rebuttal is convincing, I maintain my initial rating leaning towards acceptance.

Strengths

With all the effort that has been poured recently in diffusion models, this empirical study regarding their use outside of generative nice images is refreshing, especially since it tackle a very low level problem (keypoint correspondences) that is somewhat very remote from image generation and yet surprisingly close since the model has to generate coherent local structures. The study shows that indeed, diffusion model features are very useful for such problems and thus sheds light on what one can expect from the latent space structure of SD.

Weaknesses

- All the results are performed in zero-shot, which is a bit limiting. I am not familiar with the recent literature on correspondence problems, but I assume that a fine-tuning of the features (at the very least of the projection) is possible to see how much we can get from these models. - All the experiments were made with stable diffusion 1.5. It would be interesting to test another model to see if the same results hold across architecture and training change and thus if it is a generic property of diffusion models. - The fact that the best results are obtained using 2 really big models is a bit annoying. It means it is difficult to know if the results come from the nature of the methods employed (SSL and diffusion) or just from the sheer capacity of the models employed.

Questions

- The timestep at which features are extracted is not clearly mentioned (is it t=100 at line 214, in that case over a total time length of how long?). What influence influence does it have? Are the features more meaning closer to the denoised image ? - The swapping are impressive, but they are refined by running the diffusion process after wise. It would be great to see compositing before the diffusion process to evaluate the quality of the correspondences.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper discusses 2 limiations: namely the low resolution of the features and the computational budget to get features from a big diffusion model. The first one is probably the bigger concern for correspondences.

Summary

The paper proposes using features extracted from a stable diffusion model for dense image correspondence tasks. The paper further proposes using DINO features along with stable diffusion features for the task and empirically shows that the combination has a complementary effect--specifically SD features have good spatial localization and are smooth, whereas DINO features are sparse but accurate. Experiments are done on several datasets and the proposed feature extraction technique improves drastically over supervised, unsupervised, and weakly supervised baselines.

Strengths

1. The paper is well-written, adequately inspired, and well-executed. 2. Experiments depict the importance of these large pre-trained models for the task of correspondence learning in images. Specifically, the large gains over supervised methods are quite surprising. 3. Instance swapping application clearly demonstrates the ability to do accurate correspondences between different instances of the same category of objects.

Weaknesses

1. The use of pre-trained features from large models is a well-explored area of research [1,2,3]. These pre-trained models are known to perform well on downstream tasks such as semantic segmentation, detection, and classification. It is unsurprising that these features are also useful for correspondence learning. 2. Though the authors show limitations of the features. It should have been focussed more on the paper. Specifically categorizing the mistakes made by the approach and possible ways to alleviate the problems. This will be a good contribution to the community as it informs when to avoid using the proposed approach. [1] _Unleashing text-to-image diffusion models for visual perception._ Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. 2023 [2] _Emerging properties in self-supervised vision transformers._ Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021 [3] Open-vocabulary panoptic segmentation with text-to-image diffusion models. Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. 2023

Questions

1. Though the motivation for using these features is clear, the paper doesn't provide reasons for the complementary properties of the features from DINO and SD. It would be interesting to analyze what specifically gives rise to these complementary properties. Is it because of the training objective, architecture, datasets, etc? 2. Second and fourth rows of the right side of Figure 3 can be explained more for clarity. Specifically, if the figure also includes the original images, it will make more sense. 3. Did the authors experiment with giving different textual inputs to SD? Does including object categories in the textual categories improve the performance of correspondences?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes.