Unsupervised Semantic Correspondence Using Stable Diffusion

Summary

The authors propose an approach to establish semantic correspondences, ie correspondences between different instances of the same object, using a pre-trained stable diffusion models, and without any task-specific finetuning. In particular, they leverage the fact that intermediate attention layers of unet diffusion model respond to the semantic of the text prompt. For a source image and a particular query pixel location for which a match should be computed, they optimize the prompt embedding that leads to a cross-attention map with a high activation at the query location. Once the prompt is recovered, they can apply the reverse attention process on a target image conditioned on the optimized prompt. The intermediate cross-attention layers of the diffusion model highlight the matching location. They also introduce technical ‘tricks’ to make it work, for example averaging over multiple crops of an image or starting from different random embeddings. This process must be repeated for each query pixel location.

Strengths

- The insight that stable diffusion models inherently learn semantic correspondences without task-specific training is interesting. How to extract and use such knowledge has not been explored in the past. - The results are convincing - the approach obtains near state-of-the-art in semantic matching without any task-specific knowledge or funetuning. - The paper reads well overall

Weaknesses

1) Missing comparisons and citations in results: PWarpC [Truong et al. CVPR 2022]: - the paper only includes comparisons to two weakly-supervised semantic matching approaches, with relatively weak results, which allows the authors to claim that their approach is 20% better than the state-of-the-art in weakly-supervised semantic matching. - The authors did not include results of PWarpC, a weakly-supervised approach for semantic matching presented at CVPR 2022. The results of PWarpC significantly outperform the baseline results of this paper for SPair-71K (38.0 on SPair-71K for PWarpC-NC-Net), making the above claim invalid. - on PF-Willow: to double check since the two papers use different metrics but PWarpC (weakly-supervised) outperforms CATs, which obtains better metrics than the proposed approach. It might therefore outperform the proposed approach. - please include this comparisons and revise the claims/analysis accordingly 2) while the paper presents an interesting analysis, the proposed approach is currently very impractical and almost unusable, since as the authors say, it takes 30s to generate a single match. I believe it would be interesting to have even a small experiment training a semantic matching model based on pseudo ground-truth matches obtained from the diffusion model.

Questions

In addition to the weakness above: A) What is the resolution of the extracted feature/attention maps and at what resolution is the matching done? for example are the features interpolated to higer resolution before computing the cross-attention weights? Is a similar resolution used in competitor works like DINO? This is important because the resolution of the feature maps is of crucial importance for obtaining fine-grained/accurate correspondences. It has been the limiting factor of many previous approaches that needed to compute all-to-all correspondences, because of memory constraint. Just using a higher resolution feature map for the cross-attention would lead to an improvement in results, not necessarily related to the features themselves. B) L.230 the authors mention it takes ‘30s’ to optimize the prompt, does it include only finding the prompt? or finding the match? A real run-time analysis of time versus performance for different settings would be interesting here (depending on how many crops are used, how many initializations ect). C) clarification needed: C.1) L.178: M′ has dimension C×(h×w)×P, where is the channel coming from? According to equation, M’ is computed with the standard attention mechanism, which is just a matrix multiplication followed by a softmax. As a result, the dimensions should be (hxw)xP. C.2) L.185, 187, it would be helpful to specify the shape of M′′ and M, to better follow the different steps. What is the dimension of P for the learnt embedding? C.3) L213, is it equation 5 or equation 7? The authors say they average the optimization over multiple crops and over multiple initialization of the embeddings. Is this happening at the same time, i.e. for different initialization, the optimization is done over multiple crops? If so, then it should be equation (7). C.4) in Tab1, what is meant by ‘weakly-supervised (test-time optimization)’? DINO with NN is for me unsupervised since it only requires single images for the contrastive learning part. some typos: L.207 hyperparemter L.53-54 Note that while finding actual prompt thatcorrespond to words is a discrete problem, hence difficult to solve. -> remove while

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

they have adequately addressed the limitations an negative impact of their work.

Summary

This paper proposes a model that finds semantic correspondence between a pair of images, with a pretrained diffusion model. By optimizing the prompt embeddings, such correspondence can be read out via attention maps in the UNet. Empirical study shows the good performance of this proposed model over exising baselines.

Strengths

1. The proposed method has a simple form but is effective. 2. The presentation is clear and easy to follow. 2. The qualitative results are impressive, especially the results between different classes. 3. Some quantitative results are significantly better than baselines, or comparable to supervised results.

Weaknesses

1. More information should be provided for the optimization set up and implementation, e.g. the optimization algorithm for Eq (5) and Eq (9). 2. The complexity and efficiency are needed for better assessment of the algorithm, especially in comparison to baselines.

Questions

1. I'm wondering how would the trained prompt compare to text embeddings, e.g. how would a prompt optimized from a bird's eye compare to the text embedding of "a bird's left eye"? 2. One textual baseline that doesn't appear in the experiments is the attention map of an actual text prompt, i.e. would the prompt of "a bird's left eye" have its highest attention to the eye's location?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors has addressed valid limitations of this work.

Summary

This paper focuses unsupervised semantic correspondence and proposes to leverage the semantic knowledge within fashionable text-to-image diffusion model to accomplish this task. The method optimizes a learnable text prompt to maximize source image attention value of query location. Then, the optimized text prompt is used to find corresponding location of target image by finding the max value of target image attention value. Related experiments show the effectiveness of the proposed method.

Strengths

1.The idea of leveraging semantic knowledge of text-to-image diffusion model sounds interesting. This paper builds a new type of unsupervised semantic correspondence method based on text-to-image diffusion model. 2.The designed method is effective and achieve promising correspondence results on common datasets. 3.The design choices are reasonable, e.g., random crop for preventing overfitting when finding the text prompts.

Weaknesses

1.The usage of text-to-image (t2i) diffusion model is not well-motivated. The proposed method mainly leverages the cross-modal ability of t2i diffusion model but underuses the generative ability of t2i diffusion model. The role of t2i diffusion model in proposed method can be replaced by other cross-modal pretraining model, e.g., CLIP, BLIP, GLIP. Please claim the unique significance for adopting t2i diffusion model rather than other cross-modal pretraining model. 2.The proposed method has to execute dozens of denoising steps for each query location. As diffusion model is notorious for its inference speed and cost, the total overhead is expensive. Are there any mitigation measures? For example, grouping the query locations or modifying the diffusion model architecture?

Questions

Please see “Weakness” part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors objectively discuss the limitations of the proposed method, e.g., the slow inference time caused by the introduction of diffusion model. This paper doesn’t have negative societal impacts.

Summary

This paper explores unsupervised semantic correspondence tasks with stable diffusion model. Specifically, the authors proposed to first optimize the prompt embeddings of stable diffusion model, to maximize attention on the region of interest, then the optimized prompts are used for semantic correspondence. Experimental results on PF-Willow, CUB-200 and SPair-71k datasets show that proposed method significantly outperforms weakly / unsupervised methods.

Strengths

- This paper shows that one could explore diffusion models for semantic correspondence tasks. - the writing is clear and easy to follow. - The experiments show that the proposed method could even outperform supervised method on PF-Willow.

Weaknesses

1. How important is the optimization for prompts? Could the authors show results if the prompts are a vanilla sentence, e.g. "an image of a cat"? 2. Concurrent works: there are multiple papers presenting correspondence ability of diffusion models: [1*] Luo, Grace, et al. "Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence." arXiv preprint arXiv:2305.14334 (2023). [2*] Tang, Luming, et al. "Emergent Correspondence from Image Diffusion." arXiv preprint arXiv:2306.03881 (2023). [3*] Zhang, Junyi, et al. "A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence." arXiv preprint arXiv:2305.15347 (2023). Could the authors explain the differences if possible?

Questions

please refer them to the weakness session

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Summary

The paper proposes an approach for unsupervised semantic correspondence by employing a pre-trained Stable Diffusion generator. The basic idea is that since text-to-image diffusion models are capable of generating realistic images, they must understand the semantics of the objects they generate and thus be capable of finding semantic similarities in images. Technically, the approach feeds a real image (or, better, its latent vector) into a diffusion pipeline, recover attention matrices between pixel values and the textual prompt, and optimizes the textual prompt embedding so that the cross-attention focuses on a query pixel location. Once optimization is over, the same text embedding can be employed to find semantic correspondences in a target image. Cross-attention matrices are averaged across channels and a subset of U-Net layers, using bilinear interpolation. Further, the target attention map is simulated with a Gaussian distribution centered on the query location, and a L2 loss is employed for optimization. Authors also propose two regularization strategies to prevent over-fitting during optimization, i.e. averaging across crops of the query image, and averaging across multiple rounds of optimizations/initialization choices. Experimental results are conducted over three standard benchmarks for the task, i.e. SPair-71k, PF-Willow and CUB-200. The proposed approach achieves superior results when compared to unsupervised or weakly-supervised approaches, and is also competitive when compared to models employing strong supervision.

Strengths

- The main idea of the paper (extracting/exploiting the semantics embedded in a pre-trained a SD network) is novel and interesting. It is in-line with some recent pre-prints working on related tasks (e.g. image segmentation), but to my knowledge the proposed approach is novel wrt published papers. - The technique for extracting semantics is simple and effective, and also are the two regularization techniques. - Experimental results are solid and confirm the appropriateness of the approach. - The paper is well written, clear and a pleasure to read. The supplementary material is also interesting and comprehensive.

Weaknesses

- While the idea behind the paper is very general and could have been applied to a variety of tasks involving semantics (e.g. image segmentation, panoptic segmentation, classification), the paper only focuses on finding semantic correspondences. This limits the overall impact of the work to a single task of Computer Vision. I wonder if authors have tried to extend to other tasks, and how difficult that would be.

Questions

See above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have successfully discussed the limitations of the work in the appropriate section.