Emergent Correspondence from Image Diffusion

Summary

The authors find that intra and cross-category correspondences are implicitly learnt by diffusion models trained self-supervised on large datasets. The paper proposes an approach to extract this knowledge as features from pre-trained Unet-based diffusion models. In particular, to compute the features of a particular image, noise is added to the image to simulate the diffusion process, and then input to the pre-trained diffusion model. The intermediate layer activations from the Unet at a particular timestep are used as features. They showcase the performance of these features on semantic-matching, outperforming other unsupervised baselines and strongly-supervised approaches specifically designed for semantic matching. They also show that without any task specific finetuning, the features can be used for geometric matching and temporal matching ( video-object-segmentation throught label propagation), with competitive performance compared to state-of-the-art.

Strengths

- The proposed idea is simple and effective - The results are convincing. Having a single model applicable to many different correspondence tasks without specific architecture/training and outperform task-specific methods would be a useful contribution. The new perspectives that it opens are exciting. - The paper overall reads well.

Weaknesses

1) The authors do not comment on the run-time of the proposed approach. I expect it will be quite slow since the image needs to go through the reverse diffusion process. 2) Related to the above, relying on the diffusion model at inference time makes the method impractical in many applications. To remove the reliance on diffusion models at inference, would it be possible to train a feature predictor, using as ground-truth the features extracted from a pre-trained models? Have you done any experiments in this direction?

Questions

A) What is the run-time of the approach to extract feature on a single image? B) I think details on the ‘architecture’ of the extracted features are missing. It is only said L.113-114 that “intermediate layer activations” are extracted from the unet, and that each depends on the task. I would appreciate to have some details there - what layers are extracted (for the different tasks), what resolution and channel dimension do they have, are they interpolated? Are features from multiple levels used? If so how are they aggregated? C) What is the resolution of the extracted feature and at what resolution is the matching done? Is a similar resolution used in competitor works like DINO? The resolution of the feature maps is of crucial importance for obtaining fine-grained/accurate correspondences. Since matching is done by computing all-to-all similarities between the two images, It has often limited previous approaches to small resolution because of memory constraint. Just using a higher resolution feature map would lead to a significant improvement in results, not necessarily related to the features themselves. D) Some misclassification and missing comparisons in semantic matching: D.1) CNNGeo and A2Net are unsupervised - they only use single images as supervision, as opposed to NCNet for example which requires pairs of similar images D.2) Comparison to recent approches are missing. In particular PWarpC [Truong et al. CVPR 2022] obtained state-of-the-art results in weakly-supervised semantic matching (with extension to supervised) E) Would it be possible to extract features from multiple layers of the diffusion model such as to build a feature pyramid (similar to VGG or ResNet)? This could potentially be used as a plug-in replacement for backbones in correspondence approaches with a decoder. F) How many different passes through the network (ie different random noise) are used for an image?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

-

Summary

The paper proposes to use off-the-shelf generative networks based on denoising diffusion models to find local correspondences. The paper is extremely simple: instead of generating samples purely from random noise or doing some kind of image-based conditioning, the method just adds random noise to the input image and takes some intermediate level in the denoising U-Net as a dense feature map, from which it can extract sparse features. Feature matching is done via cosine distances. Keypoints may be provided (e.g. for semantic matching) or taken with an off-the-shelf keypoint detector, such as SuperPoint. The authors show that their approach works for high-level semantics (e.g. an eye across different animals or even species) and geometric correspondences, and can also track points across time in video sequences. Parameters such as the stage in the denoising process or which intermediate layers to use are chosen per dataset. There are some tricks (e.g. featuremaps are averaged over batches with different noise inputs), but that's about it.

Strengths

1. Cool idea. 2. Very simple. The actual method section is a single paragraph and doesn't require any math. The paper is well written and easy to understand. 3. Very good results on multiple tasks, including semantic matching and tracking in video sequences, without training or fine-tuning.

Weaknesses

1. My main complaint is that I find it hard to believe this approach will work well across arbitrary geometric changes. I have experience in this field and conclusions drawn from HPatches rarely translate to real scenarios. The paper gives very few results, which makes these experiments unreliable and difficult to trust (my guess is that this gap will disappear when the method is properly evaluated). For instance: - What is the image size? - How many points do you use? - Do you use both the illumination and viewpoint splits for HPatches (the supplementary material suggests so)? If so, why not split the results by sequence type? - Why not use the more standard MMA metric from D2-Net (which isn't perfect but is easier to understand), instead of estimating homographies? Why not both? - Why not use modern baselines? The most recent method is from 2019 (R2D2): see e.g. DISK, PoSFeat, ALIKE, or SILK (citations below). (Note: CAPS is weakly supervised and works worse than the DISK variant supervised only with epipolar geometry, as far as I know.) - Could you provide a simple precision/recall curve and compare it against traditional local feature methods? My intuition is that it would be worse. - What happens if you add RANSAC? I generally discount claims on this dataset and steer people towards evaluation benchmarks focusing on downstream tasks, such as visualocalization.net or the image matching challenge (https://image-matching-workshop.github.io/). I understand that this is likely beyond the scope of this paper, and I think it would be acceptable that it just shows these features work reasonably well for rigid matching, but if you want to claim that "though not trained using any explicit geometry supervision, DIFT outperforms prior state-of-the-art methods that utilize explicit geometric supervision signals designed specifically for this task, such as correspondences obtained from Structure from Motion pipelines" (L210-213), then you're going to have to substantiate that much better. 2. Given that the paper is so simple and the concept itself is very easy to explain, it would be nice to see more introspection. For instance, showing qualitative results using PCA or t-SNE to cluster the features (see for instance the videos from DINO v2) would help understand what's going on. 3. Important details are relegated to the supplementary material: - The amount of noise used per dataset. It's surprising that, for instance, for geometric correspondence with Stable Diffusion the best results are with t=0, which means basically no noise (if I understand the formulation correctly; I looked at the code). - The fact that for semantic correspondence per-class prompts are used. I think this is a reasonable assumption, but it should be mentioned. It also suggests the method may not work so well in non-object-centric images.

Questions

The paper is cool and novel, but given the issues above I'll mark it as borderline (BA, since I only have BA and BR) and wait on the rebuttal and reviewer discussion. - How do you deal with the loss in resolution, especially for geometric matching? The supplementary material says the method uses the 11th upsampling block out of 18 for ADM, and the 2nd block out of 4 for SD. Do you simply upscale the feature maps? And if so how do you achieve accurately localized correspondences? - Do you choose per-dataset hyperparameters on a validation set or the test set? - What is the computational cost and how does it compare with other methods? - What does "after removing outliers" mean in fig 10 in the supplementary material? Why not show the raw matches and color-code them according to correct/incorrect? - What happens without per-class prompts? Notes: - It's nice to see that the method is sensitive to the choice of t but varies smoothly (Fig. 5). It would be nice to see this in other datasets, though. And what about n? - The images in Fig. 10 in the supplementary material have the wrong colors (I guess BGR, from loading/saving with OpenCV). - Please add the dataset and metric to the headers in Tables 1 and 2 (one's missing the metric, the other two the dataset names). Citations: [DISK] https://arxiv.org/abs/2006.13566 (https://github.com/cvlab-epfl/disk) [PoSFeat] https://openaccess.thecvf.com/content/CVPR2022/papers/Li_Decoupling_Makes_Weakly_Supervised_Local_Feature_Better_CVPR_2022_paper.pdf (https://github.com/The-Learning-And-Vision-Atelier-LAVA/PoSFeat) [ALIKE] https://arxiv.org/pdf/2112.02906.pdf (https://github.com/Shiaoming/ALIKE) [SILK] https://arxiv.org/abs/2304.06194 (https://github.com/facebookresearch/silk)

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper addresses a classical computer vision problem, ie, points correspondence. The authors show that the feature maps of the decoder of a diffusion model U-Net enable robust feature matching with a simple nearest neighbor search. Semantic or geometric correspondence can be achieved, by selecting the appropriate denoiser time step. The authors report results, quantitative analysis and comparison on several benchmarks, chosen for each specific task.

Strengths

The authors observe empirically that the features contained in the U-Net decoder are powerful features for image correspondence. They also show that those features embed different levels of semantic information, depending on the time step of the denoiser. It is particularly relevant given the fact that diffusion models used by the authors are pretrained (the authors only add noise to the input image), and that the drawn conclusions are consistent over different DMs. The set of experiments reported in the paper are thorough and results are convincing. The variety of the results, quantitative and qualitative, demonstrate the reliability and flexibility of such features. I particularly enjoyed reading a paper presented with such simplicity and clarity.

Weaknesses

The drawback of the approach is probably the computational complexity that is intrinsic to any point matching approach relying on nearest neighbor search based on high dimensional features. It is not clear from the paper (nor the supplementary material), if choosing the optimal time denoising time step is critical or not. A corollary of this question would be: do we expect the semantic level contained in the features to degrade smoothly from large time step to small ones?

Questions

1) In figure 3, it seems that the semantic information is such that the right ear feature is distinct from the left ear one. Is it a general case or does this happen in this specific example only due to a very similar pose in the source and the target image? 2) Did the authors try to apply clustering based on U-Net features? 3) Could the authors elaborate on the computational time. What is the dimension of a DIFT feature?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

the authors rightly mentioned the ethical issues related to generative models .

Summary

This paper introduces DIFT, a method to yield emergent correspondence from image diffusion models without training or additional fine-tuning. The method is simple - given an image (or an image pair), DIFT adds noise to the image to simulate the forward diffusion process, and pass it to the U-Net of a pretrained diffusion model to extract feature maps. The authors discover that by simply computing the cosine similarity between the emergent feature maps, one can establish strong semantic, geometric and temporal correspondences without training.

Strengths

* The discovery of emergent correspondences from image diffusion is novel. The proposed method is surprisingly simple as well, and this opens new possibilities and research directions for future work. * Strong performances on standard benchmarks of semantic correspondence, geometric correspondence and temporal correspondence. * The writing is clear and easy to follow.

Weaknesses

* Missing evaluation of PF-PASCAL for semantic correspondence. This is not a critical drawback, as results on PF-PASCAL tend to be saturated. * Incomplete implementation details. What was the image size / feature map size used to establish the correspondences? Image size is a critical factor in many image correspondence methods. * Lack of latency and computation analysis. This is crucial to identify the applicability of DIFT to real world scenarios. * Lack of rationale or analysis on how exactly the image diffusion models can yield such feature maps easily. While some motivation is provided in the introduction, it seems insufficient for the readers to fully understand how the correspondences are emerging from image diffusion. Analyzing some failure cases could be helpful.

Questions

Please refer to the weaknesses section. The idea, simplicity, and novelty of the paper is strong, but I believe the paper lacks the experimental details and analyses to be accepted as-is.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

4 excellent

Limitations

The authors have included the ethical considerations in the paper, but not the limitations.

Summary

The authors proposed a method for semantic correspondence using pretrained diffusion model as a feature extractor of the images. Without explicitly training on the additional data/annotations, a simple feature matching based on winner-take-all strategy with cosine distance metric surpasses the previous works on three different tasks; semantic/geometric/temporal correspondence.

Strengths

- Exploration on the usage of diffusion model for correspondence tasks - Good performances just with frozen pretrained model & simple matching pipeline

Weaknesses

1. Limited novelty - Merely replacing the backbone network for feature extraction with the latest model in a straightforward manner does not captivate my interest. It also fails to provide new insights to the readers. It would be beneficial to delve deeper into why these pretrained diffusion models outperform previous backbones like DINO, CLIP, or ResNet. 2. Limited demonstrations - The paper lacks several essential ablation studies that explore architectural design choices. These studies could include comparing models with and without a decoder, evaluating different options for Q/K/V selection, and assessing the impact of finetuning on correspondence datasets. - Given the relatively low originality, the paper would benefit from showcasing more diverse pixel-level prediction tasks. This could involve demonstrating the model's performance in object detection, image/ video segmentation, as well as other correspondence tasks such as depth estimation and optical flow. These additional demonstrations would significantly strengthen the paper.

Questions

It would be interesting to see how the performances can be pushed further when sophisticated optimization techniques are applied to the raw matching similarities d(f^1,f^2).

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations and potential societal impact of this work are appropriately discussed

Official comment by reviewer 4nEG

Thanks to the authors for the detailed rebuttal and clarifications regarding experimental setup. I believe that these clarifications, together with the additional experiments, will consolidate the paper. My concerns have been addressed and I maintain a recommendation for accept.