Diffusion models have been shown to be capable of generating high-quality images, suggesting that they could contain meaningful internal representations. Unfortunately, the feature maps that encode a diffusion model's internal information are spread not only over layers of the network, but also over diffusion timesteps, making it challenging to extract useful descriptors. We propose Diffusion Hyperfeatures, a framework for consolidating multi-scale and multi-timestep feature maps into per-pixel feature descriptors that can be used for downstream tasks. These descriptors can be extracted for both synthetic and real images using the generation and inversion processes. We evaluate the utility of our Diffusion Hyperfeatures on the task of semantic keypoint correspondence: our method achieves superior performance on the SPair-71k real image benchmark. We also demonstrate that our method is flexible and transferable: our feature aggregation network trained on the inversion features of real image pairs can be used on the generation features of synthetic image pairs with unseen objects and compositions. Our code is available at https://diffusion-hyperfeatures.github.io.
Paper
References (50)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The authors propose a method to extract per-pixel feature descriptors from multi-scale and multi-timestep feature maps generated by diffusion models. These descriptors can be utilized for various downstream tasks. The framework is evaluated on the task of semantic keypoint correspondence, specifically on the SPair-71k real image benchmark. The authors claim that their method achieves superior performance on this benchmark compared to other approaches. Additionally, they demonstrate that their method is flexible and transferable, as the feature aggregation network trained on real image pairs can also be used on synthetic image pairs with unseen objects and compositions. Overall, the proposed Diffusion Hyperfeatures framework aims to enhance the internal representations of diffusion models and improve their utility in computer vision tasks like semantic keypoint correspondence.
Strengths
This paper discusses a framework called "Diffusion Hyperfeatures" for consolidating feature maps in diffusion models. Here's a substantive assessment of the strengths of the paper across various dimensions: Originality: The paper introduces a novel approach to consolidate and extract per-pixel descriptors from intermediate feature maps generated by diffusion models. While existing works focus on selecting specific layers and timesteps, this paper proposes a comprehensive framework that considers all intermediate features. This consolidation process through a feature aggregation network is a new concept, which contributes to the originality of the paper. Quality: The paper appears to be of high quality as it references relevant literature and state-of-the-art techniques in computer vision, such as ConvNets, Vision Transformers, and GANs. The authors also evaluate their proposed framework on the task of semantic keypoint correspondence using real images from the SPair-71k benchmark. The evaluation includes an analysis of different layers and timesteps of diffusion model features, demonstrating a thorough investigation. Additionally, the generalization of Diffusion Hyperfeatures to out-of-domain data is examined by evaluating them on synthetic images generated by the diffusion model. Clarity: The paper presents its ideas and contributions in a clear manner. It provides a concise overview of the challenges faced with existing methods and describes the proposed Diffusion Hyperfeatures framework step-by-step. The use of terminology and technical language is appropriate for the target audience of researchers in computer vision. However, without the full paper, it is difficult to assess the clarity of the detailed methodology and experimental setup. Significance: The paper addresses an important problem in computer vision, namely how to effectively utilize the internal representations of diffusion models for downstream tasks. By proposing the Diffusion Hyperfeatures framework, the authors aim to improve the utility of diffusion models beyond image generation. The evaluation results on semantic keypoint correspondence and the demonstration of generalization to synthetic images indicate the potential significance of this approach. If the proposed framework proves to be effective, it could contribute to advancements in various computer vision applications. Overall, based on the provided text, the paper demonstrates strengths in terms of originality, quality, clarity, and potential significance.
Weaknesses
Based on the provided paper, here are a few potential areas where the paper could benefit from improvement: Comparison with Existing Methods: While the paper mentions that features from ConvNets, Vision Transformers, and GANs have demonstrated significant capabilities, it would be beneficial to provide a more detailed comparison with existing methods that extract feature descriptors from diffusion models. This would help establish the novelty and superiority of the proposed Diffusion Hyperfeatures framework. Experimental Evaluation: The paper briefly mentions evaluating the proposed framework on the task of semantic keypoint correspondence using real images from the SPair-71k benchmark. To strengthen the paper, it would be valuable to include a thorough analysis of the experimental results, including quantitative metrics, comparative evaluations with state-of-the-art methods, and possibly visual illustrations or qualitative assessments of the generated keypoint correspondences. Generalization to Diverse Domains: The paper mentions evaluating Diffusion Hyperfeatures on synthetic images generated by the diffusion model. While this demonstrates some level of generalization, it would be useful to explore the performance and robustness of the framework across a broader range of datasets and domains. This could involve testing the framework on different benchmark datasets and real-world scenarios to validate its effectiveness in diverse settings. Clear Methodology Description: The provided text offers a high-level overview of the proposed framework, but for a comprehensive evaluation, it is crucial to have a clear and detailed description of the methodologies employed. It would be helpful to include information about the architecture of the feature aggregation network, the specific techniques used for consolidation, and any additional preprocessing steps or modifications made to the diffusion model. Potential Limitations and Future Directions: The paper could benefit from discussing potential limitations or challenges associated with the proposed framework. Identifying these limitations and suggesting future directions for improvement would enhance the depth and completeness of the research.
Questions
Clarification on Consolidation Process: Could the authors provide more details regarding the feature aggregation network used to consolidate the intermediate feature maps? How does it handle variations in scale and time? Are there any specific design choices or architectural considerations that impact its performance? Experimental Setup: It would be helpful to understand the specific experimental setup employed in the evaluation of Diffusion Hyperfeatures. Could the authors elaborate on the dataset used, the selection of keypoint correspondence task, and the metrics used for evaluation? Additionally, is there any consideration given to computational efficiency or runtime performance when applying the framework to real-time applications? Comparison with Existing Methods: The authors briefly mention that existing works select specific subsets of layers and timesteps from diffusion models for different tasks. Could the authors provide a more detailed comparison or discussion on how their approach differs and potentially outperforms these existing methods? Are there any specific limitations or challenges associated with the subset selection approach that Diffusion Hyperfeatures address? Transferability and Generalization: While the paper mentions generalization to synthetic images, it would be valuable to explore the transferability and generalization capabilities of Diffusion Hyperfeatures across other domains or datasets as well. Can the authors provide insights or experiments on how the framework performs on different benchmark datasets or real-world scenarios beyond the SPair-71k dataset?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
Yes
Summary
This paper proposes diffusion hyperfeatures, a framework for integrating different scale and timestep features to form a representative feature descriptors in dense level. Within the U-Net architecture, unlike other works that uses hand-crafted methods to select a particular subset of layers for further processing, this paper proposes a simple aggregation network that aggregates the intermediate features. The effectiveness of this method is evaluated on standard benchmark of semantic correspondence, including SPair-71k.
Strengths
1. SOTA performance is achieved with large gap to existing works. 2. One of the first attempt to tackle semantic correspondence task with diffusion. 3. Although paper is easy-to-read, each paragraphs are too long. Also it is quite hard to understand some sentences. This is a minor issue though.
Weaknesses
1. This paper is one of the first attempt to tackle semantic correspondence with diffusion concept. In this manner, this paper looks novel, since this is very new to this task. However, to me, this work severely lacks contributions. What this paper does is simply using a series of features from stable diffusion, a large-scale model, feeds to a simple aggregation network that simply performs weighted sum to obtain the final features and it is exploited for Winner-Takes-All (WTA) for finding correspondences. (1) An attempt to select features are not novel as this is already smartly done in HPF [23] and DHPF [25], while this work simply uses all the features to feed to an aggregation network. (2) Using different feature representation that is much powerful than standard backbone (resnet 101) will guarantee apparent performance boosts. This is already demonstrated in CATs++: Boosting Cost Aggregation With Convolutions and Transformers (TPAMI). (3) No novel diffusion techniques are proposed as far as i understood. 3. Lacking implementation details. What resolution was used for evaluation and training? This is very important in this task as the performance highly correlates to it. Unless this is evaluated in the same resolution and compared with other works, the comparisons are not fair at all. 4. Why did the authors use only 2 datasets for evaluation? Traditionally, existing methods would also evaluate their methods on Pf-PASCAL, PF-WILLOW, TSS as well. This needs to be included, since evaluation on 2 datasets seem very insufficient. 5. In supplementary material, section 6.1 explains computational resources. This is very abstract and not very helpful. Comparison to other works is not a must, but at least provide some measures for completeness. 6. DINOv1 is a quite old-model, since DINO-v2 is already out. DINO-v2 is a much powerful model that will yield much stronger performance. 7. Many other semantic correspondence works are not cited and compared. As far as I know, the current SOTA is Integrative Feature and Cost Aggregation with Transformers for Dense Correspondence, arxiv'22, which would be better to be included in the paper. Also when it is compared with DINO, only 1 layer is used. In semantic correspondence, multi-scale and multi-level features are highly important which would have detrimental effects on performance without them. So the comparison may not be fair here as well. However, as the authors also provided results for single layer results, this is not an issue, but a suggestion to include results that use more layers of DINOv1 or DINOv2.
Questions
See weaknesses above
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
1 poor
Contribution
2 fair
Limitations
Limitations or failure cases are not presented.
Summary
This paper proposes improving feature distillation from diffusion models for representation learning by aggregating information from the feature maps of the U-Net at varying timesteps, weighting them with a tunable aggregation network. The authors show that even at timesteps from which features are usually discarded for feature learning (i.e. early steps in the generation process) useful features can be found. After formulating their approach to extracting these features using a weighted aggregation of a set of standardized input feature maps, authors show validity of their approach by improving considerably over baseline methods and prior work that only selects a single diffusion timestep for feature extraction. Lastly, authors show the ability of their framework to generalize to unseen synthetic data (generated by the diffusion model), by extracting features from the generation process for a diffusion model trained on a different data distribution.
Strengths
- The idea of extracting features from powerful generative models such as diffusion models is very relevant, and the authors show that their approach to this extraction process obtains great results. - The paper is very well-written and reads very smoothly. Authors are very descriptive in their wording and use figures to clearly illustrate their approach. The idea is simple and seems very effective, as authors illustrate with a number of experiments and ablation over baselines. - Argumentation for design choices is very solid, and authors motivate their approach with an experiment showing the validity/salience of features found in earlier timesteps of the diffusion model. - The authors also show impressive transfer performance to an unseen dataset, highlighting possible applications in pseudo-label generation for semantic correspondence.
Weaknesses
- I have no major concerns. A minor concern I have is with regards to novelty, given that extracting hypercolumn features has been attempted before, even for diffusion models. - Another minor concern is the requirement for task-specific fine-tuning of the aggregation network. The fact that this network requires explicit supervision for a task means that the actual representation extracted from the diffusion model that can be used in arbitrary downstream tasks is the whole set of features maps across timesteps. Would it be possible to train an aggregation network on a self-supervised task?
Questions
- For figures 2, 3, would it be possible to show also the network input at these timesteps for reference? - Line 117, you mention that “these observations indicate that the diffusion model provides coarse and fine features that capture different image characteristics, throughout different combinations of layers and timesteps”. I’m wondering to what extent this is owing to the formulation of the diffusion process itself, and to what extent this is due to the use of a very specific multi-resolution processing architecture (U-Net) in the diffusion process. I.e. is the extraction of both coarse and fine image features a result of diffusion or a result of using a U-Net? Would your method also work with diffusion models that make use of other architectures? - Line 135, I have a bit of trouble understanding your reasoning for the more “trustworthy” inversion features. Could you elaborate? Why would repeated application of the model necessarily lead to more “trustworthy” features? - You apply your feature extraction specifically for semantic correspondence detection. Could the extracted features also be used in other settings? Are there any limiting factors that prevent your framework from being used in such settings? Have you tried e.g. classification of extracted features?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
2 fair
Limitations
The authors do not discuss the limitations or societal impact of their work.
Summary
This paper proposes an approach for extracting useful features for pre-trained diffusion models for application to dense visual correspondence tasks. How to do this is not clear due to presence of features both through the network, and over diffusion steps. The proposed approach is to learn which features to use using the hypercolumns framework, by passing activations through bottleneck layers then averaging over layers and time. Experiments show that this method outperforms unsupervised DINO features and supervised hypercolumns.
Strengths
- The premise of extracting useful features from diffusion models is an important one; we know that those features are present but it is not clear how to extract useful ones. - Using inversion to get features for real images is a good idea and the explanation that they are more reliable than sampling from the posterior as used in prior work makes sense. - The exploration into what feature representations are learned at different layers/times is interesting (Figures 2 and 3). - The approach is shown to outperform unsupervised DINO features as well as hypercolumns (Table 1) showing that useful features have been extracted from the diffusion model; the baselines including using a single layer from the diffusion model demonstrate the benefit of combining over layers/time; and visual examples shown in Figures 4-6 demonstrate the approach working.
Weaknesses
- Unlike using DINO features, the proposed approach requires training additional components for downstream tasks, making it much less versatile. - The baseline models in Table 1 are a poor representation of current approaches. DINO features are not trained specifically for keypoint matching and newer supervised approaches such as CATS++ (Cho et al. TPAMI 2022) and VAT (Hong et al. ECCV 2022) substantially outperform DHPF. - The approach shares bottleneck layers across time steps; this seems potentially problematic since features differ substantially across different times.
Questions
- How does it quantitatively compare against more recent supervised approaches? - Why share bottleneck layers over time steps? They contain very different representations. - Why not use attention features from the diffusion model in a similar manner to DINO? - Aggregating features over time steps is difficult, hence the trained layers on top. Could using a distilled diffusion model such as a consistency model give better representations?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
As stated in the submission details, limitations are not discussed. There are a number of limitations including having to train additional components (unlike DINO), not comparing with more recent supervised methods, and sharing layers across time steps.
Summary
This paper explores semantic correspondence tasks with stable diffusion model. Specifically, the authors proposed to first extract feature maps varying across timesteps and layers from the diffusion process and trains a lightweight neural network to aggregate them together for semantic correspondence. Experimental results on CUB-200 and SPair-71k datasets show that proposed method outperforms other baselines.
Strengths
- This paper shows that one could explore diffusion models for semantic correspondence tasks. - the writing is clear and easy to follow. - The experiments show that the proposed method could outperform other baselines on the SPair-71k real image benchmark. - the authors have shown numerous visual examples to demonstrate the correspondence capability.
Weaknesses
1. The proposed method, that "distill the information distributed across time and space from a diffusion process into a single descriptor map", seems to have more potential than correspondence tasks. Have the authors explore other tasks, like video label propagation, Homography estimation or perception tasks like classification? 2. Concurrent works: there are multiple papers presenting correspondence ability of diffusion models: [1*] Hedlin, Eric, et al. "Unsupervised Semantic Correspondence Using Stable Diffusion." arXiv preprint arXiv:2305.15581 (2023). [2*] Tang, Luming, et al. "Emergent Correspondence from Image Diffusion." arXiv preprint arXiv:2306.03881 (2023). [3*] Zhang, Junyi, et al. "A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence." arXiv preprint arXiv:2305.15347 (2023). Could the authors explain the differences if possible?
Questions
please refer to the weaknesses section.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
no
response (1/2)
Prior to my response, I would first like to thank the authors for their thorough responses. From the rebuttal, the following concerns of mine remain unresolved. (1) **Novelty**: From the authors' response regarding the novelty, authors first mention that the main contribution lies on "proposing a simple and general framework for utilizing features across the diffusion process". As I already mentioned in my initial review, I agree that bringing semantic correspondence with diffusion concept is novel. However, this will be a sufficient contribution to pass the standard of top-tier conference in this field only if sufficient, persuasive and informative quantitative and qualitative analysis, results and experiments are presented. Taking examples of the existing works, [A] and [B], I believe this paper presents an analysis that had already been investigated more thoroughly in the previous paper [A]. For example, [A] carefully visualizes feature maps of each time steps to find what semantics or what represenetations are learnred. Also, compared to the concurrent work [B], which has a solid motivation (why diffusion should be incorporated into this task) and detailed analysis with sufficient visualizations, this paper only visualizes intermediate features as if they are intermediate features of a feature backbone. I will list some of the other papers that also perform such investigation in a more thorough way: [C,D] .I got an impression that the authors are simply using diffusion model as a very heavy **"feature backbone"**. In this manner, it simply looks like A works well, let's use it for task B. Without a solid motivation, analysis and investigation, I believe this impression is unavoidable. Moreover, the authors mentioned **our final method achieves state-of-the-art performance on the SPair-71k benchmark, even when compared to DINOv2 and CATS++ (Table 1 of the global response)**. I want to first ask that is the performance contribution of this paper? From my understanding, as it is already shown in [B] as well, simply using DINOv2 feature or SD feature to find correspondences with Winner-Takes-All method already yields SOTA performance in this task. This means that it is the contribution of those backbone networks, not this paper's contribution. In semantic correspondence literature, many works have been proposing novel ideas to perform well under situations where background clutters or intra-class variations pose additional challenges. However, in this paper, SOTA is achieved by simply using the features to feed into the simplest aggregation network. To be strictly speaking, it's not SOTA as well, because a concurrent work [B], achieves higher performance with more judicious use of diffusion feature along with DINO. Now I want to talk about the technical novelty as well. I have four concerns. (1) Whether a means to develop diffusion techniques? (2) Whether a means to effectively or efficiently aggregate the selected features is proposed? (3) Whether a means to select the hyperfeatures are justified by thorough analysis and experimental results? As far as I understood, (1) is not proposed. Also investigations of feature maps or representations in diffusion models are already conducted in the referenced papers below. (2) novel aggregation approach is also not proposed, as it is just a concatentation and feeding into a network. CHM, TransforMatcher, NC-Net and many other works propose to aggregate the inputs in novel ways, while this approach looks not novel to me. (3) DHPF and HPF propose novel ways to select the features, provide sufficient analysis to justify the selections. However, in this paper, features are used as if they are just from an another backbone network. (2) **Resolution** : In semantic correspondence task, indeed evaluation resolution is important, but this is also the same for training resolution as well. This is why CATs++ provided experimental results that verify the impacts of the resolution. I understand that training on low resolution may pose challenges in this framework, as this framework needs to exploit pretrained weights. So this is just a minor concern that will not affect largely to my rating. (3) **incomplete experiments** : In the initial submission, the fact that only two datasets were used for evaluation poses a concern. It seems like the submission was an unfinished product, that was complemented by the reviews. [A] LABEL-EFFICIENT SEMANTIC SEGMENTATION WITH DIFFUSION MODELS ICLR'22 [B] A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence arxiv'23 [C] Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation [D] Diffusion Models already have a Semantic Latent Space
1. *Novelty.* We would like to re-iterate the Neurips review policy regarding comparisons to recent work: “Authors are not expected to compare to work that appeared only a month or two before the deadline.” **[B] appeared in an unofficial preprint after Neurips submission deadline, and should be considered concurrent to our work.** Nevertheless, we believe that our submission already presents “persuasive and informative quantitative and qualitative analysis, results and experiments” which is confirmed by the other reviewers: - “The idea is simple and seems very effective, as authors illustrate with a number of experiments and ablation over baselines.” (Reviewer sN43) - “Argumentation for design choices is very solid, and authors motivate their approach with an experiment showing the validity/salience of features found in earlier timesteps of the diffusion model.” (Reviewer sN43) - “The exploration into what feature representations are learned at different layers/times is interesting (Figures 2 and 3).” (Reviewer HmuP) 1a. *Whether a means to develop diffusion techniques?* We are unsure of why the reviewer thinks that previous work investigating diffusion features for other tasks diminishes our contribution. Our work shows how various naive applications of diffusion models to semantic correspondence (picking a single layer, concatenating across all layers) aren’t fully utilizing the representations present across layers and timesteps, and present an effective method to aggregate this information. We are also the first to leverage features from the inversion process, which we discuss in Figure 3 of the main paper and further ablate in Figure 1 of the global response PDF. **Reviewer HmuP notes this as a strength: “Using inversion to get features for real images is a good idea and the explanation that they are more reliable than sampling from the posterior as used in prior work makes sense.”** 1b. *Whether a means to effectively or efficiently aggregate the selected features is proposed?* While the concept of feature aggregation isn't new, our approach has distinct merits. We are the first to aggregate diffusion features into a single concise descriptor, where our aggregation of features reduces the memory consumption by 300x (1.8 GB to 6 MB) compared with naive concatenation, described further in Table 8 below. Furthermore, our network's design, including the use of shared bottleneck layers across timesteps, further reduces memory usage by 31x, as discussed further in the global response. 1c. *Whether a means to select the hyperfeatures are justified by thorough analysis and experimental results?* We analyze the selection of features learned by our model in Figure 5 of the main paper, which should be comparable to Figure 3 of DHPF [6] which similarly analyzes layer selection frequencies. We are the first to learn this selection not only across model layers but also diffusion timesteps. We provide a detailed discussion of our learned selection in L238-L256 in the main paper and Section 6.2 of the Supplementary, where we also discuss how the layers and timesteps at which the features are most “semantic” differ across Stable Diffusion variants. **Reviewer HmuP agrees that our approach “is shown to outperform [...] hypercolumns (Table 1) showing that useful features have been extracted from the diffusion model; the baselines including using a single layer from the diffusion model demonstrate the benefit of combining over layers/time.”** Overall we believe that the argument that our method is not novel because its components utilize existing techniques is unfair as most works are compositions of existing methods. We explore naive applications of diffusion models to semantic correspondence, show that these methods rely heavily on hand-selected hyperparameters and fail to fully utilize the information present across time steps and layers, and propose a method to aggregate this information in a memory-efficient manner. 3. *Experiments. In the initial submission, the fact that only two datasets were used for evaluation poses a concern.* Respectfully, we disagree with this sentiment. We evaluate SPair-71k and CUB because these were specifically proposed to overcome the limitations of previous semantic correspondence datasets which “do not display much variability in viewpoint, scale, occlusion, and truncation” [8]. It has also been stated in prior work that “PF-PASCAL [...] is almost saturated, which makes a comparison difficult” [9]. While 2 datasets may seem like a limited evaluation, we believe the value of an evaluation lies in the quality of the datasets rather than the quantity. Furthermore, we would like the point out that it is common to address reviewer feedback with additional experiments. For example, the OpenReview page for CATS [10] reveals that the reviewers’ decision was swayed positively because their “concern[s] have been addressed [...] thanks to the additional experiments.”
response( 2/2)
(4) **computational complexity** : In section 6, the paper says " our method is fast and uses a reasonable amount of memory". What I meant in the initial review was that this sentence does not provide any clues of how much memory footprint, run-time and all other computation related measurements (e.g., FLOPS). I don't understand why the authors are asking the reviewer again for the suggestion of computational metric. For now, my current thoughts are these. I am happy to have a discussion whole this week. Looking forward to author's response. Thanks.
4. *Computational Complexity. [...] clues of how memory footprint, run-time and all other computation related measurements (e.g., FLOPS).* In Table 8 we give precise run-time and memory statistics of our method, including (a) the size of the descriptor used for the method’s underlying matching algorithm and (b) the average inference time of feature extraction and matching for each pair on the SPair-71k dataset. Our method is able to consolidate the same large set of features from a diffusion model from 1.8 GB (SD-Concat-All, which uses naive concatenation) to 6 MB (Ours). While our full method explores the upper bound by utilizing all available features across the diffusion process, which takes 6.62s, one can also use the same pretrained weights to evaluate faster pruned versions of our model. Below we show running our method after stopping the diffusion process after 1, 5, and 10 timesteps. The pruned variant of our method that utilizes the first 10 timesteps performs close to our full method, with a 4\% improvement in PCK\@0.1_img over DINOv2 with an almost 2x faster inference process. Note that the inference time for our DINOv2 baseline is relatively slow because it uses the method from Amir et. al. [3] which includes a log binning algorithm to contextualize the features into descriptors. **Table 8. Memory and run-time comparison.** | Model | SPair-71k PCK\@0.1_img | Memory per Descriptor | Inference Time per Pair (s) | |:---------------------|:-----------:|:------------:|:------------:| | DINOv2 | 68.33 | 75 MB | 2.99 | | CATS++ | 70.26 | 131 MB | 0.16 | | SD-Layer-4 | 58.80 | 10 MB | 0.33 | | SD-Concat-All | 52.12 | 1.8 GB | 0.87 | | Ours (1 timesteps) | 64.61 | 6MB | 0.28 | | Ours (5 timesteps) | 69.28 | 6MB | 0.86 | | Ours (10 timesteps) | 72.00 | 6MB | 1.60 | | Ours (50 timesteps) | 72.56 | 6MB | 6.62 | [B] Zhang et. al. "A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence." arXiv 2023.\ [6] Min et. al. “Learning to Compose Hypercolumns for Visual Correspondence.” ECCV 2020.\ [7] Podell et. al. “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis.” arXiv 2023.\ [8] Min et. al. “SPair-71k: A Large-scale Benchmark for Semantic Correspondence.” arXiv 2019.\ [9] Hong et. al. “Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation.” ECCV 2022. [10] Cho et. al. “CATs: Cost Aggregation Transformers for Visual Correspondence.” Neurips 2021.
Thanks for the detailed responses. I want to clarify that my concern was not that the paper does not compare with the concurrent work, but rather, as the other reviewer commented, the SOTA performance would be because of SD backbone, and without a thorough ablations and analysis, contributions of this paper will diminish. Moreover, the reason why I thought that how other diffusion works showed the characteristics of SD features in various tasks diminish the contributions of this paper is that since it is now well known that using such a strong features will guarantee good performance even with simple aggregation networks. So I thought that the impact this paper can have on the community will be insignificant based on the initial submission draft. One last question is that since this approach requires text prompt for inversion, how does the model infer at test time? Does prompt should be fed manually? Other than above, I am now convinced by the authors' responses. One last concern is that I personally feel like the amount of revisions, including additional experiments, computation table and ablation studies conducted in this rebuttal period is quite significant. However, I believe this is something others might find different, so again, I highly appreciate the responses.
We would like to thank the reviewer for taking the time to review our submission and provide feedback. We greatly appreciate the opportunity to discuss and provide the below answers for the reviewer’s final concerns. 1. *The SOTA performance would be because of SD backbone* Per our response to Reviewer HmuP, the experiments shown in Table 9 validate that aggregation over the layers *and timesteps* of a diffusion model outperforms aggregation over layers from a strong backbone. **While training an aggregation network improves DINO’s performance over raw features, our method that operates on top of Stable Diffusion features still performs the best at 72.56\% PCK\@0.1_img, compared with 54.69\% and 68.37\% PCK\@0.1_img for DINO and DINOv2 respectively.** In a similar fashion, our final method also outperforms the variant that only aggregates over layers (Ours - One-Step), which achieves 63.47\% PCK\@0.1_img. As such, we validate that the unique combination of aggregating both over layers and timesteps over diffusion features is what leads to our superior performance. 2. *this approach requires text prompt for inversion, how does the model infer at test time* **When operating on real images using the inversion process, we feed the empty prompt “”.** We find that extracting features only from the unconditional model removes the need for manual prompting and yields high-quality results. When operating on synthetic images from the generation process, we extract features only from the conditional model, or the branch conditioned on the prompt that generated the image.
I've checked the comments, and now I think this paper is strong if these rebuttals are adequately incorporated. Thanks for the efforts to answer all my concerns.
Response to Authors
I would like to thank the authors for their responses. I still have concerns regarding the evaluation, however, the rebuttal did address my concerns to some degree and I will increase my score to borderline accept accordingly. In particular, I thank the authors for the experiment showing the impact of having different bottleneck layers per time step, this addresses my concern that sharing across all steps could be problematic. Similarly, the explanation on resblock vs attention features addresses that question. The added quantitative comparison to another supervised approach (CATS++) is very informative and it is nice to see that this approach is able to outperform existing supervised methods, going some way to address my concerns on baselines; as do the added results on other datasets/applications and generalisation to different domains. My main remaining concern is still the lack of baselines meaning that it is hard to determine concretely where performance improvements come from - e.g. is it from hyperfeatures + a stronger feature backbone, or is it from the diffusion model in particular. As such, I agree with the other reviewers’ concerns on novelty (I have not taken into account the concurrent diffusion semantic correspondence papers mentioned by the other reviewers); aggregating over time is a small novelty, but in my opinion the main contribution comes from the experiments showing that diffusion features are especially useful for this task and worth using over other faster to obtain features, which I think is lacking. While the paper does this to some degree, and the added supervised results help, as mentioned by the other reviewers, there are many more methods including more supervised (e.g. IFCAT), other generative models, aggregation networks on self-supervised networks, existing methods that extract feature descriptors from diffusion models, etc. that it would be really informative to compare/ablate against to show that the features learned by the diffusion model really are especially useful. To summarise, since the problem of extracting features from diffusion models is of interest and as added in the rebuttal the approach appears to perform well compared to recent supervised approaches, I have increased my score. But this paper would much stronger if we knew better where the improvements come from through a stronger set of baselines/ablations, so I have only increased my score to borderline accept. Happy to discuss with the authors if they disagree with any of these comments.
We would like to thank the reviewer for taking the time to review our additional results and for providing feedback regarding our evaluation. In Table 9, we ablate the effect of single layer selection, naive concatenation, and training an aggregation network for DINO and DINOv2, symmetric to the ablations we performed for our method. Note that to ensure a consistent evaluation across all backbones, we use the same input resolution of 224. Training an aggregation network noticeably improves the performance of both DINO and DINOv2 by 3\% and 8\% respectively. Interestingly, single layer selection of both DINOv2 and Stable Diffusion features (SD - Layer 4) performs comparably, but training an aggregation network on top of the Stable Diffusion features (i.e., over both timesteps and layers) yields a larger relative boost (14\% PCK\@0.1_img) than DINOv2. **Ultimately, our method that trains an aggregation network on top of Stable Diffusion features performs the best at 72.56\% PCK\@0.1_img, compared with 54.69\% and 68.37\% PCK\@0.1_img for DINO and DINOv2 respectively.** In Table 10 we also verify that our aggregation network on top of DINO features learns mixing weights consistent with the hand-selected features explored in Amir et. al. [3], where it indeed learns that Layers 9 - 11 are particularly useful for the semantic correspondence task. Since IFCAT [4] does not have publicly available code, we display the result reported in the original paper. As such, with these additional baselines we validate that the strong performance is not only from the strong backbone but also the fact that we are aggregating across the layers and timesteps of a diffusion model in particular. **Table 9. SPair-71K** | Model | PCK\@0.1_img | PCK\@0.1_bbox | |:---------------------|:-----------:|:------------:| | IFCAT [4] | - | 64.40* | | | | | DINO [3] | 51.68 | 41.04 | | DINO - Concat All | 20.17 | 13.60 | | DINO + Aggregation Network | 54.69 | 44.29 | | | | | DINOv2* | 60.14 | 46.94 | | DINOv2 - Concat All | 60.89 | 47.69 | | DINOv2 + Aggregation Network | 68.37 | 56.35| | | | | SD - Layer 4 | 58.80 | 46.58 | | SD - Concat All | 52.12 | 41.83 | | Ours | **72.56** | **64.61** | |*Since IFCAT does not have publicly available code, we take the result reported in the original paper.| |*In our previous reported experiments with DINOv2 in Tables 1-4, we used input images of resolution 770 to account for its large patch size. To ensure a fair evaluation in this table, we use input images of resolution 224 for all methods.| **Table 10. DINO + Aggregation Network Learned Mixing Weights** | Layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11| |:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---| | | 4% | 3% | 3% | 3% | 4% | 8% | 5% | 9% | 11% | **14%** | **14%** | **14%**| [4] Hong et. al. “Integrative Feature and Cost Aggregation with Transformers for Dense Correspondence.” arXiv 2022.
Response to Rebuttal
I would like to thank the authors for taking the time to answer mine and other reviewers questions and concerns. Reviewer c4Xr makes a number of solid points, especially his concerns regarding lacking experimental details seem important to resolve before finalizing this submission. The authors provide a solid rebuttal with extensive new experiments. I understand reviewer c4Xr's concern regarding the impact of these large modifications to the review process, but I personally think it's a valid way of improving the manuscript's quality. I think these modifications should be weighed in favour of acceptance in the final decision, as they strenghten the manuscript. I stand by my recommendation of acceptance, but understand the reluctance of other reviewers. I think the idea is simple and effective, and after rebuttal I think the manuscript is strong and reads well.
I acknowledge I have read the rebuttal.
Decision
Accept (poster)