Where's Waldo: Diffusion Features for Personalized Segmentation and Retrieval

Summary

The authors propose a novel approach without any additional training for Personal retrieval and segmentation tasks, which focus on identifying specific instances within a dataset. The existing methods struggle to locate a desired instance when other instances within the same class are presented. They propose to utilize the features from a pre-trained text-to-image diffusion model and design a Personal Features Diffusion Matching method. The proposed method outperforms on both existing datasets and the new dataset which is proposed in this paper.

Strengths

+ The idea of introducing the features from the text-to-image diffusion models to handle the personal retrieval and segmentation problem is interesting. + The authors propose a new approach called PDM which requires no training or fine-tuning, no prompt optimization, or any additional models. + The authors explore the hidden textural and semantic information in the layers and blocks in the diffusion model. + The authors construct a new benchmark includes videos with multiple instances from the same category.

Weaknesses

- The logical flow before the Method Section of this paper needs to be adjusted. It is hard to understand what the authors want to do and the problem they want to address, although the utilization of features from diffustion models and the PDM itself is somewhat clear. - The form of the task in the this paper is not clear. "... a dataset based on an input image and a short description of the reference instance." in Line.1-2 "this work focuses on the case where only class names are provided." Line.102-103 ... Line.103-105 As observed in the figures, the input information includes reference images, target images, and the class name (some of them are optional) - I think the SUPERVISED models for personal retrieval and segmentation, which need a large amount of annotated data, and the SELF-/WEAKLY-SUPERVISED foundation models are not in the same aspect. The authors should rearrange the related works and the motivation of their work. - The authors should list their contributions to help the readers better understand thier work. - The visualized results in both the main text and appendix do not fully capture the problem the authors aim to address. Few of them present the situation where a desired instance appears together with other instances of the same class. - There is a lack of sufficient theoretical justification for why the authors choose diffusion models and why the features of diffusion models can achieve the capabilities that the authors expect. Only using "These models have the capability to generate an infinite array of objects and instances, each exhibiting unique appearances and structures. Line.33-34" and the two examples in Figure 2 is not enough.

Questions

Please refer to the *Weakness* part. Additional comments: 1. The authors need to clearly define the form of the task and formally present the settings for differernt scenarios. 2. The Figure 3 is hard to understand. Why "dog" is not fed into the diffusion features extraction in I_t branch? As F^s_t is features from the cross attention, what information is used to calculate the cross attention with I_t? 3. The authors should explain why using the features from diffusion models and provide theoretical justification. They should figure out the differences between the diffusion models and the other models (e.g. common resnet, ViT, Vision Mamba, Transformer+KAN, ..., and the other variants, why not choose other SoTA foundation models). Why they can present the textural and semantic information?

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

NA

Summary

The paper introduces Personalized Diffusion Features Matching (PDM), a novel zero-shot approach that utilizes pre-trained text-to-image diffusion models for personalized image retrieval and segmentation without requiring additional training. PDM extracts and fuses semantic and appearance features from an off-the-shelled diffusion model to accurately identify and segment unique instances, even when multiple similar objects are present. It demonstrates superior performance on various benchmarks, outperforming both self-supervised and supervised methods. The authors also address limitations in current datasets by proposing new benchmarks that include challenging scenarios with multiple instances from the same category. Despite its reliance on image inversion, which may affect performance, PDM offers a significant advancement in the field of personalized instance retrieval and segmentation.

Strengths

1. The paper is well organized, clearly written, and easy to follow, especially for the motivation clarification. 2. The experimental observation and justification of the effect of self-attention and cross-attention in the diffusion model are insightful. 3. The proposed method is technically sound and well-motivated. 4. The contribution is sound, consisting of new observations, methods, and more practical benchmark settings. 5. The proposed method is training-free and seems to have great transferability.

Weaknesses

I didn't find a major weakness

Questions

1. It is suggested that some works related to semantic-level feature matching and their applications [1,2] be added to the related work section. 2. (Just constructive suggestions for future works). I am curious whether a similar or more profound phenomenon about attention will appear in DiT [3] and stable video diffusion [4]. Can the method be extended to these more advanced architectures [3,4]? [1] Lindenberger, P., Sarlin, P. E., & Pollefeys, M. (2023). Lightglue: Local feature matching at light speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 17627-17638). [2] Li, W., Liu, X., & Yuan, Y. (2022). Sigma: Semantic-complete graph matching for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5291-5300). [3] Peebles, W., & Xie, S. (2023). Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4195-4205). [4] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., ... & Rombach, R. (2023). Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

N/A

Summary

This paper explores the use of text-to-image diffusion models for personalized retrieval and segmentation tasks. The authors introduce a novel method called PDM (Personalized Features Diffusion Matching), which leverages intermediate features from pre-trained text-to-image models for personalization tasks without requiring additional training. PDM demonstrates superior performance across multiple benchmarks, surpassing many supervised methods. Additionally, the paper identifies deficiencies in existing datasets and introduces a new benchmark dataset that effectively addresses the challenges presented by traditional datasets in scenarios involving multiple instances of the same class. This new dataset focuses on complex scenes containing multiple instances of the same class, further validating PDM's efficiency and accuracy in precisely handling and locating multiple similar instances.

Strengths

1.This paper introduces a novel challenge in personalized retrieval and segmentation tasks involving complex scenes with multiple instances of the same class. In response to this scenario, the authors have constructed a new benchmark dataset and devised a novel method—PDM (Personalized Features Diffusion Matching), which ingeniously combines appearance and semantic features to address the issue of multiple instances within the same category. This approach is quite innovative. 2.The overall structure of the paper is logically organized and easy to understand, and the figures greatly assist in comprehension. Overall, the paper is excellently written.

Weaknesses

1.The authors simply take the average of appearance and semantic features as the overall diffusion features without considering a weighted fusion of these two features. This approach may lead to errors in segmentation or retrieval on some datasets, as it fails to balance the contribution between different features effectively. 2.There may be some issues with the process of generating the personalized multi-instance retrieval and segmentation dataset, such as the randomness in selecting video frames. While this approach improves the efficiency of data generation, it could mislead model training and evaluation if the chosen frames lack sufficient feature diversity or have quality issues. 3.Although the paper conducts extensive experiments in personalized retrieval and segmentation tasks and briefly supplements with ablation study results of key components in the appendix, and qualitatively analyzes the differences from other methods, ensuring comprehensive experimentation, there are still shortcomings. The ablation experiments are only conducted for multi-instance retrieval, which does not prove the importance of key components in multi-instance segmentation. It would be beneficial to include experiments addressing this aspect as well.

Questions

1.Given your mention that the method's performance might depend on the success of image reconstruction quality, I am curious about how you have managed to enhance image reconstruction quality to achieve such impressive results with your approach. 2.Have you considered more complex ways of combining appearance and semantic features? If so, why did these approaches not yield better results, and what are your thoughts on this? 3.Why opt for random selection of video frames without addressing potential quality issues in the generated data? How do you manage this randomness to ensure fairness and consistency across different methods tested on this data?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors briefly discuss the limitations of their method in the conclusion but do not provide any viable future solutions. Could the authors possibly offer some feasible ideas or directions for addressing these limitations in future work?

Summary

This paper introduces a novel approach called Personalized Diffusion Features Matching (PDM) for personalized retrieval and segmentation tasks. Most current self-supervised and supervised methods struggle to accurately identify specific instances within a dataset when multiple instances from the same class are present. So the proposed PDM method leverages intermediate features from pre-trained text-to-image diffusion models. PDM combines semantic and appearance cues to accurately locate and segment specific instances within target images.

Strengths

* its an innovate use of pretrained model (already available foundation models ) in a zero shot setting. This is beneficial for people without a lot of compute and good for environment. * the proposed method combines both appearance and semantic similarity, which is a well motivated design

Weaknesses

* the authors used features from stable diffusion, which is text guided image generation model. its not that fair to compare to DINOv2 or SAM since those models were not trained with text supervision. * the method depends on diffusion inversion, which is quite slower than other segmentation methods. the authors used sdxl-turbo to mitigate the latency but its unclear the impact on quality * the method seems only applicable to Unet diffusion models. its unclear if it works on DiTs

Questions

* what is the performance difference if PDM uses SDXL-turbo vs non-turbo? SDXL-turbo was notorious for not able to reproduce good quality details such as small human faces or limbs. * how does PDM performance vary wrt to the strength of the underlying diffusion model? * in table 2, the authors show openCLIP+PDM and DINOv2+PDM. what happens if you compute OpenCLIP+DINOv2+PDM?

Rating

4

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

yes the authors adequately address the limitations.

Official Comment by Reviewer JGDB

Thanks for your detailed response. I would keep my score by a comprehensive consideration of the paper's contribution and experimental results.