Summary
This paper studies a novel task of Image Copy Detection for Diffusion Models (ICDiff). Different from traditional Image Copy Detection task, ICDiff focuses on detect and evaluate the degree of image copy for images generated by text-2-image (T2I) generative diffusion models. This is an important and meaningful task for the community to study, with the emergence and influence of T2I diffusion models on downstream tasks, such as image editing, icon generation, or style transfer. Effectively and precisely detecting the generated images that might "copy" content/style/character of the artifically created images in its training data can help the end users avoid violating issues such as Image Copyright.
To study such a problem, the authors collected and proposed a new dataset called D-Rep. To build such a set, the authors first try to find text prompts from DiffusionDB that are highly similar to image titles from LAION-Aesthetics V2, using Sentence-Bert score. Subsequently, they generate 40k images using these collected prompts using Stable Diffusion 1.5 to simulate possible image copies from LAION. To obtain the degree of image copy annotation, the authors employ human annotators to score each generated-image-&-LAION-image paris from 1 to 6.
Furthermore, the authors propose and train a probablity density based embedding model (PDF-Embedding) to score the degreee of image copy for input images. 90% of the collected D-Rep dataset is used as training set for the proposed method.
In the experiments, the proposed PDF-Embedding more effectively and efficiently detect image coplies in the D-Rep test set, comparing to the compared methods. However, the quantative experiments are limited to the proposed dataset. In addition, the comparison is somewhat unfair because many of the compared methods (such as GPT-4V, CLIP, DINOv2) are not trained or fine-tuned on the training split of D-Rep, but the proposed method is.
Strengths
- [Clear writing]: The writing is clear. The sturcture of the paper is easy to follow.
- [Novel and Important Task]: The studied and proposed task Image Copy Detection for Diffusion Models (ICDiff) is novel, practical, and meaningful. It is important for the community to address the Image Copy issues caused by the training data of T2I diffusion models. This will significantly help downstream tasks avoid copyright issue.
- [Good baseline]: The proposed PDF-embedding is an efficient and good baseline to address the proposed task.
Weaknesses
- [Incompleteness]: This paper is not complete in itself. The authors largely use "refer to Section XXX in the Appendix" throughout the paper. At Sec. 5.1 (Line#185), the entire section for Training Details contains only 1 sentence "Please refer to Section E in the Appendix". I understand the space is limited in the main manuscript and the authors want to use the space for more important content. However, a paper should be complete in itself. After carefully reading the paper, I cannot know: i) how the model is trained, what is the architecture, what are the configuration; ii) how the other methods are compared, do the authors train them, how are they trained or adapted to the current task? These details are important for the main paper for the readers to understand the method and the experiments.
- [Limited experiments]: The quantitaitve experiments are only conducted on one dataset (4k images in the test set) in Tab. 1. In addition, this dataset is generated by only one specific T2I model Stable Diffusion 1.5. Different generative models might have different image copy patterns. The current experiments can hardly validate the effectiveness and advantages of the proposed method over other methods. More comprehensive experiments should be conducted. When switching prompts (data) and the generative models, maybe zero-shot methods such as CLIP, M-LLMs (GPT4-V), or DINOv2 are more generalize. Without large scale experiments, it is hard to draw a conclusion.
- [Unfair comparison]: In Tab. 1, the compared Vision-Language Models (CLIP, GPT-4) and Self-supervised Learning Models (e.g., DINOv2) are not trained or fine-tuned on the evaluation dataset's training split. However, the proposed method is trained on the training split. This makes the comparison unfair. To ensure a fair comparison, either the compared methods should be trained or fine-tuned on the same training split; or, evaluate all methods on novel datasets in a zero-shot manner. With the current unfair comparison, the advantage and effectiveness of the proposed method is not clear.
- [Duplicate content]: Sec. 5.2 and 5.3 are highly overlapped.
Questions
Q1: Do the authors study the quality of text matching using Sentence-Bert? Pure linguistic matching could be quite ambigious and subseqeuntly leads to incorrect matching. For instance, "containers" can be matched to both the huge containers for shipment at the port and the plastic containers (such as lunch box) in a kitchen context. In addition, do the authors study different scores, such as CLIP text encoder score instead of Sentence-Bert score? As observed in [1], for visual content related text, CLIP encoder seems better than Sentence-Bert.
Q2: How do you define Image Copy for Diffusion Models? What kind of "Copy" should be detected? This is important for the studied task. For instance, when a user prompts a T2I diffusion model "A portrait of the Mona Lisa.", of course, a highly similar image that copies the famous Mona Lisa potrait will be generated. But, is this an image copy? This kind of detection might not helpful for downstream tasks. The task should be defined better and thereby the method can be designed better. From the visualization in Fig. 3, I found the high-score copies are mainly "object copy". These copies might occur frequently if the prompt tasks the diffusion model to generate such an image. By conditioning and limiting the prompt text, this can be solved directly. So why is it useful? T In my opinion, a worth-to-detect image copy would be when prompting "Generate a yellow cartoon-style mouse" and a Pikachu-like image is automatically generated without explicitly mentioning it in the prompt.
Q3: How well the proposed method will generalize to images generated by other diffusion models, such as SDXL or DALL-E-3, quantatively?
[1] Chen, Z., Chen, G. H., Diao, S., Wan, X., & Wang, B. (2023). On the Difference of BERT-style and CLIP-style Text Encoders. arXiv preprint arXiv:2306.03678.