Summary
This research explores the problem of memorization in video diffusion models (VDMs), extending the concerns previously identified in image diffusion models (IDMs). This memorization can result in the generation of copyrighted content, posing legal and ethical challenges. To tackle this, the paper extends Rahman et al., 2024 by improving definitions and metrics to quantify both content and motion memorization in VDMs, referred to as GSSCD and OFS-k respectively. The authors provide a memorization analysis across various VDMs and propose an innovative detection method that leverages these newly introduced metrics, GSSCD and OFS-k, to effectively identify memorization videos. These contributions are aimed at mitigating the risks associated with memorization in video generation, enhancing the trustworthiness and applicability of VDMs.
Strengths
1. The investigation into memorization in video diffusion models (VDMs) addresses a critical and timely concern. As VDMs become increasingly sophisticated and widely used in generating realistic video content, the potential for these models to inadvertently memorize and replicate training data poses significant legal and ethical risks.
2. The paper enhances the conceptual framework and analytical tools available for studying memorization in VDMs by introducing more nuanced and practical definitions of content and motion memorization.
3. The paper proposes practical strategy for detecting memorization.
Weaknesses
1. The paper introduces a new metric for evaluating content memorization in video diffusion models in a frame-level manner but does not sufficiently justify the need for these over existing methods developed for image diffusion models. It remains unclear whether the current metrics for IDMs are insufficient for assessing memorization in videos.
2. Limited Validation: The new metrics' validation primarily relies on 1K manual annotations on a single dataset (WebVid-10M), which might not reflect the diversity of real-world scenarios. More extensive validation across diverse datasets and more automated, objective evaluation methods could strengthen the findings.
3. The paper repeatedly mentions the existence of a method in the field that addresses motion memorization in video diffusion models (Rahman et al., 2024), yet it fails to provide a qualitative comparison with this method. The absence of a direct, detailed comparative analysis weakens the argument for the proposed method's novelty and effectiveness.
4. Additional minor weaknesses:
a. Figure 3 appears to be redundant, as Table 2 already sufficiently demonstrates that ModelScope’s generations replicate the image training data.
b. In Table 2, it would improve clarity if text-conditional generations (T2V) and unconditional generations (UV) were distinguished in separate column.
Questions
Please refer to the previously noted weaknesses, and additionally, consider the following questions:
1. Could the authors discuss any limitations of their work? Including such details would provide a more balanced view and help readers understand the potential constraints or challenges in applying the proposed methods in different contexts.
2. How does the method perform under varied conditions, such as different video qualities or motion complexities? Is there a risk of the detection methods being too sensitive or not sensitive enough?
3. Given that the authors have demonstrated the separate detection of content and motion memorization, could combining these methods into an ensemble approach enhance the overall detection performance
4. Will you provide access to the implementation (code) or datasets used in the experiments? Providing these resources could greatly enhance the credibility and applicability of the research.