Evaluation of Text-to-Video Generation Models: A Dynamics Perspective

Summary

This paper proposes a new evaluation metric for the text-to-video model, and this metric in particular focus on the dynamics on the generated video. Their metric is based on three sub-scores: inter-frame dynamics score, inter-segment dynamics score, and video-level dynamics. They did

Strengths

It explores a more fine-grained protocol to evaluate the dynamics of the T2V generation. They have done extensive evaluation on the exiting T2V models, and obtained some insights. They propose the "improved metric", which would give higher weight to the large-dynamic range compared to the "existing metric". It would be useful to detect whether the generated videos of a model cover all dynamic ranges.

Weaknesses

I feel this paper over-emphasizes existing evaluation works ignore the dynamics, this is not quite what I saw. I think most of papers already paid attention to the dynamics evaluation, they are just not so detailed and fine-grained as in this paper. For example, EvalCrafter has Motion Quality section for considering the dynamics. I think there are already some existing metric to quantify the dynamics in the video, and they're simpler, such as Motion Quality in EvalCrafter. I think there should be a direct comparison between the proposed metric and the existing dynamics metric. Maybe the authors can consider to use the average of three motion quality metrics in EvalCrafter (after normalization) and show which one is aligned with human evaluation better? Another problem is for the Inter-segment Dynamics Score, I think the way to take segments is too brutal-force and sketchy, for example, just take every 8 frames. This maybe totally misaligned with the actual semantic segments, in that case, I feel the Inter-segment Dynamics Score does not make much sense. For the naturalness metric's definition, why can we trust Gemini-1.5pro? Authors may want to justify this.

Questions

- Why is this protocol called DEVIL? - What are the cross and circles in the Dynamics Evaluation plot in Figure 2? I'm confused what this plot wants to convey. - Figure 3 may need to be improved. (b) is only the word cloud for DEVIL instead of for all three benchmarks. - The reference in 136 line seems to be incorrect, it is not a paper for optical flow. - What's SIM in equation (2)? - What does I function mean in equation (14)? Typo: Line 223: overall h??

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

.

Summary

Effective evaluation protocols are essential for developing advanced text-to-video (T2V) generation models. Current protocols primarily address temporal consistency and content continuity but often neglect the dynamics of video content, which are crucial for visual vividness and fidelity to text prompts. This study introduces DEVIL, a protocol that emphasizes dynamics by defining metrics across multiple temporal granularities and creating a benchmark of text prompts graded by dynamics levels. DEVIL evaluates T2V models using metrics of dynamics ranges and T2V alignment, enhancing existing metrics from a dynamics perspective. Experimental results show that DEVIL achieves up to 90% consistency with human evaluations, demonstrating its potential as a powerful tool for advancing T2V generation models.

Strengths

1. **Introduction of New Dynamics Metrics**: The study presents a novel set of dynamics metrics that enhance the evaluation of text-to-video (T2V) generation models. These metrics focus on the dynamics dimension, which assesses the visual vividness and fidelity of video content to text prompts across multiple temporal granularities. 2. **Comparative Analysis with Existing Methods**: The research includes a thorough comparative analysis of existing evaluation methods, highlighting their limitations in addressing video dynamics. By juxtaposing these methods with the newly proposed dynamics metrics, the study demonstrates how the latter offers a more detailed assessment of T2V models. 3. **Code Submission for Reproducibility**: The authors have submitted the code used in their study. This openness is crucial for validating the proposed methods and fostering innovation in the community.

Weaknesses

1. **Difficult to Assess Dynamics from Images Alone**: Evaluating the impact of dynamics solely through images is challenging. To better understand the effectiveness of the proposed dynamics metrics, it would be beneficial to provide several real videos along with their corresponding scores. This would allow for a more accurate assessment of how well the metrics reflect the dynamics and visual vividness of real video content. Without these examples, it is hard to gauge the practical utility and accuracy of the dynamics scores. 2. **Comparison of Existing T2V Methods**: Similarly, assessing the dynamics of generated videos from existing T2V methods using only images is insufficient. To comprehensively evaluate the effectiveness of the proposed dynamics metrics, the authors are recommended to include several examples of fake videos generated by existing methods along with their corresponding dynamics scores. This comparison would help in understanding how the new metrics perform relative to current standards and whether they offer a significant improvement in evaluating video quality. 3. **Computational Efficiency**: The computational efficiency of the proposed method is not well addressed. Understanding the resource requirements and processing time for the new dynamics metrics is crucial for practical applications. If the method is computationally intensive and very slow, it might limit its applicability in real-world scenarios where quick and efficient evaluations are necessary.

Questions

Please refer to the weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

This paper discusses the limitations in Section 6.

Summary

This paper proposes DEVIL, an evaluation suite of metrics for evaluating text-to-video generation, focusing on dynamics (the authors note that many previous works on video generation focus on other aspects but ignore dynamics). Proposed metrics are computed using a number of automatically extracted values based on e.g. optical flow or autocorrelations and then a linear combination of these values are learned such that they align with human perceptions of dynamics. The authors also rate prompts (automatically using gpt4) to assign a dynamics “grade” which allows them to evaluate alignment with the level of dynamics asked for by a text prompt. They also report a “dynamic range” of a given model across the set of prompts. Finally the authors use their proposed metrics to evaluate a number of recent video generation models.

Strengths

This paper attacks a useful problem in video generation that researchers talk about / are aware of, but don’t have good metrics to measure. Overall the proposed metrics seem quite thorough and as shown in the results, the proposed measures do seem to be quite correlated with human judgements. I think this would be a good contribution to the community particularly if made publicly available.

Weaknesses

Conceptually the novelty of the paper is mostly incremental but since good measures of dynamism in video generation do not currently exist, the work should be of good value to the community. One potential weakness is that I am not sure if I agree with the decision to segment videos based on length relative to total number of frames since this ignores the fact that different models can generate widely different video lengths or at quite different framerates(e.g. 2 seconds at 8 fps for VideoPoet videos and some examples from Sora that go to 1 min at 24 fps). There is also sloppy writing in some parts of the paper. Examples: * Some tables are never referenced in the body of the text * Authors mention that human “refine” the dynamics grades but no details are given other than to say that they did this for three months (which is mostly meaningless). * The word cloud in Figure 3 is not very interpretable — is this for just the high dynamics prompts or everything? What is the take-away? Eqn 3 (which mentions 5%) does not agree with the paragraph above * The function SIM is never defined * Orphaned footnote in line 248 * Plots in Fig 5 are attractive but very hard to actually compare across the different methods

Questions

* I believe that the proposed measures are not necessarily able to separate camera motions from object motions. Similarly, it is not able to distinguish between fast moving “video textures” (e.g. roaring flame) which tend to be easy to generate using these models vs more specific motions (e.g. ninja doing a backflip). I would ask the authors to comment on whether these are real limitations or not. * Suggestion: Please add examples of prompts with high dynamics and low dynamics in the main body of the paper. * Suggestion: Report linear regression weights (given that there are just a few weights and would allow the reader to evaluate the relative importance of each factor) * Question: Will the code/prompts be made public? * Are the “improved metrics” better correlated with human judgements too? I am not sure if I saw this somewhere in the paper * Suggestion (but lower priority): I believe Runway has a control that lets you control the dynamics (by changing CFG weight) — it would be interesting to see the effect of this control on measures proposed in this paper.

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

yes

Summary

The paper presents a comprehensive study on the evaluation of Text-to-Video (T2V) generation models, with a particular focus on the dynamics of video content. The authors introduce a novel evaluation protocol named DEVIL, which aims to address the often overlooked aspect of dynamics in existing evaluation metrics. This protocol defines a set of dynamics scores that correspond to multiple temporal granularities and introduces a new benchmark of text prompts under various dynamics grades. In addition, this paper verifies a significant issue with some existing evaluation metrics: they allow models to attain high scores by generating low-dynamic videos. The authors demonstrate that the metrics provided by DEVIL align closely with human in rating dynamics, indicating its potential to significantly advance T2V generation models. The paper also highlights the current limitations in T2V models and datasets, and provides valuable suggestions for future research in this field.

Strengths

1. The paper introduces an innovative T2V model metric, DEVIL, which evaluates the content dynamics of generated videos—an aspect often overlooked by current evaluation methods but essential for realistic video generation. DEVIL assesses dynamics at multiple granularities, achieving high correlation with human evaluations. By revealing the negative correlation between existing metrics and dynamic scores, the paper challenges the current evaluation standards. 2. The study employs robust experiments to validate the dynamics metric (DEVIL), achieving up to 90% consistency with human evaluations. Additionally, the paper identifies issues with existing T2V evaluation metrics, revealing their negative correlation with dynamics metrics and the tendency to favor low-dynamic content, which misrepresents model performance. 3. The theoretical framework is detailed and thorough, covering the formulation of dynamics scores across multiple temporal granularities. The paper also highlights a reliable human rating collection process, amassing 50,000 prompt dynamic level classifications for training the linear regression model used in the metrics and evaluations of 4,800 generated videos from six T2V models for proving the consistency between the metric and human ratings. 4. The paper is overall logically structured and easy to follow.

Weaknesses

1. How to differentiate dynamic video from videos with low-quality motions, for example, flickering or temporal inconsistency? We desire videos with not only large dynamics but also high-quality dynamics, but some models which generate low-quality motion such as flickring frames and temporally-inconsistent videos may also achieve high dynamic scores under the proposed evaluation scheme. 2. Some previous works also propose metrics to evaluate dynamic levels of generated videos. For example, VBench [23] proposes dynamic degree evaluated with RAFT, and EvalCrafter [31] proposes motion quality evaluated with RAFT. The proposed metric should be compared with existing metrics from those works. 3. Frame rate of videos is an important factor that may affect the metrics. For example, a video with low fps may get higher score than a model with high fps if the evaluation is conducted on all frames without aligning their fps. Do the authors take it into consideration when designing the metrics? Are the models evaluated at their original fps or samped frames at the same fps? 4. In Section 3.3, how many videos are used to fit the human alignment dynamic scores? I am concerned that the metric is overfitting the selected videos for human alignment, but may not generalize well to other videos. Are the human evaluation results in Table 3 computed from the same videos as the videos used to fit the dynamic scores in Section 3.3? There is a risk of cheating if using the same set of videos to fit the human alignment dynamic score (in Section 3.3) and to calculate the human correlation (in Table 3). 5. The effectiveness of the improved metrics proposed in Section 4 is not validated with human correlation. 6. The relationship between the naturalness score defined in Section 4 and the rest of the paper is not clear.

Questions

Please refer to weakness section.

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors addressed the limitations and societal impact in the paper.

Official Comment by Reviewer 7oNg

Thank the authors for the rebuttal. The authors addressed most of my concerns so I raised my rating to borderline accept.