Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicality

Rebuttal (2/3)

**Q1: How does the performance of the evaluated models vary across different domains or types of prompts?** Thanks for the comments. We conducted experiments on the performance of the evaluated models that varied across different types of prompts. The results are shown in the tables below. **Setting:** - We input the human annotation results corresponding to the eight prompt categories into the Rao and Kupper model [2] to estimate and rank the model scores. **Results:** - We report the model scores and corresponding rankings obtained for the annotation results under the eight prompt categories. - Some of the scores in the table are the same and the result of different rankings is due to the retention of two decimal places in the report. **Analysis:** - Our results show that using different prompt sources does lead to slight differences in evaluation results, whereas models with a clear advantage, such as Gen2, are unaffected and have the highest ranking regardless of which prompt is used. - This result further highlights the importance of using class-wide prompts when using human evaluation protocols. **Conclusion:** - Our experimental results show that different prompt categories **do slightly affect the evaluation results** of model performance, but this effect is not obvious for models with high performance, such as Gen2. **References:** [1] Chatbot Arena. [2] Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. _Journal of the American Statistical Association_, _62_(317), 194-204. [3] Rank analysis of incomplete block designs: I. The method of paired comparisons. _Biometrika_, _39_(3/4), 324-345. [4] Vbench: Comprehensive benchmark suite for video generative models. CVPR 2024. Prompt Category - Animal | |Video Quality|Temporal Quality|Motion Quality|Text Alignment|Ethical Roubustness|Human Preference| |-|-|-|-|-|-|-| | Gen2 | 2.02(1) | 2.15(1) | 1.89(1) | 2.40(1) | 2.48(1) | 3.38(1) | | Pika | 1.18(2) | 1.41(2) | 1.52(2) | 1.54(2) | 1.04(2) | 1.35(2) | | Latte | 0.90(4) | 0.98(3) | 0.92(4) | 1.29(3) | 0.92(3) | 1.01(3) | | TF-T2V | 0.98(3) | 0.93(4) | 0.96(3) | 0.99(4) | 0.73(4) | 0.52(4) | | Videocrafter2 | 0.84(5) | 0.72(5) | 0.86(5) | 0.53(5) | 0.69(5) | 0.35(5) | Prompt Category - Architecture | |Video Quality|Temporal Quality|Motion Quality|Text Alignment|Ethical Roubustness|Human Preference| |-|-|-|-|-|-|-| | Gen2 | 2.12(1) | 1.86(1) | 1.95(1) | 2.49(1) | 3.45(1) | 2.54(1) | | Pika | 1.00(3) | 1.46(2) | 1.13(2) | 0.92(3) | 0.59(3) | 1.05(2) | | Latte | 0.81(5) | 1.07(4) | 0.93(4) | 1.28(2) | 1.30(2) | 0.95(3) | | TF-T2V | 0.82(4) | 1.10(3) | 0.94(3) | 0.85(4) | 0.50(4) | 0.72(4) | | Videocrafter2 | 1.09(2) | 0.73(5) | 0.89(5) | 0.62(5) | 0.25(5) | 0.62(5) | Prompt Category - Food | |Video Quality|Temporal Quality|Motion Quality|Text Alignment|Ethical Roubustness|Human Preference| |-|-|-|-|-|-|-| | Gen2 | 2.34(1) | 2.39(1) | 2.00(1) | 2.87(1) | 3.45(1) | 3.04(1) | | Pika | 0.99(2) | 1.03(2) | 1.16(2) | 0.94(3) | 0.59(3) | 0.82(3) | | Latte | 0.89(3) | 0.92(3) | 0.83(5) | 1.57(2) | 1.30(2) | 1.42(2) | | TF-T2V | 0.79(5) | 0.74(4) | 1.04(3) | 0.51(4) | 0.50(4) | 0.47(4) | | Videocrafter2 | 0.80(4) | 0.74(5) | 0.87(4) | 0.47(5) | 0.25(5) | 0.35(5) |

To save reviewer's time, we put a summary of rebuttal

Dear Reviewer LQLU, Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we **summarized our responses here**. 1. \[**The algorithm’s performance may vary depending on the specific prompt types and model characteristics** \]: **Response:** - The effect of different prompts and model characteristics on the algorithm is limited, and experimental results demonstrate that this effect does not diminish the validity and reliability of the dynamic evaluation component at all. 2. \[**Compare Rao and Kupper method and ELO rating** \]: **Response:** - While the ELO rating system is competitive and has been successfully applied in many domains, given the nature of our task, which involves nuanced evaluations where ties are possible and even expected, the Rao and Kupper model is more suited to our needs. 3. \[**How does the performance of the evaluated models vary across different domains or types of prompts** \]: **Response:** - We report the model scores and corresponding rankings obtained for the annotation results under the eight prompt categories. - The results show that different prompt categories do slightly affect the evaluation results of model performance, but this effect is not obvious for models with high performance, such as Gen2. Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments. Best Regards, Authors

Response to Authors

Thank you for the detailed responses from authors. I have acknowledged the strengths of your wonderful paper and I will keep my score.

Rebuttal (2/3)

**W2: Explain how Rao and Kupper model quantifies annotations results.** We appreciate your feedback on our description of the Rao and Kupper model [10]. We understand the importance of clarity and detail in explaining our methods. Here’s a more comprehensive overview of the model. **Analysis:** - **Probabilistic Model Overview:** - **Model Definition:** The Rao and Kupper model is a probabilistic framework used for pairwise comparisons, allowing us to estimate scores for each model and rank them accordingly. - **Maximum Likelihood Estimation:** We employ maximum likelihood estimation to derive the scores, which provides a statistical basis for the rankings. - **Detailed Explanation:** - **Comparison Process:** Consider a scenario with $t$ models. For any pair of models $i$ and $j$, the notation $i \succ j$ indicates $i$ beating $j$, $i \prec j$ indicates $i$ losing to $j$, and $i \sim j$ indicates a tie. The probabilities of these events are defined as follows: $$ \begin{align} P(i \succ j) &= \frac{p_i}{p_i + \theta p_j}, P(i \sim j) &= \frac{p_i p_j (\theta^2 -1)}{(p_i + \theta p_j)(\theta p_i + p_j)}, \end{align} $$ where $p_i$ and $p_j$ represent the positive real-valued scores of models $i$ and $j$. $\theta$ is a parameter indicating the likelihood of a tie between closely matched models. - **Estimation Method:** The likelihood function for our model is constructed as: $$ \begin{equation} L(p, \theta) = \prod_{i = 1}^{t} \prod_{j = i + 1}^{t} \left( \frac{p_i}{p_i + \theta p_j} \right)^{n_{ij}} \left( \frac{p_j}{\theta p_i + p_j} \right)^{n_{ji}} \left( \frac{p_i p_j (\theta^2 -1)}{(p_i + \theta p_j)(\theta p_i + p_j)} \right)^{\tilde{n}_{ij}}, \end{equation} $$ and the corresponding log-likelihood function is: $$ \begin{equation} l(p, \theta) = \sum_{i = 1}^{t} \sum_{j = i + 1}^{t} \left( n_{ij} \log \frac{p_i}{p_i + \theta p_j} + n_{ji} \log \frac{p_j}{\theta p_i + p_j} + \tilde{n}_{ij} \log \frac{p_i p_j (\theta^2 -1)}{(p_i + \theta p_j)(\theta p_i + p_j)} \right), \end{equation} $$ where $p \in \mathbb{R}^t$ is a vector of scores $(p_1, \dots, p_t)^T$. We estimate $p$ and $\theta$ by maximizing the log-likelihood function numerically. - **Application and Results:** - **Model Ranking:** After estimating the scores, models are ranked based on these scores, reflecting their relative strengths. Our detailed explanation in Appendix C.3 covers the model fitting process and the derivation of scores and rankings. **Conclusion:** - We hope this detailed explanation clarifies our methodology and addresses any concerns about the application of the Rao and Kupper model in our research. We also provide detailed parameter estimation details in Appendix C.3.

Rebuttal (3/3)

**W3: Is the human preference the only one that dominates? How can the aesthetics and evaluation systems of different annotators be consistent? How can we avoid the problem?** Thanks for the comments. We analyze this from the following perspectives. **Analysis:** - We observed similar problems in human evaluation studies in the T2I and NLG communities [8,9,12], as shown in the footnotes of the article, some NLG studies [11] achieve an IAA of only 0.14, i.e., even if constructing sufficiently objective protocols and using the most professional annotators, one **cannot avoid** the biases brought by human preferences. - At the same time, however, it has also been shown that **the existence of human preferences is a vital part of human assessment** [9], i.e., a good protocol should both effectively reduce bias due to preferences and make reservations about this, as too much agreement (e.g., IAA greater than 0.7) may suggest that there is a lack of diversity among annotators. - This is one of the core reasons our protocol distinguishes between subjective and objective metrics. For **objective** metrics, we minimize the influence of personal preferences through detailed metric definitions and annotator training, whereas for **subjective** metrics, we encourage annotators to evaluate based on their preferences. Unlike objective metrics, subjective metrics are difficult to be replaced by automated metrics and reflect the diversity of annotators. **Conclusion:** - Previous studies and our experimental results have shown that **human preferences cannot and need not be eliminated** during evaluation. By providing different guidelines for different types of metrics, our protocol achieves a balance between the pros and cons of human preferences. **W4: The annotation task with the dynamic evaluation component is too dependent on manual work, its promotion and ease of use in the open-source community are still questionable.** Thanks for the comments. We analyze this from the following perspectives. **Analysis:** - Our dynamic evaluation component was proposed to improve human evaluation protocols, i.e., this method should be compared to a traditional human evaluation protocol that **also relies on manual work**, and uses **all the video pairs**. Our component can reduce the annotation cost without additional intervention during the annotation phase, which is important for popularizing human assessment. - For example, a researcher needs to recruit five annotators, each of whom can only evaluate 200 sets of video pairs; using the same budget, employing our dynamic component would allow the researcher to recruit 10 or more people to execute evaluation, as the workload for each individual is only about half of what it would otherwise be. Not only does this allow for a more varied and reliable set of annotated results, but also as the evaluation size increases, the effect of the component becomes more pronounced. - In addition, **this component is not limited to the T2V research field**, for the T2I and NLG research fields, by simply replacing the automated indicators in the pre-preparation phase, one can directly use this component in the customized protocol framework. We provide the detailed flow of the component operation in Appendix C.2 and will open-source all the related code. We also hope and encourage researchers from different fields to conduct more comprehensive human assessment protocols at a lower cost. **Conclusion:** - Our dynamic components are proposed to improve the efficiency of the human assessment process that **already requires manual work**, without additional expense. At the same time, it can be used in other research fields with **simple adjustments**. **References:** [1] Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models. [2] ControlVideo: Training-free Controllable Text-to-Video Generation. ICLR 2024. [3] EvalCrafter: Benchmarking and Evaluating Large Video Generation Models. CVPR 2024. [4] VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR 2024. [5] AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI. TBench 2024. [6] FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation. NIPS 2023. [7] Holistic Evaluation of Text-to-Image Models. NIPS 2023. [8] Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation. CVPR 2023. [9] Evaluation of Text Generation: A Survey. [10] Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. _Journal of the American Statistical Association_, _62_(317), 194-204. [11] The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation. EMNLP 2021. [12] All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. IJCNLP 2021.

To save reviewer's time, we put a summary of rebuttal

Dear Reviewer RMCR, Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we **summarized our responses here**. 1. \[**The evaluation metrics cannot be proven reasonable or convincing** \]: **Response:** - Through extensive research and rigorous analysis, we designed an assessment system that not only incorporates the mainstream metrics used in existing T2V studies but also complements the missing parts of previous studies, making our evaluation metrics reasonable and convincing. 2. \[**Explain how Rao and Kupper model quantifies annotations results** \]: **Response:** - We provide detailed explanation in Rebuttal (2/3) and detailed parameter estimation details in Appendix C.3. 3. \[**How to handle human preference** \]: **Response:** - Previous studies and our experimental results have shown that human preferences cannot and need not be eliminated during evaluation. - By providing different guidelines for different types of metrics, our protocol achieves a balance between the pros and cons of human preferences. 4. \[**Dynamic evaluation component is too dependent on manual work** \]: **Response:** - Our dynamic components are proposed to improve the efficiency of the human assessment process that already requires manual work, without additional expense. At the same time, it can be used in other research fields with simple adjustments. Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments. Best Regards, Authors

Looking forward to the reply

Dear Reviewer RMCR, As we draw closer to the rebuttal deadline, we would like to inquire if there are any additional questions or concerns about our work. We greatly value the feedback. Thanks again for the time and effort in our work! Best wishes, Authors

Rebuttal (2/4)

**Q1: More details on how the dynamic evaluation module handles potential biases.** Thank you for your question regarding the dynamic evaluation module. We have designed our method with robustness and accuracy in mind. Here's how we address potential biases: **Analysis:** - **Robustness Against Incomplete Comparisons:** It is well-documented that the Bradley-Terry model [5] is robust against incomplete comparison data. Our approach, based on the Rao and Kupper model [4], **inherits this robustness and is specifically designed to handle scenarios with missing or incomplete data**. This property ensures that our ranking estimates remain reliable even when some comparisons are discarded. - **Handling Top-Performing Models (e.g., Gen2):** - **Bootstrap Confidence Intervals:** To mitigate potential biases in ranking models with incomplete data, we employ bootstrap confidence intervals for the score estimates of each model across various metrics. Section 5 and Figure 3 of our paper demonstrate that even in extreme cases, our rank estimation is consistent and trustworthy. - **Ensuring Fairness:** By using bootstrap methods, we can quantify the uncertainty around our model rankings and ensure that our evaluation is fair and unbiased, even when certain comparisons are omitted. - **Transparency and Detail:** We provide a detailed explanation of our dynamic evaluation module and its bias-handling mechanisms in the supplementary materials, ensuring that our methodology is transparent and replicable. **Conclusions:** - From a methodological design perspective, the Rao and Kupper model we used can **accurately estimate model scores using incomplete comparisons**. - From an experimental validation perspective, we conducted rigorous bootstrap confidence interval experiments, the results of which confirm that even after discarding many comparisons on top models such as Gen2, the final estimate of the relative strength of the model is **still accurate and credible**. **Q2: How to ensure that the reduced number of annotations still captures a representative sample?** Thank you for your insightful comment. We analyze it from the following perspectives. **Analysis:** - Before the dynamic evaluation begins, the protocol undergoes a static evaluation, in which the annotator prioritizes a group of samples that are worthy of being annotated. These video pairs contain the **full range of prompt categories and model combinations**, as shown in Figure 4, and have little difference in the scores of the automated indicators, i.e., they are **the most representative samples** according to the reviewers. - As these samples won't be discarded during the static evaluation, even if subsequent evaluations discard many comparisons, the final evaluation results can include all categories of prompts and the most representative comparisons. **Conclusion:** - The static evaluation in the protocol **retains the samples that are most worthy of being annotated**, ensuring that the evaluation results **won't be biased** by the lack of specific kinds of prompts and model combinations. **Q3: How to validate that the training provided to LRAs effectively mitigates potential biases and their potential lack of diversity?** Thank you for your insightful comment. We designed detailed guidelines and training processes to address this bias, as following points: **Analysis:** - To help annotators understand the definition of each metric, we provide multiple **reference perspectives** for each indicator. For each reference perspective, we provide **detailed examples** including a set of reference videos, a description of the analysis process, and corresponding conclusions, so the annotators from different sources can evaluate with the same mode of thinking. The specific definition and training process are shown in Table 4, Figure 1 (c), and Figures 5 to 9, respectively. - As shown in Tables 1 and 2, the consistency between the trained annotators and the crowdsourced annotators is significantly improved, i.e., the metric examples and the training process can help eliminate the inconsistency caused by the potential bias. - Although the lack of LRA's diversity cannot be completely avoided [2], by distinguishing between subjective and objective indicators, we can **preserve the diversity of annotators** as much as possible during the evaluation process. **Conclusion:** - We minimize annotator bias in assessing objective metrics by providing **detailed guidelines and training processes**, and we retain annotator diversity by introducing **subjective metrics**.

Rebuttal (3/4)

**Q4: How does the protocol account for potential biases in the evaluation of "Ethical Robustness"? Can you provide more details on how this metric is defined and measured?** Thank you for your insightful comment. We analyze it from the following perspectives. **Analysis:** - Our survey shows that while the measurement of the ethics of generated content is an important evaluation metric for generative models [6], it is missing from existing T2V evaluation protocols. Therefore, after a thorough investigation of the relevant research [7,8], we defined the corresponding reference angles according to the following criteria. - **Toxicity**. Evaluating toxicity in video generation models is crucial for ensuring user safety, compliance with community standards, and protecting brand reputation. - **Fairness.** Assessing fairness in video content ensures a diverse and fair representation of characters and subjects across different social dimensions, preventing the perpetuation of stereotypes and marginalization. - **Bias.** Evaluating bias in video generation models is essential for ensuring content accuracy, diversity of perspectives, and avoiding harm to specific groups. **Conclusion:** - After studying numerous relevant literature, we use three reference angles, i.e., Toxicity, Fairness, and Bias, to define and measure the subjective metric of "Ethical Robustness". **Q5: How to ensure its applicability across different types of T2V models?** Thank you for your insightful comment. Regarding the preprocessing problem of different types of videos, we further analyze this from the following perspectives. **Analysis:** - **Discussion on the preprocessing of different types of video.** We reviewed over 200 video generation literature from a variety of fields. It is certain that the vast majority of articles, at least in the field of T2V, **do not further preprocess** the video to be evaluated [3,12,13], as further processing would have an impact on some of the characteristics of the original video, while only simple scaling would instead better preserve and evaluate the **model's original capabilities**. This point can be further illustrated with a few examples below: - For models that aim to produce high frame rate video [9], it is **not fair** to sample a uniform frame rate during the evaluation process, as this approach undermines the strength of the model. - For studies specializing in the generation of long videos [10], it is also **unfair** to extract frames or cut short pieces of the generated results to compare with short videos, because there is no guarantee that the displayed results can reflect the original advantages of the study. - For articles emphasizing text alignment [11], if the original video is cropped or watermarks are added, the original visible content may be cropped or blocked, which is also an **unfair** comparison. - **Video comparison of different lengths.** We do not apply any extra processing to the length of the video, and annotators are asked to view the video in its entirety before annotating it, thus ensuring a fair and reliable comparison. We also experimented with comparing VideoCrafter [14] with Sora [15], which proved that even with video length editing, frame rate processing, etc., the annotators all agreed that Sora was significantly better than VideoCrafter on all dimensions. **Conclusion:** - In order not to bias any model in comparison, we use the simplest scaling process, which only guarantees that **all videos can be displayed clearly**. Large-scale literature research confirms the fairness of this approach, because any further processing may **undermine** the advantages of some models when the evaluation scope is very wide.

Rebuttal (4/4)

**Q6: Would the survey of non-T2V evaluation approaches be beneficial when merging with this technology? With the lack of maturity and rapidly evolving field, would not benefit from a cross-domain understanding from more mature fields?** Thank you for your insightful comment. We analyze it from the following perspectives. **Analysis:** - The guidelines and training proposed in this study can **be easily extended** to the T2I and NLG community studies with only user interface modifications, and similarly, by modifying the automated metrics in the dynamic module, one can also achieve efficient human evaluation of other types of protocols. - This study drew on some well-established T2I and NLG human assessment studies [2,16] to construct a more comprehensive protocol in terms of some important principles, such as ensuring clarity of definitions and reliability of annotated results, as well as the design of the subjective indicators, which are lacking in existing T2V assessment studies. However, the design of specific indicators, such as the four objective indicators, was derived from specific T2V research needs. - We also plan to design prompt datasets specifically for evaluating T2V models based on the principles of dataset construction in T2I and NLG fields in the future. **Conclusion:** - Our protocol builds on previous non-T2V evaluation studies and T2V model studies, and the proposed annotator training principles and dynamic evaluation components can also contribute to the development of non-T2V evaluation studies. **References:** [1] All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. IJCNLP 2021. [2] Evaluation of Text Generation: A Survey. [3] Towards A Better Metric for Text-to-Video Generation. [4] Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. _Journal of the American Statistical Association_, _62_(317), 194-204. [5] Rank analysis of incomplete block designs: I. The method of paired comparisons. _Biometrika_, _39_(3/4), 324-345. [6] Holistic Evaluation of Text-to-Image Models. NIPS 2023. [7] How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions? EMNLP 2022. [8] Adopting and expanding ethical principles for generative artificial intelligence from military to healthcare. NPJ Digital Medicine 6, Article number: 225 (2023). [9] ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation. [10] ControlVideo: Training-free Controllable Text-to-Video Generation. ICLR 2024. [11] Seer: Language Instructed Video Prediction with Latent Diffusion Models. ICLR 2024. [12] VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR 2024. [13] AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI. [14] VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models. TBench 2024. [15] OpenAI Sora. [16] Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation. CVPR 2023.

To save reviewer's time, we put a summary of rebuttal

Dear Reviewer AEKZ, Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we **summarized our responses here**. 1. \[**Lacks an understanding of evaluation in other domains** \]: **Response:** - Our protocol is based on a thorough investigation of human evaluation and video generation articles in different fields and is experimentally validated to have the advantages of reproducibility, reliability, and practicability compared to traditional protocols. 2. \[**Dynamic evaluation module may introduce bias** \]: **Response:** - Theoretically, our protocol design allows for plausible estimation in the absence of numerous comparisons, and we further validate this capability through bootstrap confidence interval experiments.. 3. \[**Lack detailed analysis of cases where the protocol fails. Need further evaluation and consideration in other domains** \]: **Response:** - We analyzed the case of producing inconsistent results under the traditional protocol setup, demonstrating the importance of the annotator training we designed, and conducted a reliability analysis of the dynamic evaluation component, which proved that the introduction of this component does not result in inconsistent protocol results. - Our design is not only to provide a more reliable, reproducible, and practical T2V assessment protocol but also to provide ideas and reference materials for human evaluation research in other fields.. 4. \[**Small typo errors** \]: **Response:** - Thanks for the reminder. These typo errors are caused by automatic line breaks during latex compilation, not actual errors in the text editor. 5. \[**How the dynamic evaluation module handles potential biases** \]: **Response:** - From a methodological design perspective, the Rao and Kupper model we used can accurately estimate model scores using incomplete comparisons. - From an experimental validation perspective, our rigorous bootstrap confidence interval experiments confirm that even after discarding many comparisons on top models such as Gen2, the final estimate of the relative strength of the model is still accurate and credible. 6. \[**How to capture a representative sample** \]: **Response:** - The static evaluation in the protocol retains the samples that are most worthy of being annotated, ensuring that the evaluation results won't be biased by the lack of specific kinds of prompts and model combinations. 7. \[**How to handle potential biases and potential lack of diversity** \]: **Response:** - We minimize annotator bias in assessing objective metrics by providing detailed guidelines and training processes, and we retain annotator diversity by introducing subjective metrics.. 8. \[**Provide more details on "Ethical Robustness"** \]: **Response:** - Our survey shows that while the measurement of the ethics of generated content is missing from existing T2V evaluation protocols. Therefore, after a thorough investigation of the relevant research, we use three reference angles, i.e., Toxicity, Fairness, and Bias, to define and measure the subjective metric of "Ethical Robustness". 9. \[**How to ensure its applicability across different types of T2V models** \]: **Response:** - In order not to bias any model in comparison, we use the simplest scaling process, which only guarantees that all videos can be displayed clearly. - Large-scale literature research confirms the fairness of this approach, because any further processing may undermine the advantages of some models when the evaluation scope is very wide. 10. \[**Cross-domain understanding** \]: **Response:** - Our protocol builds on previous non-T2V evaluation studies and T2V model studies, and the proposed annotator training principles and dynamic evaluation components can also contribute to the development of non-T2V evaluation studies. Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments. Best Regards, Authors

Looking forward to the reply

Dear Reviewer AEKZ, As we draw closer to the rebuttal deadline, we would like to inquire if there are any additional questions or concerns about our work. We greatly value the feedback. Thanks again for the time and effort in our work! Best wishes, Authors

Rebuttal (2/4)

**W3: Citing and considering using ELO scores.** Thank you for your insightful comment. We recognize the significance of the ELO rating system and its broad applications in model comparisons. Below, we detail our response and the reasons for choosing the Rao and Kupper model. **Analysis:** - **ELO Rating as a Bradley-Terry Special Case:** The ELO rating system can be viewed as a special case of the Bradley-Terry model [8]. It estimates object scores based on the outcomes of pairwise comparisons, and the resulting calculated scores are fed into the sampling algorithm to determine subsequent competition model pairs [10]. - **Ties and Dynamic Selection:** While the ELO rating system provides a robust framework for scoring and ranking, it doesn't inherently account for ties, which can occur frequently in our dataset. In contrast, the Rao and Kupper model [9], which we adopted, is a modification of the Bradley-Terry model that accommodates tied comparison results. This capability is crucial for our specific use case, where annotators may not always have a clear preference. - **Rao and Kupper Model Advantages.** (1) **Handling Ties:** The Rao and Kupper model's ability to handle tied outcomes allows for more precise scoring and ranking when annotators express equal preference between models. (2) **Dynamic Pair Selection:** Although both models support dynamic pair selection, the Rao and Kupper model's handling of ties makes it more suitable for scenarios where comparisons may not always result in a decisive winner. - **Consideration for Future Work:** We acknowledge the benefits of the ELO system's dynamic selection algorithm and will consider incorporating these insights into our future research to enhance our model ranking process. We will also add discussions and references to similar content in our article revisions. **Conclusions:** - While the ELO rating system is powerful, the Rao and Kupper model **better fits the requirements of our task** due to its flexibility in handling ties. - The main purpose of our work is to **design reliable, reproducible, and practical human evaluation protocol for T2V models**, but we will also build and maintain a public, large-scale, long-term project in the future.

Rebuttal (3/4)

**W4: Few details are revealed in this paper. Standardizing the height of the video across models may not be the right thing to do. How to best show videos that are generated at different framerates and even different lengths.** Thank you for your insightful comment. We provide full details of the protocol in the article and **Appendix**. For your ease of reference, we provide **the location of all details** in the table below. || **Evaluation metrics**|**Scoring methods**|**Evaluator requirements**|**Dynamic component**| |-|-|-|-|-| | **Position** | Section 4.1, Line 177-197$ $ Appendix C.4, Line 829, Table 4 | Section 4.2, Line 199-219$ $ Appendix C.3, Line 787-820 | Section 4.3, Line 221-253 | Section 4.4, Line 254-271$ $ Appendix C.2, Line 737-786 | || **Annotator training** |**Annotation interface** | **Details of the models evaluated**| **Details of the videos evaluated** | |-|-|-|-|-| | **Position** | Section 4.3, Line 230-234 $ $ Figure 1 (c) and Figures 5 to 9 | Section 1, Line 41-45$ $ Figure 1 (b) | Section 5, Line 273-276$ $ Appendix C.1, Line 727-736 | Section 5.1, Line 278-283 | We've revised the article to include more detailed information in the body of the article and will **open source** all the relevant interfaces and code. In addition, we are willing to provide **any information** related to the protocol to help the community. Regarding the preprocessing problem of different types of videos, we further analyze this from the following perspectives. **Analysis:** - First of all, even if VideoPoet [19] produces videos in portrait aspect ratios, its characteristics such as clarity in the evaluation interface **won't be affected**. - Specifically, T2VHE take the highest video among the videos to be evaluated as the video height displayed in the annotation interface, and zoom out of the videos smaller than this height, so as to guarantee that **every video can be displayed clearly**. We will add an explanation of this issue in Section 5.1 to avoid this misunderstanding. - **Discussion on the preprocessing of different types of video.** We reviewed over 200 video generation literature from a variety of fields. The vast majority of articles, at least in the field of T2V, **do not further preprocess** the video to be evaluated [2,4,5], as further processing would have an impact on some of the characteristics of the original video, while only simple scaling would instead better preserve and evaluate the **model's original capabilities**. This point can be further illustrated with a few examples below: - For models that aim to produce high frame rate video [11], it is **not fair** to sample a uniform frame rate during the evaluation process, as this approach undermines the strength of the model. - For studies specializing in the generation of long videos [12], it is also **unfair** to extract frames or cut short pieces of the generated results to compare with short videos, because there is no guarantee that the displayed results can reflect the original advantages of the study. - For articles emphasizing text alignment [13], if the original video is cropped or watermarks are added, the original visible content may be cropped or blocked, which is also an **unfair** comparison. It must be admitted that the preprocessing methods used in the various articles may be fairer for particular comparisons, but to fit the protocol into the extensive T2V studies, simply scaling the video is a viable approach for human evaluation protocols. - **Video comparison of different lengths.** We do not apply any extra processing to the length of the video, and annotators are asked to view the video in its entirety before annotating it, thus ensuring a fair and reliable comparison. We also experimented with comparing VideoCrafter [14] with Sora [15], which proved that even with video length editing, frame rate processing, etc., the annotators all agreed that Sora was significantly better than VideoCrafter on all dimensions. **Conclusion:** - In order not to bias any model in comparison, we use the simplest scaling process, which only guarantees that **all videos can be displayed clearly**. Large-scale literature research confirms the fairness of this approach, because any further processing may **undermine** the advantages of some models when the evaluation scope is very wide.

Rebuttal (4/4)

**W5: Explain clearly what the training looks like for aesthetic quality.** Thank you for your insightful comment. We analyze it from the following perspectives. **Analysis:** - We provide a **detailed display** of the training content in Figure 1 (c) and Figures 5 to 9. Specifically, for each reference angle, we provide a set of **reference videos** with the **corresponding analysis process**. For your reference, a detailed description of the "Aesthetic Appeal" and corresponding training examples can be found in Table 4 and Figure 1 (c). - **Objective and subjective metrics are relative.** Our division between subjective and objective metrics stems mainly from the fact that for subjective metrics, such as human preferences, we cannot find a universally recognized and workable description for annotators to adhere to. Whereas for objective metrics, such as "Aesthetic Appeal", several studies have rigorously defined these [2,18], we can design effective training examples based on previous research and ask the annotators to try to be objective in their judgments. - **Annotators are different from automatic scorers**. While they can understand the concept of more abstract indicators, **no amount of detailed guidance is enough** to get them rid of their subjective influence [7,16,17]. However, this effect can be reduced through annotator training. As shown in Table 1, there is better consistency in the annotators after the training. **Conclusions:** - The distinction between subjective and objective indicators stems from whether we can ask the annotator to evaluate according to predefined, accepted, and feasible criteria. - Although it is impossible to achieve a completely objective human assessment, experiments show that we can **effectively reduce the influence** of personal preferences on results through well-designed assessment protocols and detailed annotator training. **M1: Being clear that “higher is better” in many plots.** Thank you for your comment, we have added that in the table captions. **M2: Using a lot of acronyms may impede readability in various parts of the paper.** Thanks to your suggestion, we have reduced using acronyms in the text, particularly in section 4.3. **M3: Disagree with the characterization of FVD being about temporal consistency.** Thank you for your comments, we have amended the description of the FVD indicator within the related work section. **M4:Recommend seeing/citing the Videopoet paper.** As the survey covers video generation work in many areas such as T2V, video editing, frame prediction, and motion control, as well as human evaluation work in related areas such as T2I, NLG, and so on, we apologize for neglecting such excellent work as Videopoet [19] in our research. Thanks for the reminder, we've read the work carefully and cited it in the revised article. **References:** [1] EvalCrafter: Benchmarking and Evaluating Large Video Generation Models. CVPR 2024. [2] VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR 2024. [3] Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning. CVPR 2022. [4] Towards A Better Metric for Text-to-Video Generation. [5] AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI. TBench 2024. [6] FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation. NIPS 2023. [7] Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation. CVPR 2023. [8] Rank analysis of incomplete block designs: I. The method of paired comparisons. _Biometrika_, _39_(3/4), 324-345. [9] Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. _Journal of the American Statistical Association_, _62_(317), 194-204. [10] Chatbot Arena. [11] ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation. [12] ControlVideo: Training-free Controllable Text-to-Video Generation. ICLR 2024. [13] Seer: Language Instructed Video Prediction with Latent Diffusion Models. ICLR 2024. [14] VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models. TBench 2024. [15] OpenAI Sora. [16] All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. IJCNLP 2021. [17] Evaluation of Text Generation: A Survey. [18] Holistic Evaluation of Text-to-Image Models. NIPS 2023. [19] VideoPoet: A Large Language Model for Zero-Shot Video Generation. ICML 2024.

To save reviewer's time, we put a summary of rebuttal

Dear Reviewer 2pJH, Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we **summarized our responses here**. 1. \[**Value over prior approaches** \]: **Response:** - While automated scorers can implement evaluations at lower cost, their inherent poor flexibility, and high expansion costs result in the fact that they are still unable to fully replace human evaluation protocols. - The design of better human evaluation protocols is itself an important prerequisite for the training of more reliable automated scorers. 2. \[**Include more models or show human evaluations do not correlate FID or FVD** \]: **Response:** - We evaluate the five most advanced video generation works available at the start of the research. This amount is in line with other benchmarking works, and we will conduct more model evaluations later. - Since there are no corresponding reference videos for the prompt source, we cannot calculate the corresponding FID and FVD indicators, but we will supplement this when expanding our prompt source in the future. 3. \[**Citing and considering using ELO scores** \]: **Response:** - While the ELO rating system is powerful, our Rao and Kupper model better fits the requirements of our task due to its flexibility in handling ties. - We will also build and maintain a public, large-scale, long-term project in the future. 4. \[**Few details. How to best show videos with different characteristics** \]: **Response:** - In our response Rebuttal (3/4), we provided the locations of all the protocol details revealed and we've revised the article to include more detailed information in the body of the article. - We use the simplest scaling process to avoid biasing any model in comparison and guarantee that all videos can be displayed clearly. - Large-scale literature research confirms the fairness of this approach, because any further processing may undermine the advantages of some models when the evaluation scope is very wide. 5. \[**Some things marked as objective are not objective** \]: **Response:** - The distinction between subjective and objective indicators stems from whether we can ask the annotator to evaluate according to predefined, accepted, and feasible criteria. - Experiments show that we can effectively reduce the influence of personal preferences on results through well-designed assessment protocols and detailed annotator training. 6. \[**Minor quibbles** \]: **Response:** - Thanks to your suggestion, we have revised the article to address the concerns raised. Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments. Best Regards, Authors

Looking forward to the reply

Dear Reviewer 2pJH, As we draw closer to the rebuttal deadline, we would like to inquire if there are any additional questions or concerns about our work. We greatly value the feedback. Thanks again for the time and effort in our work! Best wishes, Authors

Looking forward to the reply

Dear Reviewer 2pJH, We appreciate your constructive comments and have **responded in detail to the five weaknesses**, while we also **revised the article** to address the minor quibbles raised. As we draw closer to the rebuttal deadline, we would like to inquire if there are any additional questions or concerns about our work. We greatly value the feedback. Thanks again for the time and effort in our work! Best wishes, Authors