Summary
This work introduces a novel non-autoregressive method, GLOBER, that first generates global features for comprehensive global guidance and then synthesizes video frames based on these global features to produce coherent videos. The authors propose a video auto-encoder to encode videos into global features and a video decoder to decode the global features and synthesize video frames in a non-autoregressive manner. Notably, the video decoder uses normalized frame indexes to perceive temporal information, allowing it to synthesize any video clips with predetermined frame indexes. The authors also introduce a unique adversarial loss to enhance global coherence and local realism of the synthesized video frames. Finally, a diffusion-based video generator is employed to fit the global features produced by the video encoder for video generation. The effectiveness and efficiency of the proposed method are demonstrated through extensive experiments, and it sets new state-of-the-art results on multiple benchmarks.
Strengths
1. The inclusion of Coherence and Realism Adversarial Loss is a novel approach compared to previous diffusion-based architectures.
2. Extensive experiments have been performed on various benchmarks, all demonstrating the significance of GLOBER.
Weaknesses
1. The authors identify VideoFusion as the most closely related work due to its use of non-autoregressive generation. However, there are other public models, such as ModelScope Text-to-Video, that use non-autoregressive generation in the latent space similarly to the authors' work. I suggest that the authors compare their work to these models as well.
2. It appears that GLOBER outperforms VideoFusion in all tasks, which generates videos in the pixel space. This superiority seems to result from the CRA loss proposed by the authors. Therefore, a direct comparison between GLOBER (without CRA loss) and VideoFusion would be intuitive. However, inconsistent results in Table 1 and 3 make this comparison unfeasible. Could the authors explain this inconsistency and provide justifications for GLOBER's superiority over VideoFusion, aside from the CRA loss?
3. My interpretation of Equation 9 suggests it's an estimation of the frame feature. However, this estimation might not be accurate because, like in DDPM (or DDIM), one could perform T steps of reverse denoising to generate images. What is the quality difference between these two types of images? I assume that the frame feature generated by Equation 9 will be of lower quality.
4. In Table 2, why does GLOBER use 50+50 diffusion steps?
5. The optimization objective of Equation 2 in the Video Encoding section appears to be derived from the Variational Autoencoder. Could the authors provide a justification for this design? My understanding is that video encoding trains a dataset-dependent distribution to be sampled as z_t.
6. In Line 127, "Gauss distribution" appears to be misspelled.
7. The authors seem to have overlooked specifying the dimension of C'.
My score could be revised upward if my concerns are adequately addressed.
**Reference:**
[1] ModelScope Text-to-Video Technical Report, arXiv. (Model: **https://modelscope.cn/models/damo/text-to-video-synthesis/summary**).
Questions
Please refer to the weaknesses.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
In this work, the authors mentioned limitations and broader impact.