Summary
This paper presents an improved video VAE (IV-VAE) model designed to enhance spatio-temporal compression and reconstruction quality in latent video diffusion models. It proposes a Keyframe-based Temporal Compression (KTC) architecture and a Group Causal Convolution (GCConv) mechanism. These components address limitations of conventional 3D VAEs by improving temporal compression initialization, reducing inter-frame inconsistency, and increasing the receptive field to better capture motion across high-resolution frames. Extensive evaluations on datasets such as Kinetics-600 and MotionHD demonstrate IV-VAE’s improved spatio-temporal consistency, frame fidelity, and resistance to temporal flickering compared to other state-of-the-art methods.
Strengths
* The paper’s key contributions, namely the KTC and GCConv modules, demonstrate some improvements over conventional approaches in both compression quality and temporal consistency. By dedicating latent channels for keyframes, IV-VAE provides a balanced spatiotemporal initialization that improves compression, especially when compared to standard 3D-causal VAEs that struggle with frame flickering due to forward-only information flow.
* The introduction of the MotionHD dataset is a valuable contribution for future works, as it fills a gap in testing high-resolution and high-motion content, providing an essential benchmark for video VAE reconstruction capabilities.
* Quantitative comparisons, such as FVD, PSNR, and SSIM, highlight IV-VAE’s effectiveness across various datasets, showing that it surpasses models like OD-VAE and OS-VAE with significantly fewer parameters.
Weaknesses
* Missing Comparisons with Recent Works: The paper lacks citation and comparative discussion of related work like Video Probabilistic Diffusion Models (Video Probabilistic Diffusion Models in Projected Latent Space CVPR 2024) and Hybrid Video Diffusion Models (Hybrid Video Diffusion Models with 2D Triplane and 3D Wavelet Representation, ECCV 2024). Including references to these models and their qualitative/quantitative results would strengthen the argument, especially since both focus on enhanced autoencoder backbones for video generation.
* Figure 1 Clarity Issues: Figure 1 is particularly hard to interpret due to insufficient labeling and unclear meaning of the dashed lines in various colors. For example, the significance of the green dashed arrow is not explicitly explained, which may lead to confusion regarding the model’s initialization and compression gains. Clarifying these lines would improve readers’ understanding of the proposed approach.
* Ambiguous Explanation in Introduction: The fifth paragraph of the Introduction is challenging to follow, and Figure 1(b) does not align with the accompanying text. For instance, it is difficult to reconcile why CogX-VAE’s third frame outperforms other frames or why OS-VAE’s second frame performs better than its third. This discrepancy suggests a need for clarification or additional discussion regarding these frame-level performance inconsistencies.
* Inconsistent Model References: The model names, such as OD-VAE, OS-VAE, and CogX-VAE, are not clearly linked to their respective citations until later sections, like 4.3. This misalignment disrupts the reader’s ability to match models with their attributes or methods and should be addressed earlier for clarity.
* Technical Ambiguities in Section 3.1: The notation [0, 0, W_{2D}] for W_{3D} is unclear, particularly as it relates to causal padding and convolution. The authors could improve this by specifying whether these are zero matrices or zero-value vectors, which would make the method implementation more understandable.
* Metric Reporting and Alignment with Core Objective: Given the focus on spatiotemporal quality, metrics specifically targeting spatio-temporal fidelity must be also compared, such as STREAM (ICLR 2024 STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models). As this metric evaluates both spatial and temporal coherence, its inclusion would further validate the model’s strengths in these dimensions.
* Ambiguity in Frame Group Description (Section 3.2): What does the statement, "The number of frames in the frame group is not fixed during the forward pass," mean? Doesn't the t_c = 4 setup mean a fixed number (4) of frames in the frame group?.
* Please mention the fact that the first frame acts as a keyframe in the text (Section 3.3). Currently, I believe it is mentioned only in a figure. Additionally, is Z/2 = C_{\text{out}}? Also, please clarify the exact mechanism for assigning weights to temporal upsampling and downsampling layers so that readers can understand why and how it enables the network to independently process two different frames within a frame group.
* In Section 3.3, why is it needed to maintain two branches after converting GCConv3D to 2D after two temporal downsampling stages?
* Please consider using other examples and visualization for the qualitative comparison to show the differences better. Currently, it is hard to notice the differences.
* Please use the original name format for each model. For example, "Magvit" -> " MagViT".
* Will MotionHD dataset be available for public use? Please make sure it is available for reproducibility.