CV-VAE: A Compatible Video VAE for Latent Generative Video Models

Summary

The paper introduces CV-VAE, a 3D VAE that is trained through latent space regularization using an existing two-dimensional VAE decoder. This design facilitates seamless integration with 2D VAE-based diffusion models. The concept, while simple and straightforward, is notably efficacious and offers practical utility. Moreover, relative to conventional 2D VAEs, the proposed framework significantly compresses the latent space dimensionality by integrating temporal downsampling within the 3D VAE architecture.

Strengths

1. The method is novel and delicate, which effectively compresses video latents and maintains latent distribution. 2. Experiments and visualizations demonstrate that CV-VAE performs equally to SD VAE on image-based/frame-based encoding and falls marginally behind on video encoding with little cost of finetuning SD models.

Weaknesses

1. As presented in Figure 6, CV-VAE suffers from more severe flickering problems than the original SVD, which may possibly be due to the gap between the latent distribution of CV-VAE and SD-2.1. In the paper, the author also pointed out that text-to-image generation may experience differences in color. Authors may convince readers that such flaws can be overcome through further fine-tuning, e.g., fine-tuning full parameters of SVD, or fine-tuning for longer iterations. 2. Missing details about training: The authors didn’t present details about training batch size, augmentations, video fps, etc. Also, whether the default CV-VAE displayed in the experiment section is trained on video and image data separately or jointly (as vae of Opensora) remains unknown.

Questions

1. Please refer to weakness. 2. In Table 3 and figure 6, authors may also present the metrics of SVD on 97 frames straightly or by using the frame interpolation model of SVD. 3. Authors may specify whether the decoder-based regularization is still employed when training with images as inputs for CV-VAE. 4. Authors may also present metrics comparison between SVD and SVD + CV-VAE on text-to-video generation. 5. Can the method be supported by theoretical proof? For example, can we minimize the distribution distance between video latents of CV-VAE and one of corresponding image latents encoded by 2d VAE?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper designs a 3D VAE consistent with the 2D VAE, whose output mode can losslessly switch between 2D and 3D while retaining the characteristics of both the 2D VAE and 3D VAE. This allows for obtaining performance similar to the original 2D VAE while increasing the frame count of the 3D output.

Strengths

1. Designing a 3D VAE with 2D latent consistent characteristics is very meaningful, and this paper successfully achieves this. 2. The bidirectional VAE alignment method designed in this paper is very clever. A similar design is found in [1], but the scenario used in this paper is completely different. 3. Achieving effective performance in both 2D and 3D simultaneously is of high practical value to the community, for example, supporting pre-training and frame interpolation in video Diffusion models. 4. The writing is fluent, and the experimental design is excellent. [1] Christensen, Anders, et al. "Image-free classifier injection for zero-shot classification." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Weaknesses

1. Figure 10 attempts to demonstrate the effect of the 3D VAE on frame count enhancement. I understand that the conclusion of this figure is actually the most critical value proof of this paper: retaining the 2D VAE latent consistency while having the embedding capability of 3D features, such as extending the frame count. This is quite challenging, as the 3D VAE might tend to completely collapse to perform similarly to the original 2D VAE, simply replicating the 2D results. The loss between replicated 2D frames is very small, making optimization difficult. While the distance of the horse's legs in Figure 10 might prove it's not direct replication, the other two cases are hard to distinguish differences. Can the authors provide more evidence to explain that the model is not just learning simple replication? 2. In Table 1, when comparing, which pretrained 2D VAE is aligned with "Ours"?

Questions

1. Would training the VAE with bf16 cause a performance collapse? 2. In Lines 218-219, how are the images of 256 × 256 and 512 × 512, as well as the videos of 9 × 256 × 256 and 17 × 192 × 192 organized during the training cycle? Do you train on images first, then videos, or start with low resolution and then move to high resolution? 3. What impact does training the VAE at low resolution or low frame rate have when performing SVD on high resolution and high frame rate?

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

Author added a limitation section in paper and I think there is no obvious potential negative societal impact.

Summary

This paper proposes a new video VAE, starting with pretrained image VAE. It includes several techniques that are still capable of handling images and do not suffer from significant computational overhead. First, they use pretrained weights of 2D VAE (from Stable diffusion) by using their weights as initialization with model inflation. They also used efficient 3D architecture to only half of 2D convolutions as 3D convolutions to minimize the computation increase and also applied temporal tiling to handle long videos. They also use a pretrained frozen image decoder to regularize the latent space from the 3D encoder/encoder to ensure the latent space becomes similar to the stable diffusion latent space. The paper shows the proposed video encoder can have a compact latent space with a temporal compression factor (of 4) but is very similar to pretrained image latent space, which can be used for longer and smoother video generation by finetuning existing latent video diffusion models in this latent space.

Strengths

- The paper is generally well-written and well-motivated. In particular, the paper tackles an important problem of constructing a compact latent space of videos that is similar to image space. - The performance drop from fine-tuning does not seem that large and shows a good potential for training a latent video generation model. - The paper shows a real use-case of this VAE by finetuning stable video diffusion.

Weaknesses

- The paper lacks a comparison with recent video autoencoders. Specifically, the baselines that the paper provided are quite outdated (e.g., TATS and VQGAN), considering that there are many recent attempts to design a better video autoencoder, to name a few [1, 2, 3]. The authors should discuss what's the pros and cons of this approach compared with these approaches, and if possible, the authors should compare the performance as well. In particular, [2] provides very similar intuition to this paper because they also try to construct a latent space capable of jointly handling images and videos. - Lack of novelty: The proposed method is mainly composed of components that have already been widely used to extend image models to videos. - The paper does not mention the total training time (even though it includes the number of GPUs and the total number of training iterations). [1] Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023. [2] Language Model Beats Diffusion: Tokenizer is key to visual generation, ICLR 2024. [3] Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition, ICLR 2024.

Questions

- The paper only provides video generation results on image-to-video generation tasks; I wonder if one naturally achieves more smooth video generation model by fine-tuning existing text-to-video generation models in this latent space (such as modelscope). - Are you planning to release the code and the model parameters?

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

The paper addresses the limitation appropriately.

Summary

This paper focuses on a compact video VAE suitable for both image and video generation tasks. The motivation stems from the absence of spatial-temporal compressed continuous latent 3D VAE models for video generation. To effectively utilize 2D VAE and seamlessly integrate image generation models, the authors propose a latent space regularization and initialization method to maximize the use of 2D VAE. A 3D CNN-based architecture is also proposed to compress both temporal and spatial dimensions efficiently. The evaluation of the proposed method is primarily based on reconstruction tasks and image/video generation tasks.

Strengths

The motivation is clear: the current video generation method may suffer from a lack of high-quality and high-efficiency spatial-temporal VAE if it can be used with the existing T2I model, which also makes it more general and effective. The decoder loss seems a simple and novel design, which helps the model learn spatial latent efficiently and align with the original 2D VAE. Initialization of 3D VAE with 2D ones seems to provide some insights in practice. The qualitative results are good.

Weaknesses

1. Improvements over the baseline are presented in Table 1, where the authors compare the main reconstruction results with different open-sourced VAE methods. However, I don’t think it is a fair comparison to claim superior performance. The training data is a key factor in this table. The authors use in-domain data, such as Webvid, for both training and evaluation, whereas other methods do not. Therefore, it is unclear if the proposed method can truly outperform other methods when trained on the same data. 2. Many generation models have been released based on SD VAE. Since compactness does not require further fine-tuning, it is crucial to evaluate more models to strengthen the claim. Evaluating only one model in Table 2/3 is not convincing.

Questions

The term "mapping function" seems to require clarification, as it appears to be more of a simple sampling process from video frames. In Table 7, a higher CLIP score is better. Why does random sampling work best? Providing more exploration and explanation can help readers understand this better.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes, the author discussed the limitations and potential social impact.

Thanks for rebuttal.

I have no further questions, and I stand by my rating. Personally, I believe the quality and contribution of this paper generally meet the bar for NeurIPS.

Response

Thanks for the clarifications. Many of my concerns are now addressed. Please add this discussion, particularly for a comparision with MAGVIT2 because it shares lots of similarities to this work. I have a few more questions: 1. What is a meaning of "Comp" in Table 1 in the rebuttal PDF file? 2. I checked the MAGVIT2 repository and it seems there are two versions available - the first one uses 16x downsampling and the second uses 8x downsampling factor for training and inference. Which model is used for the comparison?

Thanks for the rebuttal

The additional results addressed my concerns.