VFIMamba: Video Frame Interpolation with State Space Models

Summary

Based on the popular S6 model's advantages of linear computational complexity and data-independent modelling capability, this paper applies it to VFI. Specifically, this paper proposes a token rearrangement strategy to learn the information of adjust frames, in addition to introducing a curriculum learning strategy to dynamically learn various motion magnitudes between adjust frames through joint training of vimeo90K and X-TRAIN. The model achieves the highest performance on existing commonly used VFI datasets.

Strengths

1. This paper is the first to adapt the S6 model to the VFI. 2. Experimental results show that the proposed VFIMamba achieve performance while using competitive FLOPs.

Weaknesses

1. It is recommended that the authors can demonstrate the accuracy of the interpolation results more intuitively by visualizing the error maps. 2. The experiments on table 4 are to demonstrate the computational validity of S6 model, and it is necessary to provide flops for the different models. 3. In limitations, the author says that the VFIMamba has faster speed, I would like to know the comparison of the runtime in table 2. 4. Lack of citations essential to the field, such as VFIT, TTVFI, ABME, EDSC, etc.

Questions

I would like to see more analysis on the model effectiveness, such as the runtimes in Table 2.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss methodological limitations in the supplementary material.

Summary

This paper introduces a novel video frame interpolation (VFI) method called VFIMamba. VFIMamba is the first method that combines the State-Space Model Mamba with VFI architectures and therefore, it has the advantage of a linearly growing complexity w.r.t. the resolution while maintaining the ability to utilize global receptive fields similar to vision transformers. In order to apply the idea to VFI, architectural modifications have been proposed to handle 2 frames as input. Further, a novel curriculum learning strategy is used to increase the models performance across a large range of motions. The method has been evaluated on multiple datasets and w.r.t. to various other methods, achieving state-of-the-art PSNR values, especially improving the performance for high-resolution frames (2K and 4K).

Strengths

- Combination of an emerging alternative to transformers (MAMBA) with VFI methods that allow for higher resolution frame interpolation due to linear complexity growth. The paper is interesting to read. - Discussion and evaluation of different sequence arrangements. - Introduction of a relatively simple curriculum learning strategy for VFI with experiments confirming that this strategy is beneficial for VFI methods in general when having to deal with large motions. - Reaches new state-of-the-art performance - Exhaustive ablation studies proving the effectiveness for each of their introduced modules - Supplement contains a video with qualitative comparisons, although only a few short sequences and given the coarse time steps, it is difficult to clearly judge the temporal consistency.

Weaknesses

- It is unclear how frames at arbitrary time steps are computed to perform 8x interpolation in table 3. It should also be directly clear from the caption of table 3, that 8x interpolation is evaluated (these details are only in the appendix). - In general, a lot of important information is in the appendix. It is helpful to add at least a reference from the main paper to the appendix that there is more information, e.g., such as the evaluation and the experiment on generlalization of curriculum learning. - Unclear how the FLOPs requirement scales with resolution. A plot/table showing the FLOPs compared to the input resolution for VFIMamba and other methods would be interesting to get a better feeling for the scaling w.r.t. frame resolution and a comparison of memory footprints might be interesting. - Ideally, it should be mentioned in table 2 and 3 on which dataset the other methods have been trained on, especially for methods where the original paper proposed multiple versions such as XVFI. - Some recent methods especially for high-resolution data are missing in Table 3 such as [A] and [B]. Especially [A] has only been included in Tab. 2 although their focus is also on high-resolution datasets and code for X-Test is directly available: https://github.com/feinanshan/M2M_VFI/blob/main/Test/bench_xtest.py [A] Hu et al. Many-to-many Splatting for Efficient Video Frame Interpolation, 2022 [B] Nottebaum et al., Efficient Feature Extraction for High-resolution Video Frame Interpolation, 2022. It would be nice to have longer sequences, and playing them a bit more fluently to get a feeling for the temporal and visual consistency. Minor: - Some colors difficult to see in overlay in Figure 1. - L. 126, Check sentence “where contains”

Questions

- Table 3: There is a discrepancy between many of the reported number in their respective original papers and in this paper on X-TEST for 2k and 4k, e.g., EMA-VFI-S reported 30.89dB instead of 29.91dB and BiFormer reported 31.32dB instead 31.18. Based on the experience of the reviewer it is possible to reproduce these numbers using the correct evaluation protocol. - How is 8x interpolation performed? - Unclear why there is no FLOPs measurement and no evaluation of SNU-FILM for SoftSplat given that the source code + trained models are publicly available (https://github.com/sniklaus/softmax-splatting)

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

Discussion of limitations is only in the appendix. It would be better to discuss them already in the main paper. Additionaly, has the method similar limitations as MAMBA (the base model for this entire work). Therefore, the compute requirements are still relatively high even though they do not have to compute an attention matrix anymore.

Summary

The paper presents a novel approach for video frame interpolation using Selective State Space Models (S6). The authors introduce VFIMamba, a method designed to efficiently and dynamically model inter-frame information. This method features the Mixed-SSM Block (MSB), which rearranges tokens from adjacent frames in an interleaved manner and applies multi-directional S6 modeling. Additionally, the paper proposes a curriculum learning strategy to progressively improve the model's ability to handle varying motion magnitudes. Experimental results show that VFIMamba achieves state-of-the-art performance on various benchmarks, especially in high-resolution scenarios.

Strengths

Originality: The introduction of the S6 model into video frame interpolation tasks is a novel contribution. The use of Mixed-SSM Blocks and the interleaved token arrangement are creative solutions to enhance inter-frame modeling. Quality: The paper presents thorough experiments and comparisons with state-of-the-art methods. The quantitative results demonstrate significant improvements in performance, particularly in high-resolution and large-motion scenarios. Clarity: The methodology is clearly explained, with detailed descriptions of the proposed model components and training strategies. The visualizations and tables effectively support the claims made in the paper. Significance: The VFIMamba method addresses key challenges in video frame interpolation, such as the need for large receptive fields and efficient computation.

Weaknesses

Real-time Application: Although VFIMamba achieves high performance, it still falls short of real-time requirements. The paper could benefit from a discussion on potential strategies to improve inference speed. Performance Gap Analysis: On low-resolution datasets, VFIMamba-S fails to yield second-best scores on most benchmarks and only outperform baselines with comparable FLOPs (e.g., EMA-VFI-S) by a relatively small margin.

Questions

1. What specific optimizations could be applied to VFIMamba to make it suitable for real-time applications? Are there trade-offs between speed and accuracy that need to be considered? 2. Could the authors elaborate on the specific factors contributing to the performance gap, and suggest potential directions to address these issues?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have acknowledged several limitations of their work, including the resource-intensive nature of training VFIMamba and the current inability to meet real-time requirements. The authors suggest future work on designing more efficient SSMs and exploring the application of SSMs in the frame generation module.

Summary

The paper introduces Mamba-based video frame interpolation. To fully incorporate the power of Mamba, the paper proposes an interleaving rearrangement method. Using this method, the SSM scans the same location tokens of 2 frames together instead of processing each frame separately. The paper also proposes curriculum learning with large motion data. The experiments show the improvement of using these ideas.

Strengths

This paper includes three contributions: 1) Use of Mamba for image interpolation, 2) Interleaved rearrangement, and 3) curriculum learning. - The idea is simple but effective for video frame interpolation. All these contributions are justified throughout the paper (especially Sections 3 and 4). - The paper is easy to follow.

Weaknesses

I have comments and questions about the final comparisons (Tables 2 and 3). >Most methods training models exclusively on the Vimeo90K (Xue et al., 2019). .... (2) Sequential Learning: To mitigate the limitations of training solely on Vimeo90K, some methods (Liu et al., 2024a; Park et al., 2023) further train the model on X-TRAIN (Sim et al., 2021) after initial training on Vimeo90K. - Based on the description above, the proposed model used both Vimeo90K and X-TRAIN for training, but not all methods use both datasets. It's unclear which models are trained only on Vimeo90K or Vimeo90K+X-TRAIN. The improvement can be due to the increase of training datasets especially since the improvement is marginal compared to some models such as EMA-VFI (for low resolution), AMT-L, AMT-G, and SGM-VFI. I understand that curriculum learning improves performance compared to the simple data mix, but the current tables are not comparable if the trained datasets are not the same across the models. - I ask authors to add which models are trained only on Vimeo90K or Vimeo90K+X-TRAIN in the tables. - If some comparable models are trained only on Vimeo90K, can authors add the comparisons of these models with Vimeo90K+X-TRAIN? I'm not sure if it's possible during the rebuttal period. - As some comparing models are better or similar to the proposed model, it's difficult to compare which ones are better. Adding the average of each model on the right column would be helpful. - How can the interleave rearrangement be efficiently implemented? It would be good to add details of how it is implemented in the paper. Also, is there any speed issue of using the interleave rearrangement instead of the sequential one?

Questions

See the weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitation is discussed in the paper and is reasonable.

response to the rebuttal

Thank you for the answers. My major concern was the unfair comparisons in Tables 2 and 3 as the paper uses an extra dataset. The authors' answers and additional information in Tables 2 and 3 (also Table 6) resolve my concern. Please add these details to the revised version. Also, please move Table 6 to the main paper; it's an important ablation. As the paper includes clear contributions and improvements, I increase my rating.