Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements

Summary

The paper presents a framework for various low-level vision tasks, such as denoising, object removal, frame interpolation, and super-resolution. What sets this framework apart is that it does not require any external training data. Instead, it directly leverages the internal statistics of videos at test-time to optimize neural network modules to perform the target task. Experiments are conducted on DAVIS, UCF-101 and VIMEO90K-T.

Strengths

1) VDP presents a simple and intuitive framework for accomplishing test-time, task-agnostic video editing. 2) The evaluation across multiple video tasks lends strong evidence to the generality of the proposed framework.

Weaknesses

1) Lack of novelty: the propagation of video latent codes for editing tasks such as object removal, super-resolution, video interpolation etc. have been explored in previous works, such as [1][2]. The perceptual similarity and total variation losses have been borrowed from previous works without modification, and the spatial pyramid loss is simply a multi-level reconstruction loss, which has also been leveraged in previous works, such as [3][4][5][6]. 2) Experimental setup and evaluation results are rather weak. For specific concerns, see the following questions. [1] [Video-Specific Autoencoders for Exploring, Editing and Transmitting Videos](https://arxiv.org/abs/2103.17261v1) [2] [Temporally Consistent Semantic Video Editing](https://arxiv.org/abs/2206.10590) [3] [Loss Functions for Neural Networks for Image Processing](https://arxiv.org/abs/1511.08861) [4] [Training a Task-Specific Image Reconstruction Loss](https://arxiv.org/abs/2103.14616) [5] [Learning Multi-Scale Photo Exposure Correction](https://arxiv.org/abs/2003.11596) [6] [Masked Image Modeling with Local Multi-Scale Reconstruction](https://arxiv.org/abs/2303.05251)

Questions

1) Quantitative evaluation includes only PSNR and SSIM. However, these metrics have been identified in previous works to be problematic with respect to human-aligned notions of perceptual quality [1][2]. This problem is exacerbated for longer spatiotemporal sequences in video. Ideally, the authors could include user studies like [2][3]; in the absence of this evaluation, other metrics like FID, IS, LPIPS, etc. should be reported. 2) Fundamentally, I'm not convinced that the suite of tasks chosen by the authors is indeed the best for demonstrating the strength of their framework. Concretely, while the setting of explicitly removing dependency on external data is an interesting one, I'm wondering if this is a somewhat artificial constraint, particularly for tasks such as video super-resolution, interpolation, denoising, or object removal. Such tasks already have a well-established corpus of strong pretrained generative models which demonstrate convincing zero-shot performance on new images/videos. Further works similarly perform a few iterations of test-time optimization on single images/videos to improve performance on specific downstream tasks, as in [4][5]. Why should these video tasks be treated from a "external data"-free paradigm? 3) Questions 1) and 2) notwithstanding, the baselines used for comparison in each of the tasks are incomplete. For example, two strong prior works in [6] and [7] provide similar approaches to task-agnostic, semantic editing of videos, directly at test-time. These should be included for each of the tasks. [1] [Complex Wavelet Structural Similarity: A New Image Similarity Index](https://pubmed.ncbi.nlm.nih.gov/19556195/) [2] [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://arxiv.org/abs/1801.03924) [3] [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://arxiv.org/abs/2306.09344) [4] [Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models](https://arxiv.org/abs/2303.17599v1) [5] [MyStyle: A Personalized Generative Prior](https://arxiv.org/abs/2203.17272v1) [6] [Blind Video Temporal Consistency via Deep Video Prior](https://arxiv.org/abs/2010.11838) [7] [Learning Blind Video Temporal Consistency](https://arxiv.org/abs/1808.00449)

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Please see the above questions with respect to limitations.

Summary

Authors of the paper propose a novel framework for low-level video enhancement tasks (i.e., denoising, object removal, frame interpolation, and super resolution). The proposed framework does not require any training data by directly optimizing the neural model parameters over the corrupted test data. The key idea is to design modules and loss terms that leverage the spatiotemporal coherence and internal statistics of the test videos - fundamental properties of a video sequence. The proposed approach, the VDP model, consisting of a frame encoder and a frame decoder, performs future frame prediction. The total loss to optimize this method at the time of use (test / inference time) is a combination of (1) the typical reconstruction loss, (2) a novel spatial pyramid loss, and (3) a variational loss introduced in a prior work. VDP is more robust to spatio-temporal noises that contain in the input, and superior performance of the proposed approach was shown through qualitative results, quantitative results, comparisons to the state of the art (SOTA), and ablation studies on four low-level video enhancement tasks (i.e., denoising, object removal, frame interpolation, and super resolution).

Strengths

S1 - Novelty: A novel inference time optimization technique was proposed to learn neural module parameters / weights for performing various low-level video tasks. S2 - Significance: The proposed method eliminates the need for training data collection. It differs from the popular paradigm and is practical. S3 - Quality: Detailed analyses and experimental results have demonstrated the superiority of the proposed method. S4 - Clarity: The paper is well-constructed and easy to read.

Weaknesses

W1 - The VDP model uses LSTM as the Latent Frame predictor Network and the choice is not well justified, although the authors have acknowledged this as a limitation of their current model and encourage readers to explore more architecture choices. W2 - Oftentimes, when the input contains no noise, the proposed approach is on par with prior SOTA. W3 - The resolution involved is still quite low (e.g., 3×448×256). Not sure if the proposed method can only handle resolution of this scale, or it can actually scale to high-resolution videos. W4 - The test videos shown in the paper have relatively simple dynamics and a short time window. E.g., for frame interpolation and object removal. When the video has more complicated interactions and dynamics, will the model still perform well ? This is not clear.

Questions

Q1: Is the proposed method the first inference time optimization technique for video tasks? Q2: Is the optimization performed on the test set (rather than a single test video) ? If so, would the size of the test set be a factor greatly influencing performance of the VDP model ? What is the size of the VDP model in terms of the number of parameters ? Q3: In section A of the Supplementary Material, line 30-31: ... for the frame interpolation task, the case is reversed, and cascaded baseline approaches yield better results than only baseline approaches. Why would the denoiser wash away important details for the super-resolution task, but not frame interpolation task ? Q4: What could be other possible low-level video tasks that might benefit from ideas of this paper ? How about other video tasks that focus on pixel-level spatio-temporal dynamics, e.g., tracking and particularly tracking any points (https://deepmind-tapir.github.io/) ? Are the proposed key components of the paper mostly just for video tasks that require spatio-temporal inpainting because of the future frame prediction based formulation ? Q5: When the input contains no noise, what would be the reasons for someone to choose the proposed method over prior SOTA ? Q6: In Table 4 of the Supplementary Material, since the inference time was given as the time to process one frame of a video, would it mean that the total time (Train + Infer) for the denoising task, for example, would be 3450 frame * 12 s / 3600 ~= 12 hours, which is also the time for the VDP denoiser to be plateaued after 3600 epochs of training ? Q7: Could you explain the following sentence ? Line 245: to perform a 4x frame interpolation, we select α = [0.25, 0.5, 0.75] in the Eqn. 5.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The auhtors have discussed the limitations. However, the proposed model might be also limited in its capability to handle videos with a high resolution, complex dynamics and a long time window.

Summary

This work proposed an internal learning method for multiple video-processing tasks. Without collecting large-scale training data, the proposed method can achieve comparable or even better results than baselines. But the slow computation is one obvious limitation.

Strengths

- The proposed method is optimized individually for each video and thus does not need to collect training data. - Extensive experiments on different tasks such as super-resolution, denoising, and inpainting show effectiveness.

Weaknesses

- As the proposed method is optimized for each video, I believe that it will work on many examples and it will also fail on many examples. How about the generalization ability of the proposed method? Are results in the paper cherry-picked or randomly sampled? How about failure cases? - The writing for the technical part can be further improved. The current version just lists some equations and the procedures, and more motivation, explanation, insights would be better for readers. - One common problem of internal learning methods is that it would fail if there is insufficient information within a video. For example, if some objects keep blurry across the videos, it is difficult to recover it with internal learning. How could you solve this problem? Could the proposed method be combined with some pretrained models? - The optimization process is slow

Questions

see weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

see weakness

Summary

This paper introduces a new inference time optimization approach for low-level image enhancement tasks. The proposed approach does not need any training data and leverages the spatio-temporal coherence and internal statistics of test videos. The authors also propose a spatial pyramid loss which makes the proposed approach robust to spatial and temporal noise. The proposed approach is evaluated on multiple video enhancement tasks (denoting, frame interpolation, super-resolution, and object removal) and is shown to perform competitively or better than existing works that use a training dataset.

Strengths

The paper is written well and was easy to follow The proposed approach is applicable to many video enhancement tasks without the need for training task-specific networks on large training datasets Experimental evaluation on several datasets clearly show the effectiveness of the proposed approach.

Weaknesses

Since the proposed approach trains a network on the test video, the results depend on how long it is trained and the required training steps may vary from video to video depending on the content. If we train long enough, the model may start fitting to the corruptions in the input video to some extent (though the losses are providing impedance). It is unclear to me if the results presented in the paper correspond to using the same number of steps for all videos, or if the authors determined when to stop based on the loss curve for each video independently. The performance of the proposed approach also depends on the network architecture used. Authors haven't conducted experiments to show how sensitive the results are to the choice of architectures of LFPNet and FDNet. Different architectures may have different inductive biases and may have an impact of the behavior of loss observed in Figure 2(iii).

Questions

See my response in the weaknesses section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The main limitation of the proposed approach is that it is an offline approach. Authors have clearly acknowledged the limitations of their approach and discussed potential ways to address them (in the supplementary material).

Thanks you for the rebuttal.

After reading the rebuttal, I stick to my initial rating which is in favor of accepting the paper.