On the Choice of Perception Loss Function for Learned Video Compression

Summary

This paper studies the choice of perception loss for learned video compression and summarizes some valuable conclusions. The first one is the pros and cons of PLF-JD and PLF-FMD. The second one is the universality of MMSE reconstructions.

Strengths

This paper conduct experiments and give some theoretical analysis of the perception loss functions for learned video compression. It is valuable when designing and training a video compression pipeline.

Weaknesses

1. The datasets used in the paper are too simple. It is questionable that the conclusions still work for the datasets that are closer to the real world. It is suggested to provide more experiments on the widely used datasets in learned video compression methods, such as UVG, MCL-JCV, and HEVC Class B datasets. Most papers conduct their experiments on these datasets. So we can see more performance comparisons and verify the proposed method is not a toy example. 2. Considering this paper focus on perception loss, why not try some metrics about perceptual quality? For example, the experiments can also provide widely used perceptual metrics such as LPIPS.

Questions

In the experiments, I do not know how the proposed method precisely controls the bit rate to make the rates of different methods to be identical.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

See the weakness part.

Summary

The paper consider theoretical considerations of generative video compression. While rate-distortion-perception theory (Blau & Michaeli, 2019) explains the trade-off between realism and distortion for generaive compression, the causal processing typically introduced for video models adds constraints that need to be considered. The authors consider two settings: FMD (frame marginal distributions) and JD (joint distribution). In the FMD case, for full realism an observer cannot tell apart reconstructed frames and original frames when looking at one frame at a time, but there can be synthetic detail that is not temporally consistent. In the JD case, for full realism the joint distribution of reconstructed videos matches the originals. The main results of the paper are: Theorem 1: in the FMD setting, we can achieve full realism (“perfect perception”) by at most doubling the distortion of an MSE optimized system. This generalizes a result from (Blau & Michaeli, 2019 [16]) for the single frame scenario. Theorem 2: In the JD setting, the authors obtain a similar result but for a more constrained setting. They need the source at time j to be nearly independent of the encoder outputs upto and including time j. Interestingly, they demonstrate an example in the appendix where the factor-of-two bound is not achieable. The example can be summarized as follows: if you have some loss of information in reconstruction 1, you need reconstruction 2 to be consistent with it even if you have perfectly encoded the original frame. Hence you can’t achieve a zero reconstruction error in frame 2 despite having losslessly encoded it, violating the factor of two bound. Theorems 3-6: Here the analysis is done for the more general rate-distortion-perception setting, where the perceptual quality/realism is sacrificed to improve distortion. The authors consider both the setting of first-order Markov sources and do further analysis in the gaussian special case.

Strengths

- Relatively readable paper - Valuable insights into the generative video compression field - The theoretical results are validated in experiments on small datasets (Moving MNIST and KTH).

Weaknesses

- Intuitively, why would we only consider frame level marginal distributions? Just like pixel-level marginal distributions don’t make sense, this should lead to massive temporal artefacts/flickering in the low rate regime. While e.g. [7] used only a frame-level discriminator, they use warping for temporal consistency. - I found the proof of Theorem 1 to be quite similar to the proof of Thm. 2 in Appendix B in (Blau & Michaeli, 2019). It seems to me that proof also works as-is in the setting where you condition on previously encoded frames, as there are no assumptions on how the MSE optimized reconstruction is obtained (so whether it is using information from previous frames or not doesn’t seem to matter in the derivation). I would appreciate if the author could clarify differences here. - Experiments are limited to toy data. Minor (not affecting rating): *The supplement is very long* (30+ pages) and the main paper contains 24 references to it. It seems to me the authors have a lot more to say than fits into 9 pages. I'm not sure what to do with this but it feels slightly weird. I did not read everything in the supplement.

Questions

-

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Toy data / causal setting.

Summary

This paper examines the choice of perceptual loss for causal, low-latency video compression model with distortion-perception optimization. By using information theoretic analysis and deep-learning based experiments, the authors demonstrate that the choice of perceptual loss can have a significant effect on the decoded reconstruction, especially at low-bit rates.

Strengths

This paper demonstrates the proposed assertions through theoretical derivations and detailed experiments, which are convincing enough. In addition, these conclusions provide a good guideline basis for subjective optimization of learned video compression methods.

Weaknesses

Since this paper is oriented to the discussion of learned video compression, the experimental setup may be too simple to have generalized conclusions. For example, the used sequences (e.g., MNIST and KTH) for evaluation are too homogeneous to present the characteristics of videos. It should be better to have some comparisons on more complex sequences such as the UVG dataset. It is critical to illustrate whether these findings would be useful in practical applications.

Questions

As mentioned above, it is suggested to have some experiments on sequences with more complex motion and coded bits to examine the proposed findings in addition to those toy examples in the submission.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No.

Summary

This paper systematically studies the rated-distortion-perception tradeoff in neural video compression. Since videos are made of consecutive frames, this paper considers two different perceptual metrics, including the joint distribution of all video frames (PLF-JD), and per-frame perceptual loss (framewise marginal distribution, PLF-FDM). Some conclusions are proposed regarding the choice of PLF-JD or PLF-FDM. In addition, motivated by the universal encoded representations in previous RDP papers from the field of image compression, the authors in this paper demonstrate the video-version universal encoded representations hold when the perceptual constraint is PLF-FDM. While a similar result does not hold for PLF-JD in general, it is satisfied for a special class of encoders which operate in the low-rate regime. The universal encoded representations in video compression have several advantages for video compression such as when it needs to extract motion information from an MSE-based reconstruction.

Strengths

Originality and quality of this paper are pretty good. The PLF-JD and PLF-FDM are two different but both reasonable perceptual metrics used for optimization of video compression models. This is the first paper that studies the RDP issue in video compression and the results and conclusions are insightful. Some important conclusions regarding RDP include: (1) There is a significant penalty in distortion when using PLF-JD in the low-rate regime. (2) While PLF-JD preserves better temporal consistency across video frames, it suffers from the permanence of error phenomenon in which the mistakes in reconstructions propagate to future frames. The universal encoded representations for video compression are studied as well, which are demonstrates to be always held when using PLF-FDM (similar to compressing images sequentially). Besides, the effectiveness of RDP in video compression models is well verified by theoretical analysis in Gaussian-Markov sources.

Weaknesses

(1) (Perhaps not as a weakness) I am wondering whether these conclusions of RDP in video compression can be extended to distortion-perception tradeoff in other video generation / restoration tasks. The joint-frame loss PLF-JD and the per-frame loss PLF-FDM should hold similar conclusions for video generation. (2) The state-of-the-art neural video compression framework contains a motion encoder/decoder and a residual encoder/decoder, which are effective for high-resolution videos. It seems this paper studies the other neural video compression framework that does not explicitly transmit motion information. How to apply the proposed theory to such motion-residual-separated video compression framework?

Questions

See the abovementioned “weaknesses”.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are discussed in the final section of Appendix. This paper is in the field of video compression which does not involve strong negative societal impact.

Error Permanence on UVG Dataset

We would like to inform you that we have already obtained the results of the experiments on the UVG dataset. According to them, the error permanence phenomenon shows up in the JD reconstructions. That is, it propagates the flaws in the color tone of the previous low-rate reconstruction in the future ones. However, FMD reconstructions do not suffer from such a problem. The distortions over 3900 UVG samples are given as follows: | | MMSE | PLF-FMD | PLF-JD | :--- | :----: | ---: | --:| | Distortion (MSE) | 0.0026 | 0.0032 | 0.0168 This confirms the error permanence phenomenon for PLF-JD. Currently, we are not able to send you the visualization of our results since according to the conference timeline, the deadline for uploading a pdf file has passed. But, we are going to include these results in the new version of our work