Neural Image Compression: Generalization, Robustness, and Spectral Biases

Summary

This paper proposes to adopt spectral tools to analysis OOD issues of NIC. On the proposed corrupted dataset, the paper investigates the error of both conventional and learning-based image compression approaches and provides some interesting conclusions, showing robustness and generalization of NIC and suggesting insights of design a better NIC model in the wild.

Strengths

The paper proposes novel tools and benchmarks to thoroughly study NIC from the OOD perspective. I appreciate the delicate figures which quite help me understand the studies. The analysis and conclusions are reasonable and the findings are inspiring, rehealing missed parts of NIC design and assessment. Overall I believe this paper can technically contribute to the community.

Weaknesses

1. My major concern is the generalization of the conclusion. All the FR experiments are conducted on the hyperprior model (Ball'e et al., 2018), which is a somewhat outdated baseline. Many more powerful approaches have been proposed in the past years, involving more complex context models (Minnen et al., 2018, [40]), improved mixed quantization estimators ([e1, e2, e3]), and non-convolution network structures ([e2, e4, e5]). Shall we observe some different spectral properties when using those transformer-based models? Also, the wavelet-based JPEG2000 is neither a SOTA handcraft codec nor a worldwide popular choice. Though conventional coders don't play key roles in this paper, I suggest the authors further discuss HEIF, AVIF or VVC(IF), and JPEG. 2. As discussed by the authors, the spectral preference of NIC may be strongly related to the MSE objective. This can be explained as it corresponds to a Gaussian likelihood, according to [Ball'e et al., 2018]. However, this loss function can be replaced to MS-SSIM, GAN and LPIPS etc. For instance, refer to perceptual optimized NIC approaches [e7]. Can this OOD drawback get suppressed when consider those objectives? More discussions are expected. 3. Though the paper provides a detailed analysis with proposed tools and benchmarks, It do not provide new approaches to solve the concerned OOD issues. Without more practical suggestions, I cannot strongly appreciate the findings for above-mentioned concern about generalization. P.S. OOD in NIC is not a brand new topic. The authors may want to discuss more related works eg [e6]. [e1] Channel-wise autoregressive entropy models for learned image compression, Minnen et al. [e2] Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding, He et al. [e3] Transformer-based transform coding, Zhu et al. [e4] The Devil Is in the Details: Window-based Attention for Image Compression, Zou et al. [e5] Learned image compression with mixed transformer-cnn architectures, Liu et al. *(concurrent work)* [e6] On the Out-of-distribution Generalization of Probabilistic Image Modelling, https://openreview.net/pdf?id=q1yLPNF0UFV [e7] High-fidelity generative image compression

Questions

See above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

See above.

Summary

This paper introduces a benchmark for evaluating the out-of-distribution (OOD) performance of neural image compression (NIC) models. The authors observe distinguished spectral biases of NIC models by comparing them with JPEG2000. Then the authors listed all the spectral biases along with a theoretical analysis from the perspective of principle component decomposition.

Strengths

1. This paper proposes an interesting perspective for analyzing NIC models. 2. The observed spectral biases are potential to further improve NIC models. 3. The authors provide thorough analysis and enough details for reproducing the results.

Weaknesses

I believe the spectral perspective is important to understand NIC models. However, the theoretical analysis is of limited novelty and the analysis and conclusions provided in this paper are empirical and need to be further improved before publication. Moreover, I am afraid that classifications on several corruptions may conflict with the common sense in signal processing field. My concerns are listed as below. 1. The proposed method mainly compares with JPEG2000, which may be inappropriate. JPEG2000 is based on discrete wavelet transform. However, EBCOT and rate control of JPEG2000 could hinder an explicit understanding of the spectral characterization of JPEG2000. I suggest the authors to conduct the comparison experiments on JPEG (which is based on block-based DCT and simple quantization table) for clear comparison. Compared to JPEG2000, JPEG has been widely applied in practice. 2. The comparison presented in Figures 16 and 17 (i.e., comparison to VTM provided in supplementary material) is confusing. $\mathcal{D}$ for VTM seems to be zero but it is a lossy compression method and the reconstruction is still far from lossless (i.e., 37.42 dB). 3. I also have problem with the evaluation using power spectral density (SPD). I noticed that the authors classify Gaussian noise as a high-frequency corruption in Figure 1. However, it is well known that the Gaussian noise, i.e., white noise, has a constant SPD. In other words, the effect of Gaussian noise is expected to be constant rather than constrained on the high-frequency components. Moreover, the effect of JPEG compression is also expected to be on both median and high-frequency, as the default quantization map results in a larger quantization step on the high-frequency components. I am afraid that the classification of other corruptions may neither be sufficiently precise. I guess the gaps between above theoretical analysis and the empirical analysis in Figure 1 originate from the intensity difference on the SPD used to analyze the effect of introduced corruption. The natural images themselves are with an unbalanced SPD. Thus, the observation may be biased. 4. The theoretical analysis in Section 5 is not novel enough. The desired characteristics for transform coding are discussed in [R1]. The differences and similarity between DFT, DCT, and KLT are analyzed in [R2]. The provided analysis on linear transforms (autoencoder) may not be appropriate for nonlinear models [R3]. 5. Considering that the major contribution of this paper lies on the empirical evaluations, it is important to make thorough validations on diverse datasets and NIC models. The adopted scale hyperprior is not sufficient supporting the claims in this paper. [R1] V. K. Goyal, "Theoretical foundations of transform coding." IEEE Signal Processing Magazine, vol. 18, no. 5, pp. 9-21, 2001. [R2] D. S. Taubman and M. W. Marcellin, JPEG2000: Image compression fundamentals, standards and practice. 2002. [R3] J. Ballé, et al. "Nonlinear transform coding." IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 2, pp. 339-353, 2021.

Questions

1. Could the authors offer a comparison with JPEG compressed images? I believe it would be more straightforward. 2. What is the reason for theoretical analysis (given in comment #3 in the section of Weakness) and the given classification (Figure 1) on Gaussian noise and JPEG compression artifacts? 3. What is the difference between the theories for nonlinear transforms (Section 5) and those for traditional linear transform (such as DWT, DCT, and KLT)?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations are not discussed in the paper.

Summary

This paper analyzes neural image compression models relative to classical (hand-engineered) methods using a new dataset and new tools based on spectral analysis. Specifically the paper presents (from Section 1, "Main Contributions"): (1) a new dataset covering different kinds of distortions used to compare methods (2) an inspection tool for distortions that shows where, in the frequency domain, different distortions primarily introduce error (3) an inspection tool based on spectral analysis that provides a visualization showing which frequency ranges are most distorted by different compression methods (4) some theoretical results about neural compression, albeit limited to the linear autoencoder case The main findings are also summarized in the intro: (1) Neural image compression (NIC) produces different spectral artifacts than classical codecs (JPEG 2000 was used as the classical codec and a scale-hyperprior model was used for NIC) (2) NIC and classical codecs prioritize different frequencies as the bit rate changes (3) NIC and classical codecs fail to generalize in similar ways but succeed in different ways (4) Due to the differences, it can be hard to choose the best codec for a particular application without knowing what kind of spectral distortions will be present at inference time (i.e., out of distribution)

Strengths

Although compression systems, neural and hand-engineered, are frequently assessed in terms of visual quality, this paper is the first I've seen that tries to understand how distortions differ between neural and standard methods. The authors develop a new dataset built by applying 15 different kinds of distortions to images from standard evaluation sets (kodak and clic). They then visualize the power spectral density (PSD) of the distorted images (to understand which frequency bands are affected by each type of distortion) and they visualize the distortions caused by lossy compression for a neural method and for JPEG 2000. This analysis leads to interesting observations about how the two codecs behave differently, e.g. neural image codecs (NICs) distort high frequencies more than low frequencies (Section 4.1), and they are more sensitive to high frequency noise/distortions. I haven't previously this kind of analysis or conclusion before. Similarly, the authors analyze how classification accuracy degrades when the two types of codecs are used on images with different kinds of distortion (Fig. 5). These findings are particularly interesting because neither codec is preferable for all distortions and bit rates, e.g., JPEG2000 takes a large hit for "high" distortions at high bit rates but takes a smaller hit than NICs for "low" distortions at moderate bit rates.

Weaknesses

There are two main weaknesses with the paper. The first is whether the results from studying these two codecs generalize to other codecs in the respective classes. For example, will VCC (a modern codec still be finalized) behave similarly to JPEG2000? Will SOTA neural methods behave similarly to [9]. The latter question is further complicated because there are multiple ways the NICs differ, e.g. (1) more sophisticated models that build on [9] (like adding autoregressive priors, architectures built using transformers instead of CNNs, etc.), (2) using "implicit neural representations" instead of the "nonlinear transform coding" approach of [9], and (3) optimizing networks with an adversarial loss that targets "realism" (minimizing a divergence between distributions, not just a measure of distortion between the original and reconstructed pixel values). It's hard to say if the findings in this paper will be the same for a model that is different according to one of (or all of) these dimensions. Second, I'm not sure how important these findings are to the NeurIPS community. Neural image compression is certainly an appropriate topic for NeurIPS, but papers typically focus on new models or new applications. So the target audience may be fairly small, and the observations don't directly imply an architecture change (or loss / optimization change) to improve NICs. Since I've worked in this subfield, I find the results of this paper fascinating. And at a conference focused on compression or image processing (something like ICIP or DCC), I'd rate this paper as a 7+. I'm open to feedback from the AC or other reviewers that I'm off on my assessment of the fit at NeurIPS.

Questions

No questions beyond the implied ones in the "weaknesses" section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

2 fair

Limitations

adequately covered

Summary

This paper addressed the problem of image compression. Neural-based methods have started outperforming traditional codec methods in image compression, but there are no comprehensive datasets and informative tools to evaluate neural-based methods in real-world settings. This paper proposed a benchmark to study this that allows for evaluating out-of-distribution performance. The proposed benchmark provides some findings that are crucial to bridge the real-world adoption of neural-based image compression techniques.

Strengths

1. Comprehensive datasets for image compression. 2. Understanding of out-of-distribution performance. 3. Extensive evaluations. 4. Theoretical analyses.

Weaknesses

NeRF-based methods can also be seen as a way to do compression (compressing images into. network weights), The authors should also discuss this and benchmark them on the dataset.

Questions

Overall, this is a strong paper. I only have some minor comments. See above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The paper should address this.

Missing comparison to "implicit" methods doesn't hurt the paper

Nerf-like approaches, often called implicit neural representations (INRs), are very interesting. All of the literature I've seen (as of a few months ago, at least), show that this approach trails methods based on autoencoders on typical metrics. So comparing against a different approach to neural compression is interesting, but, in my opinion, isn't vital and so the absence of this comparison shouldn't reduce the paper's chance of acceptance.