Diversify, Contextualize, and Adapt: Efficient Entropy Modeling for Neural Image Codec

Summary

The authors proposed a novel method to improve the entropy encoding in deep image compression. The key idea is to introduce three contexts, local, region, and global, to capture contextual information at different scales while balancing the compute cost. The authors show that the proposed method achieves SOTA compression rate while being computationally cheap.

Strengths

* While introducing global context in entropy model has been explored in previous, how to efficiently capture these context information with minimal compute cost is still an open challenge. This work proposed a novel solution to fill the gap. * Experiments on real-world high resolution images show significant improvements over baseline methods.

Weaknesses

* There are quite a few deep image compression methods proposed recently. The authors only compared 4 baseline methods. It would be necessary to compare more STOA methods.

Questions

* Is the proposed model mobile-friendly?

Rating

5

Confidence

1

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Summary

This paper proposes an entropy model that achieves higher compression rates than previously proposed models in the context of learned image compression. This model combines global, regional, and local information to make better (i.e., more accurate, and therefore requiring fewer bits) predictions with respect to the latent values (quantized) that the autoencoder produces. The authors discuss the method in detail and provide a visualization. The results show an improvement over the existing methods..

Strengths

- This paper ran well designed experiments - The authors do discuss runtime, which is great. I wish more paper did this. - If I am reading the numbers correctly, the entropy model seems to provide a pretty good speedup over the existing models.

Weaknesses

- This paper does not seem to be too theoretical, but it's clearly results-focused. I wonder whether a more appropriate venue would afford this a) more visibility to the correct (i.e., compression) audience; and b) be much more well received. I don't really think that this will be of broad enough interest at NeurIPS. - When discussing runtime performance, it would be great if the authors could break it down by sub-system. For example, it's useful to know how long does the autoencoder take to encode the image/decode the image. It would be great to know how much time does the entropy model take to predict all values needed to encode/decode. It would also be interesting to know (this is highly optional, but would make this paper much more interesting from a deployment standpoint) do discuss the runtime when ANS is also interleaved in the decoding / encoding process of the image. - The proposed entropy model shows some slight improvement, but it's nothing exciting. I was hoping to see a 30% improvement given the added complexity, but we got a much smaller number than that. The biggest issue: - Since this paper is focused on entropy modeling, which I believe is a great topic, I would have thought that a primary concern would have been reproducibility. Entropy models are famously flaky, meaning that it's quite possible that an image encoded on a V100 GPU may not be decoded by any other device (including CPUs.). As such, I would have liked the authors to go into detail on what mitigations were put in place to ensure that their model can be used on variety of hardware. I truly believe, that this is the difference between a paper that slightly improves theoretical compression rates, and one which makes it possible to apply in practice.

Questions

- Do you plan to open source your code? I am asking this because (a) it would allow researchers to use it (as it's clearly a useful component of any learned image compression pipeline); and (b) determine how well it works in practice when submitting to competitions such as CLIC which will almost entirely guarantee that the decoding will happen on different hardware than the encoding. - I appreciate the experiments with the ordering of contexts, but I am very puzzled by the final order. Regional -> Global -> Local sound quite counterintuitive. I wonder why Global is not actually first, as it would "seem" like it should be the most general, and a decrease in granularity with respect to ordering would make sense. - Since the goal of this paper was to achieve the absolute best RD, have you considered backpropping into the latents once an encoding has been found (after a forward pass) in order to see whether jointly optimizing the RD loss for the particular image would yield even larger improvements?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

They seem addressed.

Summary

**Disclaimer:** My research direction is not related to image compression. I will try my best to review this work but could be biased. The work studies neural image codec. The work proposes an entropy modeling framework that uses contents for forward adaptation without comprising the bit rate. The authors build upon the quadtree partition-based backward adaptation method and introduce additional forward context to address limitations in existing approaches, particularly in the first modeling step.

Strengths

1. The work conducts in-depth analysis of the different components in the proposed method。 2. The proposed method enjoys very efficient decoding while having a better BD-rate.

Weaknesses

- There are some presentation issue, e.g. line 256-257, the links to citation are missing. Please see the questions for methodological discussion

Questions

1. While the proposed method enjoys better performance, it seems the proposed method is largely derived from B (Quadtree) + F (R) (Li et al., 2023), while uses a lager network, and therefore, slightly longer decoding time. 2. I think it will further improve the paper if the authors put some additional qualitative results say in the appendix for the readers.

Rating

6

Confidence

1

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Summary

This paper presents a method for learned image compression called DCA, which stands for "Diversify, Contextualize, and Adapt". DCA is a more sophisticated approach for building the entropy model compared to previous methods. The entropy model is a key component in compression models and is optimized to predict the distribution of quantized latents that represents an encoded image. This is crucial for compression since a more accurate entropy model directly leads to higher compression rates when used with an entropy coding algorithm. The core insight of DCA is to use multiple ("diverse") contexts for forward adaption. More specifically, the authors define a local, regional, and global context corresponding to a single spatial location in the latent representation (local), a spatial patch in the latent tensor (regional) and a compact summary of the full latent (and thus the full image) learned via cross-attention. The authors show that the three contexts are complementary (Fig. 7) and that a serial applications (architecture details in Fig. 2) outperforms a merged approach (Fig. 9 and 10). The result is a SOTA model in terms of compression rate with a moderate model size and relatively fast decoding time (see Fig. 5).

Strengths

The primary strength of this paper is achieving a non-trivial rate savings over very strong existing image codecs: 11.96% average rate savings over VTM, the current best standard codec for lossy image and video compression, and 3.73% rate savings over a strong neural method [12]. This is achieved with a well-motivated, intuitive improvement over previous entropy models along with extensive experimentation and evaluation. Furthermore, the rate savings are achieved with a moderate-size model (37.9M) and with a relatively fast decode speed. This is import because ultimately neural compression models compete with standard codecs with extremely fast decode speeds, and because it's already clear that incremental improvements to compression rates can be easily achieved at the expense of massively more compute and memory usage. Finally, the evaluation in the paper is quite thorough in terms of comparisons with appropriate baselines, runtime analysis, and investigation of importance of the different contexts used, their order, and the architecture (CNNs vs. attention). I also appreciated that the authors showed *where* DCA helped (Fig. 6) by showing the difference in the scale of the normalized latents with and without DCA. Specifically, DCA (with a global context) shows much better prediction for the first step (\bar{y}^1) where there is no opportunity for backward adaptation and the baseline only utilizes regional context.

Weaknesses

The primary weakness of the paper is modest gains (at least in the context of NeurIPS) on a version of the learned compression problem that is no longer the focus of the compression subfield. In terms of the rate gains, 3.73% over the previous SOTA method is non-trivial (for reference, papers at conferences focused on improvements to standard codecs often show a gain of less than 0.5%), but it's still incremental progress. It's also not a large enough change to garner more interest from practitioners developing new standard codecs. In terms of the problem itself, there is now more attention on modeling video and boosting subjective (perceptual) quality. Optimization for MSE can still be important since it allows clear, objective evaluation, but then we're back to the first point where I think a larger improvement is now needed to have high impact at NeurIPS.

Questions

No questions

Rating

5

Confidence

4

Soundness

4

Presentation

4

Contribution

2

Limitations

Adequately addressed.