2DQuant: Low-bit Post-Training Quantization for Image Super-Resolution

Low-bit quantization has become widespread for compressing image super-resolution (SR) models for edge deployment, which allows advanced SR models to enjoy compact low-bit parameters and efficient integer/bitwise constructions for storage compression and inference acceleration, respectively. However, it is notorious that low-bit quantization degrades the accuracy of SR models compared to their full-precision (FP) counterparts. Despite several efforts to alleviate the degradation, the transformer-based SR model still suffers severe degradation due to its distinctive activation distribution. In this work, we present a dual-stage low-bit post-training quantization (PTQ) method for image super-resolution, namely 2DQuant, which achieves efficient and accurate SR under low-bit quantization. The proposed method first investigates the weight and activation and finds that the distribution is characterized by coexisting symmetry and asymmetry, long tails. Specifically, we propose Distribution-Oriented Bound Initialization (DOBI), using different searching strategies to search a coarse bound for quantizers. To obtain refined quantizer parameters, we further propose Distillation Quantization Calibration (DQC), which employs a distillation approach to make the quantized model learn from its FP counterpart. Through extensive experiments on different bits and scaling factors, the performance of DOBI can reach the state-of-the-art (SOTA) while after stage two, our method surpasses existing PTQ in both metrics and visual effects. 2DQuant gains an increase in PSNR as high as 4.52dB on Set5 (x2) compared with SOTA when quantized to 2-bit and enjoys a 3.60x compression ratio and 5.08x speedup ratio. The code and models will be available at https://github.com/Kai-Liu001/2DQuant.

Paper

Similar papers

Peer review

Reviewer EXmU6/10 · confidence 5/52024-06-26

Summary

The paper presents 2DQuant, an innovative low-bit post-training quantization technique for transformer-based image super-resolution that significantly advances the state-of-the-art by introducing a two-stage optimization process. This process includes a novel Distribution-Oriented Bound Initialization strategy and a Distillation Quantization Calibration method, resulting in exceptional performance with minimal loss of accuracy for SwinIR-light. The 2DQuant approach demonstrates the potential to compress and accelerate transformer-based super-resolution models effectively for edge deployment.

Strengths

1. This paper is well-written and easy to understand. 2. Transformer-based SR is much popular in recent year, this method is much helpful for obtaining accurate quantized transformer-based SR models. 3. The experimental results and visualizations are sufficient.

Weaknesses

1. The proposed DOB and DQC are much similar with DBDC and PaC, but only change weight compression to symmetric quantization, could you give more details about the differences? 2. The motivation in the introduction are confusing. Usually, the deterioration of self-attention in quantized transformer causes sever performance degradation instead of the unadaptable changes in weight and activation distributions, which is also appeared in CNN-based SR networks. 3. The selected baselines in this paper are rough and the experimental results are not convinced. This paper does not compare with transformer-based post-training quantization methods, such as PTQ4ViT[1], FQ-ViT[2], NoiseQuant[3] and RepQ-ViT[4], these methods are open-source, the author should apply these PTQ methods for super resolution for that the basic transformer blocks are not different for super resolution and image classification. [1] Z. Liu et al. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092–28103, 2021. [2] Y. Lin et al. Fq-vit: Post-training quantization for fully quantized vision transformer. arXiv preprint arXiv:2111.13824, 2021. [3] Y. Liu et al. Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20321– 20330, 2023. [4] Z. Li et al. Repq-vit: Scale reparameterization for post-training quantization of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17227–17236, 2023.

Questions

Please refer to the weaknesses above.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations in this paper.

Reviewer EXmU2024-08-12

Thank you for the response and additional experiments. The specific differences about 2DQuant and PTQ4SR are clearly stated, which further shows that the differences between these two methods are marginal. DOBI on both weights and activations, one-side searching,new search object and different granularity are just engineering optimisations, the core distributions are still based on PTQ4SR. Although 2DQuant gets much better performance than baselines, the academic contributions of this paper are negligible. And I agree with Reviewer DoGX that this paper use the A+B approach to exaggerate the contribution. So I decided to keep my score.

Authorsrebuttal2024-08-13

Dear Reviewer EXmU, We appreciate your constructive reviews on the additional experiments and the positive feedback on our performance. We would like to continue addressing your concerns and resolving any misunderstandings. > Q1: The specific differences about 2DQuant and PTQ4SR ... are marginal. DOBI on both weights and activations, one-side searching,new search object and different granularity are just engineering optimisations, the core distributions are still based on PTQ4SR. A1: We clarify that the only similarity between our DOBI and PTQ4SR is the use of a search algorithm, while all other aspects are different. If these differences are merely considered engineering optimizations, then under this assumption, most search-based PTQ methods would lack any innovative aspects, which is clearly unrealistic. > Q2: Although 2DQuant gets much better performance than baselines, the academic contributions of this paper are negligible. Our 2DQuant shows an increase in PSNR of up to 4.52 dB on Set5 (×2) compared with PTQ4SR. This increase is significant in the SR field. How can such a substantial improvement be considered negligible? The SR field urgently requires lightweight and high-performance DNN-based SR models. Our method demonstrates that a DNN-based SR model with 2-bit quantization can surpass Bicubic, which is the standard for practical use. Therefore, our method can greatly accelerate the deployment of SR models in the real world, representing a significant contribution to the SR field. > Q3: ...this paper uses the A+B approach to exaggerate the contribution. Could you please specify how we have exaggerated our contributions? We believe that your request for additional experiments is intended to further highlight and strengthen our contributions. The additional experiment results show that 2DQuant surpasses all other PTQ methods you requested for comparison, as well as the state-of-the-art PTQ4SR, confirming that we have not exaggerated our contributions. We value your feedback and have addressed each of your concerns in detail. The significant performance improvements demonstrated by 2DQuant, combined with its novel approach and impact on real-world deployment, affirm its substantial contribution to the SR field. We hope this clarification resolves any misunderstandings and further underscores the value of our work. Thank you for your consideration. Sincerely, Authors

Reviewer EXmU2024-08-14

Thanks for the detailed explanation, I think I misunderstood a bit about the contributions of this paper. 2DQuant is much useful for accelerating the deployment of transformer-based image super resolution. This method is particularly beneficial to reduce the quantization error of post-softmax and post-GELU feature maps in transformer architectures. After reviewing the latest response, I decided to raise my score to 6 (weak accept).

Authorsrebuttal2024-08-14

Dear Reviewer EXmU, We extend our sincerest gratitude for your thoughtful and constructive feedback on our 2DQuant. Your recognition of the innovative aspects of our work, particularly the utility of 2DQuant in enhancing the deployment of transformer-based image super-resolution models, is **greatly appreciated**. We are honored by your decision to raise your score to a "weak accept", reflecting your positive view of our research's contribution to model quantization and SR fields. **Thank you again for your time and the positive impact your suggestions have had on our work.** Sincerely, Authors

Reviewer DoGX3/10 · confidence 5/52024-07-08

Summary

This paper introduces a novel two-stage post-training quantization (PTQ) method aimed at compressing image super-resolution (SR) models for efficient deployment on edge devices. The authors address the challenge of accuracy degradation in low-bit quantization by proposing the 2DQuant method, which combines Distribution-Oriented Bound Initialization (DOBI) and Distillation Quantization Calibration (DQC) to achieve efficient and accurate super-resolution under low-bit quantization.

Strengths

1. The writing of this paper is very clear. 2. The figures in the paper are simple and easy to understand. 3. This paper conducted some experiments to verify the effectiveness of the proposed method.

Weaknesses

1. The contribution lacks novelty. This paper does not bring any new insights. For example, the distribution of weights and activations is well studied by previous works[1][2][3]. Some papers have used searchable upper/lower bounds for quantization [2] (although these methods are based on QAT, the essence of QAT and PTQ is not different, with only slight differences in training strategies) and NO references were given in this manuscript. The proposed distillation approach is a widely used routine operation in quantization tasks [1][2][3][4][5][6]. 2. Experiments are insufficient. The PTQ for transformers has been extensively studied in many other papers (e.g. [4][5]), but it seems that the authors have not compared it with these methods. 3. Experimental settings are not consistent. Only 100 images are used in [7], which is inconsistent with this paper. References: [1] Li, Huixia et al. “PAMS: Quantized Super-Resolution via Parameterized Max Scale.” ArXiv abs/2011.04212 (2020): n. pag. [2] Zhong, Yunshan et al. “Dynamic Dual Trainable Bounds for Ultra-low Precision Super-Resolution Networks.” European Conference on Computer Vision (2022). [3] Hong, Chee and Kyoung Mu Lee. “Overcoming Distribution Mismatch in Quantizing Image Super-Resolution Networks.” ArXiv abs/2307.13337 (2023): n. pag. [4] Li, Yanjing et al. “Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer.” ArXiv abs/2210.06707 (2022): n. pag. [5] Liu, Shi et al. “Oscillation-free Quantization for Low-bit Vision Transformers.” International Conference on Machine Learning (2023). [6] Tu, Zhaopeng et al. “Toward Accurate Post-Training Quantization for Image Super Resolution.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023): 5856-5865.

Questions

See weaknesses.

Rating

3

Confidence

5

Soundness

2

Presentation

2

Contribution

1

Limitations

Yes.

Reviewer DoGX2024-08-14

Dear Authors, I hope this message finds you well. My concerns still revolve around the similarity between the core idea of this article and [1]. Although you mentioned the differences in training details between PTQ and QAT (such as the parameters updating during backward pass) and the contribution of your method to SwinIR (which I have never denied), the academic contribution of this paper is still not convincing. The difference between SwinIR and CNN models lies in the model structure, rather than the constituent modules of the model. Therefore, applying existing quantization methods to new model architectures in this paper has a certain amount of engineering contribution rather than academic contribution. If the paper considers contribution from the perspective of engineering quantity, should deployment on practical reasoning frameworks become a necessary factor? In conclusion, I choose to maintain my score. Sincerely, Reviewer DoGX References: [1] Zhong, Yunshan et al. “Dynamic Dual Trainable Bounds for Ultra-low Precision Super-Resolution Networks.” ECCV (2022).

Authorsrebuttal2024-08-14

Dear Reviewer DoGX, Thank you for your quick response and for outlining the concern that remains unresolved. > Q1: My concerns still revolve around the similarity between the core idea of this article and [1] A1: The main contributions of [1] are 1) A layer-wise quantizer with trainable upper and lower bounds, 2) A dynamic gate controller to adaptively adjust the upper and lower bounds at runtime. We differentiate ourselves from [1] in the following aspects: - **Quantizer.** We have indeed utilized the same quantize. But this is widely used and is not our main contribution. - **Different loss functions.** The granularity of the loss functions during training differs. Our DQC is conducted on the four Transformer layers within the SwinIR architecture, which represents the largest substructure. Experimentation with finer granularity, such as layer-wise (24 in total) or linear-wise (96 in total), has shown that this level of granularity is most optimal. Due to page limitations, this part of the content was omitted from the paper. In contrast, [1] applies quantization to each residual block, totaling 32 blocks in EDSR, which is slower and less effective than DQC. - **No additional modules.** [1] employs a __Dynamic gate controller__, necessitating additional floating-point (FP) modules to obtain dynamic bounds, thereby increasing the inference cost. We, on the other hand, use entirely static quantizers and do not require any additional modules, achieving the theoretical optimal speedup ratio for quantization. - **Initial values from DOBI.** [1] uses the percentile method to assign initial values, which is highly sensitive to the hyperparameters and thus requires a time-consuming search for the best parameters. Failure to do so can lead to **model collapse**, as reported in their GitHub repository issue 3. This is particularly problematic for Transformer-based models, where the self-attention modules contain many more activations. The consequence is that all elements in the attention map become the same value after quantization. Therefore, it is imperative to apply our efficient DOBI method for initial value assignment. The experimental results in Table 3 demonstrate that DOBI alone can achieve SOTA performance. Additionally, we would like to bring to your attention a **critical issue** in [1]'s implementation. They report the quantization method as per-layer quantization. But, in their github repo files(line 327-349 in model/quant\_ops.py in github repo \"zysxmu/DDTB\"), they actually uses different bounds for different channels in one activation tensor. This implement is a typical error for per-channel quantization. This is not appliable in real world GPU and will greatly **increase** the model performance. And the correct approach to quantization involves using a single pair of clipping bounds for activations and different pairs for different convolutional kernels, which is also noted in [2]. We hope you can focus on **correct** quantization approaches instead of one with implement error. We trust these clarifications address your concerns and further highlight the unique contributions of our work. Sincerely, Authors References: [1] Zhong, Yunshan et al. "Dynamic Dual Trainable Bounds for Ultra-low Precision Super-Resolution Networks." ECCV (2022). [2] Markus Nagel et al. "A White Paper on Neural Network Quantization." arXiv:2106.08295.

Reviewer Xp5V8/10 · confidence 4/52024-07-08

Summary

This paper present a practical post-training quantization method for SR transformer, namely 2DQuant. The 2dquant mainly rely one two techniques, the first is Distribution-Oriented Bound Initialization (DOBI) determine the quantization range initially, and the second one is distillation Quantization calibration (DQC) to further finetune the quantizer for accurate quantization. Both of the proposed two techniques significantly improving the performance of PTQ for SR transformer, and the quantization can even be pushed to as low as 2-bit without retraining.

Strengths

1. The proposed method is a PTQ method without full retraining instead of the usually studied quantization-aware training methods for SR tasks, which can be seen as resource-saving in real applications. Considering the targeted architectures are SR transformers is always used as well-pretrained models, the PTQ pipeline is significant and practical. 2. The proposed quantization method for SR transformer are clear and effective. DOBI allows the optimization of quantization ranges start from a statistical search-based optimal, and then use DQC to fully optimized the quantizer. It’s brings improvements of both weight and activation quantization, especially for the activation with dynamic distribution that requires more robust and powerful quantizer. 3. Experiments show that the proposed method achieve SOTA results on ultra-low 2-4 bit Sr networks, which allows SR transformer enjoys very good efficiency with little accuracy loss. And the visualization also show the good performance of the networks quantized by the proposed methods. 4. The paper is easy to follow and with good writing and presentation, and with comprehensive visualization for the distribution of both weights and distributions.

Weaknesses

1. As presented in section 3, the authors use fake quantization during the design of the method, but didn’t mention if it can be replaced by the real quant, which can be implemented and bring real acceleration in hardware. The author should discuss if the proposed method can achieve real quantized inference when deployment. 2. The algorithm of DQC is missed and make the description not very clear, I suggest to add it or merge it to the algorithm of DOBI. And Figure 3 not highlight the application of the proposed techniques, which is also suggested to revise. 3. Some details of writing should be improved and polished carefully, e.g., the subtitles of section 2 and 3, some explain of the proposed equations are missing.

Questions

Please respond for the issues raised in weaknesses part.

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors have discussed the limitation in their paper.

Reviewer 6qcS7/10 · confidence 5/52024-07-09

Summary

The authors propose a low-bit post-training quantization (PTQ) method, 2DQuant, for image super-resolution. 2DQuant is a dual-stage low-bit PTQ method. They first investigate the weight and activations. They propose Distribution-Oriented Bound Initialization (DOBI) by using different searching strategies to get a coarse bound for quantizers. They further propose Distillation Quantization Calibration (DQC) to refine the quantizer parameters with a distillation approach. They provide extensive experiments on different bits and scaling factors. When compared with SOTA methods, there proposed methods achieve superior performance with 3.6X compression ratio and 5.08X speedup ratio in the 2-bit case.

Strengths

1)The authors explore PTQ to Transformer based image super-resolution (SR), which is one of the first works in this research field. This topic is also very practical and is favored in real-world applications. 2)The authors investigate this topic with some observations and visualizations. For example, the visualizations in Figures 4, 5, 7-11, are inspiring. 3)They propose distribution-oriented Bound Initialization (DOBI) to minimize the value heterogenization between quantized and the FP models. They DOBI to search for quantizer parameters by employing customized search strategies for different distributions to balance speed and accuracy. 4)They propose Distillation Quantization Calibration (DQC) to adjust each bound to its best position finely. DQC is used to ensure the quantized model align with the FP model on the calibration set. 5)In the ablation study, the authors investigate the effect of some key components, like DOBI and DQC. 6)In the main comparisons with other SOTA methods, the proposed method obtain the best performance and visual results. I believe this low-bit quantization work in Transformer achieves excellent permance and has good potential in future research. 7)The authors give detailed derivation of their backward gradient propagation formula, which makes the method more convincing.

Weaknesses

The authors did not give results about inference time. The quatization can reduce params and ops largely. Then, how about the acceleration when deploying the quantized models into device? Some parts in the experiments are confusing. For example, in Table 3, the results of EDSR confuse me somehow. I am not very sure how the EDSR results are obtained. The original EDSR is full-precision model. Here, the authors report its quantized version. The baseline in Table 3 is not very clear. Please clarify them. In Table 4 (b), when the batch size becomes larger, the performance changes slightly. This is not very consistent with the common observation, where larger batch size usually improves performance. The writing can be further improved. Some details should be given attention and revised. For example, Section 3.2, the first letter should be capital.

Questions

In Figure 1, the proposed method performs even better than the full-precision (FP) version. What are the potential reasons behind this observation? Can this method be applied to diffusion based image SR models? If so, please give some key ideas. It will be much better if the authors can provide some preliminary results. In Table 3, what is the model size for the original EDSR? I do not which version of EDSR did the authors use in the paper. What is the motivation by providing two versions, i.e., DOBI (ours) and 2DQuant (ours), in Table 3? ------------After Rebuttal and Discussions---------- After rebuttal and disscussions, my concerns have been well solved.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors have discussion the limitation in their paper. Flag For Ethics Review: No ethics review needed.

Reviewer DoGX2024-08-08

The author's rebuttal partially solved my problem. The authors mentioned that 'not urbanization [2] is not a serious issue.' In fact, PTQ and QAT are only different in the training process, and there is no difference in quantization essence. Since there are already highly similar quantization approaches in previous works, the authors should first consider clarifying the contribution of this paper during their writing, rather than attempting to use the A+B approach to avoid citation and exaggerate the contribution. So the final rating would be 3.

Authorsrebuttal2024-08-08

Dear Reviewer DoGX, We appreciate your constructive reviews and positive feedback on our work, 2DQuant. We would like to continue addressing your concerns and resolving any misunderstandings. > Q1: The author's rebuttal partially solved my problem. First, you mentioned that we partially solved your problem. We would like to know what remains unresolved. We will do our best to address it. Please let us know the specifics. > Q2: In fact, PTQ and QAT are only different in the training process, and there is no difference in quantization essence. Second, you emphasize once again that QAT shares the same essence with PTQ. We do not agree with this point due to the differences in training cost, performance degradation, and task-specific designs. Nevertheless, we have provided detailed experimental results of [2]. We also mentioned in the rebuttal that we will cite [1] in our reversion. Our method outperforms [2] when following the PTQ settings. We think we have reached an agreement in this issue. > Q3: Since there are already highly similar quantization approaches in previous works... Third, we do not think the previous works are "highly similar" to ours. **IF so, how could our method significantly outperform other methods in the 2-bit senario?** The experimental results demonstrate that our approach is distinct from previous ones and **certainly not a simple combination of A and B**. > Q4: ...rather than attempting to use the A+B approach to avoid citation and exaggerate the contribution. Additionally, we must clarify that **we do not avoid citation or exaggerate the contribution**. As mentioned earlier, we will cite [1] in the reversion. Furthermore, the experimental results, not we, demonstrate the contributions, leaving no room for exaggeration. > Q5: So the final rating would be 3. We noticed that you scored **3** at first, but shortly after the beginning of the rebuttal, you revised it to **2** without explanation. After addressing your concerns and making efforts to resolve them, you reverted your rating back to **3**. We'd like to know why you decided to make the initial change. Could you please clarify the reason behind your initial change? Kindly list any unresolved issues so we can further improve the work. Thank you for your continued feedback and constructive engagement with our work. We look forward to resolving any remaining concerns and continuing the discussion to further improve our research. Best Regards, Authors [1] Zhong, Yunshan et al. “Dynamic Dual Trainable Bounds for Ultra-low Precision Super-Resolution Networks.” ECCV (2022). [2] Liu, Shi et al. “Oscillation-free Quantization for Low-bit Vision Transformers.” ICML (2023).

Authorsrebuttal2024-08-14

Dear Reviewer DoGX, We hope this message finds you well. In our preceding rebuttal, we made every effort to address your concerns and dispel any misunderstandings. Below is a summary of the issues we have resolved: - We have restated our primary contributions and clarified the difference between the observed distribution and those of prior studies. - We have elucidated the reasons for our initial omission of reference [1] and have now included it in our revised manuscript. - We have delineated the differences between DQC and previous distillation methodologies. - We have conducted additional experiments, comparing our approach with five other quantization methods, thereby demonstrating our superior performance in the 2-bit scenario. - We have explained the reason behind our choice of different calibration set sizes and have provided evidence of even better results when using the same set size. - We have expounded on the divergences between our proposed PTQ method and other QAT approaches. - We have clarified that we have neither avoided citations nor exaggerated our contributions. As the deadline for the rebuttal approaches, we have noticed that **Reviewer EXmU has acknowledged the merits of our work**, and we are eager to learn if there are any remaining concerns from your side that have not yet been addressed, to which we will provide further detailed explanations and clarification. We would like to express our gratitude once again for the time you have invested in reviewing our work and for the constructive feedback you have offered. Sincerely, Authors References: [1] Zhong, Yunshan et al. “Dynamic Dual Trainable Bounds for Ultra-low Precision Super-Resolution Networks.” ECCV (2022).

Reviewer Xp5V2024-08-12

After rebuttal

Thank you for the response. After reviewing the rebuttal, I can confirm that all of my concerns have been fully addressed.

Reviewer 6qcS2024-08-14

Thanks for your detailed responses, which have well addressed my concerns. I also go through other reviewer's comments, and recognize the contributions of low-bit post-training quantization (PTQ) for image super-resolution with using different searching strategies to get a coarse bound for quantizers. Overall, I tend to keep my original score as 6.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC