Binarized Diffusion Model for Image Super-Resolution

Summary

The paper introduces BI-DiffSR, a novel binarized diffusion model for image super-resolution, designed to accelerate the inference speed and reduce computational costs of diffusion models while maintaining high performance. It proposes a UNet architecture optimized for binarization, featuring consistent-pixel downsampling/upsampling and channel-shuffle fusion to address dimension mismatch and fusion difficulty, alongside a timestep-aware redistribution and activation function to adapt to varying activation distributions across different timesteps. The model demonstrates superior results over existing binarization methods, approaching the perceptual quality of full-precision models with significantly reduced memory and computational requirements.

Strengths

- The paper is well-written and easy to understand. - This paper designs a novelty 1-bit UNet for accurate binarized diffusion model, including: - New downsample module and upsample module for Dimension Consistency. - Channel shuffle module to balance the activation value ranges of two input features. - The timestep-aware redistribution (TaR) and timestep-aware activation function (TaA) - Experiments achieve the state-of-the-art in super resolution with diffusion.

Weaknesses

- The basic BI-Conv block lacks novelty, which is as the same as the binarized module in ReActNet that contains RSign and RPReLU. - TaR uses different parameters for different time steps, but in the mean while, the normal time embedding is projected into the resblock, it is also a time-aware on feature maps, what is the differences or why TaR works? - SR3 is not a new diffusion baseline for super resolution, ResShift[1], SinSR[2] should be better, and the metrics of PSNR, SSIM, LPIPS is much old, the CLIPIQA, MUSIQ, MANIQA should be better for evaluating the performance of generative super resolution. - Self-attention and MLP are common modules in diffusion, such as LDM[3] and ResShift[1], which require a lot of computation. How can the method in this paper be extended to self-attention and MLP? [1] Yue, Zongsheng, Jianyi Wang, and Chen Change Loy. "Resshift: Efficient diffusion model for image super-resolution by residual shifting." Advances in Neural Information Processing Systems 36 (2024). [2] Wang, Yufei, et al. "SinSR: diffusion-based image super-resolution in a single step." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [3] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

Questions

Please refer to the weaknesses above.

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have addressed the limitations in the paper.

Summary

This work present a novel binarized diffusion model for improving the efficiency of super resolution tasks. Compared with the existing works, this work first pointed out the specific challenges of binarized DMs for SR, including the dimension mismatch and fusion difficulty of representations. Then this work present several techniques: consistent-pixel down/upsampleing, channel-shuffle fusion, and Time-step-aware redistribution function for the aforementioned challenges. Comprehensive results show that the provided binarized DMs for SR not only significantly outperform the binarized models with existing SOTA binarization methods, but also achieve floating-point level performance. And for the efficiency, the statistics of params and flops show the advantage of proposed method, and the paper also present the real inference time on edge, which seems important and encouraged in the binarization community.

Strengths

1. As far as I know, this is the first work to present the specific binarization method for diffusion model of SR. Since the good performance has been achieved by DMs in various SR tasks, it’s important to present novel insight to compress these models, especially considering the severe drop still exists after binarizing by existing SOTA methods. 2. The motivation is intuitive and techniques are novelty, especially considering the features of DMs. The proposed CP-Up/down and channel shuffle are highly specified to the architecture of the diffusion models, which is novel and cannot be achieved by previous methods, including binarization function and binarized structures. And the computation is also small, allowing minor burden with significant performance improvement. And the proposed activation function also focus on the high dynamic of activation range during time-step, which is one of the most critical problem for the quantization of DMs. 3. The proposed method achieve SOTA results in accuracy. Comprehensive comparison has been included in this paper, including SOTA binarization methods and various evaluation datasets. The results show that the proposed outperforms than previous binarized DMs for SR with significant improvements. 4. In this paper, diverse analysis, including quantitative, statistical, and visual results are presented in detail. More important, the paper shows the efficiency evaluation based on real inference libraries and edge hardware, which is of great significance for practical application.

Weaknesses

Though it’s a good paper, some issues should be addressed. 1. The writing and presentation of the paper should be improved, including but not limited to the grammar and description. For example, some basic knowledge about quantization, SR, and DMs seems to be summarized as a preliminaries section; and let the proposed techniques be highlighted in Figure 2. 2. As for the efficiency, I suggest the authors present the computation more detailed, such as present the computation of each part in the whole network before and after the binarization. This will show the efficiency advantage of the proposed method much clearer. 3. The proposed challenge I and II are insightful, but more further discussion (such as visual, quantitative, or theoretical analysis) are presented after proposing. I suggest authors do more discussion about that. 4. Some recent binarization methods for SR [1] are suggested to be compared and some quantized DMs [2] are suggested to be discussed the differences to make the comparison more comprehensive. [1] Flexible Residual Binarization for Image Super-Resolution. ICML 2024 [2] EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models. ICLR 2024

Questions

1. Compared with the quantization of DMs for SR, can authors provide discussion about the advantage and motivation of binarization? 2. What is the type of ARM hardware for the evaluation of inference? 3. If the proposed method have potential generalized to more generative tasks and architectures?

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

The authors have addressed the limitations.

Summary

The authors propose BI-DiffSR to binarize diffusion based image super-resolution (SR) model. They design a UNet architecture for the whole binarized model structure. To maintain dimension consistency, they propose two modules, CP-Down and CP-Up, which can further help transfer full-precision information. To enhance feature fusion, they propose the channel-shuffle-fusion (CS-Fusion). They also propose TaR and TaA to dynamically adjust activation distribution cross different timesteps. The authors provide extensice experiments to demonstrate the effectiveness of their proposed method.

Strengths

The topic is very important and practical. Diffusion models have shown excellent performance for image super-resolution (SR). It is very practical to quantize the models before deploying them into devices. Binarization is an extreme tool to compress the SR model. Few works have been proposed to investigate such an important problem in image SR. The authors give several insights for the specific topic. Namely, there are some key aspects in diffusion based image SR binarization, like dimension mismatch, fusion difficulty, and activation distraction. Those problems hinder the performance of binarized image SR diffusion models. The observation and analyses given in the introduction section are insightful and motivate readers well. To alliveate the problems in binarized diffuision based SR models, the authors propose consistent-pixel-downsample (CP-Down) and consistent-pixel-upsample (CP-Up) to ensure dimensional consistency. They propose the channel-shuffle-fusion (CS-Fusion) to facilitate the fusion of different features within skip connections and suit binarized modules. They propose the timestep-aware redistribution (TaR) and timestep-aware activation function (TaA) to adjust the binarized module input and output arross different timesteps. They provide extensive ablation study experiments (including quantitative results in Table 1 and visualization analyses in Figures 6 and 7.) to show the effects of each proposed components. Those experiments are convincing. The authors provide comparions with SOTA methods. According to the main quantitive and visual comparisons, they show that their proposed BI-DiffSR achieves superior performance over others. The overall writing and organization are pretty good. I think the work is well-prepared. The supplementary file further provides more details. The paper is easy to follow and they promise to release the code, which makes this work more convincing.

Weaknesses

When binarizing full-precision model from 32-bit to 1-bit, ideally we can reduce the parameters by 32 times. But, as shown in Table 2, the authors reduce parameters from 55.41M to 4.58M (for scale 2). There is a gap between ideal case and practical one. Please give some analyese about the reasons for this gap. Also, are there any idea to further narrow the gap? The parameters and Ops are reduced obviously from full-precision to binary one. But the authors did not give results about inference time on real devices or give some analyses. I am curious how fast the binarized model will be. The writing can further refine in some cases. For example, in the abstract part (Line 9-10), “… to maintain dimension consistent” should be changed to “… to maintain dimension consistency”.

Questions

Can this method be applied to other diffusion models, like stable diffusion? If so, can the authors give some suggestions to binarize stable diffusion? Can we apply this binarization method to other related image restoration tasks? Like image denoising, deblurring? How long did the authors to train the models?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Please refer to weaknesses

Summary

This paper introduce a novel binarized diffusion model, BI-DiffSR, for image SR. A UNet architecture optimized for binarization, channel shuffle fusion, and time-step-aware redistribution and activation functions are designed. The experimental results proved the effectiveness of the method.

Strengths

1. This paper is well written, nicely presented, and well organized. 2. Binarized diffusion networks are promising. 3. The performance improvement over other binary SR networks is significant.

Weaknesses

1. Lack of discussion with some related works[1, 2, 3, 4], in particular [1] which is also for binary SR networks. Please analyze and discuss the differences with [1,2]. 2. Ablation experiments are not convincing enough. Comparisons with some other activation function or fusion methods [1, 2, 3, 4] should be included. 3. It is always known that diffusion models are slow. Although binarization will speed up the operation, can it achieve a better trade-off in performance and efficiency than a real-valued efficient SR network. It is suggested to compare with some efficient SR networks [5, 6, 7] in terms of Params, FLOPs, inference time and performance. > 1. Flexible Residual Binarization for Image Super-Resolution. ICML24. > 2. Q-DM: An Efficient Low-bit Quantized Diffusion Model. NIPS23. > 3. Binarized Low-light Raw Video Enhancement. ICCV23. > 4. Binarized Spectral Compressive Imaging. NIPS23. > 5. Efficient long-range attention network for image super-resolution. ECCV22. > 6. DLGSANet: lightweight dynamic local and global self-attention networks for image super-resolution. ICCV23. > 7. Feature modulation transformer: Cross-refinement of global representation via high-frequency prior for image super-resolution. ICCV23.

Questions

See Weaknesses.

Rating

5

Confidence

5

Soundness

3

Presentation

4

Contribution

2

Limitations

Limitations were discussed.

Further Discussion

I appreciate the author's efforts for rebuttal and some of my concerns were addressed. But I still have some concerns regarding the comparison of this work with real-valued networks. 1. PSNR as an important performance evaluation metric needs to be included in the comparison. 2. It is neither fair nor valuable to compare the per-step computational efficiency. As a binary network, it is natural for it to be more efficient per step than a real-valued network. However, for this method, single-step diffusion completely fails to achieve the claimed performance. As it stands, this work is far less efficient than an efficient real-valued network. In the future, with the development of specific hardware and frameworks, this work may be able to realize significant efficiency improvements. Overall, after rebuttal, I'm willing to raise my score to borderline accept.

Thank you for the rebuttal

Thank you for the response and additional experiments.BI-DiffSR is insteresting and promising to push the development of diffusion deployment. After reviewing the rebuttal, I decided to raise my score.

After Rebuttal

Thank you to the authors for their detailed response. After reviewing all the explanations, I can confirm that all of my concerns have been fully addressed. I would like to keep my original score.