The diffusion model has gained popularity in vision applications due to its remarkable generative performance and versatility. However, high storage and computation demands, resulting from the model size and iterative generation, hinder its use on mobile devices. Existing quantization techniques struggle to maintain performance even in 8-bit precision due to the diffusion model's unique property of temporal variation in activation. We introduce a novel quantization method that dynamically adjusts the quantization interval based on time step information, significantly improving output quality. Unlike conventional dynamic quantization techniques, our approach has no computational overhead during inference and is compatible with both post-training quantization (PTQ) and quantization-aware training (QAT). Our extensive experiments demonstrate substantial improvements in output quality with the quantized diffusion model across various datasets.
Paper
Similar papers
Peer review
Summary
This paper presents a novel quantization method for diffusion models that dynamically adjusts the quantization interval. To this end, the authors present a unique Temporal Dynamic Quantization (TDQ) module, which utilizes the time step as an input to determine the quantization interval, which significantly improves the generation quality. Extensive experiments on CIFAR-10 and LSUN Churches demonstrate the superior performance of the proposed method.
Strengths
The main idea is well-written and easy to follow. The proposed method, though simple in design, proves to be remarkably effective, notably diminishing the decline in performance. Its simplicity does not compromise its efficacy, making it a compelling solution to the problem at hand.
Weaknesses
The primary weaknesses of the paper include a lack of a vital comparison with Q-diffusion, a key baseline in diffusion models, and a need for more comprehensive ablation studies, such as investigating the impact of the number of layers in the generator module on the model's overall performance.
Questions
1. In Section 3.1, the authors state that uses symmetric quantization for weights and set $p=-n=2^{b-1}-1$. However, compared with LSQ that set $n=-2^{b-1}$, it appears that the proposed method might be under-utilizing one quantization level. This could potentially lead to a slight inefficiency in the model’s performance. 2. In Section 3.3, the authors state that previous studies have demonstrated that incorporating a quantization module … can lead to significant improvements in accuracy for specific applications. However, there appears to be a lack of citation or reference supporting this assertion. 3. Instead of using the proposed TDQ module, it is possible to directly learn distinct quantization intervals corresponding to different time steps? 4. The paper lacks a comparison in its experimental section. Specifically, the study has not considered Q-diffusion (as detailed in reference [36]) as a baseline. Since Q-diffusion also applies post-training quantization to diffusion models, it's critical for the authors to include a comparative analysis of their proposed method against Q-diffusion. This would help to further validate the efficacy of their approach and enrich the overall discussion. 5. The paper could benefit from additional ablation studies, which would provide a more comprehensive understanding of the proposed method. One specific point of interest mentioned is the influence of the number of layers in the generator module on the overall performance. By investigating this aspect, it would add depth to the analysis and potential insights into the efficiency of the proposed model.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors do not provide any discussions on limitations.
Summary
While diffusion models provide outstanding quality and diversity of generated data, they demand high storage and computation, limiting its applications on mobile devices. Further, the paper points out that current existing works on quantization are not suitable for diffusion models as diffusion models have unique temporal properties due to iterative sampling process. As such, the paper introduuces a new quantization technique for diffusion models, where quantization interval is dynamically adjusted. As a result, the proposed framework prvoides strong performance.
Strengths
- The proposed method can be easily integrated with the existing QAT and PTQ algorithms. - Clear visualizations which help the understanding of the paper and motivations. - Simple yet effective quantization method without additional inference costs.
Weaknesses
- Don't input-dependent quantization methods [31,32,33,34] provide computational efficiency benefits even after considering extra overhead? Is there a proof for supporting the following claim "dynamic approaches often incurs additional costs for extracting statistical information from activations, making it challenging to achieve performance improvements in practice."? It would be good to refer to the results or references for supporting this claim or tone down the claim a bit. . - Missing experimental comparisons against [36], which is another work on quantization of diffusion model. - The paper claims that the activation distribution is similar across inputs with Figure 2. But, Figure 2 shows the activations of an earlier convolutional layer, which is known to have low-level features that are common across diverse layers. It would be good to show that similar trends can be observed across different layers. Indeed, less clear and less time dependent trends can be observed in deeper layers (mid.attn_1.proj_out.0, down.1.downsample.conv.0, down.1.attn.0.k.0, mid.block_1.conv1.0) in Figure 7 of the supplementary material. I suspect that deeper layers will exhibit even more complex trends that may not look like time-dependent. More detailed discussions are needed. - The paper claims that the proposed method can be seamlessly integrated into existing quantization algorithms, but the paper only shows the benefit of doing that with LSQ in the case of QAT. - Non-exhaustive list of typos (a thorough proof-reading is needed): - can decipts in Line 148 - there have been limited works tried to in Line 109
Questions
Please refer to the weakness section
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
No limitations mentioned in the paper. The potential limitation of the work is not taking into account for activation distribution variation that exists among different inputs.
Summary
This paper proposes Temporal Dynamic Quantization (TDQ), a new quantization approach for diffusion models. This method utilizes a learned module to adjust the quantization interval with respect to timestep to deal with activation quantization errors. Unlike conventional dynamic quantization techniques, the TDQ approach has no computational overhead during inference and is compatible with both post-training quantization (PTQ) and quantization-aware training (QAT). Experiments have been performed using DDIM on the CIFAR-10 dataset and LDM on the LSUN-Churches dataset with TDQ and various traditional QAT and PTQ approaches.
Strengths
- The paper is well organized and written with clear motivation; the idea of addressing the rapid changes in input activation to reduce the truncation & rounding errors with a time-dependent quantization configuration is natural and interesting. - The proposed TDQ framework can be applied to both QAT and PTQ cases. - Experimental results demonstrate that the proposed framework can outperform several previous QAT and PTQ methods. Ablation studies have been conducted to show that TDQ can generalize to scenarios with fewer denoising steps and analyze output dynamics across layers.
Weaknesses
My concerns are mainly about the evaluation in this paper: - The proposed TDQ framework primarily aims to deal with the activation quantization, but it is not specified whether the act-to-act matrix multiplications (attn and v) in the attention layers are quantized. By examining the code attached in the supplementary materials, it seems like those computations still remain in FP. However, they can take a significant portion of the compute & memory overheads and are fully quantized in closely related work [1], while past literature [1, 2] has pointed out that the post-softmax attention scores are usually hard to quantize (e.g. PTQ4DM suffers from performance degradation after quantizing them according to Appendix E in [1]). It is unclear if TDQ can deal with this performance degradation without providing experimental results with act-to-act matmuls in attention layers quantized. - It is not accurate to call PTQ4DM the 'state-of-the-art study'—Q-Diffusion [1] was proposed around the same time as PTQ4DM and contained results that went down to 4-bit weights on both CIFAR and LSUN datasets, and achieved superior results on CIFAR. Its code (with quantized checkpoints) was released in April. I would highly recommend the authors include and compare Q-Diffusion results for CIFAR and LSUN-Churches experiments as it is a highly related work that natively covers many experimental settings in this work that are not in PTQ4DM. [1] Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., ... & Keutzer, K. (2023). Q-diffusion: Quantizing diffusion models. arXiv preprint arXiv:2302.04304. [2] Yuan, Z., Xue, C., Chen, Y., Wu, Q., & Sun, G. (2021). PTQ4ViT: Post-Training Quantization Framework for Vision Transformers with Twin Uniform Quantization. arXiv preprint arXiv:2111.12293.
Questions
- Can you further explain the output dynamics in Figure 9 and Appendix Figure 6? E.g. what do blue and red curves stand for and how to interpret them? - Looking at Appendix Table 1, I am curious if some simple baselines following the same principle can be helpful, such as using multiple quantizers at different time steps under the LSQ setting e.g. using a separate activation quantizer every 20 steps. - How sensitive is the TDQ module (MLP) to the hyperparameters? Are there different design choices for the training process that cause different results? For instance, it is mentioned in the paper that initialization is crucial – are there experiments showing that a different initialization (e.g. gaussian, Xavier, zero…) can make results worse?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The authors adequately addressed all limitations.
Summary
The authors present and empirically evaluate a novel quantization scheme for diffusion models. Static quantization of the evolving activation distribution in diffusion models leads to performance degradation. By using a quantization range which is a function of the timestep, the authors improve performance.
Strengths
The paper is very well structured and easy to follow and provides clear and concise descriptions of both introductory quantization literature, as well as diffusion model literature. The problem definition and proposed solution are also very well described, especially in Figures 2 and 3. Empirically, evaluation under multiple metrics (FID and ID) is always appreciated. The experimental setup is well described.
Weaknesses
- The form of the TDQ module (network size, parameters, structure) isn't provided. - Not a particularly major issue but there are a few spelling/grammatical mistakes. For brevity I won't list all the errors but online tools like Grammarly are free to use and would help in this regard. - For other weaknesses, please see my questions.
Questions
- This paper seems slightly contradictory. In section 3.3, the authors assume that the time evolution of the activation distribution is constant, which permits the use of their TDQ module. The authors also make note of the high cost of inference (clearly outweighing the training costs). So why is the TDQ network module neccessary? Why not just learn the the activation distribution directly for each timestep. Learning this function through a network (rather than simply per timestep) would impose a smoothness contraint on the function. Is this the only benefit? - Have the authors provided an ablation study where the activation variations, as in Figure 9, are used rather than the TDQ module? Again, why not just compute these beforehand? Does using this pre-computed activation variance perform better? Perhaps some more detailed ablations studies are required? - How robust is this TDQ module to training? No error bars are provided so it's unclear if there are any failure cases, or if the predicted intervals remain quite consistent across trials. Given the assumption in section 3.3, they should always converge to the same interval values. Do the authors observe this?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
- The limitations of this work aren't explicitly outlined. For example, whilst the authors show a couple of plots illustrating the time evolution of the activation distribution, the assumption doesn't appear to be investigated. What are the effects of this assumption? - In addition, it doesn't appear that the authors have outlined any limitations wrt to how text conditioning (v. common for diffusion models in practice) could break this assumption. Perhaps useful for future work, but the authors could make a comment for the sake of practitioners looking to employ this method? - I see no wider negative societal implications of this work.
Response to Authors
Dear Authors, Thank you for the comprehensive revisions and the detailed response to the previous comments. I commend your additional efforts, particularly the inclusion of new experiments and tables, which have undoubtedly enhanced the overall quality of the paper. However, upon closer examination, I still have some reservations concerning the experimental results. Specifically, in Tables 1 and 2, I noticed that the performance of QAT (w8a8) on Churches appears to be inferior to that of PTQ, despite utilizing a higher number of DDIM steps (200 compared to 100). Could you please shed some light on this observation? Looking forward to your clarification. Best regards, Reviewer ijsE
Response to Reviewer
We deeply appreciate your thoughtful feedback. Each comment we receive plays a crucial role in refining the quality of our work. In response to your feedback, we concur that the results presented in Tables 1 and 2 for the QAT require revision.Subsequent to our paper's submission, we identified an error in our source code. Specifically, when implementing QAT on the pretrained model, it is essential to construct a new PyTorch model embedded with the quantization operator and then import the pretrained parameters from the traditional full-precision model. Regrettably, during this transition, one key parameter - the pretrained scale-factor of the LDM Autoencoder - was not loaded accurately. This oversight meant that during the QAT operation, certain parameters were initialized from random values rather than from the pretrained checkpoint, leading to a decrease in accuracy. Below, we present the corrected results of the QAT church experiment, which now incorporates the accurate scale-factor. In our ongoing commitment to comprehensive and fair evaluation, especially when juxtaposed with PTQ, we have utilized a 100-DDIM step for these updated results. | | w8a8 | w4a8 | w8a4 | w4a4 | w3a3 | | --- | --- | --- | --- | --- | --- | | PACT | 9.20 | 9.94 | 8.59 | 10.35 | 12.95 | | LSQ | - | 4.92 | 5.08 | 5.06 | 7.21 | | TDQ | **3.87** | **4.04** | **4.86** | **4.64** | **6.57** | From the provided table, it's evident that QAT consistently outperforms PTQ, despite every attention layer being quantized. For those keen on replicating our results, please adjust the scale_factor to `0.2458` on `line 18` within the following file paths: `configs/lsun_churches-ldm-kl-8.yaml` and `configs/LSUN-churches_sampling.yaml`. It's also worth emphasizing that the benefits of our proposed method remain evident even in the revised results. While the entire suite of QAT experiments was impacted by the aforementioned error, a substantial proportion of the network parameters were correctly initialized. As a result, any degradation experienced was relatively minor and consistent across the board, but the overarching trend remains discernible. We deeply regret and apologize for any misunderstanding or confusion stemming from our initial erroneous experimental data. Rest assured, the corrected findings will be incorporated into the final draft of our paper.
Response to Authors
Dear Authors, I appreciate the thoroughness of your rebuttal and the additional clarifications provided. The supplementary results have addressed my previous concerns. Warm regards, Reviewer ijsE
I thank the authors for the rebuttal. One question still remains regarding W3. The claims regarding time dependency behaviours seem to be based on hand-picked observations. What is the exact ratio of layers that have time dependency behaviours in comparison to layers that do not have time dependency similar to `mid.attn_1.proj_out.0`? and also which layers tend to have time dependency behaviours and which layers tend to lack time dependency? Best, Reviewer xsTz
Exact ratio of layers that have temproal depenence
Thank you for your detailed feedback. It is important to quantify this ratio to support our claim that temporal dependence exists in diffusion models. To measure the exact ratio of layers with high temporal dependence, we calculated the *correlation* between time steps and the activation range of each time step. Specifically, we computed the absolute value of the Pearson correlation coefficient and nlcor[1], as some layers may exhibit non-linear temporal dependence. Both metrics produce values between 0 and 1, with higher values indicating stronger temporal dependence. |method | r>0.3 | r>0.5 | r>0.7 | | --- | --- | --- | --- | | absolute pearson | 71.6% | 62.1% | 38.8% | | nlcor[1] | 72.8%| 47.5% | 31.0% | The table reveals that about 60% of the layers exhibit moderate temporal dependence (r > 0.5), and roughly 35% exhibit strong temporal dependence (r > 0.7). This implies that many layers in diffusion models indeed possess strong temporal dependence. Therefore, we can deduce that employing the TDQ module for these layers is important. Additionally, we can assert that these metrics serve as proper representations of temporal dependence, supported by values `r=0.19` for `mid.attn_1.proj_out.0` and `r=0.90` for `down.1.atton.0.proj_out.0` in Figure 7 of the Supplementary Materials. --- Reference [1] Ranjan, Chitta, and Vahab Najari. “nlcor: Compute Nonlinear Correlations.” Research Gate (2020) (2020).
Response to Rebuttal
Dear Authors, First, I would like to thank you for your detailed rebuttal and additional experiments pertaining to the comments made. Whilst the response goes some way to answer the questions raised, I would like to clarify a few outstanding concerns: Q1: Whilst the TDQ module outperforms the new baselines on most tasks, I am curious about other factors involved. For example, what was the training setup? Was any regularisation across time implemented (as mentioned in my initial question) to impose the smoothness of the changes in the activation distribution. Did the authors find this simpler/easier to implement than the TDQ module? L2: Whilst I agree that the general trend stays somewhat similar with text conditioning, there are certainly subtle differences in the time evolution. Does this affect performance? I think it would be useful to present some more concrete performance metrics, especially across multiple models and more text prompts. That being said, it could perhaps make for very interesting future work. I would welcome any comments by the authors, especially with regards to Q1, which seems to be quite crucial to the proposed contribution. Many thanks.
Response to Reviewer qrnn
Dear reviewer qrnn, Thank you for bringing up important points. ### Q1 - **Training Setup**: We wish to emphasize that our training for TDQ did not use a specialized setup. We adopted the same optimizer, learning rate, scheduler, and weight decay as used for the baseline training. The TDQ module can be integrated into any static quantization activation algorithm without modifications to the existing training pipeline. For transparency, we have provided the source code, enabling easy reproduction of our results without special setups for the TDQ module. - **Smoothness Regularize**: No dedicated regularizer was employed to govern the smoothness of the activation distribution. As mentioned earlier, we adhered to the existing training pipeline without modifications. However, it is worth noting that TDQ can potentially act as an implicit smoothness regularizer due to the inherent characteristics of Multilayer Perceptrons. These networks naturally yield similar outputs for analogous inputs, consequently leading to a trend where the truncation of activations at similar time steps tends to be the same. - **Simpler Implementation**: We respectfully assert that the TDQ module we've proposed presents the most straightforward approach to addressing the temporal variation of activation distribution. As shown in Table 2 of the global response, when the TDQ dimension is adjusted for the target task's complexity, it significantly outperforms S_{N}. The TDQ module is user-friendly and integrates smoothly into existing frameworks. While there may be a need to adjust certain internal hyperparameters like the hidden dimension for specific tasks, we have provided a comprehensive implementation guideline that is applicable across various datasets. This makes it easier for researchers and developers to harness the advantages of TDQ without extra effort. ### L2 Thank you for your insightful feedback, which has certainly enhanced the quality of our work. We concur with the suggestion to explore the proposed technique for text conditioning in future work. To offer a clearer evaluation in response to your comments, we carried out further experiments using QAT for the text-conditioning diffusion model. However, due to the constraints of the rebuttal period, we regret that our results may appear somewhat preliminary. To measure the performance of TDQ Module for text conditioning, we applied our QAT-TDQ to Stable Diffusion (V1.4). We employed the CLIP score as our evaluation metric, a common metric for generative models, particularly for reference-free models like Stable Diffusion.Specifically, the LAION-Aesthetics(6.5) dataset was employed for QAT. In the evaluation of the clip_score, we randomly selected 15 texts from the MS-COCO dataset and produced 1000 images for each text. Subsequently, we computed the clip_score for both the Full-Precision (FP) Stable Diffusion model and our quantized TDQ version. | Text idx | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | |----------|------------|------------|------------|------------|------------|------------|------------|------------|------------|------------|------------|------------|------------|------------|------------| | FP | 0.315 | 0.314 | 0.310 | 0.302 | 0.311 | 0.315 | 0.316 | 0.30 | 0.320 | 0.317 | 0.329 | 0.312 | 0.348 | 0.282 | 0.347 | | TDQ_W6A6 | 0.299 | 0.309 | 0.304 | 0.291 | 0.300 | 0.305 | 0.317 | 0.290 | 0.297 | 0.294 | 0.309 | 0.292 | 0.339 | 0.276 | 0.332 | As shown in the table, the performance degradation for each text is not substantial, delivering a consistent performance drop of around 0.01 to 0.02. These results clearly demonstrate that TDQ remains minimally affected by subtle distribution variations prompted by text conditioning, making it well-suited for application in text-to-image models like Stable Diffusion or Imagen. Nevertheless, we posit that incorporating text conditioning information has the potential to enhance the quality of the output. While this might increase the overhead of step size generation owing to text-dependent features, such an increase is likely to be marginal compared to input-dependent quantization, and could be comparable to the overhead associated with the TDQ module. We view this avenue as a promising extension for future work. Once again, we are genuinely grateful for your insightful feedback.
Response to Authors
Dear Authors, Thank you for the comprehensive rebuttal and the clarifications provided. The additional results have mostly addressed my concerns. I would be inclined to raise my scores if those experiments and discussions (e.g. comparison with Q-Diffusion, learning quantization interval directly for each time step etc.) will be included in the final version of the paper. Best regards, Reviewer 7wR5
Response to Reviewer
Dear reviewer 7wR5, We will gladly include these experimental results and discussions in the final version of our paper. Thank you very much for taking the time to review our work and provide valuable feedback. We believe that these experiments and discussions have undoubtedly enhanced the quality of the paper.
Decision
Accept (poster)