BiDM: Pushing the Limit of Quantization for Diffusion Models

Summary

This paper proposes BiDM, which focuses on quantizing both weights and activations of diffusion models (DMs). Specifically, the authors introduce: - Cross-timestep feature connection to enhance the accuracy of noise prediction in binarized DMs. - Space-patched distillation, a novel variant of distillation loss that emphasizes spatial locality.

Strengths

- Binarization of diffusion models (DMs) represents a promising research area aimed at accelerating sampling processes, and this work appears to be the first to fully binarize DMs to W1A1 to the best of my knowledge. - The introduction of cross-timestep information connection is innovative in enhancing the performance of binarized DMs. - The experimental results on LSUN-Bedrooms, LSUN-Churches, and FFHQ effectively demonstrate the effectiveness of BiDM and XNOR techniques.

Weaknesses

- The experimental results lack sufficient conviction. Tables 1 and 2 compare BiDM primarily with works focused on quantizing discriminative models, which may not be the most appropriate choice of baseline. It would be more informative to include comparisons with quantization methods designed specifically for diffusion models, such as EfficientDM [cite], given its open-sourced nature and compatibility under W1A1 settings. This comparison is crucial for a comprehensive evaluation. - Figure.4 may not effectively demonstrate the advantages of paying attention to local information. To better illustrate the effectiveness of Space-Patched Distillation (SPD), the authors should compare experimental results with those obtained using vanilla distillation loss (Mean Squared Error between outputs of a full precision model and a binarized model). - The proposed binarization method introduces additional convolution computations between two matrices in floating-point, which may not be efficient for hardware deployment. Moreover, the dynamic calculation of matrix $A$ during inference introduces memory access overhead that could be significant in practical deployment scenarios. - During inference, matrix multiplications (MMs) in the attention mechanism typically consume considerable time. It appears that this work retains these MMs in floating-point arithmetic, which could impact efficiency. **minor issues** - Line 93: The citation for BinaryDM is incorrectly linked. - In Figure 4, the caption states "Ours denotes the self-attention," but in the figure, it's labeled as "SPD." - The term "self-attention" in Section 3.3 is confusing; perhaps using an alternative term would be clearer.

Questions

- Are the Q, K, V in the attention modules binarized during inference? - During inference, is $A$ in Equation 9 calculated from the inputs? - In Line 224, the authors claim that "conventional $l_2$ loss makes it difficult for binary DMs to achieve direct matching." Could you provide evidence or reference existing works that have verified this claim? - Figure 3 is somewhat confusing. Why is $L^{t-1}$ a linear scaling up of $B^{t-1}$? Is this relationship depicted by Equation 12?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

This paper includes a discussion of limitations.

Summary

The manuscript proposes a method for fully binarizing both the weights and activations of diffusion models, named BiDM. Structurally, it introduces an improved XNOR method for scaling factors of activations and high-level feature connections across time steps, based on observations of existing temporal phenomena. For optimization, the manuscript presents a patch-based attention mechanism distillation method, approached from the perspective of spatial features. Quantitative analysis and generated examples demonstrate that BiDM, as the first work to fully binarize diffusion models, surpasses existing binarization methods, showcasing the potential for deploying diffusion models in resource-constrained scenarios.

Strengths

1. This is the first work to fully binarize diffusion models, preventing them from collapsing in extreme scenarios and demonstrating acceptable generated samples. This is significant for exploring the compression potential of generative models. 2. The improvement to XNOR is not only applicable to diffusion models (DMs) but also has positive implications for the whole binarization field. Most papers and deployment frameworks, when replicating and implementing XNOR, have consistently followed the approach described in Equations 7 or 8 of this manuscript, rather than the method described in Equation 9. The authors have meticulously examined and adaptively improved the original XNOR approach within the DM field, prompting reconsideration in both DM compression and general binarization fields. 3. The cross-time step connection appears novel and well-suited to the inference structure of DMs. Unlike the cache design in DeepCache, which aims at inference efficiency, this manuscript uses connections to enhance information due to the inherently efficient nature of binarized models. These connections can be placed at multiple nodes within the model, with learnable coefficient factors providing greater adaptability. 4. Section 3.2 first designs TBS from the perspective of information enhancement and then provides further explanation from the perspective of error reduction in Figure 3, making the overall method design appear cohesive and intuitively clear. 5. The distillation strategy design is straightforward and effective, approaching patch division based on the inherent requirements of the generation task and the natural locality of the convolution module, and achieving excellent results.

Weaknesses

1. The description of the cross-time step connection during the training phase is somewhat vague. During the inference phase, the impact of this connection is iteratively attenuated. For example, when α is 0.3, the influence of step T on step T-2 is 0.3×0.3=0.09, rather than 0. However, upon reviewing the source code in the supplementary materials, I found that the authors only consider steps T-1 and T-2 during training. I would like to know if this affects the final accuracy. 2. The scenario assumed in Figure 3 is overly idealized. When only L^t changes, for instance, when L^t is at the lower right of L^(t-1), the weighted average might result in T^(t-1) being further from F^(t-1). How is this situation explained? 3. Although BiDM shows significant improvements over other binarization methods, there remains a substantial performance gap compared to full-precision models. This could still hinder its practical application.

Questions

See Weaknesses

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The manuscript only addresses diffusion models using a U-Net backbone primarily based on convolution. It appears to be inapplicable to diffusion models like DiT, which are primarily based on transformers and not related to U-Net.

Summary

This paper aims to fully binarize weights and activations of diffusion models (DMs) to achieve storage saving and inference acceleration. To this end, the paper proposes timestep-friendly binary structure (TBS), which employs learnable activation binarizers and cross-timestep feature connections to capture the correlation of the activation features over the timesteps. In addition, the paper introduces space patched distillation (SPD) to match the spatial locality of binary features with the full-precision ones during training. The method is tested on some common image generation benchmarks such as LSUN and FFHQ 256x256, CIFAR-10.

Strengths

- The paper aims to tackle a challenging problem which is to fully binarize both weights and activations of diffusion models. - The paper is well-written. - The experimental results are promising. - The proposed methods of using timestep-friendly binary structure and space patched distillation are well-motivated.

Weaknesses

- The main weakness of the paper is that the proposed method, BiDM, increases the training time of DMS compared to the original process. This issue should be more precise in the paper. The authors should compare the convergence rates of the proposed method and its competitors in terms of wall-clock time and identify which components contribute to the increased training time by breaking down the complexity of all main components of the proposed model. - The chosen baselines are quite weak as all of them were originally designed for discriminative tasks such as image classification. The authors should consider stronger baselines dedicated to generative tasks such as [1] and the baselines therein. Ref: [1] Xia et al. Basic Binary Convolution Unit for Binarized Image Restoration Network. ICLR 2023

Questions

In Table 3, it is not clear why adding SPD alone makes the results worse, whereas combining it with TBS leads to improved performance. Do the authors have any explanation for this phenomenon? This point should be addressed to clarify the underlying reasons for the observed behavior and to provide a deeper understanding of the interaction between SPD and TBS in the proposed method.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes

Rebuttal Discussion

Dear Reviewers, The authors have provided a response to the comments. Please respond to the rebuttal actively. Best, AC

Official comment from Reviewer 4HDt

Thanks the author(s) for the rebuttal. As most of my concerns have been addressed, I will increase score by 1 point.