Summary
- The paper introduces PTQ4DiT, a post-training quantization method for Diffusion Transformers. This approach can facilitate the widespread deployment of DiTs. By investigating the distribution of activation and weight of DiTs, PTQ4DiT designs the Channel-wise Balancing and timestep-aware Salience Calibration. Through these, PTQ4DiT effectively quantizes DiTs to W4A8 and reduce computational costs while maintaining image generation quality.
Strengths
- The paper identifies the two key challenges associated with quantizing DiTs and provides simple but effective methodology for effective DiT quantization.
- The description and illustration of the paper is clear and easy to follow. Detailed description of the reparamterization process of the quantization parameters are provide
Weaknesses
- **Effectiveness of Salience Calibration:** The methodology sections 4.1 and 4.3 of the paper resembles existing work on quantization [1]. The primary novel contribution of this paper is the 'Salience Calibration' technique, which addresses the specific challenge of activation timestep-variance in DiTs. It is critical to verify its importance. However, the evidence supporting the effectiveness of the Salience Calibration (SSC) is currently confined to Table 3, which presents results for ImageNet with 256x256 resolution, using a W4A8 configuration. The improvement observed with SSC is only moderate when compared to the CSB method. To more effectively emphasize the paper's unique contribution, additional evidence demonstrating the effectiveness of CSB would be beneficial."
- **Overheads for Salience Calibration:** The 'Salience Calibration' technique introduces a timestep-wise correction coefficient to refine the activation $S(X(t))$ and weight $S(W)$ distributions, resulting in the adjusted binary matrices $B_{\rho}^x$ and $B_{\rho}^w$. This refinement implies that $B_{\rho}^w$ may differ for each timestep t. However, the weights are reparameterized offline prior to quantization. Introducing a time-varying $B_{\rho}^w$ could lead to the generation of distinct quantized integer (INT) weights for each timestep, which may complicate efforts to reduce the model's weight memory cost. This could involve either storing multiple sets of INT weights or repeatedly offloading weights for each timestep.
[1] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." International Conference on Machine Learning. PMLR, 2023
Questions
- Is the 'Spearman’s ρ-guided Salience Calibration (SSC)' method capable of generalizing across a different timesteps and solvers? It seems that the calibration process depends on the number of timesteps. Does this imply that recalibration is necessary when employing SSC with different models, timesteps, or solvers?
- In the context of reparameterization, the paper specifically addresses the 'Post-adaLN' and 'Post-Matrix-Multiplication' scenarios. However, within the Feed-Forward Network (FFN) layers of DiTs, there are instances where two consecutive linear layers do not conform to either of these cases. It raises the question of how to approach reparameterization in such circumstances."
- Some unclear details about the quantization process:
- Are all layers in the DiTs quantized? e.g., time embedding linear layers, attention matmuls.
- In Line 237, the authors state that " the optimization of quantization parameters follows the implementation of Q-Diffusion". Does this mean that the quantization process involves gradient-based optimization of scaling factors, and Adaround of zero-points? What is the cost of the PTQ process.
- The statement “the lower the correlation between activation salience s(X(t)) and weight salience s(W), the greater the reduction effect in overall channel salience” in Sec. 4.2 should be more clearly explained. What is the definition of "correlation salience"?
Limitations
- The authors have discussed the limitations in the appendix.