Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers

Summary

This manuscript proposed a layer-wise post-training quantization (PTQ) algorithm called aespa aiming at quantizing large transformer models, and refined quantization objectives for the attention layer which can accelerate the quantization process by pre-computation. The authors gave clearly descriptions for the formulas which are used as quantization objectives and showed analysis results that aespa has a lower computational cost. The authors also showed stable experiment results comparing aespa with other quantization methods. Overall, I would suggest an Accept.

Strengths

1. This PTQ algorithm is proved by mathematical methods, so it is easy to be applied to other transformer models. 2. The proposed algorithm considers the cross-layer dependency by targeting attention modules. In detail, it quantizes query, key, and value in an attention module separately: when quantizing one projection, the other 2 projections are fixed with full precision, decreasing the heavy computation of block-wise quantization. 3. Compared with conventional block-wise quantization and layer-wise PTQ schemes, the perplexity of aespa showed good results

Weaknesses

1. This quantization algorithm only focuses on the attention module, targeting the cross-layer dependency within an attention module, not cross-layer dependency within the full model. 2. In Experiments and Appendices, most tables compare the performance (perplexity), time, and memory cost. There is only one table (Table 10) showing the accuracy of zero-shot performance. There is no explicit result showing how aespa is more accurate than other methods. How much is the decrease in the accuracy compared with the original floating point models?

Questions

1. When quantizing a transformer model with the proposed algorithm, will all weights and activations stay the same quantization precision? Or only the weight will be quantized? 2. Will it be possible to discuss more on the accuracy of aespa compared to other methods?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors dicusse dthe limitation of aespa: this algorithm only focused on the attention module as a block.

Summary

The paper introduces 'aespa,' a novel post-training quantization (PTQ) algorithm designed to balance accuracy and efficiency in quantizing hyper-scale Transformer models. This method quantizes layer-wise for efficiency and employs attention-wise reconstruction to account for cross-layer dependencies. Extensive experiments and complexity analysis demonstrate that aespa outperforms existing PTQ methods, especially in low-bit quantization scenarios.

Strengths

1. Aespa, which combines layer-wise quantization with attention-wise reconstruction, is a significant advancement in the field of model quantization. 2. This paper emphasizes a pre-computation-based loss computation strategy, demonstrating its significant speed advantage over traditional block-wise PTQ methods like BRECQ. 3. Extensive experiments on diverse language models and datasets provide a comprehensive evaluation of aespa's performance. 4. Enhancing the credibility of this method would benefit from conducting thorough ablation studies.

Weaknesses

1. Further experiments on LLMs are crucial, particularly with the LlaMa 2 & 3 family. 2. How were other layers updated within a transformer block, apart from the attention module? Were they done layer-wise or block-wise? 3. The implementation details require clarification. Specifically, in the attention module, Q, K, and V are updated separately. How many iterations are necessary for each layer? Does the order of updating Q, K, and V impact performance?

Questions

please refer to Weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

A more in-depth discussion on implementation details would be beneficial.

Summary

As a cost-effective alternative, learning-free PTQ schemes have been proposed for LLMs. Still, the performance is somewhat limited because they cannot consider inter-layer dependency within the attention module, a significant feature of Transformers. This paper propose a PTQ algorithm that balances accuracy and efficiency. The key idea of the proposed algorithm called aespa is to perform quantization layer-wise for efficiency while targeting attention-wise reconstruction to consider the cross-layer dependency. Through extensive experiments on various language models and complexity analysis, it demonstrates that aespa is accurate and efficient in quantizing Transformer models.

Strengths

The scheme aims to reconstruct the attention output to consider cross-layer dependency while quantizing models layer-wise to pursue efficiency. It is different from previous layer-wise error minimization. From extensive experiments on language models, it demonstrates that the approach outperforms conventional schemes by a significant margin, particularly for low-bit precision (INT2). To accelerate the quantization process, it proposes refined quantization objectives for the attention module. Through a complexity analysis, it demonstrates that about 10 times faster quantization than existing block-wise approaches can be achieved by exploiting the proposed objectives.

Weaknesses

I am a bit confused about the results of baselines. For OmniQuant, in the original paper, the perplexity on wikitext2 for 2bit weight quantization on llama-7B, 13B, and 30B are 15.47, 13.21, 8.71, respectively. However, in this paper, they become 24.28, 22.85,13.04. The differences are too large. Though the authors mention that the difference is due to different calibration dataset, the ablation study in OmniQuant demonstrates that different calibration datasets (wikitext2, C4, and pile) do not lead to significant perplexity difference with very small variance, see table A10 of OmniQuant. So the performance of OmniQuant using another calibration dataset such as C4, should also achieve a similar performance close to the original paper. However, the results in this paper are significantly different from the original paper. It is better to provide more discussions about this problem. The OmniQuant and AffineQuant methods demonstrate the performance of quantization in groups. Smaller groupsize leads to better performance. It is better to include some ablation study with different groupsizes. LLaMA3 is published. It is better to have some results for recent models such as LLaMA2. Most of the results are shown in the appendix. It is better to organize the experimental part better to include the main results in the main paper. Currently there are only block-wise perplexity comparison in the main paper. It may be better to put zero-shot evaluation in the main paper.

Questions

See the weakness.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper propose a new quantization strategy that balances accuracy and efficiency that reconstruct the attention output to consider cross-layer dependency. To accelerate the quantization process, this paper proposes refined quantization objectives for the attention module. This approach gets better results in low-bit precision.

Strengths

1 This article does a lot of experiments, which are also in the appendix section. 2 This paper obtains higher accuracy at ultra-low bit than any other work.

Weaknesses

1 Formula derivation is too redundant. 2 Paper does not make clear the trade-off between accuracy and efficiency in introduction.

Questions

1 In Figure 2, what is Layer, what is Block? 2 Following the method in paper, will the training time longer than ordinary PTQ?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

I did not identify any issues related to limitations and potential negative societal impact.

Summary

This paper presents a post-training quantization approach for transformers. The proposed algorithm finds a better balance between accuracy and efficiency. The method is to perform quantization in a layer-wise way for efficiency and the optimization objective is to maintain the reconstruction of the quantized model to obtain good accuracy. Experimental results on 3 datasets over 2 transformers (OPT and LLaMA) are reported at the low bit widths (3 Bits and 2 Bits). The proposed approach outperforms other baselines at the low bit widths.

Strengths

- This paper is well written and organized in general. - The mathematical formulation and derivation are clear and easy to follow. - Results on low bit widths outperform several baselines including BRECQ, OmniQuant, and AffineQuant. - Experimental results on two sota LLMs (OPT and LLaMA) are reported.

Weaknesses

- The novelty in this paper is limited, as similar formulation and optimization methods have been widely used in prior papers. - Experiments are not solid. Some important results and comparisons are missing. - The paper lacks analysis and results to support the claims of the contributions. Please check the questions section for details.

Questions

- The idea of reducing reconstruction error for quantization is not new - several prior works have already proposed this idea and adopted it in their works. The optimization method (iterative optimization - updating one variable and fixing others) is not new. - Actually, this paper only considers the reconstruction of the current quantization layer, which is sub-optimal. A better way is to optimize the reconstruction of the last layer (the reconstruction of the whole network). Because output errors could accumulate, small output errors at the first layers may change to large ones when going to the last layers. Only the reconstruction of the last layer's output can reflect the impact on the network. - This paper only reports results at low bit widths (4, 3, and 2 Bits). Although it outperforms the three baselines (BRECQ, OmniQuant, AffineQuant), the accuracy loss is noticeable at low bit widths. How about the results at higher bit widths (6 bits or 8 bits)? Is that possible to maintain the accuracy at higher bit widths? People want to see quantization to a certain number of bits without losing accuracy. - One important contribution claimed by this paper is the capability to handle situations where frequent model updates and multiple hyperparameter tunings are required. However, we haven't found practical results about the analysis and comparison of such situations. Although the authors gave the time complexity of the algorithm and provided the time to run the algorithm, I think this is different with the real situation when weights are frequently updated. Moreover, some baselines also show faster speed (OPTQ, Z-FOLD) than the proposed method. - I suggest that the authors cite the latest related post-training quantization works for generative models or diffusion models [1][2][3][4][5], discuss the differences with them, and compare with them if possible. [1] Towards Accurate Post-training Quantization for Diffusion Models [2] TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models [3] Post-training Quantization on Diffusion Models [4] Q-Diffusion: Quantizing Diffusion Models [5] PTQD: Accurate Post-Training Quantization for Diffusion Models

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

N.A.

Response to Authors' Rebuttal

Thanks for the authors further work. I would maintain my scores.