MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization

Summary

- The authors propose an optimization-based preprocessing technique called Weight Magnitude Reduction (MagR) to improve the performance of post-training quantization (PTQ) for large language models (LLMs). They motivate their method from previous work, showing that linear transformation of weights can render more quantization-friendly, allowing for sub-4bit quantization. However, such methods normally require a post-processing step during inference to undo that linear transformation, introducing a run-time overhead. - Instead, they propose a non-linear transformation without needing an additional post-processing step. That transformation aims to minimise the $\ell_{\infty}$ norm, or the maximum value of the weights per channel or column while preserving the layer's output. By making the weight distribution more compact around zero and removing outliers, they are able to quantize weights to 2 or 3 bits with little accuracy drop compared to competing methods. - They formulate the optimization as an $\ell_{\infty}$ -regularisation and solve it using proximal gradient descent. In fact, the problem reduces to computing the projection of weight vectors onto the $\ell_1$ ball, for which there are several efficient algorithms available.

Strengths

- MagR achieves state-of-art weight quantization without introducing any additional computation overhead. This is a very important contribution as all other competing methods with the linear transformation of weight, such as QuIP, require a linear transformation of the transformer block input/output. Having said this, the inference runtime comparison to QuIP is missing from the experiments section - The authors extensively evaluate the methods, evaluating a large corpus of datasets and benchmarks. They also prove that they outperform strong competing methods, such as OmniQuant, in almost all cases. - The idea of regularizing the weight's $\ell_{\infty}$ norm is quite novel and very effective. In fact, I am surprised this has not been established before in PTQ literature.

Weaknesses

- The paper is very focused on ppl and accuracy results on a wide range of benchmarks and datasets at the cost of other important ablation studies. For example: - ablation study on the penalty factor $\alpha$ and its impact on the final accuracy. - Inference runtime comparison is missing compared to competing methods, such as QuIP. In fact, the main claim of the paper is that it is faster than QuIP, but there are no studies supporting it. - Results with activation quantization beyond FP16 and runtime improvements vs baselines. - The connection between the rank-deficient feature matrix argument and $\ell_{\infty}$ regularisation requires further theoretical justification. There must be a theoretical guarantee for the optimality of that optimizaiton. In fact, the method would be stronger if it was formulated more generally as an $\ell_p$ constrained, and the authors show that $\ell_{\infty}$ is better constrained.

Questions

- Table 1: It is unclear to me how all these numbers are relevant in supporting the method. One can see from the maximum value that all Llama Models are rank-deficient based on the maximum value being below 100%. How are the rest of the statistics informative? - Line 124-125: From all the solutions available, why choose the lowest infinity norm? Why is this optimal? What is missing there is the optimality of this norm for quantization purposes. There, in fact, a discussion about quantization noise as a perturbation bounded by $\ell_{\infty}$ -norm in a paper by M. Alizadeh et. al called "Gradient $\ell_1$ regularization for quantization robustness" - Table 3: activation quantization bitwidth missing - Please provide ablation studies about the penalty factor $\alpha$. What is the effect of it not being tiny? How sensitive are the results? - To fully support the claims of the paper, it would be nice to see an ablation study on other norms as regularization. Otherwise, a theoretical justification is required about the optimality of the $\ell_{\infty}$ -norm. - Paragraph starting at line 210: it is unclear how you establish the optimality of the shrinking factor $\beta$. In fact, I cannot see why you would not search for the optimal clipping threshold based on minimizing the MSE per column or group. This is a standard method for weight clipping in LLM quantization literature. - Section 5.3: the section's title is misleading. Maybe rename it to preprocessing runtime? Most commonly, runtime refers to the inference runtime - A section on inference runtime comparison to QuIP is required. The main claim of the paper is that the method is faster than QuIP due to the lack of transformation, but no experimental evidence is provided to support it.

Rating

8

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The paper lacks a thorough discussion of each limitation. The only reference I found was in lines 245-6 about extending MagR into incoherence processing.

Summary

The paper proposes a novel approach called Weight Magnitude Reduction (MagR) to improve the performance of PTQ for LLM. The MagR reduces the magnitude of the weights in PTQ. The experiments demonstrate the effectiveness of MagR.

Strengths

1. The idea is clear. 2. The paper is easy to read (except typos/errors).

Weaknesses

1. There exist some typos/grammatical errors in the paper and should be revised. 2. The presentation is not clear. For example, this paper aims to reduce the parameters of LLM using PTQ, however, the title and abstract did not reflect the contribution of LLM compression. In the abstract, the author claims MagR can diminish the maximum magnitude of the weights and smooth out outliers, however, the title only focuses on Weight Magnitude Reduction. 3. Provide a more thorough discussion of the generalization ability, robustness, and potential applications of the proposed approach. 4. The format of references is not correct, i.e., [1], [3], [9],[14], etc. 5. The main limitation of this paper is that the proposed method lacks theoretical analysis. 6. While the abstract provides an overview of the proposed method, some aspects of the methodology could benefit from further elaboration in the main paper. Providing step-by-step explanations and intuitive visualizations for key components such as L infinity norm, and L infinity-> L 1 norm, would enhance the reader's understanding. 7. The paper claims that the method outperforms state-of-the-art methods, but a more comprehensive comparative analysis is needed. Detailed comparisons with other existing approaches, along with discussions about the reasons behind the performance differences, would strengthen the argument for the superiority. 8. Exploring the reasons behind the success of these techniques and providing intuitive explanations would contribute to the overall scientific contribution of the work.

Questions

1. The paper contains numerous hyperparameters, such as \(\alpha\). Including more ablation studies would enhance the readers' understanding. 2. In Table 3, why did MagR+OPTQ not achieve better results compared to peer competitors? 3. How can the impact of outliers on PQT for LLM be reduced?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

NO

Summary

Authors propose an optimization-based preprocessing technique called MagR to enhance the performance of post-training quantization. MagR adjusts weights by solving an l1-regularized optimization problem, reducing the maximum magnitude and smoothing out outliers. As a nonlinear transformation, MagR eliminates the need for additional post-processing, thus avoiding overhead during inference. Experiments demonstrate that MagR achieves state-of-the-art performance on the Llama family of models, such as significantly reduced perplexity on Wikitext2 for the LLaMA2-70B model.

Strengths

1.The experiment results are good. 2. The idea is novel, which is using preprocessing technology before the quantization process.

Weaknesses

In fact, I like the work but not in authors' view. I think this work is improve the generalization of model via norm. If this paper presents their work in this view, I would like to accept more. 1.Some value is lack of explanation, like " minimal the singular values of the feature matrix are less than 0.01 times the maximum singular value." 2. The applicability and the nature analysis of methods should be discussed.

Questions

1. In Section 4.1, the singular values of the feature matrix are less than 0.01 times the maximum singular value. How is 0.01 calculated? If this value is changed, relaxing the constraint, will the results change?" 2. If the feature matrix is not actually rank-deficient, could the performance of the MagR method potentially suffer? 3. Proximal gradient descent may require more computational resources. 4. The model performance after preprocessing and before quantization should be presented. I wonder that the nature of this method is improve the generalizability of model before quantization.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The paper is lack of discussing the nature of preprocessing, i.e., what the preprocessing do for quantization.

Summary

This paper introduces Weight Magnitude Reduction (MagR), a technique designed to smooth out outliers before LLM quantization. MagR adjusts pre-trained floating-point weights by solving an ℓ∞-regularized optimization problem. This preprocessing step reduces the maximum weight magnitudes, making the LLMs more suitable for quantization and resulting in better perplexity/accuracy results compared to models without preprocessing. Importantly, the MagR technique does not introduce any additional computational overhead during inference.

Strengths

1) Since MagR is a technique for preprocessing LLM weights before quantization, it can be used in conjunction with other quantization methods (e.g. OPTQ). 2) MagR introduces proximal gradient descent steps, and it addresses these through layer-wise optimization using the Hessian, resulting in a shorter quantization runtime similar to OPTQ. 3) This method does not require additional operations for scaling activation, as needed in AWQ and QuIP, thereby avoiding extra overhead during inference.

Weaknesses

1) MagR does not show significant improvement in perplexity/accuracy over previous method - QuIP. Additionally, DecoupleQ [1] presents strong results for 2-bit quantization. While I understand that DecoupleQ was released recently, allowing insufficient time for a thorough comparison, I recommend the authors review DecoupleQ. 2) The paper emphasizes that MagR has no inference overhead, unlike AWQ and QuIP, which require scaling activations. However, it does not provide any latency information to highlight the significance of removing this additional inference overhead. 3) The runtime cost comparison between MagR and other methods is insufficient. The paper only provides the runtime of MagR, mentioning roughly that it is half the runtime of OmniQuant on page 8, lines 252-254. Since MagR is a preprocessing method, it is unclear if this comparison holds when considering the entire quantization process (MagR + OPTQ). Additionally, the runtime of QuIP is not addressed. 4) Some prior works have regularized weight distribution to make the model more suitable for quantization [2]. However, this paper does not mention or compare the differences between the proposed method and these earlier approaches. 5) The paper does not include any information about the calibration dataset. [1] Guo, Yi, et al. "decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points." arXiv preprint arXiv:2404.12759 (2024). [2] Kundu, Arnav, et al. "R2 Loss: Range Restriction Loss for Model Compression and Quantization." arXiv preprint arXiv:2303.08253 (2023).

Questions

1) What is the inference latency of the proposed method compared to QuIP? 2) What is the runtime for QuIP and OmniQuant? 3) Why did you not provide perplexity results for the 7B/13B models for QuIP in Table 2? 4) Is QuIP incompatible with group-wise quantization, or can it be adapted to group-wise quantization with minor modifications, as done with MagR and OPTQ? 5) For the results in Table 3, do all the methods use channel-wise quantization, or do some use group-wise quantization? 6) What type of calibration dataset did you use, and how many data points were included for calibration? 7) What happens if you apply Omniquant/QuIP after processing LLMs with MagR?

Rating

7

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

The proposed MagR is a promising approach, as it regularizes weight values to produce LLMs that are better suited for quantization. However, this paper lacks sufficient evaluation to convincingly demonstrate the superiority of the proposed method compared to state-of-the-art techniques.

QuIP quantization error

Dear authors, Thank you for your thorough response to our comments and questions. I have one more question. In the attached figure, you compare MagR's quantization error with OPTQ. Do you have time to add the quantization error with the transformed weights of QuIP? This would be quite useful in understanding whether your optimization is theoretically superior to the random linear transformation of QuIP (beyond QuIP's additional compute overhead). Also, is the quantization error calculated before or clipping, namely, is $\beta$<1 for these results? Thanks

QuIP Minimizes a Different Quantization Error

Thank you for your questions. In response to the second question, we evaluated the quantization errors for INT4 quantization, and we did not apply weight clipping at 4-bit (i.e., $\beta=1$). This means that the reduction in quantization error is entirely due to the MagR preprocessing. Regarding the quantization error with QuIP, it targets a different quantization error in a distinct parameter space compared to our method or OPTQ. Specifically, we solve $\min_{W_q\in\mathbb{Q}} ||X(W_q - W)||^2$, where the focus is directly on minimizing the error in the original weight space. In contrast, QuIP first applies orthogonal transformations $U$ and $V$ to the weights and activations before quantization: $\min_{W_q} ||XV V^{\top} (W_q - W)U||^2 := \min_{\tilde{W_q}\in\mathbb{Q}} ||\tilde{X} (\tilde{W_q} - \tilde{W})||^2$. Here, $\tilde{X} = XV$, $\tilde{W_q} = V^{\top} W_q U$, and $\tilde{W} = V^{\top} W U$ represent the transformed activations and weights. QuIP only quantizes the transformed weights $\tilde{W_q}$, and their corresponding $W_q$ is actually float. In contrast, we directly quantize $W_q$ as is done in standard OPTQ. As a result, the quantization errors between our method and QuIP are not directly comparable. We hope we have addressed the reviewer’s questions.

Reviewed score

Having reviewed the responses, ablation studies and latest results in the conversation with reviewer jujD, I have decided to increase my score to strong accept, provided the additional results and analyses are included in the camera-ready version. From my initial review, I found the method itself innovative but concerns about the presentation and certain claims. The authors have been diligent in addressing the points raised by me and other reviewers.