DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs

Summary

The paper introduces a method called DuQuant, a new quantization method specialized for LLMs. The paper notes that "massive outliers" cause previous quantization approaches to be less effective or powerful, and then proposes a new quantization method which ameliorates the effect of such massive outliers. Theoretical analysis shows that the method has good properties, such as successfully spreading out the outlier mass directly before quantization, which facilitates better outcomes. Experiments show state-of-the-art performance across standard language modelling benchmarks on Llama and Llama-2 class models, preserving most of the base model's performance.

Strengths

- The exposition/writing is very clear. - All major steps of the method are motivated, and the theoretical calculations straightforwardly apply to the method's real-world implementation. The design choices are either made straightforwardly or chosen via experiments/ablations. - Experimental results show that the overall model performance is better than other (previously state-of-the-art) quantization methods. The time and memory costs are also reduced, owing to the simple construction of the matrices in the construction (block diagonal, orthogonal, or permutation matrices).

Weaknesses

Several potentially desirable properties of the model may not be completely covered by the analysis. For example: - The memory consumption reduction seems good, but the presentation of this result is postponed to the appendix E. Could you find a way to put this in the main body (maybe moving back the presentation of one of the several pure performance metrics such as PPL, or one of the ablation studies), and compare to other state-of-the-art models? In my opinion, since people usually only do quantization for better memory consumption or speeding up inference, measuring the performance on both of these axes is crucial. - The speedup is only measured for pre-filling. Is it possible to compute end-to-end speedup for open-ended generation? It would be great to see the impact of the method on some more realistic workloads (again, across the axes of performance, memory consumption, and speedup), to supplement the thorough analysis of each component in the paper. To address both these issues, along with an empirical study, an analysis of the asymptotic runtime/memory complexity of the quantization procedure would be great if possible. Also a slight nitpick: at the bottom of page 2, massive activations are not the same phenomenon as attention sinks, though they can be related.

Questions

- Previous work on Massive Activations have shown their ubiquity in many types of LLMs, not just Llama, and even on certain types of vision models. Is it possible that this work can be done on other types of LLMs/transformers? Does the backbone model you pick significantly change the effectiveness of the method (e.g., Phi vs Llama vs Mistral)?

Rating

8

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors are generally precise about the method's strengths and weaknesses, and include a limitations section.

Summary

This work proposed a transformation (composition of orthogonal and permutation transformation) that makes LLMs more quantization-friendly (accounting for the presence of outlier features). The approach is validated on several modern LLMs from Llama-1,2,3 families.

Strengths

The introduced method makes sense and targets the specific case of Massive outliers, that is not accounted in previous weight+activation approaches. The obtained results are pretty strong and achieve state-of-the-art at W4A4 quantization. This approach is pretty fast and takes a couple of minutes even on large LLMs. This work conducts a study on location and impact of specific types of outliers. Speedups are quite significant.

Weaknesses

The idea of applying rotation transformation for simplifying weight/activation quantization is not novel and (to my knowledge) was first introduced in QuaRot. The introduced method proposes a specific form of orthogonal/permutation matrices. QuaRot results for Llama-2-7b quantization reported in QuaRot paper are significantly better than the one presented in Table 7 and only marginally inferior to DuQuant numbers. I would suggest comparing with the numbers from original paper for fairness.

Questions

* Do the learned rotation possess incoherence property [1]? * Can one boost GPTQ performance further (as it is strictly stronger method compared to RTN) with DuQuant rotation matrices? * What is the context length for Llama-3-8B evaluation? Typically, one uses the training context length for comparison (8k for Llama-3 model family). [1] Chee, Jerry, et al. "Quip: 2-bit quantization of large language models with guarantees." Advances in Neural Information Processing Systems 36 (2024).

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

See Weaknesses.

Summary

The paper presents a new post-quantization method (DuQuant) that targets low-precision (4-bit / 6-bit) weight and activation quantization. The authors show how the presence of massive outliers affects quantization when using existing methods (smoothing is not sufficient with SmoothQuant / OmniQuant training is not stable for layers exhibiting massive outliers). They propose their method for better handling both normal and massive outliers - by utilizing orthogonal rotation and permutation matrices, with a simple zigzag permutation scheme for a better / more even distribution of outliers. They provide theoretical proofs for both their rotation and zigzag-permutation operations grounding their proposed algorithm. They showcase the strength of the propsed methods as it can be enabled with simple quantization, and not rely on expensive quantization methods like GPTQ to achieve new state of art quantized models across a range of different models.

Strengths

1. The paper presents a theoretically grounded approach to low-precision (4-bit / 6-bit) quantization of LLMs. 2. The paper showcases how massive outliers are challenging for existing methods to adapt to (for e.g., SmoothQuant just fails and OmniQuant sees unstable gradients in training layers with massive outliers). The authors also particularly show that the down-projection layers in FFNs particularly face massive outliers - which inhibit effectively their quantization. 3. The authors propose a new RAP (rotation and permutation) based method for enabling more even distribution of outliers from the activations to weights. - They first start with the smoothing operation proposed in SmoothQuant, through ablations they show this helps get better post-quant model quality. - They follow this up with a rotation operation, with the constraints that it should be able to as evenly distribute the outliers through the matrix multiplications. They show that a single rotation process cannot handle this effectively and design an iterative but greedy process that solves this using block-diagonal rotation matrices. - Followed by this, they propose a new zigzag permutation operation to evenly distribute large outliers across different blocks. Finally, they apply another rotation to ensure that the outliers are maximally reduced and spread across the weights. 4. They follow the proposal of the method through theoretical analysis, showing how their methods will results in either optimal results / have bounds in each phase of the smoothening to ensure maximal quality. 5. They propose two variants of the algorithm - one standard with specific activation / weight clipping coeffs and another with LWC enabled from the OmniQuant algorithm. They follow these with 4-bit and 6-bit results for the LLaMA-1,2,3 and Vicuna-1.5 models, showcasing that their method outperforms other existing methods in these settings. 6. They provide a comparison with the recently proposed QuaRot method and how their proposed algorithm performs better than the algorithm, while being competitive for implementation performance.

Weaknesses

1. Most of the evals (except MT-Bench) are logit-based evals (and not generative). This has a side effect of hiding some of the inherent limitations of low-precision quantization algorithms (i.e., error accumulation across generated tokens). 2. One limitation of the proposed benchmarks (for implementation performance) is that they measure the prefill performance, but do not show any generation performance. This typically dominates over prefill performance. 3. One thing is the reporting of results is not consistent across the models - for eg. some tables use Atom for reporting model quality at 4-bits, but some models do not report this performance. While the benefits of the method are clear given the higher accuracies on downstream tasks, it is difficult to judge how the differences translate on models of higher quality (e.g. Llama-2 vs Llama=3 70B).

Questions

1. Many times the perplexity numbers and downstream performance are not 1:1 correlated. This most likely has to do with the standard error of the downstream tasks? If so, it will be good to clarify this for the reader. 2. Figure 5 is not clear in explaining the different settings - what do Perm 0, 1, and 2 correspond to? Also in the paragraph above the figure, in lines 312-313, it is not clear which Figure is being referred to. 3. For Table 6, it will be better to explain how to read the table. It took multiple passes to understand the full setup of the ablations and the associated results in the table. 4. It is surprising that the method is largely calibration free - indicating that the outliers are more a property of the model weights, and activations are suffering as a by-product of this? can the authors clarify their intuition around this.

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors address the limitations and broader impacts of their work. One thing they do mention in the checklist is reporting of statistical signficance - which I do not see anywhere in the paper. Can authors point to where these results are? Or equivalently change the checklist to reflect this.

Summary

The paper explores new approaches in LLM quantization. The work tries to address the performance degradation due to massive outliers in the weights. The work show competitive performance across different settings, up to 4-bit weight activation quantization. The work also provides solid experiments and visualization on the effectiveness of the proposed methods.

Strengths

1. I think the paper is well-written. The experiments, plots, and tables are well clearly tied to the story on large/abnormal outlier in the activation. 2. The work is easy to follow, especially in the methodology. 3. The results in the tables show that work offer very competitive performance comparing to previous methods.

Weaknesses

1. I am a bit puzzled by the results in Table.1 vs Table.2. It is shown that lower perplexity does not necessarily reflect in higher benchmark accuracy. This conflict exists within DuQuant (DuQuant vs DuQuant_LWC) and other methods like AffineQuant,OmniQuant vs QLLM. The authors might want to consider adding some justification to this conflict to further improve the soundness of the tables.

Questions

From the benchmarks, it is shown that the proposed method almost matches the fp16 results (unquantized). It would be interesting to see the mt-bench of the quantized vs fp16 model to see if such an observation still holds.

Rating

8

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

NA

Summary

The paper proposes a new LLM quantization method named DuQuant (for Dual transformations Quantization). This method is able to quantize the weights and activations of an LLM to 4 or 6 bits without losing significant precision. The paper identifies the issue of Massive outliers in the activations of a LLM. These are outliers within the outliers, having magnitudes in the order of 100-1000. Traditional quantization methods, such as SmoothQuant, cannot deal with these outliers, because they attempt to address the errors locally. DuQuant, on the other hand, uses a sequence of rotation, permutation, and another rotation to spread these outliers out across many weights, resulting in a more accurate capture. The results are supported with theoretical results as well as experiments. DuQuant outperforms the baselines on perplexity, QA and MT-bench.

Strengths

The paper is well-written and easy to understand (apart from a few minor typos). I especially like figure 2 that give a strong intuition to how the method works and also why it eliminates Massive outliers. The algorithm, the justification and the results are easy to follow. The experimental results are very convincing. They demonstrate an improvement in prediction quality in W4A4 across the board. The paper shows extensive experiments looking at various Llama models as well as Vicuna. Compared to the baselines (SmoothQuant, AffineQuant, OmniQuant, Atom, QLLM), DuQuant always performs the closest to the uncompressed model. This result is shown across various tasks: perplexity, QA and MT bench. The paper also includes ablations and runtime costs of the required transforms: the transforms incur an additional 8.9-9.3% computational cost at inference time compared to standard W4A4.

Weaknesses

A shortcoming of the paper is the lack of close comparison to QuaRot. QuaRot is a very similar technique and it deserves to be in the related works section as well as be a baseline in the experiments. While there is a detailed comparison in the appendix, the work should also be mentioned in the main body. The paper could be improved at places, but I don't consider any of these shortcomings a major issue: * The paper has numerous typos (eg.: Line 11: Typo, Line 188: typo, Line 313: Figure reference missing). Please proofread. * The paper is extremely dense, its sometimes difficult to distinguish the figure captions from the main body. Please use proper paddings for tables and figures where possible. * The description of how the rotation matrix is constructed could be more detailed (Lines 141 - 147). Given that this is a core algorithmic contribution of the paper, it should be clear to the reader how it works. Perhaps a figure or pseudocode would be helpful here.

Questions

No outstanding questions.

Rating

7

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

No limitations.

Thank you for your reply

Thank you for the detailed reply and for addressing my concerns. I think the construction of the rotation matrix will be an important clarification in the revised manuscript.