QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

Summary

The paper focusses on improving post-training-quantization for LLMs. Specifically, they combine insights into incoherence processing from the two #QUIP papers and rotational invariance from SliceGPT to improve accuracy after quantization. The insight is that LLMs are, or can be made, invariant to rotations on the residual, which can then be exploited by multiplying the residual with random rotation matrices (and it's inverse), which approximately reduce outliers in the activations and weights. A part of this method comes at the cost of no extra overhead, as the weight matrices are absorbed into the layers. The paper also introduces a specific form of rotation matrix, the Hadamard matrix (which was also used in QUIP# in several other parts of the network. These Hadamard matrices cannot be absorbed, but can be efficiently executed with the Walsh-Hadamard matrix multiplication algorithm, incurring only little overhead. The paper does extensive experiments on the LLamaV2 model showing the quantization is significantly improved after the rotation operations. The authors also combine the method with other methods that improve performance, such as GPTQ and grouped-quantization, showing that the method can be combined with other quantization-improvement methods. The authors also show actual speed-ups from their 4bit kernels.

Strengths

The idea of combining SliceGPT and QUIP was out there hanging in the ether, as the two papers compliment each other very neatly. I might expect more authors in this field to do similar work. Regardless, I believe this work is novel at this point. The impact of this work is potentially quite large, as it allows LLMs to significantly improve in speed by having lower-bit-width quantization much more available. On top of this, the algorithms are pretty easy to apply. The paper reads as a full state-of-the-art quantization pipeline that can be applied on LLMs in the PTQ setting. I can see this being applied in many practical LLM settings. The paper is well-written and easy to follow. It does not claim anything it doesn't do, and is factual without scruples. The experiment section is solid, and the ablations in the appendices good. The code has been released, and I have been able to verify some of the numbers from the authors. The authors went the extra mile showing not only their theoretical gains, but also speed-ups in practice.

Weaknesses

- Emoji's in titles should get desk-rejected ;) - Table 2 does not really compare to anything. There's plenty of papers that do 4 bits, you have the original GPTQ paper, LLM-QAT, ATOM, OmniQuant, SmoothQuant, simple RTN. it's pretty tough to contextualize the results here; but zero-shot is quite important for a proper evaluation

Questions

Is it possible to put more comparisons with other methods into the tables? In Table 7, you show that the GPTQ algorithm for 7B gives worse performance than RTN. I also validated that this happens in your codebase. How is it possible that GPTQ gives a worse performance than RTN? Theoretically, it should reduce the MSE for each layer, and never really give a worse performance. Is your implementation correct? Editorial notes: - 140 consider linking computational invariance to the relevant section

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

They have adequately addressed the limitations.

Summary

This paper applies the random Hadamard transform (RHT) in strategic places in the GPT architecture to improve the quality of weight and activation quantized models. The RHT is a fully invertible (up to machine epsilon) transformation that effectively concentrates matrices. The authors claim that applying the RHT to activations reduces the effect activation outliers have on activation quantization performance. The authors show that their method can be used for INT4 weight and activation quantization without significant degradation, which enables higher throughput inference in compute-bound scenarios.

Strengths

The empirical results appear to be pretty good. Quantizing both the weights and activations to INT4 using QuaRot results in minimal degradation and enables up to a >3x speedup during prefill.

Weaknesses

My main concern with this paper is that it combines methods from existing works (RHT from QuIP#, GPTQ from GPTQ, "Comptuational Invariance" from SliceGPT) instead of introducing something actually novel and new. Those works independently showed that their respective ideas worked, so it is not surprising that combining them also works. Fusing the Hadamard transformation into the weight matrix and rotating the activations is also not something "new", as this is actually how QuIP# does inference (it is cheaper to rotate the activations than the weights in small-batch settings). Regarding the actual method, QuaRot essentially does GPTQ + Incoherence Processing on the weight matrix, which QuIP showed was optimal within a certain class of rounding methods. However, to the best of my knowledge, there is no optimality result for doing Incoherence Processing + RTN, which is what QuaRot does on the activations. In fact, the distortion for quantizing a Gaussian (the result of the RHT) with a uniform scalar quantizer (INT4) is actually pretty bad relative to the distortion for quantizing other common distributions with a uniform scalar quantizer.

Questions

1. What is the comptuational cost of each of the steps in QuaRot? The matrix multiplication is the dominating factor, but the Hadamard transformation is not completely free. If you used the RHT as presented in QuIP#, the RHT can actually be quite slow for "bad" matrix dimensions where the non-power-of-2 component is quite large (eg 172 in Llama 2 7B). How would you solve this in practice? 2. The RHT requires "mixing" activations across the input dimension, making it costly to use with tensor parallelism since you need to send lots of data across a potentially slow bus. Have you run any experiments on the effectiveness of QuaRot when the RHT applied "per slice" (eg 1/8 of the activation dimension for 8 GPUs)? How well does the RHT work here? 3. RE the uniform scalar quantizer comment above, FP4 would probably work better for quantizing a Gaussian. Can you run an experiment with FP4 activations and weights and see if the quality improves?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

The authors provide a framework for W4A4KV4 quantization of LLMs, leading to computing and peak memory improvement while maintaining good performance on language generation and zero-shot tasks. They achieve this by introducing randomised Hadamard transformations at both weight and activations of transformer blocks. They effectively build on the ideas of incoherence processing and computational invariance that have shown that applying orthogonal transformation to weight matrices in LLMs allows them to be quantized with smaller errors while ensuring the network output is unchanged.

Strengths

* Achieving good accuracy in W4A4KV4 LLM quantization is a challenging problem. It is commendable that the authors achieve good accuracy in zero-shot and language generation tasks in such an aggressive quantization regime with significant compute and peak memory savings. * The paper is very well written, with a concise and clear background section and readable graphs (although some colour-coding of the boxes would be helpful). * Applying the online Hadamard transformation to all activations is an interesting idea that improves the quantizabilty of activation and KV cache. Given the complexity of rotational embeddings, the authors have also taken great care in consistently applying these transformations. * The authors have conducted very good and extended quantization ablation studies. In the current landscape of numerous quantization hacks and choices for LLMs, this research is crucial in understanding the effect of each choice.

Weaknesses

* The motivation for using Hadamard matrices as orthogonal matrices is not well-established. Given that incoherence processing and computation invariance have established that orthogonal matrices can be used to improve quantization/pruning, I would like to see a more coherent and concise motivation for using Hadamard matrices. I can tell from section 3.1 that they are computationally efficient, but this is mentioned as a side note rather than the reason for using them. Building on this, it is still unclear to me from sections 3 & 4 if Hadamard matrices are the authors' contribution or taken from previous LLM quantization work. * The comparison to other relevant work lacks details. In Table 1, the authors should take better care to ensure that the quantization granularity of weights, activations, and cache is coherent between their method and the other benchmarks. For example, for a fair comparison, they should use SmoothQuant-O1 (see Table 2 of the paper). Even if the authors use the correct quantization scheme, they should detail any differences between their method and the competing ones either in section 5.1 or the appendix. LLM quantization methods are becoming increasingly complex. Therefore, it is important that there is transparency when comparing perplexity to other methods. * Why not include the same benchmark methods from Table 1 to Table 2?

Questions

- Figure 3. What does the (α) mean in the box? Is it supposed to be a diagonal? - Have you considered listing all the quantization bitwidths/granularities weights/activations, etc., mentioned in section 5 in a table? See Table 2 of SmootQuant. - What operation does the Hadamard block do in Figs. 3&6. Based on my calculations, it is a post: $Y = X H^T$ . Please add more information about what these blocks do for each graph. - Are all results in Table 1 for all methods with W4A4KV4 and the same granularity everywhere? - Is the perplexity in Table 1 for the test or validation set? - Line 280 explicitly mentions that this is about a single transformer block rather than saying it later in the paragraph. One needs to look in the appendix or the brackets below. - Figure 4: Please mention that these are the averages of 100 runs as you do in the appendix. What is the STD of these runs? - Table 9: Are weight and activation quantized to the same bitwidth? What about KV cache? Please be more explicit about the bit width choices per tensor in all tables and figures. - Appendix A.6: why is this ablation study there? What is the take-aways?

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors recognise the limitations of the method and provide a roadmap of improvements in the conclusion section.

Summary

The paper introduces QuaRot, a novel quantization approach for Large Language Models (LLMs) that utilizes Hadamard transformations to address the challenge of outlier features in activations, weights, and KV caches. By incorporating these transformations, QuaRot enables the entire model, including activations and caches, to be quantized to 4 bits without significant loss in performance. This method improves both the computational efficiency and memory usage during model inference, preserving up to 99% of zero-shot performance in tests with the LLAMA2-70B model.

Strengths

QuaRot addresses quantization of both weights, activations and KV caches, making the approach practical in the real-world scenarios. The application of randomized Hadamard transformations helps in effectively managing outlier features, allowing for lower bit representation without performance degradation. The method achieves substantial efficiency improvements, evidenced by up to 3.33× speedups in prefill operations and significant memory savings during the decoding stage.

Weaknesses

Although the method sounds good in terms of the computational invariance, the experimental results are not impressive to some extent. Based on Table 2, there is a clear margin between the FP16 baseline and the proposed QuaRot on 7B, 13B and 70B settings. The distinction between the proposed randomized Hadamard transformations and the Hadamard quantization method in Xi et al “Training Transformers with 4-bit Integers” should be elaborated.

Questions

How to extend your method if the model adopts different normalization methods, like layer normalization or batch normalization. Also, why INT4 but not INT2 or INT1 as QuaRot can handle the latter two too? Why the RMSNorm is not quantized? In the case of normalization layer quantization, how much the acceleration performance of the model and its performance on the datasets will be affected? Furthermore, are the biases in the linear layers and the residual connections quantized to INT4? Please specify your meaning of online Hadamard transform on line 38. The consideration of the Hadamard transformations increases the difficulty of implementing the quantization kernel. Please explain more about how the quantization kernel is implemented based on the CUTLASS library to achieve real speedup? In Fig. 3, (\alpha) should be diag(\alpha)? Also, line 122, the Q^T on output matrix becomes Q in Fig. 3?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The INT4 QuaRot still incurs significant drops from INT8 and FP16.

Rebuttal incomplete

Did you per-chance forget to post an answer to the 'weaknesses' indicated by the reviewer? o_O

Comments

My point regarding SliceGPT, GPTQ, and QuIP# is that you're basically taking the RHT from QuIP# and fusing it into an adjacent layer where you can, and then quantizing the activations with nearest rounding. This is a nice implementation detail that improves throughput in compute bound scenarios. However, the only contribution here is fusing the Hadamard transform, which just doesn't seem to be that novel (this essentially amounts to saying linear operations can be fused). Nearest rounding is also very simple, and the results you posted above with MXFP4 and INT4 suggest that nearest rounding is suboptimal for the activations. Post-RHT distributions are very close to Gaussian (you can verify this with a QQ plot), so MXFP4 should result in lower distortion for the activations, but perplexity is higher. I would have liked to see some analysis on what the best way to round the activations is. Are there any rounding algorithms that can improve over direct rounding while still being fast enough to not have large overhead during inference? Regarding tensor parallelism, running the RHT per batch of data would be emulating data parallelism, while running the RHT on slices of input channels would be tensor parallelism. The reason why I asked is because the main benefit of QuaRot seems to be in compute bound scenarios where we can benefit from INT4 hardware, such as large batch inference. In memory bound scenarios where the activations are small, the memory savings from quantizing activations aren't that large. However, large batch inference usually means doing some sort of sharding like DP, TP or even both. Using TP with the RHT would mean having to use a smaller Hadamard matrix, which would reduce quality. The incoherence bounds from applying the RHT depend on the matrix dimension. Can you run an experiment where you quantize the weights as 8 separate slices along the channel dimension? I would like to see how much degradation there is by doing this. If the matrix dimension is large enough, this might not actually make a difference. I may have overlooked the contribution on using the RHT to help quantize the KV cache, which I think is new and hasn't been done before. For that reason, I will raise my score to a 5. The paper is still more of an implementation optimization paper on existing methods, so outside of the KV cache part the contribution is limited.

Thanks for your reply.

I am happy with the authors' reply and will keep my score.

Final review

I have read all reviewer comments and rebuttals, and could not find anything that would change my mind. Because of this, I'm keeping my score of strong accept. I think this method is very worthwhile, and likely going to be much cited and used in the field of LLM quantization. I believe it is significantly novel, and not just a trivial combination of two methods.