Training Transformers with 4-bit Integers

Summary

This paper presents a 4-bit training method mainly to speed up the training process. In the forward propagation, Hadamard quantizer is proposed to suppress the outliers whereas structure sparsity and bit splitting are used for quantization of gradients. It has been shown that the proposed training method can successfully train transformer models across different datasets with some performance degradation w.r.t. the baseline.

Strengths

-- The main strength of this paper is theoretical analysis of their proposed quantization method for both forward and backward steps. -- Use of different models and tasks for evaluation purposes is another strength of this paper.

Weaknesses

-- The main concern of mine is the limited speedup of this paper. The main goal of quantized training in this paper is to speed up the computations and not to reduce the memory. However, the authors have only shown up to 35% speedup w.r.t. the baseline (FP16). First, there is no comparison with Int8 training speed. It's not clear why I should choose Int4 over Int8 while I know the accuracy of Int8 is better than Int4. Second, the speedup amounts are only measured across specific configurations which are not the ones used for evaluation purposes in Table 1. Measuring the training time for the tasks listed in Table 1 can make the contribution of this paper clear. -- Fig. 4 and Fig. 5 are hard to read. Again, why not simply reporting the speedup for each task in Table 1. -- I have a hard time understanding Table 3. What does "epoch 1-5" mean? Which dataset was used for the measurement? What is the accuracy performance?

Questions

In summary, the idea of Int4 training to accelerate the process is interesting. The authors have shown that Int4 training can obtain a comparable accuracy w.r.t. the baseline and Int8 with around ~1% to ~2% degradation, which is fine. However, the speedup amount is not provided for the benchmarks used for accuracy evaluation in Table 1. Given the above explanations, I have the following questions: 1) What is the speedup for tasks listed in Table 1? 2) How do you compare your method w.r.t. Int8 in terms of training speed? In other words, why should I select Int4 over Int8? 3) How was the accuracy performance measured? Did you report the accuracy in Table 1 using quantized inference (Int4) or full-precision? 4) Please provide more details for Table 3.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

NA

Summary

The authors propose 4-bit training methods for Transformers. It first proposes a Hadamard quantizer with activation outliers issue being solved by Hadamard Transformation. Optimization for backpropagation is proposed by leveraging gradient sparsity with bit splitting and score sampling. The proposed 4-bit training methods achieve competitive accuracy on NLP, Machine translation, and Image classification.

Strengths

* Achieves competitive or superior accuracy compared with existing works on 4-bit training. * Optimize backpropagation with hardware-efficient structural sparsity. * Hardware-friendly INT4 training method is runnable on GPUs with up to 2.2 times faster than the FP16 MM.

Weaknesses

* The activation outlier handling in Transformers is not new, as in SmoothQuant. And the comparison is lacking in the paper. Are those techniques not able to be used in 4-bit training? And what is the efficiency comparison between those methods when handling outlier activations? * Could you elaborate more on why the proposed methods cannot be applied to Conv layers? What is the bottleneck?

Questions

* It would be helpful if you could explain why the speed is not comparable with the baseline in Fig.5 when the hidden layer size is small.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

* Conv cannot be supported * INT4 training on extremely large models remains to be an open question.

Summary

The research paper presents a framework for training transformer-based neural networks using 4-bit integers, offering enhanced computational and memory efficiency. The proposed method includes a dedicated Hadamard quantizer for forward propagation, designed to suppress outliers which typically cause degradation of model accuracy. The Hadamard quantizer works by transforming the activation matrix, spreading outlier information across nearby entries in the matrix, and reducing the numerical range of outliers. In backpropagation, the authors exploit the structural sparsity of activation gradients. They note that a few token gradients are extremely large, while most are much smaller, even below the quantization residuals of larger gradients. To make efficient use of computational resources, the authors propose a technique called bit splitting. This technique divides each token's gradient into two parts: the higher 4 bits and the lower 4 bits. They then identify the most informative gradients using a technique known as leverage score sampling, an importance sampling technique in the field of RandNLA. The combination of these techniques results in an algorithm that employs 4-bit integer MMs for all linear operations in transformers. This algorithm has been evaluated and found to perform competitively across a wide range of tasks, including natural language understanding, question answering, machine translation, and image classification.

Strengths

- Unlike many other low-precision training methods, this proposed method does not require custom numerical formats and is compatible with contemporary hardware, such as GPUs. This opens the potential for widespread adoption and usability. - The proposed method demonstrated competitive or superior accuracy across a range of tasks, compared to existing 4-bit training methods. This indicates the practical effectiveness of the method in diverse applications. - The authors have developed specialized quantization methods for both forward and backpropagation. These techniques manage outliers and leverage the structural sparsity of activation gradients, maintaining model accuracy despite the ultra-low numerical precision.

Weaknesses

My main concern is that the paper chose to compare the performance and computation complexity of their approach with FP16 rather than examining its efficiency against INT8 or other INT4 alternatives. This could potentially limit the depth and breadth of the performance analysis. The key contributions of this research are primarily focused on the implementation of a Hadamard quantizer and bit splitting techniques. Both techniques leverage INT4 matrix multiplication (MM). However, the HQ-MM has the 4 steps procedure to leverage the efficiency of INT4 computation and the bit-splitting technique deploys two INT4 MMs as a replacement for one INT8 MM. This can potentially induce more overhead than simply utilizing INT8, particularly considering that INT4 does not straightforwardly translate to half the computational, memory access, or energy costs associated with INT8 computations. While the aim of utilizing INT4 computations is to increase efficiency, it's crucial to recognize that the benefits of moving from INT8 to INT4 are not always linear. According to a report by NVIDIA, INT4, in an optimal scenario involving Convolutional Neural Network (CNN) inference in image classification tasks, resulting in a throughput improvement of approximately 59% with a negligible accuracy loss (around 1%) on NVIDIA T4. Meanwhile, on TITAN RTX, the speedup was about 52%. These improvements, although significant, don't necessarily equate to halving the computational resources or doubling the performance, and this is also applied to the transformer-based models scenario of this paper. Hence, a more thorough comparative study involving INT8 and other INT4 counterparts could potentially provide a more nuanced and comprehensive understanding of the computational advantages and trade-offs of the proposed techniques. Additionally, if there are any misinterpretations or inaccuracies in my understanding of the paper, I would appreciate it if you could provide corrections to ensure my perspective aligns accurately with the content and intentions of the research.

Questions

The concept of using a Hadamard matrix to decompose matrix multiplication is a thoughtful and fitting approach in low-bit quantization scenarios, especially for extreme cases such as binary and ternary methods. The paper meticulously analyzes the time complexity of the Hadamard Quantizer Matrix Multiplication (HQ-MM) procedure in Section 3.3, and in Appendix A.5, it further elucidates on how to select an appropriate Hadamard matrix size. I am intrigued to learn more about the finer details of this procedure. Given the significant shift in distribution during training, it is indeed crucial to reconfigure both the Hadamard matrix size and the Learned Step Size Quantization (LSQ) step size. This brings up a few questions: Is the re-initialization process executed only once, or is it a recurring process? Does the system stabilize after approximately two epochs? Furthermore, is X representative of the activation of a batch of input data? These details will enhance the understanding and application of the proposed methodology.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Implementing advanced quantization techniques such as the Hadamard quantizer and bit splitting may be technically challenging and potentially difficult to integrate into existing neural network training pipelines. The method exhibits limitations in its scalability, as it is not equipped to support exceedingly large models like OPT-175B. This underscores a broader, unresolved issue in the realm of machine learning, where even the application of INT8 training to these substantial models remains an open-ended problem.

Summary

This paper deals with 4-bit training (fine-tuning) of transformers. For forward propagation, a Hadamard quantizer is proposed to solve the problem of outliers. For backpropagation, the authors leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling techniques to quantize gradients accurately. Experiments show up to 35.1% speedup for 4-bit training with

Strengths

1. The proposed Hadamard quantizer is simple while useful for transformer quantization with outliers. 2. The authors illustrate the structural sparsity of gradients and propose bit splitting (BS), which splits a full-precision matrix as higher and lower 4 bits. The similar idea is used in quantization for inference. This is the first to use it for training. 3. The experiments are extensive. The training speed on GPUs are reported.

Weaknesses

1. The proposed method introduce extra FP16 computations. The BS with selection needs data rearrangement, which could slow down the speed. 2. The title is somewhat over-claim. The experiments are mainly about fine-tuning (FT). It seems that the proposed method works for fine-tuning (FT) of a converged model. For pre-training (PT), there is large accuracy loss. 3. The speed-up of up to 35.1% is misleading. It is better to use the average speed-up in the abstract.

Questions

Please see the weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Rebuttal Comments

Thank you for the clarification and responses to my comments. My main concern here is the speedup gain from the proposed algorithm. I can understand that INT4 training is a hard task but the main goal of such a method (that results in an accuracy degradation without any memory saving during the training process) is to speed up the training process. Therefore, it's expected to show a superior speed compared to existing INT8 training methods. If there is no open-source INT8 training method, then you can at least compare with INT8 inference methods such as I-BERT which is available on HuggingFace.