OneBit: Towards Extremely Low-bit Large Language Models

Summary

This paper presents OneBit, a framework for quantizing large language models (LLMs) to 1-bit weight matrices. Unlike existing methods that rely on 4-bit or 8-bit quantization to avoid severe performance degradation, OneBit introduces a novel 1-bit parameter representation and an effective parameter initialization method based on matrix decomposition.

Strengths

- This paper proposes an aggressive compression method, exploring the feasibility and challenges of compressing LLMs to 1-bit. This is a highly meaningful research direction, and similar work should be encouraged for publication. However, I have some concerns about this approach, which I will detail in the questions section. - The paper is well-organized and well-written. - The experiments compare the proposed method with current popular low-bit quantization methods and demonstrate superior results. Additionally, the experiments address an important question in the current LLM research field: whether to use quantized large models or directly use smaller models.

Weaknesses

- Entropy Encoding: Intuitively, 1-bit compressed models should be suitable for entropy encoding. Previous work has demonstrated that quantized LLMs still have compressibility [1,2]. Have the authors tried using popular entropy encoders to further compress these weights? - Inference Calculations: During inference, do g and h participate in the calculations? Does this mean that 1-bit net is not entirely integer-based computation? Does each layer have its own G and h? - Dequantization: Is dequantization required between layers during inference? - Knowledge Distillation (KD): In line 191, it is mentioned that KD does not use LM loss. Why is that? - Comparison with 1.58bitnet: Have the authors considered comparing their method with 1.58bitnet [3]? - Table 2 Details: Does Table 2 show the performance of W1A16 or W2A16? If it is W2A16, where is the performance of W1A16 reported? Why are there no experimental results for 16B and 60B models in Table 2? I suspect that extreme quantization has a more detrimental effect on larger models. - Comparison in Figure 3: Figure 3 compares the performance of 1-bit quantized 7B models with smaller models. Are these smaller models in full precision? How would the results compare with 8-bit smaller models? - Code Availability: Will the code be open-sourced? - Minor Issues: Figures 3 and 4 are not clear when printed in gray-scale. References: [1] Mao Y, Wang W, Du H, et al. On the compressibility of quantized large language models. arXiv preprint arXiv:2403.01384, 2024. [2] [2] Hershcovitch M, Choshen L, Wood A, et al. Lossless and Near-Lossless Compression for Foundation Models[J]. arXiv preprint arXiv:2404.15198, 2024. [3] Ma S, Wang H, Ma L, et al. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits[J]. arXiv preprint arXiv:2402.17764, 2024.

Questions

See above.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See above.

Summary

This paper explores an innovative 1-bit quantization framework for Large Language Models (LLMs) to significantly reduce their memory and computational demands. Traditional methods face severe performance drops with reduced bit-width; however, this paper introduces a novel quantization and initialization approach that maintains at least 81% of the original model’s performance, even with extreme bit-width reduction. The proposed method, which includes a specialized matrix decomposition and parameter initialization, demonstrates strong performance and robustness in experiments, establishing a new direction for deploying LLMs on resource-constrained environments.

Strengths

The paper is easy to follow. The proposed method shows good performance, achieving at least 81% of the unquantized model’s efficacy, a significant achievement given the drastic reduction in model complexity and size.

Weaknesses

A significant issue discussed in the paper is the lack of a specialized CUDA kernel for optimizing binary operations, which hinders accurate evaluation of the additional computational costs associated with the two FP vectors $\mathbf{a}$ and $\mathbf{b}$. This limitation complicates the assessment of their impact on overall performance. Furthermore, despite the inclusion of $\mathbf{a}$ and $\mathbf{b}$, there remains a considerable performance decline compared to FP16 models, challenging the practical applicability of this approach in real-world settings.

Questions

1. The experiments described in Section 4.3 may not provide an appropriate comparison. Assessing the proposed method alongside directly training smaller models or utilizing low-rank decomposition to minimize parameter counts involves fundamentally different approaches to reducing model size. Additionally, the proposed method incorporates knowledge distillation, which is not employed in the baseline methods being compared, potentially skewing performance comparisons. 2. An essential ablation study is notably absent from the discussion. The proposed method incorporates two additional floating-point vectors, $\mathbf{a}$ and $\mathbf{b}$, for binary quantization. Yet, the impact of these vectors on performance enhancement remains unclear, highlighting a gap in the evaluation of the method’s effectiveness. 3. To offer a more comprehensive evaluation, I recommend including assessments on generative tasks, such as code generation. This would provide deeper insights into the versatility and practical applicability of the proposed method across different domains.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations of their study.

Summary

This paper proposes OneBits, a novel quantization-aware training methodology for 1-bit large language models (LLMs). OneBits introduces two key contributions for training 1-bit models. First, it presents a new 1-bit binary quantization linear design that separates the weight matrix into sign and value components. The sign is packed into INT1, while the value is decomposed using a 1-rank decomposition factor added to the linear operation. Second, to train the 1-bit models in the linear layers of BitNets, OneBits modifies the traditional quantization-aware training (QAT) method by augmenting the cross-entropy loss function with an additional term for the reconstruction error of each layer, resulting in the final objective loss function. Using the proposed approach, OneBits is applied to various decoder-only LLM models. The comparisons between OneBits (W1A16) and other methods like LLM-QAT, AWQ, and OmniQuant (W2A16) demonstrate that OneBits achieves superior performance in common sense reasoning tasks.

Strengths

- The paper proposes a final objective loss function that combines the final cross-entropy loss with the reconstruction error of each layer using a Quantization-aware Knowledge Transfer method. The effectiveness of incorporating the reconstruction error is demonstrated through an ablation study (Table 6). - Unlike the traditional 1-bit linear design in BitNet, the authors introduce a new 1-bit binary quantization linear design that includes scaling factors (g/h) for each input/output channel of the weight matrix. They also propose an initialization method from a pretrained model using Sign-Value Independent Decomposition (SVID). - To initialize the scaling factors (g/h) for each input/output channel, the paper explores various 1-rank decomposition methods for value in SVID, including Singular Value Decomposition (SVD) and Non-negative Matrix Factorization (NMF). Experimental results indicate that the 1-rank decomposition of value using NMF is more effective than SVD.

Weaknesses

- While the paper demonstrates zero-shot performance in terms of PPL and CSR, it lacks experiments on how the same model maintains performance in few-shot scenarios, such as the MMLU benchmark. - If the quality of this generated data is poor, it could negatively impact the performance of the OneBit LLM. The paper does not clearly explain why self-generated data was used instead of public datasets like C4. - The analysis of OneBit LLM's benefits in terms of inference latency and throughput relative to accuracy is insufficient. A detailed examination of these metrics would provide a more comprehensive understanding of the advantages of using OneBit LLM.

Questions

- While the OneBit method has demonstrated independent evaluation of zero-shot and few-shot performance, showing effectiveness compared to LLM-QAT and OmniQuant, it does not provide evidence of a single OneBit model performing well in both zero-shot and few-shot scenarios simultaneously. It would be valuable to present combined performance results, such as including MMLU results in Table 2 for a comprehensive comparison. - When comparing the quality of output generated by OneBit LLM to other models using metrics like AlpacaEval, what trends or patterns emerge regarding the quality of generated data?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Since OneBit LLM is applied only to weights, this method is likely to be effective in improving latency and throughput, particularly in scenarios involving small batch sizes during the generation phase. In this paper, activation quantization has not been considered, and further research in this area is necessary to optimize performance.

Summary

This paper proposes OneBit, which quantizes the LLM weight matrices to 1-bit and achieves good performance and improved convergence speed by using two additional vectors with FP16 per one linear layer.

Strengths

1. This paper is generally well-written and easy to follow. 2. The memory required for the model part is less than other methods. 3. Their methods nicely outperforms other methods in many tasks.

Weaknesses

1. It will be great if they compare with other one-bit based quantization methods such as BitNet. 2. I recommend authors to add empirical results for larger models like Llama-70b.

Questions

1. Would it be possible to compare the inference speed of this method and other methods? I am curious about the potential delay in inference caused by using additional FP16 vectors.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Invitation to Participate in the Discussion Period

Thank you very much for your review. We have provided detailed responses to your question. If you could participate in the discussion period, we would be very grateful again.

Response from Reviewer gg38

Thank you for your considerate response. While most of my concerns have been addressed, I still believe it is important to examine how the gap between Zero-shot and Few-shot performance changes before and after applying OneBits to the public LLM models in Table 2. When performing QAT from scratch, as with OneBits-7B, I believe that training with at least a similar number of tokens to what is suggested by the Chinchilla-optimal is necessary to observe a reliable trend. However, considering that Figures 3(a) and 3(b) show that OneBit-7B achieves performance comparable to the 1B-scale model, which was trained with a larger amount of data, and that OneBit demonstrates a meaningful performance improvement over other quantization methods on public LLMs, I have decided to raise my score from 4 to 5.