Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization

Summary

This paper presents a method PEQA that combines quantization and parameter-efficient fine-tuning. It takes both advantages, including updating only a tiny fraction of model weights and saving the memory by quantization. The model weights will be decomposed into a matrix of low-bit integers and a scalar vector. During the fine-tuning, only the scalar vector will be updated. The paper has done comprehensive experiments to measure its effectiveness. The proposed method outperforms PTQ and has an acceptable degradation compared to LoRA, but which significantly saved the memory.

Strengths

1. The paper is well-written and easy-to-follow. 2. The idea and the method are clean. 3. The evaluation is comprehensive.

Weaknesses

1. The degradation of the accuracy of the proposed method compared to LoRA is non-negligible. 2. The paper combines two existing techniques rather than inventing a new technique.

Questions

1. Table 2: Why does PEQA have an even lower perplexity in some cases than QAT? 2. How does PEQA compare to the direct combination of LoRA and quantization (e.x. compress to 4-bit, then LoRA on the 4-bit matrices)? 3. What is the intuition behind the decomposition used in PEQA? 4. Table 4: Why PEQA degrades the performance of Five-Shot?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

2 fair

Limitations

1. The degradation of the accuracy of the proposed method compared to LoRA is non-negligible.

Summary

This paper introduces a new framework, PEQA, for efficiently tuning Large Language Models (LLMs). PEQA uniquely fine-tunes the scale parameters during the instructing tuning phase while keeping the backbone parameters frozen. This approach allows PEQA models to maintain advantages during both the training and inference phases. The primary experiments are performed on the Alpaca datasets. The paper also compares PEQA's performance on various datasets with other efficient tuning methods, such as Lora.

Strengths

The paper is strongly motivated, emphasizing the importance of memory efficiency in tuning methods. The authors highlight the role of quantization methods in reducing memory usage beyond what is achievable with parameter efficiency methods alone [1]. The paper is well-written and easy to follow. [1]. https://github.com/huggingface/peft

Weaknesses

Table 1 could be improved by emphasizing that other parameter-efficient tuning methods can also apply post-training quantization methods, such as GPTQ and AWQ. The baseline should include the post-training quantization methods applied before or after fine-tuning (e.g., Lora and Adapter), and report on the memory usage during inference and training. This is particularly relevant as the HuggingFace PEFT library has incorporated Int8 quantization. Reference: [1] QLoRA: Efficient Fine-tuning of Quantized LLMs

Questions

See Weaknesses

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

See Weaknesses

Summary

The paper presents a novel approach named Parameter-Efficient and Quantization-aware Adaptation (PEQA) to address the challenges of efficiently fine-tuning and deploying large language models (LLMs). The paper demonstrates PEQA's effectiveness and scalability through extensive experiments, comparing it with competitive baselines across a range of tasks from natural language understanding to generation benchmarks.

Strengths

* The paper centers on the topic of efficient model compression and fine-tuning, framing its approach through a unified method referred to as PEQA. * The authors' proposed method offers benefits, most noticeably in substantially reducing the memory usage associated with optimizer state saving and the final model size. * The authors have undertaken a comprehensive evaluation setup which encompasses several benchmark tasks and a variety of model sizes.

Weaknesses

* The overall presentation of the paper could be improved for clarity. Specifically, the meaning of statements such as "Notice that s0 and z0 have nothing to do with a choice of downstream task" on Line 148, and the term "integer quantization indices" on Line 149, are not clear. * The calculation of memory usage during fine-tuning is not sufficiently explained. While it's understood that the proposed method lowers optimizer state storage by only updating scaling factors, the paper does not address memory usage during gradient calculation, a crucial aspect that often dictates peak memory usage during optimization. * It's unclear why the choice of only updating scaling factors is advantageous. The paper only compares the proposed method with LoRA, limiting the assessment of its effectiveness. For instance, the BitFit method (Ben Zaken et al., 2022) demonstrates that fine-tuning the LLM can be achieved by only updating biases. Moreover, the paper does not discuss the potential outcomes of optimizing zero-points only or optimizing both zero-points and the scaling factor.

Questions

* The methodology section of the paper leaves some key details unaddressed. Specifically, it's unclear how the gradient of the scaling factors is computed in the proposed model. The paper PACT by Choi, Jungwook, et al. [1] presents two different ways to compute the gradient for the scaling factor/clamp_range for the quantization function. It would be beneficial for the authors to clarify which method was adopted in their work. [1] Choi, Jungwook, et al. \Pact: Parameterized clipping activation for quantized neural networks.\ arXiv preprint arXiv:1805.06085 (2018).

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

see above.

Summary

This paper introduces a model compression method called Parameter-Efficient Quantization-Aware Adaptation (PEQA) to tackle the size challenge of Large Language Models (LLMs) while enhancing their task-specific fine-tuning. The PEQA technique quantizes the fully-connected layers into quantization scales and integer values for initialization, and during downstream tasks, only the quantization scales are fine-tuned, leaving the integer matrix fixed. This approach significantly reduces the parameters required for gradient updates and the memory for loading pre-trained model weights, making fine-tuning more memory-efficient than other techniques such as Post Training Quantization (PTQ) and Quantization-Aware Training (QAT). However, there are several concerns regarding novelty and effectiveness of the proposed methods.

Strengths

- The paper provides an easily understandable explanation of the existing research trends and exhibits good readability. - By quantizing the pre-trained weights to sub-4bits, the memory cost for LLM fine-tuning has been significantly reduced. This enables a wider range of individuals to fine-tune LLMs according to their desired tasks efficiently. In particular, the existing similar approach, PEFT + PTQ technique, has been studied for models of size smaller than 1.3B. However, PEQA proves to be effective even for models as large as 65B and reduces memory costs. - Through various experimental results, the paper demonstrates the performance of the PEQA approach in fine-tuning and its application of quantization compared with PTQ and QAT results. This effectively shows how well the method works for the purpose of LLM fine-tuning.

Weaknesses

- This paper's significant contribution, fine-tuning only the quantization scales (termed PEQA), seems similar to AlphaTuning [22]. The paper does not thoroughly discuss the limitations of AlphaTuning, only noting its lack of evaluation on large LLMs. - The baseline settings in the evaluation setup are unusual. For instance, the authors claim PEQA matches or exceeds QAT's performance. However, QAT's accuracy appears lower than state-of-the-art methodologies: e.g., LLM-QAT[a] demonstrated that 4-bit weight quantization almost reproduces the full-precision accuracy, but QAT in PEQA (Table2, Table3) shows noticeable degradation (LoRA 10.63 vs QAT 11.07). Such understated baselines could potentially mislead readers regarding PEQA's true performance. [a] LLM-QAT: Data-Free Quantization Aware Training for Large Language Models - The authors state that PEQA maintains generalization capabilities, yet the results are inconsistent. Table 4 indicates PEQA's zero-shot accuracy matches LoRA, but this improvement disappears in five-shot accuracy. Therefore, it is not clear if PEQA really preserves generalization capability or not. - There is little understanding of why fine-tuning only the quantization scales is sufficient. Therefore, it is hard to be convinced that the proposed method would work as expected. For example, Table 5 shows that PEQA even noticeably outperforms LoRA (without quantization) LoRA for zero-shot MMLU (and even LoRA+OPTQ outperforms LoRA for LaMMA-7B), which counters the accuracy trends of "LoRA > PEQA > LoRA+OPTQ" consistently shown in all the other tables. It would be desirable to provide insights into such unexpected evaluation results. - The problem setup in Section 3.1 doesn't align with PEQA's main objectives. The memory-bound issue during text generation inference discussed in this section seems less relevant to PEQA's fine-tuning memory efficiency benefits. Since forward computation isn't performed in the generation style during training, and text generation's memory-bound issue can be resolved by quantization (e.g., LoRA + OPTQ), including this issue in the problem setup may be misleading. - Comparing 3-bit weight quantization with OPTQ is valuable, but considering OPTQ's known limitations at this level of granularity, a fine-grained quantization comparison might be more convincing. Furthermore, a comparison with recent state-of-the-art weight quantization methodologies, such as LoRA + AWQ, could provide a clearer understanding of PEQA's effectiveness.

Questions

- What could be the reason for PEQA outperforming FP fine-tuning with LoRA (+LoRA) as shown in Table 5? Additionally, the performance of applying OPTQ to LoRA actually increases in LLaMA-7B. Could this be interpreted as the LoRA Fine-Tuning approach not being optimally applied in this experiment? - From the Five-Shot results in Table 4, we can observe that the performance of PEQA on LLaMA-7B actually decreases, or its fine-tuning effect is insufficient compared to LoRA. Could this result be interpreted as the In-Context Learning ability of LLM not being well-preserved in PEQA Fine-Tuning? - How does the number of learnable parameters or model size change with a decrease in group size in Table 8? Only perplexity is provided, making it difficult to observe the trade-off together.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors did not explain the limitations of this paper.

Thanks for the response

Thanks for the response. My questions are well addressed. I raise my score accordingly.

Thank you for the responses.

I appreciate the authors for the detailed responses and additional experimental results. Although some of the answers do not sufficiently answer the questions, I believe the additional experimental results, as well as the in-depth discussion, would be valuable for future research in this field. Therefore, I raise my original rating.