Discussion about the value of knowledge-preserved mode and answers to remaining questions
> (2) The knowledge-preserved mode is not very compelling given this method will hurt the target task performance. I don’t know why the paper emphasizes this mode in parallel with the instruction-previewed adaptation mode. [1] may be a pointer for reference as it also studies how to preserve the forgetting of general knowledge in LMs when continual training them.
**Please note that the knowledge-preserved adaptation does NOT hurt the target task performance**. As shown in Table 1, compared with LoRA, our method has better performance than LoRA in both worlds (world knowledge benchmarks and downstream tasks) in most cases. We **quote some results of LoRA and CorDA in Table 1 here** for your reference.
|Method|Trivia QA | NQ open | WebQS | GSM8k | Math | Avg. |
|---|---|---|---|---|---|---|
|LoRA|44.17 | 1.91 |6.64| 42.68| 5.92| 20.26|
|CorDA | 44.30 | 9.36 | 7.14 | 44.58 | 6.92 | 22.46 |
|Method|Trivia QA | NQ open | WebQS | MTBench | Avg.|
|---|---|---|---|---|---|
|LoRA|47.46 |10.28| 7.73| 4.60| 17.52|
|CorDA | 50.34 |14.43| 8.17| 5.05| 19.50|
More importantly, as we respond above, existing PEFT methods rarely consider or support finetuning with knowledge better perserved. There are some studies on the continual training of LLMs [1] and [14, 21] (ref. in our paper), but they are not PEFT methods. We will cite [1] in the revised version of our paper.
As shown in Table 1, when using the average performance of the both worlds to measure the comprehensive ability, CorDA in knowledge-preserved adaptation achieves the best results among all the compared methods in all the three tasks. **Therefore, we respectfully do not agree with your comment that "knowledge-preserved adaptation is not compelling" and "it hurts the target task performance".**
Admittedly, CorDA in knowledge-preserved adaptation is not stronger than the instruction-previewed adaptation when only evaluating the downstream task performance. But please note that maintaining pre-trained knowledge better and pursuing a better finetuning performance is inherently a tradeoff (also mentioned in [14] and [21]). A method may be better than another one in both worlds, just like CorDA v.s. LoRA in the results above, but for the method itself, the two worlds are still a tradeoff. We would like to use an analogy to support. The advanced neural architecture (_e.g._ Transformer) may be better than a traditional MLP/CNN architecture in terms of both accuracy and parameter efficiency. But for the better architecture itself, higher accuracy still brings more parameters (BERT_large is better than BERT_base with more parameters).
Therefore, when knowledge maintenance is not a concern, we introduce our instruction-previewed adaptation, which puts all its efforts for the downstream task, surpassing the competitive studies DoRA and PiSSA in the three finetuning tasks, Math, Code, and instruction following, as shown in Table 2.
In conclusion, the knowledge-preserved adaptation and instruction-previewed adaptation highlight the comprehensive performance and the specialized ability of downstream task, respectively. We think this is also a feature of our PEFT method, allowing for customized selectivity based on the actual need.
That is why we emphasize the two modes in parallel.
[1] Continual Pre-training of Language Models, Ke et al., ICLR 2023
> (3) Questions: Does the major difference between CorDA and LoRA lie in the initialization? Have you compared with other PEFT methods that focuses on initialization technique? Currently, the baseline comparison is not very comprehensive.
Yes, CorDA brings task context into the LoRA adapter initialization.
Adopting the same LoRA structure not only facilitates fair comparison, but also enables to restore the orignal LLM architecture after finetuning without architectural change or introducing inference burden.
Yes, we have compared with PiSSA, which also focues on the LoRA adapter initialization but does not consider task context.
DoRA builds the adapter with a normalization and a learnable magnitude, and also does not consider task context.
It is noteworthy that both DoRA (ICML 24) and PiSSA (released on Arxiv in April 2024) are recent studies and are strong baselines. Besides, full parameter finetuning is the most direct reference because it usually has the best finetuning performance without considering parameter efficiency.
For downstream tasks, as shown in Table 2 and Table 3, our method achieves finetuning performances on par with full parameter finetuning, and better performances than the compared PEFT methods LoRA, DoRA, and PiSSA.
For comprehensive ability (with knowledge benchmarks included), as shown in Table 1, our method has the best average performance among full finetuning and the PEFT methods.
Therefore, our experimental results are already able to demonstrate the effectiveness of our proposed method.