Summary
The authors present CA-LoRA, a compression-aware parameter-efficient finetuning method based on LoRA. They train LoRA modules on an uncompressed LLM and then adapt them downstream to be used with a compressed version of the same LLM, with a relatively small number of additional parameters introduced in addition to the LoRA modules. They explore two model families -- T5 and Llama 2 -- across both SuperGLUE and instruction tuning tasks, and see noticeable improvements in nearly all scenarios they tested.
The experiments are reasonable, the idea is straightforward yet novel, and motivation is sound. The significance of this work lies in their stated motivation: enabling strong, static (or rarely updated) task-specific models to be deployed in resource-constrained environments, such as mobile devices. As far as I am aware, this is a novel method for adapting compressed language models to downstream tasks. They are also clear about limitations: this method requires extra compute to adapt the full sized model before being compressed. Whether this tradeoff is worth it depends on the specific training and deployment scenario.
The structure of the paper is good, but the clarity could be improved. For example, I am not sure why the extra parameters introduced are called “knowledge recovery modules” – based on my reading, they are essentially extra parameters (+ a nonlinear function) that allow the compressed model + LoRA weights extra freedom during finetuning, but they do not explicitly encode any knowledge from the original LLM. Additionally, I believe that many of the equations written are not vital to understanding the work, and replacing them with prose and/or diagrams would improve the overall readability of the paper.
Reasons to accept
This paper demonstrates an interesting and useful strategy for building small but performant task-specific LLMs. They show applicability to real world use cases, and ground their work in an important motivation: deploying strong, task-specific models in resource constrained environments. Additionally, their method is theoretically straightforward, and adds little latency to inference. Overall, the method described seems like a very useful way to adapt compressed LLMs in certain scenarios.
Reasons to reject
The first experiments focus on older models and tasks, which are less relevant for real world scenarios as described in the motivation. SuperGLUE tasks are fairly outdated, and I would prefer to see the instruction tuning experiments expanded and highlighted, because those results are also promising.
The paper could also be written more clearly. The amount of equations is – in my opinion – unnecessary, and the paper would be more straightforward to read if some of these were changed to prose (and/or diagrams) instead.
Questions to authors
While the T5 experiments show value in CA-LoRA for task-specific finetuning, I believe the strength of the method lies in the instruction tuning experiments; I think they’re the most interesting and applicable to real world scenarios.
If compute allows, I have a few suggestions for extending the IFT experiments:
* Adding more evaluation domains, to cover other important properties of instruction-tuned LLMs:
* Safety (XSTest, ToxiGen, etc)
* Instruction-following ability (AlpacaEval 1 and/or 2, IFEval)
* Math (GSM8K)
* If your compute budget allows, I would recommend adding another baseline:
* Llama 2 13B finetuned (with LoRA, most likely), on Alpaca. This would be a fairer baseline vs your method than Llama 2 13B without instruction tuning.
* Try instruction tuning on a domain-specific IFT set, such as CodeAlpaca-20k, and then evaluating on domain-specific evaluations (e.g. for coding, HumanEval+ and MBPP+).
I don’t think any of these are strictly necessary, but I think they’d help strengthen your argument (especially the evaluations I mentioned in the first bullet point).
Additionally, if you have the space, I’d recommend moving Table 6 from the appendix to section 4.
Overall, I really like your work!