Summary
The authors present PiSSA, a relatively simple change to the LoRA framework leading to a large amount of demonstrated benefits. PiSSA proceeds by adjusting the initialization step of standard LoRA; rather than freezing the original weight matrix W and learning a low-rank perturbation W' = W + BA, PiSSA instead considers the SVD W = USV^T = U[:,r:]S[r:,r:]V[:,r:]^T + U[:,:r]S[:r,:r]V[:,:r]^T = W^{res} + AB, s.t. A=U[:,:r]S^{0.5}[:r,:r], B = S^{0.5}[:r,:r]V[:,:r]^T. Due to this backwards compatibility, PiSSA enjoys the fine-tuning speed/memory savings of LoRA while also being easily implementable to existing PEFT packages. Most importantly, the authors demonstrate across a large number of models, fine-tuning recipes, and benchmarks that PiSSA greatly improves on downstream accuracy compared to LoRA. Furthermore, the authors show that this initialization + fine-tuning strategy also leads to significant decreases in quantization error, outperforming both QLoRA and the recently proposed LoftQ both in terms of training loss (e.g., faster loss reduction) and performance for GSM8K and MATH tasks.
Strengths
**Originality** - the specific algorithm is an original, lightweight alteration to the LoRA framework. In particular, the fact that there are no additional hyperparameters and, after the initialization phase, PiSSA is essentially swappable with LoRA implementations are all significant benefits of the method. Furthermore, PiSSA's use of only fine-tuning the principle components in a LoRA fashion lends itself to strong intuition as to why the method leads to such strong downstream fine-tuning performance, and why LoRA is less successful in comparison. The latter point is also true when considering quantization.
**Quality** - the extensive experiments across models and fine-tuning evaluations well convey the benefits of this method over standard LoRA. This is also true of the extensive quantization experiments.
**Clarity** - for the most part, the paper is well written. However, there are specific points where the paper could improve clarity (see weaknesses below).
**Significance** - the performance benefits and backwards compatibility lead to significant opportunity for future impact of this work.
Weaknesses
# Clarity
There are several areas the paper could improve in clarify. In particular, the literature review requires more granularity, e.g., "AdaLoRA [42, 41, 43]," only one of these works is AdaLoRA. Furthermore, there is overlap between AdaLoRA and PiSSA in both their reliance on the SVD. It is necessary to discuss exactly how AdaLoRA leverages SVDs, and how this differs from PiSSA.
There are several experimental details missing, e.g., are the GSM8K evals 8-shot? What are the shots used for the other benchmarks? Were the benchmarks run using the Eleuther eval harness? Please include these details in the paper.
Paper at times states it is similar to LoRA and, at other times, seeks to differentiate itself:
- "Unlike LoRA and its successors, which focus on learning low-rank approximations of weight updates, our PiSSA approach directly tunes the essential but low-rank parts of the model while keeping the noisier, high-rank, and nonessential parts frozen. Although our approach differs in philosophy from LoRA, it shares most of LoRA’s structural benefits and can be extended by these methods to enhance its performance"
- "Since PiSSA shares the identical architecture with LoRA, it inherits most of LoRA’s benefits."
This was confusing on an initial pass of the paper. A short, succinct list enumerating the similarities and differences early on may help readers understand exactly how PiSSA distinguishes itself from LoRA, but how it is able to inherit all of LoRA's PEFT benefits.
# Quality
The authors consider the weight matrix of a single attention layer in Llama-2 in Figure 2:
"Since the residual model has removed the large-singular-value components, Wres has a narrower
distribution than that of W, as can be seen in Figures 3a and 3b (comparing the singular value distributions of W and Wres), as well as Figures 3c and 3f (comparing the value distributions of W and Wres), which is beneficial for reducing the quantization error" <- However, since this only looks at a single layer of a single model, it hard to accept this as evidence for a general claim. Can you include a comparison of the distributions of mu/sigma for W and W^{res} (e.g., mean absolute differences between means and ratio between sigmas) across all layers for Llama2 and another model (e.g., Mistral) to show that, in general, W^{res} has a narrow distribution than W?
Questions
See weaknesses.
Minor comment:
"loftQ" in Table 3 <- "LoftQ"
Limitations
The authors have addressed potential limitations of the demonstrated method.