The Impact of Initialization on LoRA Finetuning Dynamics

Summary

The paper investigates the impact on training dynamics of two initialization schemes for LoRA. For this purpose the authors investigate the asymptotic behaviour of activations and weights for LoRA adapters. The authors find that init[A] where A is intialized randomly and B is initalized with zeros leads to more efficient learning dynamics but leads to some numerical instabilities. This initialzation method is the one most commonly used for LoRA-style methods. Further, the authors found that init[B] leads to suboptimal learning dynamics but is less prone to numerical instabilities. Finally, init[A] performs better with larger learning rates than init[B], which is supported by experiments on NLU as well as NLG finetuning tasks.

Strengths

The paper is well motivated and presents theoretical analysis accompanied by empirical experiments. Claims are supported by theoretical analysis as well as relevant experiments. Furthermore experiment hyperparameters as well as hardware requirements are well documented.

Weaknesses

**Novelty** Most of the theoretical Analysis of the LoRA finetuning dynamics in this paper have already been presented in [1]. The authors should have made it more clear how their work differs from the Analysis done in [1], especially since most formulas and notation is identical. Further, both of the methods evaluated in this paper have also been investigated in [1]. The paper would greatly benefit from a clear explanation on how the analysis in this work differs from results published in [1]. Finally, it is not clear if the contribution of individual LoRA layers to feature learning can be derived from a setting where only a single LoRA layer is trainable while all others are frozen (as claimed in chapter 3.1). **Experimental results** Since the main gist of the paper is to investigate the learning dynamics of LoRA, it would be good to also investigate different initialization schemes aside from init[A] and init [B] prevalent in the litearture, e.g. gaussian init [2], kaiming init [3], principal components [4], etc. In [1] only different learning rates for A and B have been investigated. **Significance of results** While the main finding that init[A] has a better optimal performance is interesting, this initialization scheme is already the common initialization scheme used by LoRA. Therefore, the impact of the main findings is marginal. [1] Hayou et al., LoRA+: Efficient low rank adaptation of large models., ICML 2024 [2] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022 [3] He et al., & Sun, J. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification., ICCV 2015 [4] Meng et al., PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models., arXiv 2024

Questions

- Did the authors investigate other commonly used initialization schemes? - Will the authors provide code that can be used to reproduce the results in the experiments section?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have properly addressed limitations.

Summary

This paper investigates the impact of initialization schemes on the finetuning dynamics of Low Rank Adaptation (LoRA) in large language models. The authors compare two initialization approaches: Init[A], where A is initialized randomly and B to zero, versus Init[B], where B is initialized randomly and A to zero. Through theoretical analysis and empirical experiments, they demonstrate that Init[A] allows for larger learning rates and leads to more efficient feature learning compared to Init[B], albeit at the cost of some internal instability. The paper provides a rigorous theoretical framework based on large width limits and supports the findings with experiments on toy models and real-world language tasks.

Strengths

S1: The paper addresses an important and understudied aspect of an usually handwaved-away aspect of LoRA, namely the impact of initialization schemes. This is a relevant topic given the widespread use of LoRA for efficient adaptation of large language models. S2: The theoretical analysis is rigorous and well-grounded, using asymptotic analysis in the large width limit to derive principled insights about the dynamics of LoRA finetuning under different initialization schemes. S3: The empirical results on both toy models and real-world language tasks provide strong support for the theoretical findings, demonstrating the practical relevance of the initialization choice. S4: The paper presents a clear trade-off between feature learning efficiency and internal stability, providing nuanced insights that go beyond simply recommending one initialization over the other. S5: I would like to commend the authors for putting great amount of work on both the main document as well as the appendix. Overall not only does the paper read well and seems theoretically and experimentally sound, but the manuscript includes an excellent degree of experimental details, and it's thus likely that future work will be able to cleanly build on it.

Weaknesses

W1: While the theoretical analysis is sound, the paper could benefit from a more intuitive explanation of why Init[A] allows for larger learning rates and more efficient feature learning. This would make the insights more accessible to some practitioners. W2: The experiments focus primarily on language models and NLP tasks. This is definitely acceptable and overall a very sound choice, however the inclusion of experiments from other domains (e.g., finetuning an image classification model, or some other architecture that relates to a vision transformer) would strengthen the experimental section significantly, and increase the overall impact of the paper in the broader NeurIPS community.

Questions

Q1: How sensitive are the results to the choice of LoRA rank? Do the theoretical predictions and empirical findings hold across different rank values? Q2: The paper focuses on vanilla LoRA. How do the authors expect these initialization schemes to interact with variants like QLoRA or LoRA+? Would the trade-offs and recommendations change? Q3: Given the internal instability observed with Init[A], are there any potential negative consequences for downstream task performance or generalization ability? How might this instability manifest in practice (and would there be a difference depending on input domain and/or other parameters of the model + problem space)? Q4: The theoretical analysis seems to assume a single LoRA layer, for simplicity. How well do the insights generalize to more realistic scenarios with multiple LoRA layers throughout a network?

Rating

8

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

L1: The theoretical analysis seems to rely on assumptions about the asymptotic behavior in the large width limit. The paper could benefit from a more detailed discussion of how these assumptions may or may not hold in practical scenarios with finite-width networks. L2: The empirical evaluation, while comprehensive, is limited to a subset of NLP tasks and model architectures. Expanding the evaluation to a broader range of tasks, model sizes, and architectures would strengthen the generalizability of the findings.

Summary

The paper analyzes the impact of different initialization techniques for A and B matrices in LoRA adapters. Typically, either A or B matrix is initialized with zero while the other is initialized form a Gaussian distribution. This is done so that fine-tuning starts with LoRA adapters ($A \times B = 0$) having zero effect. However, the paper argues that it matters which of A and B initialize to 0. As per the results, $B = 0$ init allows higher optimal learning rates which leads to better overall performance. However, there is still instability during training which could be mitigated with a method like LoRA+.

Strengths

It's useful to know that the default initialization scheme in LoRA happens to be the better alternative. The experiments are convincing.

Weaknesses

Perhaps I am not the right person to review this paper, but I don't fully understand the impact of this paper. It shows that there are two ways of initializing AB matrices and the default one that we have been using until now is the better option. Additionally, it says that the default initialization is not good enough but doesn't propose anything new. Maybe the theory part is important, but I couldn't understand it fully in the limited time I had. Having said that, I still feel that the paper has something worthwhile to contribute to the community. So, I vote to accept it for now.

Questions

None.

Rating

6

Confidence

2

Soundness

2

Presentation

2

Contribution

2

Limitations

The limitations section in the paper is adequate.

Summary

Finetuning large language models has become a common technique among practitioners, however due to the memory footprint of practically viable LLMs, there is a dire need of techniques that allow memory-lightweight finetuning (PEFT). Among those techniques LoRA has gained immense popularity. In LoRA a dense matrix is modified via addition of a low-rank update consisting of matrices A and B. In order for this update to initially conserve the model’s outputs while preserving the ability to learn, exactly one of the matrices needs to be initialized to zeros. “The Impact of Initialization on the Finetuning Dynamics in LoRA” proposes that the B matrix should be initialized with zeros. It hypothesizes that the reason for that is that this initialization (Init[A]) allows using greater learning rate without sacrificing stability. The main points of the paper are supported by theoretical derivations stemming from the perspective of infinite-width networks. Furthermore, experimental evaluations serve as proof of the soundness of the theoretical approach and as practical justification of the method.

Strengths

- Mathematical support combined with experimental results - Potentially significant due to how widespread LoRA adoption is and due to how easy the technique is to implement - The paper is quite original - even though LoRA is a popular technique and it is a fundamental question whether the A or the B matrix should be initialized with zeros, this work seems to provide the first attempt at answering it.

Weaknesses

- There is virtually no discussion of why the training hyperparameters are set to those exact values, in particular with regard to the number of epochs and weight decay = 0. I suspect changing those HPs might mitigate the need to choose the initialization schema so carefully. Also the r parameter seems to be chosen arbitrarily. - The gains provided by the method seem to be small and would be better visualized if they were put in the perspective of comparison with other methods such as differing r or even, if possible, full finetuning. - The Toy Model setup is not explained clearly.

Questions

- There are other works that try to improve LoRA’s performance (LoRA+) or are inspired by it (VeRA, AdaLoRA, etc.). Do you believe your work could be used with those techniques? - Do you feel that there is some simple, informal intuition suggesting why the Init[A] outperforms Init[B]? - AdamW seems to be among the most popular optimizers, however the paper focuses on rigorous derivation in case of SignSGD claiming that the result should extend to the typical choice of the Adam optimizer. Even though the authors claim they are using the AdamW optimizer in the experiments, they set weight decay = 0, effectively replacing AdamW with Adam. Would the method perform as well while using weight decay? Would it be justified theoretically the same way Adam is? - Can you justify the fixed hyperparameters’ values? - Can you explain the toy model setup more clearly? As I understand it now, the same model is being adapted and is used to create the dataset which does not seem to make sense.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

- I do not believe the authors have sufficiently explained the limitations of the fixed hyperparameters in the evaluations