Joint Response
Here we respond to questions raised by multiple reviewers in more detail.
### Theoretical framework / expressivity of ELoRA (Bg6K, iJDt, rHtj)
Similar to the original LoRA paper, our approach in this paper is predominantly empirical, focusing on demonstrating the practical effectiveness and efficiency of ELoRA. While a comprehensive theoretical analysis of ELoRA is not provided, we do draw upon existing theory and evidence supporting the use of random matrices in similar contexts: random matrices being almost always full-rank (rank r<N matrices form a Lebesque measure zero set in the set of NxN matrices) and the Johnson–Lindenstrauss lemma which shows that random matrix projections approximately keep low-dimensional structures [1] and that the effective dimensionality of LLMs activations is very low [2].
Another perspective is that the random matrices in our method are just a convenient and efficient way to span part of the hyperspace of possible LoRA updates. Theory-wise, our method works the same way as LoRA, but we obtain more flexibility with our 'mixture-of-random-vectors' setup. To this end, we've conducted a new analysis on the expressivity of LoRA vs ELoRA on the task of fitting random square matrices.
We find that ELoRA performs as well as LoRA in this task for given number of trainable parameters, yet at the same time giving more flexibility, e.g. by allowing going to much lower parametrisations (below LoRA's rank=1). The resulting figure and analysis will be added in the new appendix. Thank you for this fruitful suggestion!
### Experiments on vision models (JAXk, rHtj)
We conducted additional experiments by adapting Visual Transformers (pretrained on ImageNet-21k), base and large variants, on image classification tasks of CIFAR100, Food101, Flowers102, RESISC45.
For each dataset we train on a subset of 10 samples per class, and evaluate on the full test set (CIFAR100, Food101, Flowers102) or on all the remaining samples (for RESISC45, as it does not have a test set).
We evaluated LoRA and ELoRA methods applied on the query and value layers of ViT, along with two baselines - fully-finetuned model (*Full*), and training the classification head only (*Head*). Similarly to the GLUE benchmark, we use rank 8 for LoRA, and rank 256 for ELoRA (i.e. ELoRA uses <10% of LoRA's parameters). We conducted grid-search of learning rates for all methods and reported results after 10 epochs. The reported parameter count excludes the classification head, which has to be additionally trained in all methods.
**Table 1 Finetuning ViTs on image classification datasets**
|Method | Parameters | CIFAR100 | Food101 | Flowers102 | RESISC45 |
|-------|-----------:|---------:|--------:|--------:|---------:|
| **ViT-B**
| +Head | | 77.7 | 86.1 |98.4| 67.2 |
| +Full | 85.8M |**86.5** |**90.8**|98.9|**78.9** |
| +LoRA | 294.9K | 85.9 | 89.9 |98.8| 77.7 |
| +ELoRA | **24.6K** | 84.8 | 89.0 |**99.0**| 77.0 |
||||
|**ViT-L**|
| +Head | | 79.4 | 76.5 |98.9| 67.8 |
| +Full | 303.3M | 86.8 | 78.7 |98.8|**79.0** |
| +LoRA | 786.4K | 87.0 |**79.5** |99.1| 78.3 |
| +ELoRA |**61.4K** |**87.5** | 79.2 |**99.2**| 78.6 |
From Table 1, we find that ELoRA approaches performance of LoRA on the *Base* model for three datasets (84.8 vs 85.9, 89.0 vs 89.9, 77.0 vs 77.7) and outperforms it for Flowers102 (99.0 vs 98.8), despite using over 10x less trainable parameters. For ViT-*Large*, we even **outperform** LoRA for three datasets: CIFAR100 (87.5 vs 87.0), Flowers102 (99.2 vs 99.1) and RESISC45 (78.6 vs 78.3). Notably, for ViT-L, ELoRA trains only a tiny fraction of 0.002% of the backbone's number of parameters.
These results overall confirm our findings in the NLP domain that ELoRA can match LoRA performance while being far more parameter-efficient. We will add these results to the paper.
**References**
- [1] Lindenstrauss et al. Extensions of Lipschitz mappings into a Hilbert space. 1984
- [2] Aghajanyan et al. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL 2021