Summary
The authors suggest that low-rank updating mechanisms, as employed by LoRA, may limit the ability of large language models (LLMs) to effectively learn and memorize new knowledge, particularly in memory-intensive tasks. To address this limitation, they propose increasing the rank of the update matrix used in LoRA by employing a square matrix with a larger rank, while keeping the total number of trainable parameters constant. To achieve this, the authors introduce non-parametric operations that compress the input and decompress the output, ensuring dimensional compatibility for matrix multiplication. Additionally, they propose a rotation operation to preserve positional encodings during input compression, maintaining compatibility with matrix products.
Strengths
The authors observe that LoRA’s low-rank updates (e.g., using a rank as low as 8) may not fully leverage the model's learning potential. To address this, they propose restructuring LoRA updates by compressing the input dimension, increasing the rank of the update matrix, and then decompressing the output to match the original dimensions. This approach maintains the same number of trainable parameters while increasing the rank, which is an interesting idea. However, while this method enhances learning capabilities in terms of memorization, the performance improvements on downstream tasks do not directly translate as significantly.
Weaknesses
1. Unclear Compression and Decompression Operations: The compression and decompression operations described in equations (8) and following are not clearly explained. The rationale behind choosing these specific operations is unclear and requires further elaboration.
2. Rotation Operation: While the rotation operation is likely intended to preserve positional information after compressing the input, the explanation provided in the paper is not sufficiently clear. The authors need to clarify why rotation is added and how it enhances the matrix's ability to distinguish input information. The current description, especially the statement "rotation information can help the square matrix to distinguish the input information," is vague and needs more technical justification.
3. Performance Discrepancy in Table 1: In Table 1, MoRA with rank 8 achieves 100% accuracy, whereas LoRA with the same rank only achieves 52%. This discrepancy suggests that the LoRA model may have been undertrained or improperly tuned. The paper should address this discrepancy to ensure fair comparisons between methods.
4. Limited Improvement Across Tasks: MoRA shows clear improvements only in continual pre-training tasks, with little or no improvement in other downstream tasks. This suggests that MoRA is a task-specific method, which contradicts the goal of parameter-efficient fine-tuning methods that are expected to generalize across various tasks. This task-specific nature makes it challenging to justify its broad application without further experimentation, which defeats the purpose of an efficient fine-tuning method.
5. Uncertain Advantage of Rotation: The paper mentions that rotation helps the square matrix distinguish input information, but this benefit is not clearly demonstrated, particularly in higher-rank scenarios. At rank 256, the performance differences between the Sharing, Decouple, and Rotation methods are minimal. This raises the question of whether rotation is truly necessary at higher ranks, and the paper should clarify under what conditions (if any) rotation provides a real advantage.
6. For table 5, could you also provide the number of trainable parameters for both the methods under both ranks?
Questions
Please see above.