FouRA: Fourier Low Rank Adaptation

Summary

This paper presents a PEFT method (mainly for text-to-image tasks) called FouRA. FouRA learns the LoRA projection in the frequency domain. This idea helps solve the problems of data copying and distribution collapse and thus improves the generated image quality. The effectiveness of FouRA is verified on both CV and NLP tasks.

Strengths

1. The paper is well-organized. 2. The introduction of the FouRA method is clear and easy to follow. 3. Among the issues that this article focuses on, adaptive rank selection is an important issue in the field of PEFT.

Weaknesses

First, ranking has nothing to do with importance. 1. Efficiency is an important property that PEFT methods should have. Compared to LoRA, FouRA introduces additional computational operations, where multiple 1D-DCT transform will be involved in Eq.1. For every token, FouRA requires to perform this transform. Will these 1D-DCTs take too much time? Can the authors show the time required for **each training epoch**? 2. Also, for GPU cost, can the authors please report GPU peak memory required **during fine-tuning**? Comparison of both time and GPU cost can follow two fair settings. (a) FouRA and LoRA achieve similar accuracy, (b) FouRA and LoRA have (roughly) the same rank. 3. In my opinion, except for the methods section, the paper is not very easy to follow. The main reason is that the authors attempt to claim too many arguments, but not all problems are fully analyzed and solved. Generally, the three core points of this article are (a) the Fourier low rank adaptation, (b) adapter rank selection strategy and (c) enabling flexible mixure of multiple adapters. The following 4, 5, 6 are my questions about these three points. 4. ***About Fourier.*** First of all, I don't quite understand how to get $\Delta W_{foura}$ from Eq.5, can the authors provide a derivation? For Lemma 4.1, my understanding is that $\Delta W_1$ and $\Delta W_2$ are actually two potential fitting targets, and you can judge the error of LoRA's r-rank approximation by their eigenvalue distributions. In short, these two variables are approximate targets rather than the fine-tuning results of LoRA or FouRA. So what is the specific meaning of the eigenvalues ​​calculated in Figure 4? There should be a simpler and less confusing way to verify the error of the FouRA approximation, such as fitting a random target matrix (see Figure 6 in the Vera[1] paper) or designing some simple classification tasks (see Figure 7 in the FourierFT[2] paper) with your method . 5. ***Adaptive Rank Selection.*** I reserve my opinion on flexibility. Increasing flexibility may not always lead to high generalization, and may even make convergence difficult (the author could provide a comparison of the convergence speed of fine-tuning with and without the gating module). The claim about flexibility (i.e., input dependent selection) is too strong without evidence or reasonable intuitions. On the contrary, the data-agnostic selection paradigm is probably more concise and elegant because we do not need to learn selection strategies on new data sets. If the authors insist on making this claim, then they can show sufficient experimental results, such as data-dependent selection is better than selection that relies only on the model. In addition, it seems that no ablation study results on rank selection were found. 6. ***Multiple Adapters.*** Not sure what the purpose of section 3.5 is. I understand that the PEFT method has many scenarios and therefore has many excellent properties that should be met. However, it seems that many important metrics are not evaluated, such as efficiency, the number of trainable parameters, and the storage memory occupied by the adapter, etc. 7. It is recommended that authors focus their writing on the text-to-image task. Although there are experimental results on GLUE, this does not seem to be sufficient to verify that FouRA is a general PEFT method. If the main claim of the FouRA paper is to propose a PEFT method for text-to-image generation, I personally believe it will be more readable and the contribution will be more prominent. [1] VERA: VECTOR-BASED RANDOM MATRIX ADAPTATION. ICLR 2024. [2] Parameter-Efficient Fine-Tuning with Discrete Fourier Transform. ICML 2024.

Questions

In Line 248, is "RoBERTA-Base" a typo? The results in Table 3 are more like the performance of using RoBERTa-Large. Moreover, can the author provide the code or demo only for reproducing the result (70.6) on the CoLA dataset? It would be cool if one could reproduce this result with FouRA regardless of whether you use base or large RoBERTa models.

Rating

4

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

None.

Summary

This paper address a fundemental diversity limitation of any LoRA fine-tuned diffusion model. More specifically, we can observe distribution collapse with these fine-tuned models in the setting of limited data. The authors propose to address this problem by applying LoRA in the frequency domain. The fourier transform provides a disentangled orthogonal basis, which is a more suitable space for low-rank adaptation, especially in the diffusion setting.

Strengths

The method is very well motivated and directly addresses a main limitation of LoRA in the generative setting. There are many qualitative and quantitative ablations showing the superiority of FouRA v.s. vanilla LoRA. The disentangled low-rank space is clearly very effective for concept sliders. It is very interesting to see FouRA does not degrade in performance when generalised to language tasks too. The authors also average over 3 seeds for these runs.

Weaknesses

My only concern is with the additional memory overheads induced by having to perform the forward and inverse fourier transform. The primary practical interest of LoRA is to make fine-tuning large models possible on lower-grade GPUs, with lower memory. To me, parameter efficiency alone is more of a theoretical interest. I can see the authors have shown that the training time is not much higher than vanilla LoRA, however I would like to see the memory overhead and how this scales with the batch size. small points/spelling: The main contribution and focus of this paper is on diffusion/generative models. Although the authors do show generality to discriminative tasks, I think it may make more sense to have "diffusion models" or some variant in the title. L77 "denosing" L113 "gradined" L699 "Computaional"

Questions

L888: where are these numbers coming from (1.15, 8.0, 2.3, 4.15)? are the results very sensitive to these parameters and are they used for all experiments presented here?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

It would have been nice to see a commitment to open sourcing an official implementation, but other than this yes, the authors have adequately addressed all the impacts and limitations of their work.

Summary

The authors propose FouRA, a novel low-rank adaptation for pretrained diffusion models that can successfully handle data copying and distribution collapse problems observed in previous works. FouRA performs low-rank adaptation in the frequency domain and incorporates input-dependent adaptive rank selection during inference by the help of learnable gating function. The authors show FouRA learns decorrelated projections which is effective when merging multiple concepts of adapters. The paper demonstrates the superiority of FouRA through extensive experiments and analysis.

Strengths

1. The proposed FouRA, which applies low-rank adaptation in the frequency domain with input-dependent rank selection, is well-motivated and novel. 2. The proposed FouRA-trained multiple adapters can be combined without further training and produce better-quality images than LoRA adapters. 3. The authors support their claim thoroughly with extensive experiments and analysis throughout the paper, which makes their work solid. The experimental results are convincing.

Weaknesses

1. One favorable property of LoRA is that it can be merged into the pretrained weights, due to its linearity. If my understanding is correct, the proposed FouRA cannot be merged with the base models’ weights due to the intermediate gating function, which will consequently increase the latency of the model. Authors only provide training time in computational analysis in appendix, and I am curious how FouRA would affect the overall inference time. 2. It seems the ablation study of each component of FouRA is missing. Further study would help readers to understand how each component affects the performance of FouRA. Also, direct comparison between FouRA and FouRA with fixed dynamic rank would further highlight the efficacy of proposed adaptive rank gating method.

Questions

Please see the weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

I do not see any serious societal impact in this submission.

Summary

This paper proposes a new parameter-efficient fine-tuning method that operates in the frequency domain, termed FouRA. Specifically, The method operates in the frequency domain, learning low-rank adapter transforms to Fourier-transformed input features. It also incorporates an adaptive rank selection strategy that can vary during both training and inference. The authors provide theoretical analysis and extensive experimental results across multiple tasks, demonstrating FouRA's effectiveness in text-to-image generation, concept editing, and language understanding.

Strengths

- Overall, this paper is well-written. All the contents are organized properly. The proposed method is described clearly with details. - The idea of operating in the frequency domain is novel. It provides a reasonable way to interpret the learned LoRAs and to control the generated images. - The authors provide theoretical analysis and proofs for their claims, including lemmas on singular value decomposition and sparsity. - The paper includes pretty comprehensive experimental results on multiple tasks, including text-to-image generation, image editing, and language understanding. They compare FouRA to existing methods like LoRA and provide both quantitative and qualitative results.

Weaknesses

- While the qualitative results are appealing, it could be great to include more quantitative evaluation and more baselines. The improvement in GLUE tasks is not that significant. - The paper does not provide a detailed analysis of the computational overhead of FouRA compared to LoRA. While there is a brief mention in the appendix, a more thorough discussion would be beneficial. - Minor: Please consider enlarging the fonts in Figure 3.

Questions

- The proposed method effectively addresses LoRA's "data-copying" phenomenon. I am wondering whether this "data-copying" effect is caused by the overfitting of LoRAs and can also be eliminated by early stopping.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

There is no societal negative impact.