Spectral Adapter: Fine-Tuning in Spectral Space

$\textbf{Weaknesses 1.}$ There are some typos that need to be corrected. For example, a spelling mistake in the word "Apppendix" in line 32, it should be corrected to "Appendix." Additionally, the word "digged" in line 20 should be corrected to "dug" to use the proper past tense of the verb "dig". $\textbf{Answer to Weaknesses 1.}$ Thanks for pointing out these issues, we've corrected the spelling and the tense of the word in our revision. $\textbf{Weaknesses 2.}$ It is not common to put the whole literature review in Appendix. $\textbf{Answer to Weaknesses 2.}$ The decision of putting literature review into appendix is due to the page limit, we will move it into the main content with more page budget in camera ready version if our work gets accepted. Moreover, since we already involve baseline method explanation and all its relative properties whenever a prior method is compared to, e.g., bottom part of page 6 and middle part of page 8 (with Table 3 summarizing the key features), we believe that postponing literature review into appendix doesn't affect reading and understanding. Also, our introduction part (Section 1) and methodology part (Section 2) already draw connection to most relevant work, the literature review section (Appendix A) includes additional works that are less relevant and is presented for completeness and credit for all related work in related research fields. $\textbf{Weaknesses 3.}$ In line 53, the phrase "orthogonal rotating the top singular vector space." This phrase should be corrected to "orthogonally rotating the top singular vector space" to properly use the adverbial form "orthogonally," which modifies the verb "rotating." $\textbf{Answer to Weaknesses 3.}$ Thanks for the suggestion, we've corrected the phrase in our revision. $\textbf{Limitations.}$ Instead of fine-tuning the singular values of the weights, this paper proposes fine-tuning the singular vectors of the weights. However, the motivation behind this approach is not convincingly presented. For example, The motivation for fine-tuning the singular value vectors is not clearly articulated. As we know, U is an orthogonal matrix representing the left singular vectors, S is a diagonal matrix of singular values, and V is an orthogonal matrix representing the right singular vectors. A critical question that arises is whether U and V maintain their orthogonality after fine-tuning. Orthogonal U and V matrices provide an optimal basis for representing the weight matrices. Losing orthogonality results in a suboptimal basis, which can lead to less efficient representations of the neural network weights. Also, without orthogonality, the interpretability and distribution of the singular values will also be affected. Furthermore, orthogonal matrices are numerically stable and well-conditioned, meaning small changes in the data lead to small changes in the results. Without orthogonality, the resulting matrices may become ill-conditioned, causing numerical instability and issues in the training and optimization algorithms. $\textbf{Answer to Limitations.}$ Thanks for raising this point. In our paper, we propose two spectral fine-tuning paradigms: an additive version and a rotational version. Our Spectral Adapter$^R$ applies a trainable rotation to the top-$r$ left and right singular vectors and preserves the orthogonality. This is achieved by using the differentiable Cayley parameterization. Our Lemma 4.1 proves that Spectral Adapter$^R$ with Cayley parameterization maintains orthogonality of the singular vectors. This follows by parameterizing a skew-symmetric matrix $Q=(A-A^T)/2$ and letting the rotation matrix be $(I+Q)(I-Q)^{-1}$. In Appendix C, we provide a proof that $(I+Q)(I-Q)^{-1}$ is orthogonal. On the other hand, our second version, Spectral Adapter$^A$, drops the orthogonality constraint. However, after fine-tuning, the weights are close to orthogonal since only the top-$r$ gets updated and the gradient updates are of small magnitude. We find in our experiments that both adapters work very well, and outperform low rank adapters. In this version, the reviewer is correct that dropping orthogonality will invalidate the optimality of the SVD decomposition in low-rank matrix approximation problems. However, in fine-tuning foundational models, the requirement of orthogonality of the fine-tuned weights is hard to justify. Meanwhile, we agree with the reviewer that interpretability of the fine-tuned model is improved when we use orthogonal fine-tuned weights. This is provided by our Spectral Adapter$^R$. If one wants to perform interpretability tasks which require singular values with our Spectral Adapter$^A$, just re-SVD the tuned matrix and continue analysis with the new set of basis.

Please let us know whether our replies address all your concerns. If so, could you please kindly consider increasing the score? If not, we are willing to address any other question in more details. Thanks!

$\textbf{Additional Answer to Weaknesses 1.}$ consider the same two-layer ReLU model trained for minimizing squared loss, i.e., $\qquad\qquad\qquad\qquad\qquad \\min_{W^{(1)},W^{(2)}} \\|(XW^{(1)})_+W^{(2)}-y\\|_2^2+\\beta(\\|W^{(1)}\\|_F^2+\\|W^{(2)}\\|_2^2$, where $X\in\mathbb{R}^{n\times d},W^{(1)}\in\mathbb{R}^{d\times m},W^{(2)}\in\mathbb{R}^{m}, y\in\mathbb{R}^n.$ Now we decompose each first-layer neuron $W_j^{(1)}\in\mathbb R^d$ (there are in total $m$ of them) into two parts $W_j^{(1)}=w_{j1}+w_{j2}$ where $w_{j1}$ lies in row space of $X$ and $w_{j2}$ is perpendicular to row space of $X$, i.e., $w_{j1}\in\mathcal R(X)$ and $w_{j2}\perp\mathcal R(X).$ This indicates $Xw_{j2}=0$ and thus $w_{j2}$ has no contribution to reduce the first quadratic loss term above. Since we have a second non-negative weight decay term, which would then result in $w_{j2}=0$ when trained to optimality. Therefore, when minimal loss is achieved, all first-layer neurons should lie in the row space of $X$, i.e., $W_j^{(1)}=w_{j1}\in\mathcal R(X),~\forall j.$ However, it might happen that due to optimization errors, some neurons are not exactly aligned with row space of $X$ and may have small value in the perpendicular direction. For demonstration, consider a toy example where we have two pieces of data, each of dimension three, i.e., $X\in\mathbb R^{2\times 3}$, and takes value $$X=\begin{bmatrix} 1 & 0 & 0 \\\\ 0 & 1 & 0 \end{bmatrix}.$$ Consider the case when we have three neurons, with first and second neurons being in row space of $X$ and third neuron perpendicular to row space of $X$ with a small value: $$ W = \begin{bmatrix} 5 & 0 & 0 \\\\ 0 & 7 & 0 \\\\ 0 & 0 & 0.1 \end{bmatrix}. $$ Then, using torch.svd on $W$ results in $$ U = \begin{bmatrix} 0 & 1 & 0\\\\ 1 & 0 & 0\\\\ 0 & 0 & 1 \end{bmatrix}, S = \begin{bmatrix}7 & 5 & 0.1\end{bmatrix}, V=\begin{bmatrix}0 & 1 & 0\\\\ 1 & 0 & 0 \\\\ 0& 0 & 1 \end{bmatrix}. $$ The top, second, and third spectral decomposition components would be $$\\text{top:} \begin{bmatrix}0 & 1 & 0\end{bmatrix} \begin{bmatrix}7\end{bmatrix} \begin{bmatrix}0\\\\1\\\\0\end{bmatrix}=\begin{bmatrix}0 & 0 & 0\\\\ 0 & 7 & 0\\\\ 0& 0 & 0\end{bmatrix}, \text{second:}\begin{bmatrix}1 & 0 & 0\end{bmatrix}\begin{bmatrix}5\end{bmatrix}\begin{bmatrix}1\\\\0\\\\0\end{bmatrix}=\begin{bmatrix}5 & 0 & 0\\\\ 0 & 0 & 0\\\\ 0& 0 & 0\end{bmatrix}, \text{third:}\begin{bmatrix}0 & 0 & 1\end{bmatrix}\begin{bmatrix}0.1\end{bmatrix}\begin{bmatrix}0\\\\0\\\\1\end{bmatrix}=\begin{bmatrix}0 & 0 & 0\\\\ 0 & 0 & 0\\\\ 0& 0 & 0.1\end{bmatrix}.$$ Therefore, if we focus on tuning the top two directions, we only need to deal with the ones aligned with row space of $X$ and are thus protected against small optimization errors, which we consider to be more robust and we address as "noiseless''. $\textbf{Weaknesses 2.}$ The paper's organization might benefit from some restructuring. Consider moving certain content, such as Lemma 4.1 and Table 3, from the experiments section to the methods section. Additionally, restructuring the experiments section could improve clarity. $\textbf{Answer to Weaknesses 2.}$ Thanks for the suggestions, our current structure follows: methodology (Section 2), theory (Section 3), experiments (Section 4), where the experiment part is splitted further into: language model (Section 4.1), diffusion model with additive spectral adapter (Section 4.2), diffusion model with rotational spectral adapter (Section 4.3), runtime comparison (Section 4.4). We've tried hard on optimizing the layout of the paper and we want to keep methodology part (Section 2) short since we want to demonstrate what are the newly proposed method more clearly and succinctly. We have separate baseline discussions in Section 4.2 (bottom of page 6) and Section 4.3 (middle of page 8, including Table 3) since we are comparing with two different sets of baseline methods, that's why we haven't put Table 3 in the methodology section. The divergence in baselines being compared is because we are studying different characteristics of the fine-tuning model, i.e., we focus on model fusion in Section 4.2 and parameter efficiency in Section 4.3, thus we compare with different prior methods which are more proficient with respect to each of the aspect. Lemma 4.1 is more about methodology, but our primary concern is that it's specific to rotational adapter thus we put it in Section 4.3. We'll consider moving this part to the methodology section or we'll create another separate implementation section to discuss this implementation detail. We thank the reviewer again for bringing up the structure issue which suggests that our current paper layout might still cause come confusion, we'll do more optimization on this in our revision.

$\textbf{Weaknesses 3.}$ While spectral adaptation is proposed, it would be valuable to more clearly demonstrate in the methods section how this approach is superior to prior works like SVDiff and OFT. $\textbf{Answer to Weaknesses 3.}$ Comparison/Advantage to SVDiff: idea wise, SVDiff proposes only to tune singular values, which is more constrained compared to our proposed method that considers tuning singular vector space. To see this, first of all, tuning singular values can be achieved by tuning singular vector space (just tune the scale of each row/column pair in $U$ and $V$); Secondly, tuning singular values would result in the same row/column space while tuning singular vector space has more freedom. Practically, SVDiff requires to store all $U,S,V$ in their full format while our spectral adapter only tunes the top spectral component, thus the bottom part can be merged. For example, consider a weight matrix of dimension $n\times n$ with full rank, we have $U\in\mathbb R^{n\times n}, S\in\mathbb R^n, V\in\mathbb R^{n\times n}$ after singular value decomposition. Then SVDiff requires storing all $U,S,V$ and thus takes $(2n^2+n)\*$sizeof(float) GPU storage. For our spectral adapter, say we are tuning the top-$r$ spectral components, then we only need to store top-$r$ components of $U, S, V$ and we merge the rest as $W_2=U[r:]S[r:]V[r:]^T$ that we can sum up with the top part. Thus our method requires $(n^2+2rn+r)\*$sizeof(float) storage, which is two times smaller than SVDiff. Note we can not reduce beyond $n^2*$sizeof(float) since that is required to store the original weight matrix. Comparison/Advantage to OFT: OFT proposes to tune the row space of weight matrix by rotating them orthogonally. This is connected to our rotational spectral adapter which rotates the top columns of singular vector matrices. Intuitively, OFT is preserving the distance between neurons while our rotational spectral adapter is preserving the orthogonality of singular vector basis. These are two lines of logic thus it's hard to say our method is strictly superior to OFT in a rigorous sense. However, our method does have advantage in how it tackles the parameter efficiency issue. Vanilla OFT method is not parameter-efficient since for any weight matrix $W\in\mathbb R^{n\times n}$, it needs to have another orthogonal permutation matrix $A\in\mathbb R^{n\times n}$ of the same size and it uses $AW$ in place of $W$. In OFT paper, the authors manually inject parameter efficiency by constraining $A$ to be only block diagonal with orthogonal blocks, and the authors claim that this constraint is not affecting much in their empirical experiments. Moreover, with such block diagonal $A$, the trainable parameter budget is constrained to be $r^2\*(n/r)$ where we assume the block size is $r\times r$ and it requires that $r|n$. On the contrary, our rotational spectral adapter orthogonally rotates the top $r$ columns of $U$ and $V$ matrices thus we are guaranteed with more parameter budget choices, i.e. our $r$ can take any value between $1$ and $n$, and our method is naturally parameter-efficient.

We hope that our response has positively influenced your perception of our work. Please let us know if your queries have been addressed satisfactorily. If so, could you please kindly consider increasing the score? If you require further clarifications of any points, we are enthusiastic about engaging in further discussion. Please do not hesitate to contact us. We highly value the generous contribution of your time to review our paper. Thanks!

$\textbf{Strengths.}$ A similar idea was presented recently in the PiSSA paper. $\textbf{Comments on Strengths.}$ Thanks for mentioning PiSSA, which is a very recent and concurrent work to ours. This work explores a similar idea as our additive spectral adapter. We noticed it after completing our work. To distinguish our work from PiSSA and explain to other reviewers who may not be familiar with this line of work, we make several notes here. While PiSSA focuses only on LLM tasks and frames their method as a specific initialization method of classic LoRA model, we treat it quite differently and consider the fine-tuning procedure from a spectral decomposition angle. Here are some of the main components we consider which are not involved in PiSSA: 1) PiSSA does not consider rotational fine-tuning and does not maintain the orthogonality of SVD. 2) PiSSA does not explore adapter fusion, i.e., merging fine-tuned models. 3) PiSSA has no vision or diffusion model experiments. 4) PiSSA does not contain theoretic explanations. In contrast, we cover both additive and rotational SVD fine-tuning, adapter fusion by combining models fine-tuned via different spectral spaces, and we experiment with both LLM and generative diffusion models. Moreover, we have a theoretical analysis of adapter rank capacity (Lemma 3.1), weight subspace alignment (Section 3.2) and orthogonality via Cayley parameterization (Lemma 4.1). We believe there is still room for future research on NN weight spectral decomposition. Despite, our rotational spectral adapter has pushed parameter efficiency to certain extreme, in Figure 15 for example, we observe that our rotational adapter with $r=2$ is already performing quite well, which means each weight matrix is attached with only $2*2^2=8$ trainable parameters. As far as we know, this is the first PEFT model with trainable parameter budget scales with $r^2$ where $r$ is the rank, which is fully independent of weight dimension. Prior models such as VeRA and LiDB focus solely on reducing trainable parameter budget but their number of parameters still scales with original weight dimensions. $\textbf{Weaknesses 2.}$ I wish the authors had focused more on LLM evaluations. I may be biased as I work on LLMs. $\textbf{Answer to Weakness 2.}$ We'll consider involving more LLM tasks in our revision. We also feel LLM tasks seem more convincing since the quantitative evaluation metrics are more well-established. $\textbf{Weaknesses 3.}$ Figure 8 of supplementary (above like 586) is interesting and the authors should consider moving it to the main body of the paper. $\textbf{Answer to Weakness 3.}$ Thanks for the suggestion, we'll consider moving them to main text in our revision. We have also provided newly generated test scores for Figure 8 experiments, see the left and middle panel in ``Section B of our attached one-page pdf`` for the results. It kind of shows that increasing the rank can close up the gaps between different methods.

$\textbf{Limitations.}$ 1. The limitations section is missing. One limitation that I can think of is: is it possible for a model to overfit since we are always tuning the most important eigenvectors. What if the fine-tuning dataset is structured so that only a select non-top eigenvectors need to be updated. Datasets like GSM-8k restrict the model output to a narrow pattern that is significantly different from what the model outputs normally. Hence, updating the top eigenvectors makes sense. However, what if the fine-tuning dataset is supposed to modify only a few things that the model has learnt. For instance, there's a lora that switches the model output from Joe Biden to whoever is the next US president. I believe that updating the top eigenvectors is actually a problem here, and, as the authors note in lora merging section (lines 210-214), there may need to be some eigenvector scheduling. $\textbf{Answer to Limitations.}$ Sorry about the missing of limitation section. We'll add a discussion about the limitations in our revision. We feel the main potential limitation of the proposed method is still the overhead for computing SVD of weight matrices. However, for most modern networks (Stable Diffusion, Llama 3 and Mistral), this is tractable since the dimensions of layer weights are moderate. See Figure 7 which shows the SVD time/memory overhead is negligible. For certain other models with larger weights, our method may require more resources. With respect to the reviewer's concerns, we first would like to thank the reviewer for their careful considerations and detailed reading. We have actually thought about similar ideas around using non-principal spectral spaces before, we would like to explain some of our thoughts about this. First of all, if we have a fine-tuning task which requires modification of non-top eigenvectors and the exact space needs fine-tuning is known a priori, we can apply spectral adapter to that particular subspace. In Figure 10 and 11 of Appendix F.5, we illustrate fine-tuning the bottom-$r$ eigenvectors for diffusion models, we have added the results for fine-tuning starting at the $20$th column of $U$ and $V$ for Llama3 in the right panel of ``Section B of our attached one-page pdf``. We also experiment with fine-tuning different singular vectors in our adapter fusion experiments (Section 4.2), where we assign different objects different spectral spaces (e.g., singular vector index 1 to 8 for object A and singular vector index 9 to 16 for object B). This prevents the overlap of the learned adapters in the weight space and improves the visual quality (Figures 5, 12 and 13). We agree with the reviewer that some fine-tuning tasks may require modifying certain non-top eigenvectors, and knowing exactly which eigenvector to tune is usually pretty hard. A simple approach would be to run spectral adapter fine-tuning for different bands of singular vectors and evaluating them on the validation loss, then one can dynamically choose the best basis for fine-tuning. This can be done once at the beginning, or also after a fixed number of training iterations to accommodate for spectral shift that may happen. Second of all, while the training loss of fine-tuned models can be comparable for different adapters, their robustness levels can be different. Capacity wise, if there is a large change $\Delta W$ of low dimension required for weight $W$ while fine-tuning, then both LoRA component and our spectral adapter's component can be dedicated to model $\Delta W$. Thus, these methods would have close training loss landscape. We feel that the difference between these adapter models would lie more in their robustness and stability in their optimization procedure. As shown in our analysis in Section 3.2, our spectral adapter can be more robust against optimization error in certain cases. Regarding the connection to eigenvector scheduling, we propose to schedule different tuning tasks over different spectral spaces thus to better utilize their orthogonality. We believe that modification happening within different orthogonal basis would affect each other less, this is also the underline logic of [42]. But say if we have an extreme case that task 1 is modifying exactly the $5$-th column of $U$ and $V$, and task 2 is modifying exactly the $10$-th column of $U$ and $V$. Then in this case, we agree that it would be ideal if one could recognize a priori which indices need to be tuned, though this is a nontrivial job. Our hope here is that, with rank $r=1$ for example, tuning the top component would capture a bit about task 1 (the top part may change from $U_1S_1V_1^T$ to some combination of $U_1S_1V_1^T$ and $U_5S_5V_5^T$), and tuning the second top component would capture a bit about task 2. These differences would happen more independently with respect to each eigen basis. Thus the integrity of each task is better preserved.

Please let us know whether our replies address all your concerns. If so, could you please kindly consider increasing the score? If not, we are willing to address any other question in more details. Thanks!

Dear reviewer, given that this is the last day of author-reviewer discussion period, we would like to for the last time check whether our above responses have addressed your prior concerns satisfactorily. Hope that given our (newly) incorporated quantitative scores for our diffusion model experiments, which demonstrate improvement of our method, and different models/benchmarks for LLM tasks we've already considered, your prior concerns with respect to evaluation problems have been alleviated. If our responses effectively tackled all your questions, could you please kindly consider raising your score? If there is any point requires further classification, please let us know and we'll be here to address any existing questions until the final second of the discussion period. Thanks again for spending time reviewing our work! We highly value the generous contribution of your time to review our paper.