Towards Optimal Adapter Placement for Efficient Transfer Learning

Response to Reviewer Dkam

We thank the Reviewer for the evaluation of our work and are happy to see that the Reviewer recognized the extended search space as a novel contribution that enhances flexibility and improves fine-tuning results. We are also glad to hear that the construction of the gradient rank metric was appreciated. Overall, we are grateful for the positive feedback. Below, we address the concerns raised by the Reviewer: > the reviewer is curious about its performance on larger and more diverse datasets, such as ImageNet. The datasets that we have selected for the empirical evaluation were based on those used in related studies on transfer learning and parameter-efficient fine-tuning (e.g., iNaturalist18 and Places were used in [1], VTAB in [2,3], SST2/MNLI in [4]). Following reviewers suggestion, we conducted some experiments on transferring models pretrained on Imagenet-21K to the ImageNet-1K, exploring the placement of a single parallel adapter and the influence of long-range connections (see the new Appendix L). Our results match the results presented for iNaturalist18 and Places365 in Figure 1, where we observe adapters added around middle blocks to achieve higher transfer accuracy. > The reviewer is curious if the added complexity of long-range and recurrent adapters could lead to instability during training or inference, particularly in large-scale applications Each adapter includes a Layer-Normalization module. Additionally, standard transformer architectures also incorporate such normalization layers. As a result, the inputs are always normalized, regardless of the structure of the adapter connectivity graph. We also employ learnable scale parameters (as described in Equation 1), which are initialized to zero to ensure stability. During experiments conducted in the Extended Search Space, we found that training hyperparameters, such as the learning rate, could be reused from the optimal configuration for Parallel Adapters. Even when running experiments with 40 adapters in the large-scale ViT-g/14 model, no stability issues were observed, as evidenced by the standard deviations shown in Figure 6c. > The reviewer wonders if there is potential to reduce the task-specific nature of adapter placement, making the method more widely applicable without requiring custom placement strategies for each task. Our experiments show that the best adapter placements are clearly task-dependent (consider the difference between e.g. Clevr-Count and SVHN in Figures 1 and 3). Therefore, the same set of adapters for every dataset and modality won’t in general be optimal. Nevertheless, we observe that the best adapter placements can transfer to some extent (recall Figure 4 where, for instance, correlation between best placements for iNaturalist18 and Places365 reaches 0.88). To further explore the idea of transition, we have conducted an experiment where the best random placement (Extended Max from Table 1) identified for SST2 was applied to the MNLI task, and vice versa. The results are promising, yielding test accuracies of $86.72 (-0.24)% $ for MNLI and $94.48 (-0.52)$ for SST2, where the performance reduction in comparison to the results for Extended Max (Table 1) is given in parenthesis. While these results roughly match the performance of PA, they fall short of the performance achieved by using the best placements tailored to each task. We added this experiment to the Appendix (Table 8). We once again thank the Reviewer for taking the time to evaluate our work and for the positive feedback. We believe we have addressed the raised concerns and remain available for any further clarifications if needed. Meanwhile, we kindly request the Reviewer to reassess our work in light of our responses. *References* [1] Mercea, Otniel-Bogdan, et al. "Time-Memory-and Parameter-Efficient Visual Adaptation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [2] Jie, Shibo, and Zhi-Hong Deng. "Convolutional bypasses are better vision transformer adapters." arXiv preprint arXiv:2207.07039 (2022). [3] Zhang, Yuanhan, Kaiyang Zhou, and Ziwei Liu. "Neural prompt search." arXiv preprint arXiv:2206.04673 (2022). [4] He, Junxian, et al. "Towards a unified view of parameter-efficient transfer learning." arXiv preprint arXiv:2110.04366 (2021).

Response to Reviewer BPzc

We appreciate the Reviewer's feedback on our work. We're glad that the Reviewer finds the topic of optimal adapter placement interesting and recognizes the potential of the GGA algorithm. We've addressed the specific concerns raised below: > The recurrent adapters require two forward processes, further deteriorating the efficiency of pre-trained models The motivation for introducing the recurrent adapters was to explore all potential pathways for information adaptation within a network. As we have discussed in Chapter 3 (line 269), the recurrent adapters introduce at most 50\% increase of the computational cost but do not introduce any additional parameters to the network. This makes them a compelling option for scenarios where an increase in computational time can be traded for improved accuracy, without requiring additional parameter resources. One can always opt to only use the diagonal and upper triangular parts of the connectivity matrix. To demonstrate this, we restricted the extended search space to not include any recurrent connections. Table 7 in the Appendix shows that the naive random search can still uncover placements that match the performance of Parallel Adapters using half of the parameters without increasing inference cost. > Results in table 1 lack the comparison with related works. The aim of the experiment in Table 1 is to demonstrate that there exist adapter placements within the extended search space that can outperform the standard PA approach. This is why we compare our method with other adapter placement strategies. For a comparison of how adapters perform in relation to other PETL approaches such as LoRA or Prefix Tuning, please refer to the work that focused on that topic (e.g. [1,2]). > The paper lacks a clear and actionable conclusion. While the findings are interesting, it remains unclear how these results can be directly applied to practical scenarios. The key takeaways for practitioners from our paper are as follows: 1. Performing a random search within an extended adapter search space can effectively reduce the parameter count. 2. Since an adapter’s performance is often linked to the rank of its gradient, this metric can be leveraged to identify optimal adapter placements (as demonstrated by the GGA algorithm). 3. Adapter selection needs to be dataset-specific, as the best placements are unlikely to generalize across unrelated datasets. Crucially, our work establishes a strong connection between adapter placement and performance, paving the way for future research. This insight encourages further exploration into how this relationship can be harnessed to develop new guidelines for designing adapters. **Questions:** >The GGA algorithm shows promise, but it only works on the use of a limited number of adapters [...] By the use of GGA we can match or surpass the performance of the baseline approach with less adapters - consider, for instance Figure 6a, in which we outperform the PA approach with as few as 3 adapters, or Figure 6b in which we only need 12 adapters to match the performance of PA (that uses 24 adapter by default). In consequence, we are able to further reduce the parameter count. Note that even though adapters are inherently lightweight, further reducing their parameter count is far from pointless; it remains an active area of research with substantial ongoing work. Consider, for instance, scenarios where adapters need to be optimized and stored separately for a large number of disjoined tasks using the same pre-trained model. Additionally, reducing the number of parameters could facilitate fine-tuning on a single GPU, enabling a greater number of simultaneous runs or improving the accessibility of transfer learning to researchers and developers without access to extensive computational resources. We again thank the Reviewer for their valuable feedback and believe we have addressed their concerns. We remain available in case of any further questions and kindly ask the Reviewer to consider our responses and reassess our work. *References* [1] He, Junxian, et al. "Towards a unified view of parameter-efficient transfer learning." arXiv preprint arXiv:2110.04366 (2021). [2] Zhang, Yuanhan, Kaiyang Zhou, and Ziwei Liu. "Neural prompt search." arXiv preprint arXiv:2206.04673 (2022).

Response to Reviewer 6CsQ [Part 1/2]

We thank the Reviewer for the time and effort dedicated to evaluating our work. We are pleased to learn that the Reviewer considers our construction of the searching graph as solid, and has recognized the significance of the improved performance achieved through the strategic placement of adapters, as demonstrated in our experiments. We greatly appreciate this positive feedback. Below, we address the issues raised by the Reviewer in detail. > 1.How many adapters would achieve optimal performance? In Figures 6a and 6b we examined the effect of the number of adapters on the transfer performance. We observe that while adding more adapters initially enhances the results, the improvement eventually plateaus for GGA—the best-performing method—after 12 adapters. Due to this saturation we did not explore configurations with more than 24 adapters, which is the number used by the baseline Parallel Adapters (PA) approach. Moreover, adding more adapters often does not improve the test accuracy. For instance, on the SST2 dataset, using all $N^2=625$ adapters from the extended search space achieves a test accuracy of 93.81$\pm$0.76, which falls short of the 95.00$\pm$0.25 obtained by the best random selection of just 24 adapters (see the new Figure 15 in Appendix). This raises the question of whether peak performance can be achieved with fewer parameters. Our study demonstrates that by strategically selecting adapter locations, it is possible to reduce the number of adapters required. > is there a mathematical formalization that explicitly demonstrates how many adapters are sufficient? We agree that the topic of how many adapters are needed to achieve optimal performance is very interesting, and in general touches on the broader problem of generalization capabilities in the context of transfer learning [1,2]. To the best of our knowledge, no existing research has theoretically explored the relationship between adapter capacity and its impact on transfer generalization. Even in the context of linear probing and full fine-tuning, establishing solid mathematical foundations remains an open area of inquiry [3,4,5]. Predicting the optimal number of adapters in advance is generally difficult and, as demonstrated by the experiments in Figures 6, may depend on various factors, such as the underlying task or the chosen search space. > Is it possible that adding too many adapters could lead to an over-parameterized model, potentially causing a drop in accuracy? As with any deep learning framework, introducing too many trainable parameters relative to the available training data can lead to issues with overparameterization. We do, in fact, observe one such situation in our study, as shown by the results for the Clevr-Count-1K dataset in Figures 1 and 3. Here, the full fine-tuning (FT) method, despite achieving the lowest training loss (below $10^{−4}$, see Table 6 in the Appendix), yields the worst test accuracy among all approaches. Except for this example, the datasets studied by us do not have this issue. Nevertheless, even in PETL, maintaining an appropriate balance between model complexity and the available data is an essential task of the researcher. > The authors suggest that GGA is effective with a few adapters (as shown in Figure 6, where 12 GGA-based adapters achieve similar or higher performance). I believe it would be beneficial if the authors provided comparisons of computational complexity and memory usage for these methods. Specifically, what is the training complexity compared to the compressed approach in this paper? Calculating gradients may introduce additional time and memory costs, as it requires retaining the graph during training. Please note that we compute the GGA scores (Equation 7 and 6) **only once**, after which GGA uses discounting to sample a set of adapters (see Algorithm 1 in the Appendix). Calculating the GGA scores for a network with $N^2$ possible placements involves computing the gradients from Equation 6 and evaluating Equation 7. For the ViT-16/B model from the experiment in Figure 6, this cost amounts to approximately $110.0$ TFLOPS (see the new Appendix J for details). A forward pass of ViT-16/B model with batch size 128 is approximately 4.5 TFLOPS. Since we perform 20000 iterations during the adaptation (that amount to a total 90 PFLOPS, and this is considering only the forward passes, without the cost of backpropagation) $F_{GGA}$ introduces only a negligible (0.12%) overhead in comparison.