Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing

Summary

This work presents an efficient adapter design for transfer learning of large retrained models. The key idea is to enable adapter to be shared across layers and imposing low-rank constraint, so that the overall trainable parameters can be reduced. Given the re-composed linear adapter, existing reparameterization methods can be further used to avoid extra computation cost.

Strengths

- Parameter efficient transfer learning is an important approach for making large pertained model useful in many applications. - The paper is written to be easy to read.

Weaknesses

First of all, the motivation of this work is weak -- Existing adapters are already lightweight, e.g., less than 0.5% of the pretrained model, and the scope for further reducing the size of adapter is marginal. That being said, this adapter efficiency problem is not significant. In introduction, it is hard to read out the novelty of this method since more related works such LoRA is not discussed and compared at all. Limited novelty with the proposed method: Low-rank constraint is not new, as already used in LoRA [24] although not in the same way, and parameter sharing across layers is also not novel idea. This work combines the two in a single place, which could be considered as being not significant. Limited performance gain: - In most cases, the proposed method cannot achieve a good margin over previous methods (e.g., SSF). - Another important metric, FLOPs, would be useful to report.

Questions

See the weaknesses above.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Summary

This paper introduces a parameter-efficient transfer learning method named Adapter Re-Composing (ARC), which mainly focuses on investigating the reusability of adapted parameters. The authors propose to apply a shared adapter to all the layers (blocks) of the pre-trained model, and they use different Re-Scaling Coefficients (diagonal matrices) in different layers to ensure the diversity of parameters in different layers. The motivation behind this design is the adapter module's low-rank property, shown in Fig.3 in the main text. Moreover, the authors conduct extensive experiments to demonstrate the effectiveness of their designs, where they train fewer parameters and achieve competitive or even better performance compared to prior arts.

Strengths

1: This paper is well-written and well-organized. 2: The ARC design in the paper is well-motivated. 3: Although the technic is simple and easy to implement, the performance is impressive. 4: The experiments are comprehensive.

Weaknesses

1: The ARC leverages learnable re-scaling coefficients in different layers to maintain diversity. However, the authors didn't disscuss the numerical difference among the re-scaling coefficients across different layers. If they are also similar, a share-weight adapter can replace the ARC. 2: In Tab. 1 and Tab. 2, the SSF* applies additional techniques (e.g., data augmentation) during training and gains non-trivial improvements. Why did the authors not use those techniques to improve the performance of ARC further? Will these techniques further benefit the ARC? 3: I wonder about the computational overhead of the ARC, e.g., the GPU memory usage during training, because the number of learnable parameters may not necessarily be positively correlated with the GPU memory usage. Could authors provide the additional GPU memory usage comparison between ARC and other related works? 4: How to use ARC in Hierarchical Vision Transformers such as Swin Transformer is unclear in Lines 277-279. Adding more details here may be helpful. Overall, my major concern is weakness 1. Happy to raise my rating if my concers are well adressed.

Questions

See the weakness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors disscuss the limitations and boarder impact.

Summary

The paper explores ARC, which is a novel parameter-efficient fine-tuning method which uses a similar architecture as adapters but introduces inter- and intra- layer weight sharing. Some down- and up- projection weights are shared but every adapter position uses an independent set of per-channel scaling factor on the channel-reduced intermediate features. The proposed method obtains competitive results on various vision transformer adaptation benchmarks in terms of recognition performance and number of parameters. Moreover it can be re-parameterized into adjacent fully-connected layers so no overhead is incurred during inference.

Strengths

* The paper is generally well written. The figures are clear and the text is easy to follow. * The experiments are comprehensive and cover a wide range of visual recognition datasets and the results are competitive.

Weaknesses

* The effectiveness of the proposed method is not well justified. Section 3.3 does not make much sense to me: Having long-tailed singular values only implies that the weight difference can be approximated well by a low-rank matrix and thus justifies the bottleneck design. Shared projections further requires the adapters to have largely overlapped kernel and image spaces, which, instead of the singular values themselves, are determined by the direction of the singular vectors corresponding to the top singular values. It also doesn't theoretically justify the intra-block sharing design (i.e., using the transpose of down-projection as up-projection). * In multiple places (e.g., caption of Table 1 and Line 229), the paper's claim of using simple augmentations on baselines for 'fair comparison' is questionable. Due to the vast differences in nature of different methods, it is expected that their optimal training configurations are different, and each method, including this paper's own, should have the right to choose its optimal training configuration as long as it does not violate some principal rules of machine learning (e.g., leak of test data): Conversely, it is also improper to request that the paper's method being run under SSF's data augmentation for 'fair comparison'.

Questions

* The paper cited [7] in multiple tables for visual adapter results but [7] seems to not include any results on computer vision tasks (actually the paper title is *Parameter-efficient transfer learning for **NLP***). Are those results reproduced by the authors using the same architecture as [7]? If so, could the authors point to a place where more detailed settings (e.g., the bottleneck dimension, the activation function, the training hyper-parameters) of this baseline can be found? * In Table 3, it is strange to me why the proposed methods obtain lower performance on ViT-Huge than ViT-Large. Is it possible that the proposed method or some baselines are over-fitting in extremely large backbones? * Following weakness 1, an understanding of why the proposed method works fairly well can also be provided by experiments: For example, are features from adjacent layers similar due to the identity connections? What if the projections are progressively shared in groups of increasing sizes among consecutive layers before reaching the global sharing setting as reported in the paper?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

2 fair

Limitations

There are no unmentioned limitations in the paper to my mind.

Summary

This paper proposes to further reduce the parameters of the adapter by introducing a weight-sharing scheme between different layers. To accommodate the variations across different layers, re-scaling coefficients are learned to re-compose the layer-adaptive adaptation matrices. Experiments are conducted on 24 downstream image classification tasks using various Vision Transformer variants.

Strengths

+ The observation that “learned adaptation matrices naturally exhibit low-rank characteristics” as shown in Fig. 3 is quite interesting, making the motivation for the weight-sharing design compelling. + This paper is well-written and easy to follow. The figures are well-prepared and illustrate the core idea clearly. + The experiments are extensive.

Weaknesses

- The comparisons to SSF shown in Tables 1 and 2 are re-implemented by the authors and data augmentations are removed. Their original performances of SSF are much higher than the proposed approach. It would be better if the authors reported the performance of the proposed approach with these advanced data augmentations. - Fig. 3 shows the singular value distribution of the original MHA and FFN adapter. It would be better to show how the proposed approach alleviates such a problem by visualizing the value distribution after using the proposed approach. - Though the observation of “learned adaptation matrices naturally exhibit low-rank characteristics” and the proposed weight-sharing scheme is very interesting, the motivation to further reduce the parameters of the adapter is not very compelling as the parameters of the adapter are already relatively small.

Questions

Please see the weaknesses

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This paper introduces a novel parameter-efficient fine-tuning method called Adapter Re-Composing (ARC). ARC effectively reuses parameters across different layers, resulting in remarkable improvements in performance across 24 image classification datasets while utilizing fewer learnable parameters. The experimental evaluation conducted on various downstream datasets provides compelling evidence of ARC's superiority. It outperforms existing methods and establishes a new benchmark for parameter-efficient fine-tuning techniques.

Strengths

1. The proposed ARC method is simple yet effective, achieving superior performance on multiple downstream datasets while utilizing fewer trainable parameters. 2. The experiments conducted provide compelling evidence, as they encompass various datasets, attention-based architectures, and ablations, ensuring the robustness and reliability of the findings.

Weaknesses

1. In Table 3, it is observed that the performance of ViT-Huge is lower than ViT-Large. It would be beneficial if the authors could provide an explanation for this disparity. 2. The utilization of symmetric matrices ($W_{up} = W_{down}^T$) in the bottleneck design helps reduce the number of learnable parameters. However, it would be interesting to explore whether further improvements in performance can be achieved by making the downsampling and upsampling matrices independent. It would be valuable if the authors could provide a comparison of performance and parameter statistics to address this potential enhancement. 3. The experiments conducted have demonstrated the effectiveness of the proposed method. However, it would greatly enhance the strength of this paper if the authors could supplement these empirical results with theoretical analysis.

Questions

Please see Weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please see Weaknesses.

Thanks for the rebuttal

I appreciate the rebuttal and the clarification. The responses address most of my concerns. Therefore, I would retain my score.