Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference

Summary

The paper proposes Conditional Adapter (CODA), which not only enables parameter-efficient fine-tuning but is also beneficial for the inference speedup. The main idea of CODA is to pass all tokens to a lightweight routing adapter path while selectively passing only certain tokens to the original path. The selection process, named the “Soft Top-K” operation, is conducted by solving an optimization problem that utilizes a generalized entropy function and an iterative searching algorithm. Experimental results on Encoder-Decoder Transformers in NLP, Vision, and Speech applications demonstrate significant inference acceleration with only minimal performance degradation.

Strengths

* The idea of CODA is simple yet powerful. The authors support the effectiveness of the proposed method through sufficient experiments. Although the baseline (Parallel Adapter) achieves higher performance than CODA, I believe the inference speedup can compensate for the performance loss. * The derivation and implementation of the “Soft Top-K” operation are particularly unique. This operation outperforms simple sigmoid gating or Top-K truncation. Furthermore, this differentiable function has the potential for a broader range of neural network research.

Weaknesses

* Although the authors briefly mentioned CoLT5, it would be better to discuss more the differences between CoLT5 and CODA. The current explanation (L78-79) is somewhat insufficient.

Questions

* How much is the burden of the iterative searching process (T=20) during training? In other words, how does the training time/performance change as iteration steps T are reduced? * Regarding Figure 5(b), how could the decoding be accelerated? As I understand, CODA does not remove the number of tokens, so the input #tokens to the cross-attention would also not be reduced. Additionally, the readability could be improved if Figure 5 is enlarged in the manuscript (minor issue). * (Suggestion) I expect CODA can also reduce memory consumption for both training and inference. It would strengthen the paper’s value if there were a comparison regarding memory usage. * (Suggestion) CODA currently fixes the reduction ratio r for all layers. I understand that it is a simple and structured way, but how about setting different r for different layers? For example, small/large r for low/high layers. This could provide additional insights.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have addressed the limitations in Section 7.

Summary

This paper proposes a unique modification over parameter-efficient transfer learning by employing a conditional adapter parallel to the pre-trained model. In doing so, a dynamic token selection mechanism is proposed for each block. The selected tokens undergo heavy pre-trained operations, while all the tokens undergo light-weight adapter operations. This greatly improves the inference efficiency of the target dataset. Experiments from language, vision and speech recognition are shown to test the efficacy of the proposed method.

Strengths

1. Wide variety of experiments provided show the superiority of CoDA as compared to parallel-adapter among others. 2. The overall idea of improving the inference speed of large models during the transfer learning stage has huge practical value. 3. A large chuck of questions have been addressed in the main text including - iteration wise performance, the importance of routing, etc.

Weaknesses

1. The presentation of the work can be improved; after section 3.2, the paper seems unorganised, and the scale and location of tables and plots also require some work. 2. Thorough comparison with existing works is missing. There are two streams of comparison possible - [1] Parameter-efficient fine-tuning methods [2] Dynamic/conditional inference methods. Even though experiments are shown on a variety of tasks, it is still unclear how this work compares with existing methods. 3. Final latency with respect to the baseline model is not thoroughly presented, also additional parameters due to parallel branches are not given.

Questions

1. Is there a simple softmax baseline instead of the top-soft-k operation? 2. Is the iterative process for soft top-k done for each forward pass and each layer? If yes, how much overhead does it add? 3. What is the parameter overhead of CoDA and absolute latency numbers for different tasks? 4. There are a variety of standard benchmarks for parameter efficient fine-tuning - VTAB-1k (vision), GLUE (language), among others; why did the authors choose specific benchmarks presented in the paper? If possible, can the authors present results on these standard benchmarks?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes

Summary

This paper proposed Conditional Adapter (CODA), which can achieve good trade-offs between speed and accuracy. CODA achieves 2x to 8x inference speed-up compared to the state-of-the-art Adapter approach while maintaining moderate to no accuracy loss and the same parameter efficiency across language, vision, and speech tasks.

Strengths

1. The proposed method can greatly speed up inference, while keeping competitive performance. 2. The writing of the paper is clear and easy to understand. 3. The experiments are extensive, which can also well support the arguments of this paper.

Weaknesses

1. This paper proposes a new adapter and claims to accelerate model inference. However, there is no direct relationship between the adapter and the inference acceleration. The token pruning approach appears more like an independent module. Therefore, this paper is more like a combination of token pruning and traditional adapters, and both approaches are not new in the community. 2. The token pruning is not new, as there have been many methods applied to ViT. Some methods can even achieve inference acceleration without fine-tuning. The author should disscuss the differences with these methods. Token Merging: Your ViT But Faster, ICLR, 2023 Dynamicvit: Efficient vision transformers with dynamic token sparsification. NeurIPS, 2021 A-ViT: Adaptive tokens for efficient vision transformer, CVPR, 2022 3. Based on 1, I'm not quite clear whether this article proposes an adapter method or a pruning method. From the author's descriptions, i lean towards the former. However, in the experimental section, the author lacks a comparison with newly proposed parameter-efficient transferring learning (PETL) methods, such as AdaMix, MAM adapter, and so on. Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models, EMNLP, 2022 TOWARDS A UNIFIED VIEW OF PARAMETER-EFFICIENT TRANSFER LEARNING, ICLR, 2022 4. It is unclear whether this method can be applied to the currently popular autoregressive models.

Questions

See weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See weakness

Summary

The paper introduces a novel technique called Conditional Adapters (CODA), which aims to enhance transfer learning and improve inference efficiency. CODA surpasses the state-of-the-art Adapter approach by achieving a remarkable 2x to 8x acceleration in inference speed while maintaining moderate accuracy loss and identical parameter efficiency. The paper provides examples of language, vision, and speech tasks where CODA has been extensively evaluated, showcasing the promising results obtained through its application.

Strengths

1. This paper addresses an intriguing problem of accelerating inference speed through efficient parameter tuning. 2. It is commendable that the proposed method is evaluated across three different modalities, indicating its versatility. 3. The model's simplicity and effectiveness are notable. The overall design is straightforward, and the method is technically sound. 4. The paper is well-written and easy to follow.

Weaknesses

1. The availability of the source code is unclear, and it would be beneficial to know if it will be made publicly available for further exploration and reproducibility. 3. Can not be applied to current main-stream auto-regressive models. 2. While the paper covers text and speech modalities, for the visual modality, it would be valuable to benchmark the proposed method on tasks such as image recognition and object detection, in addition to OCR-VQA and Doc-VQA.

Questions

See weakness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Discussions are required for the submission

Dear all reviewers， Thank you very much for your great efforts in reviewing the referred submission. Now the authors have provided responses regarding your concerns. Would you please read the authors response and see whether your concerns have been addressed or not. You are welcome to raise further concerns if necessary so that the authors can respond to them timely. Your great service would be very important for the community to make final decisions. Best regards， Your AC