Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models

Summary

This paper aims to achieve both parameters and computation efficiency for the transfer learning of pre-trained VLMs. They combine the previous ladder-side tuning adapter with the proposed dynamic architecture skipping technique to achieve such a goal.

Strengths

It is a good idea to achieve both parameters and computation efficiency for the transfer learning of VLM.

Weaknesses

Although it is a good idea to achieve both parameters and computation efficiency for the transfer learning of VLM, the author does not show the pros and cons of the proposed methods clearly. 1. The training cost of the DAS and the benefit of the DAS is not clear. In Table 1, the authors do not show the running time in practice. FLOPs are not equal to the inference speed, especially the reduction of FLIOPs is marginal. Plus, please provide the training cost of DAS. 2. Why only take experiments on classification tasks? The authors claim they propose a new problem, PCETL, but lack experiments on import VL tasks - image captioning. Is the parameter pruning of VLM not suitable for the image captioning tasks? Does the layer dropping break the text generation ability of pre-trained VLM? 3. Why only provide retrieval results on Flickr30K, it is small and easy. A large model may be unnecessary on Flickr30K. Please provide retrieval results on MSCOCO. 4. How does the proposed DAS compare to the pruning method in EfficientVLM[1] or other layer pruning methods in the pruning area? Will DAS be necessary? A random baseline in Figure 3 is not enough. [1] https://arxiv.org/pdf/2210.07795.pdf

Questions

1. Why the method is called "Dynamic Architecture Skipping"? Where the dynamic comes from? From my point of view, the skipped modules are fixed after training and it will not change with the input during inference.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

As I discuss in the Weaknesses, the authors do not fully address the limitations of the proposed mehtods.

Summary

The paper proposes a dynamic architecture skipping (DAS) method for the parameter and computation efficient transfer learning (PCETL) problem. DAS explores the optimal short-cut pathway in VLP models. Extensive experiments show the effectiveness of DAS both in reducing computation and parameters.

Strengths

The paper introduces a novel and intriguing approach by considering network jumps as k-armed bandit sampling. It highlights the significance of reducing computational complexity in Visual Language Pretraining (VLP) models, providing strong motivation for the proposed methodology. The paper is well-written and effectively communicates its ideas. Experiments are solid, with necessary analysis and ablations for different parts of the method.

Weaknesses

Compared to the number of parameters and FLOPS, it would be better to add the training time and GPU memory cost as metrics.

Questions

N/A. Already written in the weakness section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes.

Summary

The paper presents dynamic architecture skipping (DAS), which can drop some transformer layers during inference. The routing of DAS is learned by reinforcement learning. To make the training parameter efficient, every layer has an adapter for training. After training, DAS drops several redundant layers and replaces them with adapters to reduce the inference FLOPs.

Strengths

1. Most parameter-efficient training methods add extra cost in inference, and it is interesting to explore how to reduce the cost. 2. The design of the proposed approach, which replaces some transformer layers of the backbone model with adapters, is reasonable.

Weaknesses

1. Several baselines [2] and related works [1, 2] are missing. I think the paper would be stronger if it can compare to [2]. 2. I am not sure what is the uniqueness of the approach only applies to the VL domain. If the approach is general, I would expect applying the approach to some other models and tasks too (LLM or ViT). [1] Fan, Angela, Edouard Grave and Armand Joulin. “Reducing Transformer Depth on Demand with Structured Dropout.” [2] Din, Alexander Yom, Taelin Karidi, Leshem Choshen and Mor Geva. “Jump to Conclusions: Short-Cutting Transformers With Linear Transformations."

Questions

1. The training pipeline is more complex than the other approaches since the training involves search, redundancy observation, and final fine-tuning. I wonder how much cost (e.g. training time and memory) is needed to train the method compared to other approaches? 2. Are the results on one run or multiple runs? If it is on one run, I would suggest using the average of multiple runs to justify the robustness of the approach. 3. Given the GFLOPs saving, how much the approach can improve the inference speed and inference memory? --- **Post-rebuttal** Thank you for the authors' response. I have read it and it addressed my questions.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The limitation in the paper is sufficient.

Summary

This work focuses on the problem of transfer learning in the context of vision-language pre-trained (VLP) models. Existing works that adapt VLP models primarily address the issue of parameter efficient transfer learning (PETL). However, these methods do not effectively reduce the computation complexity of VLP models. Therefore, this work takes PETL a step further and introduces a new setting called parameter and computation efficient transfer learning (PCETL). The goal of PCETL is to reduce the computation complexity of pre-training models, enabling faster inference, while only tuning a fraction of parameters. To achieve PCETL, a method called dynamic architecture skipping (DAS) is proposed. DAS assists in finding the optimal subnetwork routing of VLP models for downstream tasks. Although the concept of dynamic architecture skipping has been extensively explored in transfer learning for image spaces, it has not yet been investigated in the era of large-scale pre-trained models. The proposed approach demonstrates promising results on two VLP models and three vision-language benchmarks."

Strengths

1. The paper is well-written and easy to follow. 2. The concept of dynamic skipping, as yet unexplored in the era of large-scale VLP models, holds considerable promise. Its introduction to VLP models could significantly reduce computational and time complexity. 3. As the daily release of pre-trained large-scale models continues to increase, methods for efficiently transferring learned features become increasingly significant. 4. The experimental results compellingly illustrate the potential of the proposed methodology.

Weaknesses

1. The concept of dynamic architecture skipping is not new and is already well-established in the case of transfer learning for standard imagery (see references below). It would be beneficial if the authors could provide reasons why existing approaches are unsuitable for the current problem and setting. 2. The proposed approach is not end-to-end differentiable and models the problem as a k-armed bandit. A more comprehensive end-to-end approach would be ideal (refer to references 2 and 4). 3. The paper lacks a comparison with existing dynamic architecture skipping methods, which is a crucial element for an encompassing evaluation. 4. The authors acknowledge that the number of layers to be skipped must be manually determined. Some of the transfer learning methodologies listed below are capable of handling this automatically, borrowing ideas from there can benefit this work. **References** 1. [BlockDrop: Dynamic Inference Paths in Residual Networks](https://arxiv.org/abs/1711.08393) 2. [SpotTune: Transfer Learning through Adaptive Fine-tuning](https://arxiv.org/abs/1811.08737) 3. [Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?](https://arxiv.org/abs/2106.02890) 4. [$\Delta$-Networks for Efficient Model Patching](https://arxiv.org/abs/2303.14772)

Questions

1. Looks like random skipping is a very strong baseline for this work (Line 256-257). The performance difference between random skipping and the proposed approach appears marginal, yet considering that random skipping is less costly, are these gains still substantial? 2. Could you please explain the rationale for modeling this as a k-armed bandit problem? Additionally, what would be the primary challenges in solving this problem in an end-to-end differentiable manner?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations Section discusses the limitations of this work clearly

Respone to the rebuttal

Thanks for your response. - **Comment 1** Does the inference speed mean the model can process 100+ samples in one second on VQA Test-Dev? - **Comment 2** It is surprising to see the method can also achieve good performance after dropping 6 layers. However, there are still a few questions. - Why the #updated parameters is more than LaVIN-7B? What is the performance of the PEFT baseline with roughly the same #updated parameters? - The Mem and Time should also be reported with LLaMA. - So, METER + DAS will not work for image captioning? - **Comment 3** The performance of DAS on COCO is not as good as the performance of DAS on Flickr30k. So, there is a relationship between model size, the difficulty of the task, and the pruning choice. - **Comment 4** We can consider the work as pruning + Adapter. Not many parameters have been pruned (compared to previous pruning methods), so the performance can be recovered by tuning the adapter. A big problem here is that the paper does not discuss and compare previous pruning methods thoroughly. Why do we need Dynamic Architecture Skipping? Why we cannot use some similar pruning methods like [1,2,3]? I do not see any discussion of previous pruning methods in the paper. Personally, I think this is not respectful of the previous pruning works. - **Comment 5** The authors are talking about the generalization of the proposed methods. What does "dynamic" mean? Why use the word dynamic? [1] https://aclanthology.org/2021.emnlp-main.829.pdf [2] https://arxiv.org/pdf/2111.15127.pdf [3] https://proceedings.mlr.press/v202/shi23e/shi23e.pdf