Summary
This paper tackles the inference efficiency and entangled representation aspects of parameter-efficient transfer learning (PETL) . It proposes Dynamic Adapter (Dyn-Adapter) to add prediction heads for intermediate features. During inference, if the confidence score of a prediction head is less than a threshold, the features will continue to be processed by later layers, if the confidence exceeds the threshold, the inference process exits. It also proposes to use a higher dropout rate for shallow layers and a random masked gradient update for better generalization.
Strengths
- The paper tries to address important problems in PETL
- The proposed method improves the throughput by a large margin while maintaining accuracy performance.
Weaknesses
- Considerable improvements are needed in the writing and presentation quality.
- Figure 2 should show what the dashed lines mean
- "supervision is performed with a stable interval T by classification targets Y" and "The total number of introduced supervision S". It's quite confusing. I can only get the idea when I read the later parts of the paper. What does mean by introduced supervision S. T is an integer, not an interval, right? What value can T be selected? T should be factors of N?
- "However, in the past early exit scenarios, the heavy misalignment of path to obtain Y^_i and Y^_{i+1} may cause optimization direction interference intrinsically" I don't get the main point of this sentence.
- "while the shallow features are used for late classification and early decision both", what do late classification and early decision mean?
- theorem should be rigorously proved but theorem 2 is just a statement
- "generalization ability of shallow layers should be enhanced". generalization of deep layers should also be enhanced right?
- typos: Dyn-Adpater should be Adapter
- There are two rows for Dyn-XXX in table 1. The author should explain what they are in the caption.
- Some statements in the paper are not clear and confusing.
- "By freezing the backbone and only updating the adapter to assume the task-related semantic abstraction, we guarantee that the low-level feature in the backbone will not be interfered by loss signal, realizing explicit decoupling of low level feature and high level semantics." Updating adapters of shallow layers will change low-level features. I don't see how updating adapters can guarantee that the low-level feature in the backbone will not be interfered by loss signal. How can updating adapters fix the problem that early classifiers
force intermediate low-level features to encapsulate high-level semantics and be linearly separable?
- "Nevertheless, current PETL methods are only capable of tuning the high-level semantic representations and can not directly utilize low-level information in case of the demand of down-stream tasks, thereby undermining their capacity to adapt to diverse datasets." Because of the skip connections in transformer blocks, prediction can also use the features from the shallow layers [1].
-Because of theorem 1, the authors proposed to use MLP for early stages and light-weight head for later stages. What’s the logic and rationale behind it?
- For Adaptive Prioritization, does it mean that the lambda needs to adjust manually during the training? The description of the adjustment is very vague, "progressively increase, gradually declines". How much does it increase/decrease, and how often? The authors need to make this part more clear.
- The increase in the number of parameters is quite significant. 75% for adapter, 59% for LoRAm, and 73% for Repadapter in table 1. In table 5, the number of parameters is doubled.
- Related work. Multiple works have proposed to leverage intermediate features in PETL or transfer learning [2, 3, 4]. The authors should discuss them in the paper.
[1] Revisiting Vision Transformer from the View of Path Ensemble
[2] Visual query tuning: Towards effective usage of intermediate representations for parameter and memory efficient transfer learning
[3] Head2toe: Utilizing intermediate representations for better transfer learning
[4] Lst: Ladder side-tuning for parameter and memory efficient transfer learning
Questions
- Since the number of epochs is fixed, the reported numbers are accuracies at the end of 100 epochs?
- This paper shows that ReAdapter is accepted at ICCV23 but I can't find them in the ICCV accepted paper repository.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.