Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation

Thank you for your response. I will carefully consider your clarifications, and if other reviewers also find the innovation of the paper to be sufficient during discussion, I will raise my score.

Dear Reviewer Zpyf, Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarize our responses here. 1. **[Difference between existing dynamic token pruning methods and the proposed method]** **Response:** - Our model learns where and how many tokens to skip. - Our model allocates computation in a data-dependent manner. - Our model preserves complete feature maps and can handle denste prediction tasks. - Our model aims to improve efficiency during cross-domain adaption. Since the discussion stage is already halfway through, may we know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments. Best Regards Authors

Dear Reviewer uJS2, Thanks so much for the support. We sincerely appreciate the valuable feedback. We will include all experimental results and analysis in the revision to facilitate future work. We will clarify the differences with other studies highlighted by reviewers in the revised paper. We hope our response resolves your concerns. We are more than willing to answer any further questions you may have. Your support is appreciated and helps us improve our work. Best regards Authors

Dear Reviewer fmWb, Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarize our responses here. > ### 1. [MoE improves performance only in video classification, not image classification, and is absent from most experiments] **Response:** - MoE-adapter is primarily designed to enhance the adapter’s capacity to address challenging tasks (Lines 180-183). - We also validate its effectiveness in semantic segmentation (Appendix Table 10) and object detection (In reponse to Reviewer uJS2). - We do not employ it in the VTAB-1K experiments as it does not outperform the model without it, likely due to the extremely limited training data > ### 2. [Table 6 shows that using MoE results in fewer FLOPs, which is theoretically implausible] **Response:** - The FLOPs of our models depend on **learned token dispather** during fine-tuning and may slightly **fluctuate around the target FLOPs**. - The extra computational cost of the adapter and the MoE adapter is nearly equivalent (Lines 202-204). > ### 3. [MoE might be better suited for the appendix] **Response:** - In the revision, we will move the introduction of the MoE adapter and its corresponding experiments to the Appendix. > ### 4.[The token dispatch mechanism is similar to existing dynamic neural network] **Response:** - DynamicViT and EViT prune a **certain number** or percentage of tokens in fixed layers, while our token dispatcher **learns the number of tokens to skip** before each MLP block. - DynamicViT and EViT maintain the **same FLOPs for all samples**, while DyT achieves **data-dependent computation**. - DyT to preserve **complete feature maps** > ### 5.[The paper should report the performance of DynamicViT and EViT's full fine-tuning and their combination with other PEFT methods] **Response:** - Combining DynamicViT and EViT with AdaptFormer enhances performance. - DyT outperforms “DynamicViT+AdaptFormer” and “EViT+AdaptFormer” by learning to skip an appropriate number of tokens at each MLP block. > ### 6.[It is unclear why MLP Dispatch has lower FLOPs than Layer Dispatch] **Response:** - The actual token activation rate during inference depends on the **learned token dispatcher**. - We set the activation rate $r$ to 0.5 for “Attention Dispatch” and “MLP Dispatch” variants, and to 0.7 for “Attention-MLP Dispatch” and “Layer Dispatch” variants, ensuring **similar** average FLOPs. > ### 7.[The authors should explain why their method outperforms traditional adapters] **Response:** - The dynamic architecture in DyT enhances generalization. - The distillation loss in DyT improves performance. - Dynamic architectures outperforming static models is also observed in previous works. > ### 8.[Discuss the increased training time caused by the need for two forwards] **Response:** - Two forward passes takes 1.8 times longer than only one pass - Given that our primary contribution focuses on improving inference efficiency, the additional training time would be acceptable. > ### 9.[The paper should include an ablation study on the loss function] **Response:** - Removing $L_{distill}$ or $L^\prime_{cls}$ negatively impacts performance. Since the discussion stage is already halfway through, may we know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments. Best Regards Authors

Dear Reviewer gv1w: We appreciate the reviewer’s feedback and would like to address the points raised below: ## Part1/2 **Compared with conditional adapter** |Question|Answer |-|- |"DyT selects an appropriate number of tokens in each layer, while the conditional adapter selects a fixed number."It is indeed different in this regard.| We appreciate that the reviewer acknowledges this contribution. The soft-topK token dispatcher serves as a core component in 'conditional adapter', which is indeed different from our approach. This verifies the distinction between our approach and 'conditonal adapter'. |"Conditional adapter is primarily designed for encoder-decoder models, while DyT is for ViT." Although experiments were conducted only on the encoder-decoder model, the conditional adapter is not limited to a specific model architecture, just like LoRA.| The 'conditional adapter' can be applied into ViT but does not work well. In our response, we conduct experiments and apply the 'conditional adapter' to ViT by replacing our token dispatcher with the soft-topK operation. However, DyT outperforms the 'conditional adapter' significantly, emphasizing the significance of our approach in ViT. |"Conditional adapter skips the whole layer, while DyT skips the MLP blocks." Similarly, although the placement is different, this alone does not provide sufficient novelty unless a theoretical or intuitive motivation for doing so can be provided.|In Vision Transformers (ViT), the self-attention and MLP blocks serve distinct roles in token mixing and feature projection, respectively.However, the impact of token skipping at different blocks on adaptation performance remains unexplored in previous works. Its potential impacts encourage us to explore. In our main paper (Lines 155-178), we analyze four variants, carefully considering their advantages and disadvantages. Additionally, in Figure 6 (Appendix), we conduct experiments to explore this and demonstrate the superiority of the “MLP dispatch” method. These analyses and experiments underscore **the non-trivial nature of the placement of token skipping** and further validate the significance of the exploration in DyT. **Compared with AdaMix** |Question|Answer |-|- |"AdaMix lacks a learning-based router and fuses experts" There is still a lot of work being done on MoE adapters. For example, MoLoRA [1] uses a learnable router as an adapter and does not fuse experts.| The MoLoRA [1] is adopted to reduce the additional parameters when converting a dense model into a Mixture of Experts (MoE) model. Our MoE-adapter aims to improve the performance of DyT on challenging tasks, while avoiding introducing additional computational cost. Notably, our MoE-adapter adopts weighted summation at both down-projection $W_{down}$ and up-projection $W_{up}$ in adapters, in contrast to MoLoRA [1], which aggregates all LoRA blocks. Hence, our architecture is also different from MoLoRA [1]. While MoE is a widely used and effective technique in previous works, our MoE-adapter offers a unique design and is **tailored to improve the performance of DyT**. Finally, we would like to clarify that we pay more attention to achieving both parameter and inference efficiency for ViT adaptation and the MoE-adapter is **not the main contribution** in our work. **Compared with DynamicViT and token compression** |Question|Answer| |-|- |"DynamicViT and token compression methods prune a certain number or percentage of tokens in fixed layers, and cannot complete feature maps" This issue existed in early token compression work, but later efforts, such as DiffRate, have addressed it. Combining token compression methods with existing PEFT approaches, such as AdaptFormer and LoRA, could serve as a strong baseline.| In the Diffrate model, token pruning is **inherently data-independent**. After training, a transformer layer **prunes or merges the same number of tokens across all input data**. Additionally, the prune/merge operation **can not preserve complete feature maps**, which poses challenges for dense prediction tasks. In comparison, **DyT is a data-dependent approach**. The router in DyT learns to skip varying numbers of tokens before each MLP block based on the input data. The visualization in Figure 8 and Figure 9 (Appendix) demonstrate that the learned token skipping strategy in DyT **selects different numbers of tokens to skip based on different inputs**, which is more flexible and reasonable. By avoiding prune/merge operations and only conducting token skipping, DyT maintains complete feature maps, enabling it to **effectively handle dense prediction tasks**. Furthermore, in our response to Reviewer fmWb, we also explore combining token compression methods with AdaptFormer together, and find that our method can surpass them obviously, verifying the superiority of designs in DyT.

## Part2/2 In addition to distinguishing DyT from existing works, we summarize the contribution of our work below: - **Verify the feasibility:** The proposed method DyT proposes to learn a data-dependent architecture in a parameter-efficient manner during the ViT adaption. It verifies the feasibility of achieving both parameter and inference in ViT adaptation, which may direct an avenue for efficient fine-tuning and adaptation. - **Elaborate design:** To achieve better efficiency and performance, DyT explores the optimal placement of token skipping (Section 3.3 Model variants), through extensive analysis and experiments. We also design a MoE-adapter to improve the performane of DyT in challenging tasks. - **DyT can handle various adaptation tasks:**. DyT achieves both parameter and inference efficiency in various tasks, including image classification (Table3 and Table6), video classification (Table 6), semantic segmentation (Table 10), and object detection & instance segmentation (Our response to Reviewer Reviewer uJS2).