Summary
The paper contributes a new prompt-tuning framework to adapt pre-trained models to new downstream tasks using a mixture of expert methods. For this, authors utilize a dynamic token dispatching strategy at different prompt layers to optimize the contribution of each domain expert. They conducted a wide range of experiments on both general and medical domains, including classification, segmentation, etc, and demonstrated improved results overall.
Strengths
**Methodology**:
- Ideas of using a mixture of experts to combine different kinds of prompt tuning are interesting ideas.
- The author also proposes using dynamic gating to selectively choose and integrate tokens from various experts, which is essential to (1) allow communication among experts for better information exchange and (2) determine which information should be retained.
**Experiments**:
- Reviewers appreciate the efforts of the paper to conduct several experiments and compare with several relevant models. Ablation studies on Expert Prompt Tokens and Dispatcher type, etc.
- Figure 2 illustrates how the mixture-of-expert path is interesting in understanding how the model likely works in practice.
Weaknesses
The reviewer found the following major weaknesses in this paper:
**Methodology**:
- The writing part for Section 3.3 is very convoluted and is difficult to grasp ideas beyond. For e.g., the following sentence is *These tokens are the added EPTs for all experts $P^{l}$, accumulated prompts from the last layer $Z_{P, expert_{k}}^{l}$, and patch tokens of the current expert $Z_{expert_{k}}^{l}$*. So the question is, what is **these** here? and what is exactly the equation for using $\hat{P}_{expert-k}^{l}$? Will it be used to define the matrix $P^{l+1}$ for the next layer?
- Another important question is what the Expert K represents. Does it mean the k-th domain expert is a specific pre-trained model? (like what authors write below equation (3) with DINO and LVM-Med), or each k-th domain expert is just another prompt vector?
- Figure 1 is also confusing. For example, Reviewers don't understand the block output of Dispatcher Layer l, which has several rectangles and colors that are not defined in the notion. Furthermore, why do we have the same notations for the two first rectangles for Expert $k$, Expert $1$, and Expert $K$, which contradicts the equation (5)?
- The authors mentioned the *Mixture of Expert Prompt Tokens* and *Mixture of Experts Prompt Layers*. While the first term is presented in the paper, there is no discussion about *Mixture of Experts Prompt Layers*.
In summary, the lack of details and unclear presentation in Section 3.3 about how the *dispatch* and the meaning of *expert* make the Reviewer unable to follow and understand contributions in experiments.
**Experiments**:
- The first and also the most important questions authors did not provide in their experiments are *how many experts are used in their settings*, i.e., the number of $k$? and *what are they (in terms of architectures)?* It is crucial to understand and analyze the results in the reported table.
- For instance, in the Medical Domain Segmentation task (line 416), authors follow the setting of [1], which integrated two ViT pre-trained models, one from the general domain and the other from the medical one. Given this, the question is how do authors adapt their method in this case with two pre-trained models? This question is indeed relevant to the above question (methodology) about the meaning of $k$-th domain expert. Do you choose K=2 for this case, or does each pre-trained ViT have different K-experts?
- **Repeat the same baselines with different results**: In particular., in Tables 2, 3, 4, and 5, why do baselines like VPT, GaPT, and LSPT repeat two times, but their total params and other results are different? The reviewer could not find experimental description information for these results, which raises a concern about the accuracy of baseline results.
- Another problem with Table 9 in Ablation studies. There are two columns *Model 1* and *Model 2* in the Table. Does that mean that the first expert is *Model 1* and the second expert is *Model 2*, or k = 2?
[1] Shentong Mo, Xufang Luo, Yansen Wang, and Dongsheng Li. A large-scale medical visual task adaptation benchmark, Arxiv 2024.
Questions
As mentioned above, the Reviewer doesn't understand the right setting of the expert-k (and how you apply it to a single model and different models) and how dispatch learning works, which causes disconnections in most experiment parts. So can you please provide a details explanation for those questions as well as some other things in experiments (repeat the same baseline with different numbers)?