Model Spider: Learning to Rank Pre-Trained Models Efficiently

Summary

This paper investigates how to select the most suitable PTM given a target task efficiently and accurately. A novel approach called Model Spider has been proposed. It learns to encode both PTMs and tasks into vectors and measures their similarity, which is further used to rank the PTMs. It can also incorporate task-specific forward results of PTMs for more accurate re-ranking when resources budgets allow. Extensive experiments have been conducted to verify the effectiveness of the proposed method.

Strengths

1. The idea of encoding PTMs and downsteam tasks into vectors for PTM ranking is well-motivated. 2. The overall presentation of the proposed method is well-organized and generally easy to follow.

Weaknesses

1. The generalization ability of the proposed method has not been well-verified. The supervised training method is known to result in models with poor generalization ability. It seems that the proposed method heavily relies on the frozen encoder $\psi$ to capture the relevance between different tasks. What if the new task is quite different from the tasks used in training? When evaluating the proposed method, all the downstream tasks are about image classification. The authors should evaluate the generalization ability of the proposed method with more diverse downstream tasks. 2. The proposed method is highly related to the classification task as it uses the class centers as the task token in Equation (5). However, many tasks do not have such "classes", such as regression tasks and generation tasks. How to adopt the proposed method to such tasks has not been elaborated. 3. Only 10 PTMs are used when evaluating the proposed method on a single-source model zoo, which is not enough. It is recommended to evaluate the proposed method in the NLP domains, where there are numerous pre-training models and diverse downstream tasks, which can be used to better verify the generability of the proposed method to a new task.

Questions

In addition to the questions above, I have the following questions: 1. What's the motivation for using Equation (4) to train the model to capture the ranking order? There are many other methods that can train the model to learn to rank, such as the pairwise BPR and listwise ListMLE. It is recommended to verify the impact of different ranking loss functions. 2. Does the proposed method require re-training whenever a new PTM comes? If so, the proposed method seems costly when applied in real-world scenarios.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

No, the authors have not discussed the limitations of the proposed method. The authors are recommended to explain whether the proposed method is limited to classification tasks and how to adapt it to other tasks.

Summary

This paper introduces Model Spider, a unique method to efficiently and accurately rank Pre-Trained Models (PTMs) for a specific task within a model zoo. Model Spider innovatively creates tokens for both PTMs and tasks, encapsulating their characteristics in a manner that facilitates an efficient selection process. It utilizes a separate set of training tasks to learn how to construct these tokens and calculate the fitness score between a model-task pair. The paper also presents a strategy to update the task tokens based on the semantics specific to the top-ranked PTM candidates, improving the final selection. The key contributions of this paper are the innovative method of tokenizing tasks and PTMs for easy ranking, and the ability of the system to incorporate task-specific forward results of certain PTMs within resource limitations. Through rigorous testing across various model zoo configurations, the authors demonstrate the efficacy of Model Spider, showing significant improvements in PTM selection and efficiency. The work represents an innovative solution to the challenge of sifting through the proliferating number of PTMs to find the most suitable model for a given task.

Strengths

1. Proposed methods show quite significant improvements over the strong baselines. 2. Methodology of tokenizing PTM looks new. PTMs are tokenized by mapping them to unsupervised trained task tokens. The method looks sensible. 3. Applying learning to rank to rank the model fitness is also new and interesting. 4. Extensive experiments with ablation studies.

Weaknesses

1. “Hyperparameter k” in Figure 1 is not explained at all. In the main text, notation k is also not clearly defined, only top-k mentioned here and there. Readers have to guess k means the number of top ranked Pretrained Models. 2. Some important results and discussions are included in the Appendix, but no reference in the main text. Please refer readers to the appendix for the useful information. 3. Figure 3 is a bit hard to read. What information should I draw from the figure, besides correlation number?

Questions

1. How good can the method be applied to the pre-trained large language models?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1. Good tokenization of pre-trained models (PTMs) depends highly on the number of high-quality PTMs. How the Model spider perform when the number of PTMs M and the number of tasks varies? 2. The training of Model Spider depends on the RankAgg. As authors explained (in the appendix), RankAgg introduces a significant computational burden. How could one get a RankAgg model to be used for training Model Spider in the first place?

Summary

This paper introduces a very interesting approach named "model spider", to address the challenging problem of selecting suitable Pre-Trained Models (PTMs) from a large number of options to fit the target tasks. Instead of relying on time-consuming and computationally heavy forward or backward passes over all PTMs, the model spider tokenizes both PTMs and tasks, summarizing their characteristics into vectors for efficient PTM selection. Experiments show that model spider performs well in various configurations of model zoos, providing a balance between efficiency and selection accuracy.

Strengths

1. The problem and idea of this paper are quite interesting. 2. The proposed method is simple yet effective. 3. The results are quite encouraging.

Weaknesses

1. The results are tested on vision tasks. It is not clear whether the proposed method can be generalized to tasks in other modalities. 2. The model size used in this work is relatively small (only up to tens of millions of parameters). It is not clear whether the proposed method can handle larger models, such as ViT, BERT, and GPT. It would be more exciting if larger models (especially large language models) can be easily evaluated. 3. The proposed method still requires some samples to train on new tasks. It would be better to consider using some methods (e.g., meta-learning) to generate initial tokens for tasks so that new tasks can be handled without additional training samples. Minor: The term "token" is a little confusing since the meaning of token in this paper is different from the common meaning of tokens in PTMs.

Questions

See weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have discussed the limitations.

Summary

This paper proposes a method to select the "best" pre-trained model for a given task. This problem is important given the large number of available pre-trained models. The key behind Spider relies on tokenizing both the models and the tasks by summarizing their characteristics into vectors. More specifically, the authors use a general encoder and measure the similarity between tokens in a supervised manner: the ranking of models are obtained through some historical tasks. Normally, one would take a list of pre-trained model, (optionally) freeze the feature-extractor part, add a randomly-initialized head on top, fine-tune on the dataset, and then measure their transferability. This process is computationally intensive. The proposed model Spider first randomly sample training tasks and assume that we can compute the transferability for M pre-trained models and thus, their ranking. Given this dataset, the model is trained to learn a similarity function to mimic this ranking. The only features that are used are task tokens and tokens from the models. A model token consists of a representation to reflect how good the pre-trained model is in general. A task token is an embedding that represent a class in a dataset. Finally, different re-ranking strategies are proposed with efficiency-accuracy trade-off. Transferability assessment is not my domain. Nevertheless, the proposed method seems sound to me. The experiment section is really complete and includes many datasets and baselines for single-source and multi-source model zoo. Finally, the ablation study emphasizes the importance of RankAgg. Overall, the paper is well written, the approach seems novel, and again, given that this topic is not my domain, I don't see any reason to reject this paper.

Strengths

- Efficient model selection method for a given task - The method is novel - Good performance in the experiment section

Weaknesses

- Some concrete examples of Task & Model tokens would be appreciated

Questions

I have no question

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

There is not a limitation section.

Summary

The paper proposes a method called MODEL SPIDER for selecting the most suitable pre-trained models (PTMs) for a given downstream task. The proposed method aims to maintain a balance between efficiency and accuracy in the selection of PTMs. To achieve this, the authors tokenize all PTMs and tasks into vector representations that capture their general properties and their relationship with each other. During the training process, they dynamically select a partial set of PTMs and incorporate the specific tokens into the sampled tasks. During deployment, the authors employ a coarse-grained PTM search to narrow down the candidate PTMs and then fine-tune the selected PTMs for downstream use. The proposed approach is evaluated on several benchmark datasets, and the results demonstrate that it outperforms existing PTM selection methods in terms of efficiency and accuracy. As part of the analysis of the effectiveness of the proposed method, the authors conduct ablation studies. They find that incorporating PTM-specific features and prompts improves the performance of the proposed method significantly.

Strengths

* This paper presents a clear and well-motivated problem statement in the Introduction: how to select the best pre-trained model (PTM) for a given task. The approach section introduces the necessary preliminary and explains almost parts of the proposed method in detail. * The method is novel and interesting, as it constructs tokens for both the PTMs and the target task, and then measures their similarity to find the optimal match. This way, the method can leverage the rich information encoded in the PTMs and adapt it to different tasks. * The evaluation is comprehensive and detailed, covering 10 PTMs from five architectures and 9 downstream datasets for classification and regression tasks. The paper also compares the method to several strong baselines and shows that it outperforms them in selecting the most suitable PTM for each task

Weaknesses

* The fitness function is a neural network that maps the PTM and task tokens to a scalar score, but the paper does not specify how this score is calculated or which threshold is compared with. * The paper lacks details on the design and training of the task encoder, which is a crucial component of the method. In line 42-43, the authors mentioned that they use a Transformer module and refer to the reference [72] "Attention is all you need". In my opinion, this is an important detail to explain how the tokenization process is performed. * The paper does not explain how the authors handle noise and irrelevant data in the task tokenization process."

Questions

Do we need to represent all data instances, or can we use a sampling strategy to select the most representative examples from the data for encoding?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper does not include a limitation section that explicitly discusses the drawbacks or challenges of the proposed approach.