Foundation Model is Efficient Multimodal Multitask Model Selector

Summary

This paper proposed an efficient multi-task model selector (EMMS) to address the inapplicability in a multi-modal multi-task scenario. Specifically, the proposed method achieves a new state-of-the-art in performance and speedup through the incorporation of design elements such as the F-label, Weighted Linear Square Regression, Fast Computation by Alternating Minimization.

Strengths

1. The motivation for addressing “a unified representation to represent diverse label formats” is clearly presented and validated. And use foundation model is an impressive way. 2. The detailed derivation and experiment in this paper are comprehensive, providing strong evidence of the validity of the proposed model.

Weaknesses

1. It is recommended to include a complete demo in the code address of the paper, rather than a short python file. 2. The phrase "5.13×, 6.29×, 3.59×, 6.19×, and 5.66× speedup in wall-clock time" appears to describe the speed efficiency of the EMMS (One) method. Could you clarify why EMMS's performance varies, being faster than LogME in some instances and slower in others, as indicated in Table 1? 3. In Table 1, there seems to be some ambiguity about the effect of F-label on transferability assessment, since the perforamnce of the EMMS(one) and EMMS is identical in some rows. 4. It is recommended that the paper include a broader array of foundational models for F-label. Is CLIP currently the model with the highest effects?

Questions

na

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

na

Summary

This paper focus on an under-explored problem of estimating neural network transferring capability without actually fine-tuning the multi-modal multi-task model on individual downstream tasks. The problem is well-motivated and is of great practical importance. The solution proposed in this paper is straightforward, which essentially is to encode the target label (in the text form) with a foundation model. The embedding of the label text is then treated as the target and a model's transferability can be estimated through a simple weighted linear regression, which is then solved by an alternating minimization algorithm. The paper experiments with 5 downstream tasks and 24 datasets, and show that the proposed estimation method (EMMS) is fast and effective on most tasks.

Strengths

1. The task is practically important and under-explored in previous work. As the authors mentioned, the previous estimation is usually limited by the classification tasks and the proposed work is more flexible as it directly encodes the label as a text sequence, therefore it can be used for multiple different types of tasks. 2. The proposed method is intuitive and straightforward, and the algorithms for solving the problem are clear. The theoretical proof looks reasonable although I didn't carefully check all the details in the equations. 3. The approximation with algorithm significantly speed up the computation when estimating the performance. 4. This paper performs extensive experimentations by fine-tuning the expensive models on many downstream tasks (to obtain ground truth) and the proposed approach shows superior performance over other methods with regard to correlation and the wall clock speed.

Weaknesses

1. The Figure 1 bottom is not very informative -- the radar chart is most effective for showing multivariate data with quantitative variables, but in this setting the variables are all binary (applicable/inapplicable), therefore it's actually carrying limited information. Probably replace the figure with a table of checkbox to illustrate the advantage of the proposed approach. 2. As I mentioned in the limitations, some important assumptions are not validated in the paper. Specifically the key assumption that directly leads to the weighted linear square regression problem and alternating minimization algorithm is the linear mapping from model feature space to the F-label space. More justifications about the assumptions are needed. 3. I'm also concerned about the choice of evaluation metric of weighted Kendall's $\tau_w$. If I understand correctly, as a ranking metric it focus more on the relative orders of the elements in the rank, but ignores the actual number. In other words for A>B>C>D, it doesn't care whether A is just marginally higher than B or is much higher than B. It seems wasteful since we already obtain the ground truth downstream task model performance through expensive fine-tuning, but with the $\tau_w$ metric only the relative orders of the ground truths are useful. Therefore I'm wondering if it's possible to directly estimate the actual performance. I might be wrong or missed some important considerations so any clarifications or rationales are appreciated. 4. Some minor issues: in the tables with actual results, some numbers are bolded as the best performance, but in many cases there is a tie between two numbers in the table. Probably it's better to highlight both.

Questions

I'm curious if there are other metrics that can be used to compare the proposed methods with other approaches, e.g. other than $\tau_w$ and wall-clock time. For the former as I stated in the weaknesses section, it seems that it only cares about relative orders of candidate pre-trained models; for the latter the wall-clock time is easily affected by multiple factors (e.g. multi-threading, concurrent execution of other tasks in multi-core environments, etc.) I would like to learn more about the rationale for choosing these metrics. Also it seems in this paper only the linear mapping is considered and it is used as an assumption without further justification. This may need more clarifications as one can easily come up with alternative approaches for estimate the mapping from features to label embeddings.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

This paper has a dedicated section to discuss the limitations and I agree with the authors that the quality of the foundation model (which is mainly responsible for the encoding of the label text) plays critical roles in the effectiveness. However I think there exists some other limitations which are also worth mentioning. Specifically, some important assumptions in this paper is in L164 and L168, where the label embedding is assumed as a linear mapping of the model feature. This assumption seems harsh and I think a more thorough discussion about the assumption will be useful. There is no obvious potential negative societal impact with this paper.

Summary

This paper proposes to utilize large-scale foundation models for efficient multi-task model selector (EMMS). Concretely, the authors utilize foundation model to transform different label format (category label, text, bounding boxes) into unified noisy label embeddings. EMMS then could measure the compatibility between the models’ features and corresponding 59 label embeddings via weighted linear regression. Experiments on 5 downstream tasks with 24 multi-modal tasks shows the proposed method's effectiveness and efficiency.

Strengths

- the motivation of this paper is important: investigating how to evaluate foundation model on a set of multi-modal tasks without finetuning all the target tasks. - the writing is clear and easy to follow. - the authors also provide code for reproducibility. - The experiments show improvement on the 5 downstream tasks with 24 datasets.

Weaknesses

As the authors mentioned in the limitation section, the proposed method is bottlenecked by the capabilities of the chosen foundation model. One could further ask, if the foundation model is good enough, why don't we just use the foundation model for the intended downstream tasks? For example, use CLIP for image classification?

Questions

please refer to the weakness section

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Summary

This paper introduces EMMS, an efficient multi-task model selector for predicting the performance of pre-trained neural networks on multi-modal tasks without fine-tuning. EMMS employs large-scale foundation models to transform diverse label formats into a unified noisy label embedding. Through weighted linear regression and an alternating minimization algorithm, EMMS accurately estimates transferability. Experimental results demonstrate superior performance and significant speedup compared to existing methods.

Strengths

1. Generic Transferability Estimation Technique: The proposed method, Efficient Multi-task Model Selector (EMMS), offers a generic approach to estimate transferability. By utilizing a unified label embedding derived from foundation models and employing a simple weighted linear square regression (WLSR), EMMS becomes a fast and effective method for assessing the transferability of pre-trained models across different tasks. 2. Novel Alternating Minimization Algorithm: The paper introduces a novel alternating minimization algorithm specifically designed to solve the WLSR problem. This algorithm ensures efficient and accurate estimation of transferability within the EMMS framework. 3. The authors did extensive experiments across different tasks and the results demonstrate the efficacy and effectiveness of the proposed method.

Weaknesses

1. The proposed method is complicated and difficult to follow. I can barely understand how to use WLSR to maximize the log-likelihood, which may bring extra complexity for re-producing the method. 2. One of the simple baselines could be estimating the mutual information between the $\hat{x}$ and $Z$. I doubt the effectiveness of this baseline, but it could be nice to have. 3. More experiment details need to be provided like, what are the foundation models used in different tasks. Another ablation could be studying the effectiveness of different foundation models. I guess CLIP could be the strongest model to use in those tasks if we just use one single foundation model. Minor: Line 111, `denote finetuning score`. Please provide more description of Figure 3 (b) as it is not straightforward to understand the confusion matrix of image captions.

Questions

Please address the comments in weakness section.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

n/a

Summary

This paper introduces the Efficient Multi-task Model Selector, which utilizes foundation models to convert diverse labels into unified label embeddings. These embeddings are then used to calculate a transferability metric within a weighted linear square regression (WLSR) framework. The proposed method achieves impressive results in multi-task scenarios.

Strengths

1. The paper presents a novel and unified strategy for model selection. The use of label embeddings enables the capturing of fine-grained semantics, leading to improved estimation in downstream tasks. 2. The effectiveness of the proposed method is supported by a comprehensive set of experiments covering various downstream tasks. 3. The derivation of equations is provided, establishing a solid foundation for the main contribution of the paper.

Weaknesses

No questions.

Questions

No questions.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No questions.

Did not receive responses from Reviewer vJKA, Reviewer z1Vu, and Reviewer 4U2K so far

Dear ACs, We thank all reviewers for their valuable comments, which have substantially improved our work. We highlight **the contributions of EMMS** as follows. - Our work tackles an important problem of model selection in transfer learning. The proposed EMMS is the first transferability estimating method which can be used in multimodal multitask scenarios. - Equipped with a unified label embedding provided by foundation models, we show that multitask model selection can be simplified into a problem of weighted least square regression (WLSR). We propose a novel alternating minimization algorithm to solve WLSR efficiently with theoretical analysis. - Extensive experiments on 5 downstream tasks with 24 datasets demonstrate the effectiveness of EMMS. Specifically, EMMS achieves 9.0%, 26.3%, 20.1%, 54.8%, and 12.2%, performance gain on image recognition, referring, captioning, visual question answering, and text question answering, while bringing 5.13×, 6.29×, 3.59×, 6.19×, and 5.66× speedup in wall-clock time compared with the state-of-the-art method LogME. Reviewer vJKA, Reviewer z1Vu, and Reviewer 4U2K raised some concerns about our manuscript, to which we have added sufficient experiments as well as detailed explanations during rebuttal. Unfortunately, we have not received any feedback from Reviewer vJKA, Reviewer z1Vu, and Reviewer 4U2K so far. **Reviewer vJKA's** suggestions include: (1) conducting robustness analysis experiments for the effectiveness of the single foundation model. (2) adding a baseline that measures the performance using mutual information, and (3) providing a clear description of the WLSR process. In response to these suggestions, we (1) provide detailed explanations of our method EMMS, (2) compare EMMS with TransRate which is a transferability estimating method based on mutual information, and (3) investigate the effect of each single foundation on EMMS. **Reviewer z1Vu** raised a crucial question regarding why we don't directly use the foundation model for all tasks instead of performing the model selection. We addressed this concern by noting that EMMS could be a useful tool for task completion as foundation models still achieve suboptimal performance on certain tasks. Moreover, foundation models are computationally expensive and require significant computational resources for training and deployment. **Reviewer 4U2K** requested (1) further experiments that involve a broader range of foundational models for F-Labels, echoing the issue raised by Reviewer vJKA, (2) the assessment of computational complexity of EMMS, and (3) improvements to the open-source code. In response, we (1) explain the time complexity differences between EMMS and LogME and highlighted the importance of F-Label for enabling the scalability of EMMS to multiple tasks, (2) conduct more experiments to demonstrate the enhancing effect of F-Label on EMMS, and (3) release the open-source code. We believe that these explanations and additional experiments can answer their questions well. Hope our clarification helps you make a decision about our work. Best Regards, NeurIPS 2023 Conference Paper5953 Authors