The growing popularity of transfer learning, due to the availability of models pre-trained on vast amounts of data, makes it imperative to understand when the knowledge of these pre-trained models can be transferred to obtain high-performing models on downstream target tasks. However, the exact conditions under which transfer learning succeeds in a cross-domain cross-task setting are still poorly understood. To bridge this gap, we propose a novel analysis that analyzes the transferability of the representations of pre-trained models to downstream tasks in terms of their relatedness to a given reference task. Our analysis leads to an upper bound on transferability in terms of task-relatedness, quantified using the difference between the class priors, label sets, and features of the two tasks. Our experiments using state-of-the-art pre-trained models show the effectiveness of task-relatedness in explaining transferability on various vision and language tasks. The efficient computability of task-relatedness even without labels of the target task and its high correlation with the model's accuracy after end-to-end fine-tuning on the target task makes it a useful metric for transferability estimation. Our empirical results of using task-relatedness to select the best pre-trained model from a model zoo for a target task highlight its utility for practical problems.
Paper
Similar papers
Peer review
Summary
This paper considers transfer learning in a general cross-domain, cross-task setting. It introduces the concept of "task-relatedness" into measuring whether transfer learning can be successful. Specifically, task-relatedness consists of (1) a reference loss term for measuring the difference in class prior distribution between the target and source tasks; (2) a label mismatch term for measuring the conditional entropy between the label distributions; and (3) a distribution mismatch term for measuring the difference in feature distributions. To make computing task-relatedness practical, the paper proposes several approximation schemes. It shows the metric is highly correlated with model transferrability in practice and can be used for selecting pretrained models.
Strengths
1. Practical significance: The topic studied is important to the field of machine learning, as fine-tuning large-scale pretrained models gradually becomes the to-go option for model development. The idea of decomposing transferability into prior, feature, and label distribution differences is clean and interesting. 2. Complete pipeline: the paper not only proposes task-relatedness as a transferability metric but also shows how it can be transformed into a training objective for effective transfer learning. This makes the paper more comprehensive and the proposed method more valuable. 3. Presentation is clear and code for reproducing the experiments is provided
Weaknesses
1. Limited novelty: the idea of using distribution distances to measure transferability is not new. As the related work section has pointed out, using the Wasserstein distance to estimate domain difference has been studied by OTDD [1]. Works like OTCE [2] also uses the proposed metric to do pretrained model selection. One argument the paper makes is that the proposed task-relatedness works without requiring target labels. While this does add some practical significance, the proposed method is a simple pseudo-labeling scheme without theoretical justification. 2. Limited evaluation: the baselines considered in the experiments are not comprehensive, e.g., OTCE is not evaluated, and other recent OT-based works like [3] are not discussed. There's also no comparison for different pseudo-labeling schemes for the missing-label setting. Besides, there's no ablation study showing the contribution of each of the three terms proposed for computing task-relatedness. [1] Geometric Dataset Distances via Optimal Transport. David Alvarez-Melis. Nicolò Fusi. [2] OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations. Yang Tan, Yang Li, Shao-Lun Huang. [3] Wasserstein Task Embedding for Measuring Task Similarities. Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea Soltoggio, Soheil Kolouri.
Questions
1. Class-prior transformation assumes the reference task has more classes than the target task. Is this too strong an assumption? Does the proposed method applies to the settings where the target task has more labels? 2. In Figure 3, the rankings of transferability for different target tasks are the same across all models. This is perhaps because the same reference dataset is used, and the experiments do not show the effect of varying model backbones. It would be better if the paper can evaluate multiple reference datasets on multiple models. (Figure 4 does look at different reference datasets but the model is fixed here.) 3. Does the proposed metric applyy to non-classification tasks like regression or auto-regressive prediction? 4. What's the max cross-domain distance the theory holds? I’d imagine that when the domain gap is large, the theory breaks. Have you studied that?
Rating
5
Confidence
2
Soundness
2
Presentation
3
Contribution
2
Limitations
See weaknesses.
Summary
This paper proposes an analysis that analyzes the transferability of the representations of pre-trained models to downstream tasks in terms of their relatedness to a given reference task. It aims to understand when the knowledge of these pre-trained models can be transferred to obtain high-performing models on downstream target tasks. Their analysis leads to an upper bound on transferability in terms of task-relatedness, quantified using the difference between the class priors, label sets, and features of the two tasks. The experiments results demonstrate the effectiveness of the proposed method.
Strengths
(i) This work analyzes transferability for classification tasks and provides the first upper bound on transferability in terms of task-relatedness in a cross-domain cross-task setting. (ii) This work proposes an optimization problem to compute task-relatedness using a small amount of target labels and shows that it can even predict performance after end-to-end fine-tuning without requiring target labels. (iii) This work conduct multiple experiments to demonstrate the effectiveness of the proposed method.
Weaknesses
(i) This work mentioned that it is the first to analyze transferability for classification tasks under cross-domain cross-task settings. However, there are multiple works [1-5] that have focused on the transferability and generalization of learned representations. Meanwhile, transferability is mainly evaluated by utilizing the performance differences between different domains and tasks with distribution shifts. This is also the case in previous works, but this is not the case for the introduction of motivation in this paper.\ [1] Understanding few-shot learning: Measuring task relatedness and adaptation difficulty via attributes.\ [2] Otce: A transferability metric for cross-domain cross-task representations.\ [3] On the theory of transfer learning: The importance of task diversity.\ [4] An information-theoretic approach to transferability in task transfer learning.\ [5] Task relatedness-based generalization bounds for meta learning. (ii) (Minor) Although this work focuses more on theoretical exploration, empirical verification on real-world datasets is also necessary, such as ablation studies on the three proposed terms, trade-off experiments (performance vs. efficiency vs. memory footprint) due to the introduction of additional computations, and comparisons with baselines on transferability. (iii) (Minor) The formatting and layout of some formulas and tables need to be adjusted to make the reading clearer and more intuitive while some equations with writing errors, such as Eq. 3, L527-528, the proof of Corollary 1. At the same time, due to the large number of formulas and physical quantities, it may be better to construct a table to explain the symbols and definitions. (iv) (Minor) The track chosen by the paper is not that matched to its content.
Questions
Please see Weaknesses.
Rating
5
Confidence
3
Soundness
2
Presentation
3
Contribution
3
Limitations
N/A
Response by Authors
We thank the reviewer for their response to the rebuttal. While we agree that transfer learning is a broad research topic, our paper specifically works in the inductive transfer learning setting where the focus is to leverage an inductive bias (a pre-trained model) to improve performance on a target task. Due to this, our analysis specifically proposes a model-dependent way of measuring the relatedness between a reference and a target task. The works such as [1,3,5] provided by the reviewer deal with the problem of task transfer learning. Their goal is to identify the relationship between tasks, regardless of the model, to explain the transfer performance. E.g., the task attribute distance proposed by [1] is independent of the model being used and only depends on the tasks. (This terminology of inductive and task transfer learning is based on a previous work LogMe[54] which has the same setting as ours.). Thus, the setting of our work differs significantly from the setting in [1,3,5]. We thank the reviewer for bringing this up. We will add a discussion to the paper and clarify the type of transfer learning being studied in the paper along with the mention of the relevant papers from the area of task transfer learning. We would be happy to address any further concerns the reviewer may have.
Summary
This paper analyzes transfer learning from the perspective of task relatedness, a model for transforming from a reference task (pretraining task) to the target task is proposed, which consist of prior transform, label transform, and feature transform. The task relatedness is then measured as the distribution mismatch of the transformed distribution and the target distribution. Experiments have shown that the proposed task relatedness tightly upperbounds the transferrability to a range of architectures and datasets.
Strengths
1. The motivation and the derivation of the proposed task relatedness make senses to me. 2. The empirical results have shown the advantage of the proposed task relatedness.
Weaknesses
1. The datasets for evaluation and empirical studies seem to be small in scale, for example, MNIST and CIFAR which is small in the image resolution, or Aircraft and DTD which is small in the number of images.
Questions
The appendix has answered most of my questions. My only question is that for algorithm 1, does this has guaranteed convergence? and practically how long does it take to reach convergence? Minor: A possible related work: Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective, ECCV 2022.
Rating
6
Confidence
4
Soundness
4
Presentation
4
Contribution
3
Limitations
The paper has discussed its limitations
Summary
The paper presents a way of assessing the transferability of representations from a pre-trained model to a target task by assessing the impact of the pre-trained model's representations on a reference task. By "transforming" the reference task into the target task, the paper produces a bound on the training loss of the target task
Strengths
1. The paper presents both theoretical and empirical analysis of a problem 2. Some of the provided experiments are reasonably comprehensive 3. The idea of tackling the transferability problem by introducing a reference task seems novel AFAIK
Weaknesses
My main issues with this paper are as follows. 1. **Utility of the method** : I was initially excited by the following sentence in the introduction of the paper -- *"Moreover, unlike previous SbTE metrics, task-relatedness can be estimated even without labeled target data, making it suitable for unsupervised transferability estimation, highlighting the advantage of a reference task as used in our analysis"*. However, it seems that Algorithm 1 requires access to the target labels $\mathcal{Y}_{T}$. This calls into question this stated advantage over the SbTE methods. Specifically, SbTE methods like TransRate and LEEP do not require training on a reference task which not only comes with extra overhead but whose selection requires a-priori meta-intuition about which tasks are similar to the target task. Also, none of the empirical evaluations presented actually show that the proposed method is superior to SbTE approaches for choosing models form a zoo. Specifically, the paper presents hard to interpret correlation measures in Table 1 as proof of transferability -- when a superior test would be compare deltas in final training performance of the models the models that each method select from a zoo. 2. **Correctness / Impact of claims**: [a]. From line 238 - 241: *However, since finding data semantically related to the target task may not always be possible we choose a reference task with the same number of classes as the target and fix that matrix B to a random permutation of identity (making the label mismatch term zero) and D to the prior of the reference task, learning only the transformation A, in our experiments.*. This seems to undercut the utility of the analysis presented. Why does this work ? Why even discuss these terms if they are effectively fixed ? [b]. It feels like the insight of *Highly related reference–target task pairs, based on task-relatedness, achieve higher transferability coinciding with the semantic relatedness between tasks.* is just punting the problem of transferability estimation to task similarity estimation (and now calling on practitioner intuition to decide which tasks are highly related). Thus it does not seem that the authors are actually effectively solving the problem of transferability estimation -- they are just transforming it into a different problem. [c]. The authors claim "tight bounds / small gaps between task related and transferability" but this is based on (imo) just eyeballing the graphs in Figure 3 and 9. We are not provided with a rigorous way to interpret the gap. Eg -- a very simple way to artificially present a small gap is to just plot the the losses at random init of the classification head for the the reference and target tasks that have the same number of class labels. 3. **Strength of underlying technical assumptions**: [a]. the K_r > K_t seems like a strong assumption that limits the methods applicability. What if the most similar task to the target task has K_r < K_t. [b]. Also as mentioned above in 238 - 241 it seems the label and class-prior matching component of the algorithm effectively has no influence on the final algo results. Score updated after discussion
Questions
1. Work focuses on classification based tasks -- could you discuss how this applies to generative tasks 2. Can you discuss how faithful the estimates / method is when K_r < K_t ? 3. The analysis is for training error -- how is transferability estimation affected if there is overfitting to the reference task ?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
N/A -- paper presents some limitations
Response by Authors
We thank the reviewer for their response to the rebuttal. > Are ... of B. The reviewer’s intuition is correct that the pseudo labels are computed using the $B$ matrix provided at the initialization of Alg. 1 for the first time. But as Alg. 1 progresses the matrix $B$ will also be updated. As a result, the pseudo labels of the target data are also updated. This can be thought of as an additional step in Alg 1 as follows: Step 2a: If target labels ($y_T^j$) are not available then compute $y_T^j = arg max_{y \in Y_T} Bh_R(A^{-1}x_T)$ for all $j = 1, …, n_T$. For computing the results in Table 2, the matrix B is fixed to a permutation matrix (similar to our other experiments). Thus, there is no variability due to the matrix B. > I would ... mistaken. The reviewer's intuition is correct. The choice of reference task affects the upper bound as we demonstrated, in detail, in Sec. 4.2. Specifically, when the reference and target tasks are more related (in terms of our task-relatedness metric) the upper bound is smaller. However, finding the most related reference task for every target task may not be possible in practice. Since our analysis does not impose any restriction on the choice of the reference task to use for computing task relatedness, a practitioner can estimate the upper bound using a few benchmark/publicly available datasets and select the reference task as the one that minimizes the upper bound, similar to the approach suggested by the reviewer of treating the reference task as a variable. > I would like ... problem. This seems to be a misunderstanding. SbTE methods as per [54] produce a score for each model (ideally without end-to-end fine-tuning on the target) that correlates well with end-to-end accuracy. This allows selecting the top-performing model by simply evaluating these scores. Thus, all prior works report correlation coefficients similar to our results in Table 1. Moreover, scores produced by SbTE methods do **not** numerically approximate the accuracy after end-to-end fine-tuning. For example, the results in Fig. 4 of [54] show that the value of SbTE metrics (y-axis of the plot) such as LEEP [34], NCE [50], and LogMe [54] lies in the range [-0.8,-0.3], [-0.45, -0.25], and [0.935, 0.95] respectively for the Aircraft dataset whereas the end-to-end fine-tuning accuracy (x-axis of the plot) lies in the range ~[72.5, 87.5] for the various models. It would be great if the reviewer could clarify their question if we have misunderstood them. > It is very ... suffice. The experiment in Appendix C.1 is a small-scale experiment as it is in the same setting as in Sec 4.1 with the ResNet-18 model. It is intended to be an ablation study to understand how different transformations affect the upper bound. The label mismatch term of Theorem 3 is minimized with a sparse B matrix corresponding to a one-to-one mapping between the classes of the reference and target tasks. However, due to the combinatorial nature of the problem finding the exact permutation is infeasible. Regardless of that, the linear transformation $A$ can align the distributions of the representation of the target to any permutation of the reference classes quite accurately. This is primarily due to the high dimensionality of the representation space where the linear transformation is applied. As a result, the value of the bound differs only by little for the different permutations of the reference task’s classes. This is indeed a very interesting outcome that makes our bound usable without requiring the exact semantic matching between the classes of the reference and the target classes. Below, we present an empirical evaluation to show that the difference in the bound computed with true vs random permutation is small. We use MNIST as the reference task and USPS as the target task (and vice-versa). We compare our results in a setting where only $A$ is learned and $B$ is set to an identity matrix and when $B$ is set to a random permutation matrix. Note that the identity matrix corresponds to the correct mapping between the classes of MNIST and USPS tasks (both contain digits from 0 to 9). We find that the upper bound obtained when $B$ is fixed to identity is only marginally better than the case when $B$ is a random permutation. Specifically, the difference between the bound when $B$ is fixed to a random permutation and when $B$ is an identity matrix is 0.10 for the MNIST→USPS task and 0.17 for the USPS→MNIST task. The primary reason for the decrease in the upper bound comes from the reduced distribution mismatch term. While the upper bound improves slightly when the ideal matching between the labels is known, such a mapping may not be known when the labels of the tasks are not related, e.g., for FMNIST and MNIST. Thus, fixing $B$ to a random permutation matrix yields a reliable estimate of transferability in most cases. We would be happy to address any further concerns the reviewer may have.
Response
Thanks for your detailed response to my review. Please see my responses below > `A: While Alg. 1 requires labels of the target task, we discuss in Lines 329-347 how to use Alg.` Hmm -- this is very confusing / seems contradictory. First, note that Algo 1, uses a random permutation matrix as B. Next, in lines in Lines 329-347 requires you to use B to compute the pseudo-labels. But 156 - 166 requires using target labels to specify B. So are you saying that to compute pseudo labels you use a random B ? Please clarify if that is not the case, and if that is the case, what are the error bars on the results in Table 2 ? I would imagine some non-trivial sensitivity to the choice of B. > `B: .. We emphasize that the reference task does not need to be similar to the target task to use our methodology for estimating transferability. Any benchmark/publicly available dataset can serve as the reference dataset and its use is not a bottleneck for our analysis/approach.` This is also confusing to me -- and seems a bit "free-lunchy". Theorem 3 on line 213 clearly gives **an upper bound** that depends on the label and distribution. I would imagine that the choice of the reference tasks affects the looseness of this upper bound -- we can simply think of the reference task as a variable that we are minimizing the upper bound over to make it tighter. Thus, the quality of your transferability estimate does depend on the *appropriate* choice of reference task. Please clarify if I am mistaken. > `While the best way to evaluate transferability is to fine-tune a model end-to-end on the target task, it is computationally prohibitive.` I would like to respectfully push back on this. You could have set up much smaller scale experiments to validate that the transferability scores are actually good for a reasonably sized model selection problem. > `We ablate the use of different transformations and measure the effect of learning each of them in Appendix C.1. Our experiments show that minimizing the conditional entropy by learning transformation B prefers a sparse mapping between the classes of the reference and the target tasks. .... Based on this, we use a reference task with the same number of classes as the target task and fix the matrix B to be a random permutation of the identity matrix.` As you mention in Appendix C.1 this is done on a toy problem -- it's a big leap to assume that this is the right thing to do for a more realistic problem without a bit of preliminary / small scale experimentation. Next, even if there is a sparse 1-1 mapping between classes, it is very strange to me that you decided that the exact mapping is not important and rather a random one would suffice. Basically, I understand your restriction to the set of `permutation matrices` but not your assumption that any random one of these matrices would suffice
Invitation for further discussion
Dear reviewer, We sincerely appreciate your time and effort in reviewing our manuscript. Through our rebuttal we clarified your concern around making Alg. 1 suitable for transferability estimation when target labels are unavailable. We also justified the reason for reporting the correlation coefficient for the problem of transferability estimation in line with all prior SbTE works [5, 34, 25, 54, 49]. Lastly, our latest response also provided clarification and more empirical evidence showing that fixing some of the transformations (such as fixing the matrix B to a random permutation) eases the combinatorial nature of the optimization problem in Eq. 3 without significantly affecting the value of the transferability estimate. As the discussion period is about to end, it would be appreciated if you could let us know of any other concerns we can address at this point. If you believe our responses sufficiently addressed your concerns we would humbly request you to re-evaluate our paper in the light of our discussion and update your rating of our work. Thank you immensely for your contributions and thoughtful consideration. Regards Authors
Response
Thanks for your clarifications >`For computing the results in Table 2, the matrix B is fixed to a permutation matrix (similar to our other experiments). Thus, there is no variability due to the matrix B.` I think the target label free case is clearer now, but I do still think it's important to evaluate the sensitivity of the final result to the initial (random) value of B. I think I see the value of the method at least in the case of no target labels present. >`to use for computing task relatedness, a practitioner can estimate the upper bound using a few benchmark/publicly available datasets and select the reference task as the one that minimizes the upper bound, similar to the approach suggested by the reviewer of treating the reference task as a variable.` As I mentioned in my initial review, this seems like punting the problem to a new problem of reference task selection and requiring practitioner intuition in selecting the reference task. And then there is the case of the computational overhead (if you have to run the algo on several tasks ) of this approach compared to other methods that do not require reference task My questions have been sufficiently clarified by the authors. I will raise my score from 3 --> 6.
Response by Authors
We sincerely appreciate the reviewer for engaging in the discussion and are glad that our responses clarified their concerns. We are encouraged that they find practical value in the ability of our method to estimate transferability without target labels. Following the suggestions of the reviewer, we will include the discussion and experiments for the initialization of transformations, presented here, in the paper to further improve the quality of our work.
Thank you for the responses. But my main concerns still exist, for example, the author mentioned that transfer learning is a setting uniquely considered in this article, but meta-learning can be considered a branch of transfer learning; the concept of task similarity is difficult to distinguish from the one in this article, and the descriptions like "can be used or not" and "may not" is vague. Therefore, the confusion in Weakness 1 is difficult to eliminate from the response. At the same time, by reading the responses of other reviewers, I found some issues that were previously overlooked. Therefore, after carefully checking all the responses, I tend to maintain my score.
I appreciate the authors' detailed response. I'll raise my score to a 5.
Response by Authors
We appreciate the reviewer's response to our rebuttal. We would be happy to address any other concerns the reviewer has.
Public comment
I congratulate the authors on the acceptance of their paper. The problem of transferability estimation is highly relevant in the field, and I appreciate the effort to propose a new method that advances this research area. After reading the paper and reviewers’ feedback, I am still left with some questions that I would like the authors to clarify: **Questions** **Use of Reference Task:** The proposed method requires access to a reference task, which could limit its practicality. Many off-the-shelf foundation models are now trained on proprietary datasets that are not publicly accessible. 1. I believe this is a limitation of the proposed method, as the majority of the compared transferability estimation methods do not require access to the source data. What is your insight on this? 2. How does the method perform in scenarios where source datasets are unavailable? 3. Even if the source data is available, self-supervised models are trained without using labels. How does the proposed method handle self-supervised models, which are trained without labels, differently from supervised models? In my opinion, the two major components of the proposed method, that is Reweighted Reference Loss and Label Mismatch become irrelevant for SSL methods. What is your opinion? __Practical use__ From Table 1 and Figure 5, it appears that LEEP often performs well, even surpassing the proposed method in Table 1. Yet, LEEP offers some advantages: 1) it does not require a reference set, 2) simply accounts for task differences, and 3) avoids optimization. What practical advantages does the proposed method offer over LEEP to motivate its use for me as a practitioner? __Target Datasets Without Labels__ The paper mentions using pseudo-labels generated by the transformed classifier as a proxy for target labels. However, off-the-shelf clustering methods could similarly generate pseudo-labels on the target data, enabling other transferability estimation methods to be applied on unlabeled target. Could the authors clarify how this is a distinct advantage of their method? Further, how this step can be handled if one is transferring an SSL model, which does not directly produce class labels? **Suggestions** __Technical statements__ The claim, "Since end-to-end fine-tuning is costly (takes almost a day to fully fine-tune a single model on a single target task...)," seems overly broad and potentially inaccurate in many scenarios. Fine-tuning time varies significantly depending on factors such as the size of the model and the target dataset, among others. What is your insight?
Response to the public comment
Thank you for your interest in our work; we appreciate your questions. We have answered your questions below. Please feel free to reach out if you have any further questions. > The proposed ... this? Our work's primary contribution is understanding on which tasks transfer learning can lead to high-performing models. To answer this question, we consider a reference task that achieves high performance after fine-tuning. Through our upper bound in Theorem 3, we show that performance on any target task can be provably related back to this reference task. Crucially, the reference task may or may not be the source task used to train the pre-trained model, and any publicly available dataset can be considered the reference task. Lastly, various previous transferability estimation methods such as NCE[56] and OTCE [55] have also utilized the source task for computing their metric for transferability. In our empirical section, we show that task relatedness is also an effective metric for the pre-trained model selection problem. We also emphasize that, unlike our work, an analytical understanding of transferability in the cross-domain cross-task setting is not the focus of other score-based transferability estimation methods. > How ... unavailable? As mentioned above, our analysis does not restrict the choice of the reference task to use for computing task relatedness. Any benchmark or publicly available dataset can serve as the reference task. > Even if ... your opinion? In Figures 3 and 9, we show experiments with various self-supervised models such as MOCO, SWAV, and MAE. We used Imagenet as the reference task for these experiments. Since the reference task can be any task, all terms in the bound are still meaningful. > From Table 1 and Figure 5, ... as a practitioner? We see in Figure 5 that LEEP performs worse in scenarios where there is less target data available. Comparatively, our method produces consistent correlation at different percentages of target data. While we agree that LEEP indeed offers the practical advantages you mentioned, it is sensitive to the amount of target data available for transferability estimation, and it only provides limited insights into analytically explaining transferability, unlike our approach. > A) The paper mentions ... of their method? B) Further, how this step can... class labels? A) While we agree that practically various alternatives such as off-the-shelf clustering methods could be used to obtain pseudo-labels, our task transfer approach does not require using these to handle unsupervised transferability estimation, unlike other score-based methods. B) For pre-trained SSL models, the predictions from $h_R$, which is the classifier trained on the reference task, can be used to obtain the pseudo-labels using $\arg \max_{y \in Y_T} Bh_R(A^{−1}(z_T))$ as mentioned in Sec. 4.3. > The claim, ... What is your insight? While end-to-end fine-tuning depends on various factors, an in-depth analysis from previous works such as Table 4 in LogMe [61] has shown that it is extremely costly to find the best end-to-end fine-tuned model for a single target task along with the best hyperparameter search. This has been the basis of various score-based transferability estimation approaches we cited in our paper.
Decision
Accept (poster)