Transfer learning with affine model transformation

Summary

The paper provides a comprehensive investigation of transfer learning and proposes an affine model transfer framework. The authors summarize various transfer learning approaches under the class of affine model transformation and establish theoretical properties such as generalization error and excess risk. In the experimental evaluation, the proposed methods demonstrate comparable performance to other baseline methods on tasks such as robot arm trajectory prediction and scientific document score prediction.

Strengths

The paper studies an important topic: the theoretical properties of transfer learning. The proposed theoretical framework for affine model transfer and the derived generalization bound provide valuable insights into transfer learning, domain generalization, and model adaptation. Overall, the paper is well-organized and effectively conveys the intricacies of the proposed theoretical framework.

Weaknesses

My main concerns revolve around the experimental evaluation conducted in the paper. The evaluation is narrower in scope compared to most transfer learning studies. It would be beneficial to validate the proposed method on standard transfer learning benchmarks, such as the MIT Indoors dataset, Caltech 256 dataset, or other commonly used regression datasets. This kind of evaluation can be found in references [1,2]. Furthermore, in the experimental evaluation, the observed improvement is limited. Table 1, specifically in the Torque 7 prediction, shows that the proposed method performs worse than Fine-tuning in most cases. While Theorem 2.4 provides a nice property of transformation functions, the paper lacks a discussion on the specific properties of $g_1, g_2$, and $g_3$ functions. It would be helpful to provide suggestions or insights into the types of $g_1, g_2$, and $g_3$ functions. [1] L. Xuhong, et al, “Explicit inductive bias for transfer learning with convolutional networks,” ICML, 2018. [2] S. Myung, et al, “PAC-Net: A model pruning approach to inductive transfer learning,” ICML, 2022.

Questions

I would like to know that why were the experiments not conducted on commonly adopted transfer learning datasets, such as MIT Indoors, Caltech 256, and CelebA? Considering that the proposed method performs worse than Fine-tuning in the case of Torque 7 prediction according to table 1, it is important to clarify the advantages of the proposed method over Fine-tuning. What unique benefits or improvements does the proposed method offer in comparison? Are there any suggestions provided regarding the design of $g_1, g_2$, and $g_3$ functions, or any specific neural network architectures? Providing guidance on the selection and design of these components would greatly assist readers in implementing the proposed method effectively.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See [Weaknesses] for limitations.

Summary

The authors consider the transfer learning setting by affine models adapting source features to the target domain by separately estimating cross-domain shift and domain-specific factors. They motivate the choice of such models deriving the affine form as an optimal class based on the squared loss for the target task. The authors show that the considered affine model encompasses several existing transfer learning methods, including neural network based transfer learning. The authors provide an estimation algorithm when modeled by kernel methods. The authors provide a generalization bound and an excess risk bound for their method. The bound show that when the source feature is strongly related to the target domain, their transfer learning approach results to be advantageous in comparison to the vanilla learning approach. Such an advantage is confirmed also by numerical experiments, in which the proposed method outperforms widely applied transfer learning methods lacking of theoretical foundation, such as re-using features from pre-trained models. More specifically, they consider two experimental settings: (i) the prediction of feed-forward torque at seven joints of the robot arm, and (ii) the prediction of review scores and decisions of scientific papers. Additionally, two case studies in materials science are presented in the appendix.

Strengths

The authors consider an interesting problem in a rigorous and formal way. The authors provide both theoretical and experimental validation for their method. The derivation/motivation of the affine form as an optimal class based on the squared loss for the target task is interesting. The affine model transfer is model-agnostic and it can be easily combined with pre-existent models. In addition, since domain specific factors are involved in incorporating the source features, the affine transfer can also deal with domain common and unique factors simultaneously and separately. Overall, the paper is written in a clear way.

Weaknesses

Considering only the squared loss is quite limiting. The authors comment about this, but it would be nice to understand if there is possibility to extend their framework to more general loss functions. For instance, what about the optimality of the class of affine functions with different loss functions? The authors say they can also derive the same optimal function by minimizing the upper bound of the estimation error in the TL procedure, as discussed in the appendix. I guess this could be related to my question. Some technical aspects could be better explained and intuitively motivated in my opinion, see for instance my comment below.

Questions

The algorithm the authors propose applies a coordinate descent minimization procedure on the objective function F. Is F jointly convex? Is the algorithm globally convergent? Usually, the study of global convergence for coordinate descent algorithms is problematic. The authors require the Lipschitzness of the loss function, but the squared loss is Lipschitz if and only if the labels are bounded and this is quite restrictive. From Thm. 4.1 it seems the authors get an improvement w.r.t. n by transfer learning. This seems to be strange, since, usually, the advantage observed in previous work by transfer learning is by constant factors. Could you please comment about this with reference to the papers 'Stability and hypothesis transfer learning' and 'Fast rates by transferring from auxiliary hypotheses'?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

I do not see any potential negative societal impact related to this work.

Summary

This paper presents a simple yet effective approach to transfer learning in regression problems by introducing an affine transform model. The proposed model incorporates the source feature function and the product of the source feature function and the function associated with the target input augmentation. Through the utilization of a least square loss function and a transformation function, this paper establishes the optimality of the affine model as the architecture of choice for addressing transfer learning challenges. Furthermore, the paper extends upon previous research by deriving generalization bounds and excess risk bounds in the proposed framework.

Strengths

1. This paper is impeccably written and exceptionally comprehensible. 2. The proposed method is rigorously derived and its performance is systematically evaluated from a theoretical standpoint. 3. The proposed algorithm can incorporate and generalize previously proposed algorithms while showing better performance.

Weaknesses

1. The existence of inverse functions appears to be of significance. However, it can be considered a potential limitation as it may not always be feasible to obtain inverse functions in practice. 2. I would suggest that the authors conduct evaluations of the proposed method using widely recognized benchmark datasets. It is particularly crucial to assess the algorithm's performance based on the proximity or disparity between the source and target, in order to ascertain its effectiveness under varying conditions.

Questions

1. Would it not be an overstatement to claim that the proposed algorithm encompasses all existing algorithms? This assertion arises from the fact that weight-based algorithms (such as L2-SP and PAC-NET) are not incorporated into the proposed algorithm. 2. Could we possibly expand the experimentation to include a broader range of experimental datasets?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

This reviewer thinks that the algorithm proposed in this paper has limitations that can only be applied to regression problems.

Summary

This paper focuses on the theoretical perspective on the affine model transfer learning. Specifically, this paper presents the optimal class of models on transfer learning with l2 loss takes the form of an affine decoupling. The decoupling consists of three functions, \$g_1 + g_2g_3\$, where \$g_1, g_2\$ encodes the domain gaps between source and target domains, and \$g_3\$ encodes the domain-specific information. Further, for each of the functions, this paper proposes a kernel method to approximate. Additionally, the generalization and risk bounds are presented. **___________________________________________________ Post Rebuttal __________________________________________________** The reviewer has read the rebuttal. In the rebuttal, parts of the theorems are restricted to \$ \mathbb{R} \$. This may degrade the contribution of the paper since we are handling the high-dimension data for most of the cases. The reviewer will maintain the score of 7 since there appear to be no technical flaws to the reviewer, and the reviewer gained some insights from the paper. However, the reviewer won't champion the paper, since I am not familiar to the literature.

Strengths

1. This paper presents universal results to the transfer learning on linear regression, which looks interesting to the reviewer. Specifically, it is shown that the optimal solution under l2 loss falls into a certain class of models. And the analysis is not limited to machine learning models. 2. This paper also presents the generalization and risk bounds, making the analysis more solid. 2. Empirically, this paper experiments on two datasets with extensive ablation and baselines, making the paper complete and thorough.

Weaknesses

1. Since the reviewer is not from the area, it is hard for the reviewer to judge the novelty and significance of the work.

Questions

1. All the discussions are limited to the \$\mathbb{R}\$. It remains elusive whether the analysis can be extended to higher dimensional spaces naturally. 2. What is d in table 1, is it the source domain feature dimension?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

While the paper's analysis is primarily limited to the regression task, it provides valuable insights with a commendable depth of exploration, despite limiting its general applicability. The complexity of a broader analysis would undoubtedly be a challenging undertaking, but the paper sets a strong foundation for future exploration.

Summary

The authors focus on the hypothesis transfer learning (HTL) problem. They show that under some hypothesis the data and model transformation is given by an affine transformation. They consider an associated kernel regression problem to identify the transformation and then characterise the generalisation bound which reflects the importance of the target-source relation. Finally, the method is benchmarked in 2 settings.

Strengths

* The proposed method allows for the theoretical characterisation of generalisation and excess risk bounds. * The method is versatile and can be used in very different settings.

Weaknesses

* The model is not clear (sec.2.1). Maybe adding a scheme or introducing the linear models as a first example before moving to the general version would help. Given this first setting, the rest flows nicely. * I find assumption 2.2 a strong assumption. The authors comment saying that this is common in HTLs. However, it implies no information compression which is hard to believe in many practical contexts. * In tables 1 and 2, the improvements are only marginally better than the other methods even if significant according to the authors' test. Minor comments: * assumption 2.2 (Invetribility) is instead (Invertibility) * fig.1 is misplaced. * proposition A.1 refer to the proof of Th.2.4 in the main paper, but this is instead in the SM sec.D.

Questions

* In the formulation of sec.2.1, it seems that $f_t(x)$ and $f_s(x)$ have the same dimension of $y$, i.e. they both belong to $\mathbb R$. However, in the examples below $y$ has the same dimension as the scalar product $\langle \theta, f_s(x) \rangle$. Can you clarify?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

I find assumption 2.2 a limitation. The authors seemed not concerned by this and pointed out that this is standard in the field. However, the intuition is that if the target representation is not included (or equal) to the source representation we shouldn't be able to invert the function.

Appreciation for the Author's Response

I extend my gratitude for the author's comprehensive response, which has indeed addressed some of my concerns. However, the observed inferior performance of the proposed method in certain settings, as exemplified by the Torque 7 prediction scenario in the paper, raises a pertinent question. When faced with a specific dataset, what criteria should guide the decision between employing the proposed method versus other available approaches? There's a genuine concern that an inadvertent application of the proposed approach might result in outcomes worse than simple fine-tuning. Hence, I kindly request the authors to offer specific guidelines or criteria for determining when the proposed approach is most suitable.

Thank you for the author's response.

Thank you for the author's response. I've fully reviewed the author's response, but I would like to keep my score. The reason is that the contribution of this paper is limited; i) it seems to be applicable only to regression problem and ii) the assessed problems are not a well-known benchmark problem. I think it is essential to verify the proposed algorithm in well-known problems.

Response to the authors

I thank the authors for their response. They answered my questions.