Auxiliary-Task Learning (ATL) aims to improve the performance of the target task by leveraging the knowledge obtained from related tasks. Occasionally, learning multiple tasks simultaneously results in lower accuracy than learning only the target task, which is known as negative transfer. This problem is often attributed to the gradient conflicts among tasks, and is frequently tackled by coordinating the task gradients in previous works. However, these optimization-based methods largely overlook the auxiliary-target generalization capability. To better understand the root cause of negative transfer, we experimentally investigate it from both optimization and generalization perspectives. Based on our findings, we introduce ForkMerge, a novel approach that periodically forks the model into multiple branches, automatically searches the varying task weights by minimizing target validation errors, and dynamically merges all branches to filter out detrimental task-parameter updates. On a series of auxiliary-task learning benchmarks, ForkMerge outperforms existing methods and effectively mitigates negative transfer.
Paper
Similar papers
Peer review
Summary
This paper considers how to best use auxiliary tasks to improve performance on target tasks. Specifically, a "ForkMerge" procedure is proposed which consists of two parallel optimization procedures, one on the target task, and one which includes auxiliary data, and the resulting weights are synchronized at regular intervals. Compared to grid searching for an appropriate interpolation factor, the proposed approach can dynamically alter the interpolation between the two sets of model parameters. Experiments on diverse set of benchmarks and a wide array of baselines show that the proposed approach is effective at improving performance using auxiliary tasks, relative to the state of the art.
Strengths
* This paper tackles a difficult problem of improving target task performance using auxiliary data. * The approach is well-motivated and the experiments are quite convincing. * The justification in terms of reducing distribution shift relative to test data is quite interesting.
Weaknesses
* The task selection with multiple auxiliary tasks is computationally expensive, and the impact of the pruning procedure is not completely clear. * There is no adequate discussion of limitations.
Questions
None.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
See "Weaknesses"
Summary
The paper tackles the problem of learning multiple tasks together which is known to lead to "task inteference" or "negative transfer" issues. This is usually tackled by automatically scaling the task weights or gradients based on training statistics (e.g. GradNorm or uncertainty weighing of losses). In particular, the paper studies an asymetric version of the problem, Auxiliary Task Learning (**ATL**) where one task may not be important at inference but can greatly improve performance when jointly trained with the main target task. The paper first explore potential causes of task interference under the lens of train/target distribution shifts. Then, the paper proposes **ForkMerge**, a novel optimization method to avoid task interference: For each update of ForkMerge, first the current parameters are duplicated. Then, the weights are updated separately; one on the target task, while the other is jointly trained for the task and auxiliary tasks. Then, the optimal task weights are found by finding the linear combination of these two sets of weights that maximize validation accuracy (**Equation 12**). The algorithm can be extended to multiple auxiliary tasks which requires an additional separate optimization branch for each new auxiliary task. The method is then evaluated on the NYU dataset (3 tasks) and DomainNet (6 domains) and compared to previous MTL and ATL approaches
Strengths
- I liked the analysis section of the paper and it contains interesting insights on the problem of task interference: For instance, gradient conflicts is often named as a cause of negative transfer but its effect/strength is rarely actually measured in practice. Although it would have been interesting to extend the analysis to more diverse scenarios. - I also like the insights that task weighing methods should take generalization into account, which is captured in the proposed algorithm by finding the optimal task weights via evaluation on the validation set - The paper presents extensive experiments on two classical multitask/multidomain benchmarks, which are further completed in the supplementary material (e.g. additional backbone)
Weaknesses
- **Training cost**: the paper should discuss the training cost in terms of memory efficiency more concretely/quantitatively: ForkMerge requires a separate set of parameters for each task. In addition these parameters are also updated independently hence the number of forward/backward passes will also increases with t he number of tasks. Finally, the cost of search for optimal $\Lambda$ in Equation 13 will also increase with the number of tasks. These costs may remain reasonable in the application scenario with only one auxiliary task, but it's not clear how practical ForkMerge becomes when dealing with more than two tasks like in Section 5.2 - **Finding 2 does not seem very significant**. Finding 2 states that negative transfer is likely to occur when the added auxiliary task increases the train/test distribution shift for the target task: This seems like a pretty classic statement from statistical machine learning theory: For instance generalization bounds for domain adaptation typically include a term measuring the discrepancy between the source / target domains. - **Chosen baselines might not all be a fair choice**: the multi-task optimization baselines (MGDA, GradNorm, PCGrad... etc) typically aim to optimize performance on **all tasks**, as opposed to auxiliary task learning where there is a clear bias towards the target tasks: For instance in ForkMerge, every branch includes the target task objective. For that reason, I do not think applying MTL methods "as-is" to the problem of ATL is the best baseline. A stronger baseline could be to finetune the task weights in the "Equal Weights" (**EW**) baseline for performance on the target task; this might be to costly for DomainNet but would be reasonable for NYU as there are only two auxilair ytasks.
Questions
- **Suggestion of writing**: I feel like the paper sometimes overuse abbreviations. For instance only mentioning the tasks **P** and **Q** for DomainNet (Painting and Quickdraw domains ?) assumes the reader is very familiar with the dataset. In additional, there are many abbreviations introduced in the paper which often make the text hard to read (for instance lines 154-159) - **Distribution shift and hypothesis shift**: I did not understand the point that *lines 213-221* try to convey and the difference between Equations 12 and 13 (outside of renaming $K$ to $B$ and allowing $\Delta_t$ time steps instead of only 1). Similarly, I'm not sure what insight **Figure 6** tries to convey. More specifically: the task weighting $\lambda$ is initially defined at the loss level (Equation 1); However, it seems that in later sections $\lambda$ is rephrased as a parameter mixing the distribution of the target and auxiliary tasks; while there are links between these two views, I found that switching between the two interpretations in the text a bit confusing. It is also not clear what it means for multi-task applications: There is only one input distribution (the input images are the same for all tasks) so which distribution shift are we referring to ? - From the appendix **C.3**, it seems that the test set of DomainNet was split into two parts: one for validation, and the other one for test performance. I think this should be clearly stated in the main paper directly as it means the reported results are not comparable to other DomainNet works
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
There is no dedicated limitations section but the paper discusses the limitation of dedicating independent optimization branches for every auxiliary task (e.g. branch pruning to avoid handling too many branches in the DomainNet)
Summary
The authors conduct an analysis of negative transfer in auxiliary task learning, finding that gradient conflicts are not necessarily tied to negative transfer, but that auxiliary tasks that induce large distribution shifts from the new training distribution to the test distribution tend to cause negative transfer. The authors then propose ForkMerge, which repeatedly forks a model into branches trained on just the target task and trained on both the target task and auxiliary task(s), and then uses target task validation set performance to determine how to merge the forked models.
Strengths
1. The authors share an interesting and thoughtful analysis, investigating both gradient conflicts and induced distribution shifts as potential causes of negative transfer and concluding that gradient conflicts are not necessarily tied to negative transfer. 2. The ForkMerge approach that the authors introduce is intuitive and simple, and the results are consistent across a variety of tasks. 3. Well-motivated and significant problem
Weaknesses
1. In general, some critical details are omitted altogether in the main text, sometimes with no proper reference to the Appendix when details are missing. In particular: - In Section 3, please spell out Painting and Quickdraw (and later, Real in Section 5) and describe DomainNet when first introducing the tasks. If it needs to be brief, the authors can at the very least include a pointer to Appendix C for dataset and task details. - I also believe it is worth mentioning "We use ResNet-18 [8] pre-trained on ImageNet [3] for all experiments." in the main text rather than only noting it in the appendix. 2. I would have liked to see more overlap between works cited in the Experiments section and the works discussed in Related Work, especially discussion about and comparison of adaptive auxiliary- and multi-task learning strategies and settings. In particular, [36] and the original meta-learning paper [13]. A couple additional related works the authors may consider adding: https://arxiv.org/abs/2205.14082 and https://arxiv.org/abs/2212.01378
Questions
1. Can the authors describe in more detail the computational overhead of ForkMerge, especially wrt time and memory, and in relation to the existing approaches from Section 5? I am most curious about the comparison to meta-learning. 2. Do the authors predict that these results and the practicality of ForkMerge are limited to settings where one starts with a pretrained initialization? 3. re: definition 3.5, are there alternative measures of distribution shift that the authors have considered? I am personally somewhat skeptical of confidence scores as a basis for measuring distribution shift given that many models are poorly calibrated in the first place.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The authors test their method using only one model, ResNet-18 pre-trained on ImageNet, on image tasks. I won't push for additional experiments, but I would like to see the authors discuss what assumptions from this paper's experiments are expected to be critical for generalization of findings to other settings.
Summary
Auxiliary-Task-Learning (ATL) has been studied from the perspective of optimization, which aims to improve the performance of the target task by leveraging similar tasks. However, ATL can sometimes suffer from negative transfer, where the performance of the target task actually decreases when auxiliary tasks are added. In this paper, the authors take a more holistic perspective by considering both optimization and generalization. They propose a new method called ForkMerge that is able to resolve negative transfer and improve the performance of the target task.
Strengths
* ForkMerge is a simple and practical algorithm that demonstrates good empirical observation. * Many analytic figures help readers to understand the negative transfer better although they are limited to DomainNet dataset.
Weaknesses
* There seems to be some logical jump between the two observations listed in Section 3 and the proposed algorithm in Section 4. It is quite unclear how they are connected directly, and it’ll be great to have some theoretical formulation for the connection (at least in high-level). For example: why dynamically adjusting \lambda is superior to the previous method? How does this address the generalization problem? I think this limits the contribution of this work. * ForkMerge essentially optimizes a fork of the given model and trains them separately multiple times; hence, it does require significantly larger compute during the model training.
Questions
* As discussed in the weaknesses section, the connection between the two main observations to the ForkMerge is not strong. Can you provide a high level theoretical analysis of how ForkMerge can make better generalization not just based on the empirical analysis? * Forking a large model is very expensive. Did authors consider a simple ensemble based approach which should be also similar in their compute (not the actual model weight merging rather than prediction level)?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
1 poor
Limitations
I don’t see the negative societal impact of this work.
Summary
To fully leverage the knowledge from auxiliary tasks and mitigate negative transfer issues, this paper introduces ForkMerge, which automatically searches for varying task weights for auxiliary tasks by minimizing target validation errors. ForkMerge is evaluated under various settings, including multi-task learning, multi-domain learning, and semi-supervised learning. The results demonstrate that it outperforms existing methods and proves to be effective.
Strengths
This paper demonstrates a well-organized structure. The authors have conducted experiments using diverse datasets and tasks, resulting in promising outcomes. Furthermore, the paper provides theoretical analysis of each component, offering valuable insights into their impact on performance. Overall, the clear structure and in-depth analyses make it an engaging and compelling read.
Weaknesses
The paper lacks clear explanations for certain sentences, training details, and figure information. For example, there is a lack of clarity regarding the data division strategy for each branch and the effectiveness of the learned weights. Additional elaboration and detailed analysis in these areas would enhance the overall understanding and impact of the proposed approach.
Questions
W1. The paper does not provide a clear explanation of how the method filters out harmful parameter updates to mitigate negative transfer after merging and synchronizing the parameters of each branch. W2. Figure 3 would benefit from improved clarity, using a distinct color or different marks would be helpful. Additionally, why does the number of auxiliary task data points increase with the increasing $\lambda$? W3. Lacks clarity regarding the data division strategy for each branch. It is not explicitly explained whether each branch is trained with a part of the data or full data. Moreover, the impact of increasing the number of branches on computational cost and its trade-off with efficiency and accuracy is not thoroughly addressed in the paper. W4. I am curious about the comparison between the learned weights by ForkMerge and the optimal weights. For instance, in Figure 3 (b), for WNT tasks, do the learned weights resemble the optimal weights shown in the figure? W5. It would be valuable to investigate and illustrate the trajectory of the learned weights over the course of training to understand their dynamics and convergence patterns. To gain a better understanding of the effectiveness of the ForkMerge approach, it would be insightful to compare the performance of fixed learned weights by ForkMerge with dynamically learned weights during training.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Limitations could be more clarified in the paper.
Summary
This paper strives to mitigate negative transfer in auxiliary-task learning by optimizing the coefficients assigned to auxiliary tasks. By conducting an empirical investigation into the factors contributing to negative transfer, this paper reveals two interesting findings. Based on the findings, a new approach named ForkMerge is proposed to mitigate negative transfer and boost the performance of auxiliary-task learning. Extensive experiments demonstrate the effectiveness of the proposed approach.
Strengths
+The problem is well-defined and well-motivated. +The findings seem interesting +The proposed approach is reasonable and well presented.
Weaknesses
-It is not clear how the new findings motivate the proposed ForkMerge. -I'm curious about the performance when directly trying various lambdas (grid search) during multiple training sessions. Although the proposed method is more efficient, there is a concern about potential accuracy trade-offs. I seek to understand if the proposed approach sacrifices accuracy compared to the direct lambda variation method. -The simple post-train method leads to superior performance over other complex approaches, which makes me doubt the significance of research efforts in this area over the past years. This paper claims that the main drawback of post-train method is that it fails to consider the task relationship in the pre-training phase, and suffers from forgetting during fine-tuning. I’m wondering the performance of the post-train method if considering the task relationship in the pre-training phase and preventing forgetting during fine-tuning such as by distillation. In this scenario, if the post-train method manages to surpass the proposed ForMerge approach, considering ForMerge’s slight improvement in most tasks compared to Post-train. -The experimental evaluation is limited to small datasets, such as NYUv2 and CIFAR-10, and employs a relatively small network like ResNet-50 (DeepLabV3+). As a result, there is uncertainty regarding the efficacy of the proposed method on larger datasets and powerful networks like transformers, which already excel in single-target tasks. Consequently, assessing the true significance of the proposed method in the current era of deep learning becomes challenging. -Too may abbreviations make paper hard to follow in some paragraphs.
Questions
See Weaknesses
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Not sure
Thanks authors for the detailed response. I have looked at the rebuttal (for my review and also the global), and the rebuttal does not fully address my concerns; hence I'll be keeping my current score. To be more specific, the logical connection still seems to be a bit hazy to me. I looked at Global Rebuttal #1, but I am not fully convinced. For example, "hyperparameter to be properly tuned -> let's dynamic change" is not well connected. Why is the model so sensitive with the hyperparameters? What model property is inducing this? Can we make modifications for the modeling assumptions so that this can be handled differently? These are not discussed well.
Replying to Reviewer
Thanks again for your dedication to reviewing our paper. We will provide additional clarification in this response. Firstly, it's important to clarify that **ForkMerge is inspired by, but not a direct extension of, the findings presented in $\text{Section 3}$.** ForkMerge is rooted in two fundamental principles of machine learning: the linear combination of losses in auxiliary task learning (as detailed in $\text{Equation 1}$) and the utilization of stochastic gradient descent for optimization (as outlined in $\text{Equation 5}$). Through these foundational principles, we naturally arrive at the concept of linearly combining model parameters (elaborated upon in $\text{Appendix A}$). Hence, the insights in $\text{Section 3}$ offer an intuitive explanation for the approach in $\text{Section 4}$, and even in the absence of $\text{Section 3}$, ForkMerge could still be derived through theoretical reasoning. **Q1:** Logical connection between our findings and proposed method. - $\text{Section 3.1}$ revisits the issue of gradient conflict and concludes that it's not necessarily correlated with negative transfer. **Thus, unlike prior methods, our approach does not initiate from the standpoint of gradient conflict.** - $\text{Section 3.2}$ underscores the importance of considering generalization. Drawing from these two points, ForkMerge leverages target task validation error for $\lambda$ selection and deduces the form of model parameter interpolation, as illustrated in $\text{Equation 7}$. Building upon these findings, our approach further: - Extends to dynamically adjusting $\lambda$ due to the evolving importance of different tasks during training (refer to $\text{Appendix D.1 Figure 3}$). - Employs longer time intervals for $\lambda$ estimation to reduce noise during the estimation process.
Replying to Reviewer
**Q2:** Why is the model so sensitive to the hyperparameters? What model property is inducing this? The sensitivity of the model to the weights assigned to different tasks is an inherent aspect of auxiliary-task learning. These task weights serve as crucial hyperparameters that directly influence the optimization objectives of the model. In the realm of auxiliary task learning and multi-task learning, numerous studies focus on tuning the weights of different tasks to attain optimal model performance. For instance: - **Uncertainty Weighting (UW)** [1] employs task uncertainties to weight the loss functions, effectively balancing the significance of various tasks. - **Dynamic Weighted Averaging (DWA)** [2] utilizes the decreased rate of task losses over time to weight the loss function dynamically. - **Gradient-Cosine Similarity (GCS)** [3] and **Auto-$\lambda$** [4] estimate dynamic task weights within a single iteration, using gradient cosine similarity and finite difference approximation, respectively. Furthermore, $\text{Section 3}$ of our paper provides a comprehensive exploration of the considerable impact that task weights have on the model from different perspectives: - **Multitask Optimization Perspective:** The task weights govern the optimization goal. Setting the auxiliary task weight to $0$ disregards the auxiliary task, while an infinitely large weight would cause the auxiliary task to dominate the primary task. This establishes the existence of an optimal equilibrium. - **Distribution Perspective**: The task weights influence the training distribution, as illustrated in $\text{Figure 3}$. A proper selection of task weights can better fit the testing distribution of the target task. [1] *Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, 2018.* [2] *Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In CVPR, 2019.* [3] *Yunshu Du, Wojciech M Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:1812.02224, 2018.* [4] *Shikun Liu, Stephen James, Andrew J Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. In TMLR, 2022.* **Q3:** Can we make modifications to the modeling assumptions so that this can be handled differently? The model's sensitivity to task weights stems from the fundamental intricacies of auxiliary-task learning, where careful consideration is required to strike a balance between mitigating overfitting and managing potential negative transfer. An intuitive solution could involve employing larger models to address this challenge, yet this approach brings forth its own set of complexities. Apart from the additional computational costs and memory overhead that come with larger models, a critical concern arises in the form of increased susceptibility to overfitting. Thus, a delicate trade-off between mitigating overfitting, a challenge ingrained in single-task learning, and the introduction of potential negative transfer due to auxiliary tasks must be meticulously weighed. Indeed, during the rebuttal phase, we delved into the prospect of employing larger models to examine this hypothesis. Specifically, we substituted the backbone network with ViT-Base [5], which has been pretrained on ImageNet 21K. We then conducted experiments on the DomainNet dataset, as outlined in $\text{Section 5.2}$. The results, as presented in $\text{Table 2 of our global rebuttal pdf}$, yield insightful observations: - ViT-Base demonstrates enhanced average accuracy through the equal weighting (EW) method, as compared to single-task learning. We posit that this improvement can be attributed to the data-hungry nature of vision transformers [5], wherein the advantages of auxiliary tasks in alleviating overfitting could potentially outweigh any interference introduced by the auxiliary tasks themselves. - ForkMerge consistently outperforms the comparison methods across all tasks, with an average accuracy of $73.3$ for ForkMerge, as opposed to $70.0$ for Post-train. This robust performance across various network architectures further substantiates the effectiveness of the ForkMerge approach. [5] *Dosovitskiy, Alexey et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2021.*
thanks for your reply
Hello authors, thanks for your reply and clarifications! Regarding the difference between distribution and hypothesis shift, I still think the way it is described in the paper introduces more confusion than necessary (which ties in with the weakness I listed about finding 2) although it is more clear now thanks to your reply. I would like to keep my original rating of (6). **Minor comments, mainly about writing clarity** **a. About the difference between $\lambda$ and $\Lambda$** In the response, you explain that *"we consistently employ the symbol to represent the mixture weights of distributions, while employing to denote the mixture weights of model hypotheses"*; however $\lambda$ is initially defined as the weights in the task losses: Building on this, Equation 11 only defines the vector $\Lambda$ to be essentially equal to $\lambda$; similarly Equation (7) and (8) both define mixture of weights using $\lambda$ and not $\Lambda$. Going further, it seems that instead, the intuition of $\lambda$ as distribution mixture coefficient is the one that should be redefined ? For instance, in lines 216, you write *Thus, we transform the problem of mixture distribution into that of mixture hypothesis*. However the previous paragraph(s) only show how yo transform the problem of mixture of losses to that of mixture hypothesis (equations 9 to 12): As a reader, I found that the link between "mixture of losses" and "mixture of distribution" should be better formalized/motivated; it's briefly mentioned in Section 3.2 (equation 3) but does not really tie in with the rest of the paper and the method of Section 4 **b. regarding Equation 13**, I understand the points you mentioned, but I think the writing should be rephrased to motivate Equation (13) better: For instance in the paper, $B$ is only introduced as "number of candidate branches" while in your response, you clearly mention its use for pruning/efficiency. The **extended time for lambda estimation** is not mentioned at all when introducing (13) (but it makes sense from a practical perspective). And similarly, the **enhanced flexibility** wrt to Equation 12 is also not mentioned, and is not apparent from the equation itself.
Thanks for the Reviewer's Reply
We'd like to thank Reviewer pM6c again for providing an impressively insightful pre-rebuttal review, which has enabled us to make an effective response. We'd also thank you for carefully judging our feedback and acknowledging our work in the final review. Following your suggestion on writing clarity, in the next version, we will delve deeper into the relationship between the concepts of "mixture of losses" and "mixture of distributions." We will also provide more explanation on $\text{Equation 13}$ to highlight its distinctions and advantages in comparison to $\text{Equation 12}$.
Thank you for the detailed response and additional context! Updating my score to 7 assuming the authors will incorporate the discussions and clarifications into the camera ready if accepted.
Thanks for the Reviewer's Reply
We'd like to thank Reviewer uS5H again for providing an impressively insightful pre-rebuttal review, which has enabled us to make an effective response. We'd also thank you for carefully judging our feedback and acknowledging our work in the final review. **We will guarantee to incorporate the discussions and clarifications into the next version.**
Thank the authors for their comprehensive response, which has effectively solved some of my major concerns. However, I do have an additional query regarding the performance of the post-train method. In Table 2 of the primary manuscript, the post-train method exhibits superior performance compared to EW and Auto_lambda. Nevertheless, when employing the more robust ViT-Base model, an intriguing shift occurs as shown in Table 2 of the global rebuttal. The post-train method's performance deteriorates, whereas the performance of the other two methods experiences a significant improvement. Could the authors kindly provide further elucidation regarding these observed outcomes?
Replying to Reviewer
We sincerely appreciate your feedback and are pleased to see that our response has addressed many of your concerns. We are grateful for the opportunity to provide further clarification on the additional question regarding the performance of the post-train method. In $\text{Section 5.2}$, our experiment results on DomainNet show that the Post-train method has consistently exhibited a slight advantage over STL. This trend can be attributed to several factors that contribute to the generalization ability of Post-train: - **Influence of Single-Task Fine-Tuning**: In datasets with a substantial volume of data, such as DomainNet, the final performance of a model is **notably influenced by the last stage of training**, which involves single-task fine-tuning. Consequently, when STL performs well, the Post-train method also benefits from single-task fine-tuning, resulting in better performance. - **Implicit Regularization from Pre-training**: The initial pre-training phase serves as a form of parameter initialization, offering implicit regularization that aids in optimizing the model. This initialization effect contributes to the enhanced generalization capability of the Post-train method compared to STL. However, we acknowledge the intriguing observation you have made when applying the more robust ViT-Base model, as highlighted in $\text{Table 2 of global rebuttal pdf}$. In this case, there is a shift in performance dynamics, where the Post-train method's performance deteriorates while both the EW and Auto-$\lambda$ methods experience significant improvements. This phenomenon can be attributed to the interaction between auxiliary tasks and the target task, which has varying implications across different model architectures: **Impact of Auxiliary Tasks**: The introduction of auxiliary tasks brings about a dual effect on the model's performance. On one hand, the additional supervision signals contribute to a reduction in the risk of overfitting, enhancing generalization. On the other hand, the joint distribution shift between the auxiliary tasks and the target task can lead to negative transfer, impacting performance adversely. In different scenarios, both of these effects are present, but one effect might be more pronounced. **Impact of Model Capacity:** The influence of the dual effects of auxiliary tasks is dependent on the specific scenario and model capacity. - For instance, with limited model capacity, as is the case with the ResNet101 architecture on DomainNet, the influence of task conflicts induced by auxiliary tasks becomes more pronounced. This results in the performance of the EW method being inferior to both STL and subsequently Post-train. - Conversely, when employing the Vision Transformer model, which boasts increased capacity, the risk of overfitting with limited data becomes more pronounced. This makes STL less effective and consequently leads to the EW method outperforming STL, causing the Post-train method to fall short of EW and Auto-$\lambda$. Across different scenarios, there exists an equilibrium point between EW and STL, driven by the dynamic interplay between the positive and negative effects of auxiliary tasks. **This equilibrium point represents the central aim of our ForkMerge algorithm: to uncover the optimal balance between these effects.** We believe that the consistent patterns observed across various scenarios underscore the utility and potential of the ForkMerge algorithm in identifying this equilibrium and leveraging the strengths of both STL and EW methods. We hope that this additional explanation provides a clearer understanding of the observed outcomes, and we remain open to any further inquiries you may have. Thank you once again for your valuable insights and the opportunity to enhance the clarity of our work.
Decision
Accept (poster)