Domain generalization (DG) methods aim to maintain good performance in an unseen target domain by using training data from multiple source domains. While success on certain occasions are observed, enhancing the baseline across most scenarios remains challenging. This work introduces a simple yet effective framework, dubbed learning from multiple experts (LFME), that aims to make the target model an expert in all source domains to improve DG. Specifically, besides learning the target model used in inference, LFME will also train multiple experts specialized in different domains, whose output probabilities provide professional guidance by simply regularizing the logit of the target model. Delving deep into the framework, we reveal that the introduced logit regularization term implicitly provides effects of enabling the target model to harness more information, and mining hard samples from the experts during training. Extensive experiments on benchmarks from different DG tasks demonstrate that LFME is consistently beneficial to the baseline and can achieve comparable performance to existing arts. Code is available at~\url{https://github.com/liangchen527/LFME}.
Paper
Similar papers
Peer review
Summary
This paper proposes a simple framework called LFME for learning from multiple experts in domain generalization. LFME introduces a logarithmic regularization term to enforce similarity between the target model and expert models, allowing the target model to acquire expertise from all source domains and perform well in any testing domain. Experimental results demonstrate that LFME outperforms existing techniques not only in classification and segmentation tasks but also reveals through in-depth analysis its ability to implicitly use more information for predictions and to discover difficult samples from experts, thereby enhancing generalization capability.
Strengths
This paper proposes a novel idea on the basis of a new knowledge distillation paradigm for DG. The motivation of using multiple experts for improving generalization is clear and reasonable. Meanwhile, I also found the paper well-presented and easy to understand. After checking the pseudocode and the provided code, I found the implementation extremely simple. Given the impressive results computed with rigorous evaluation protocols, I believe this paper can offer valuable contributions to the literature by providing an accessible and effective solution. The provided deep analyses that explains why their method works are appreciated and, in my opinion, crucial for a paper, especially in the era of deep learning. Coupled with the empirical evidence in their supplementary material, I think most of claims made in their paper can be well supported.
Weaknesses
Regarding the proposed method, although large improvements over the baseline ERM are observed, their method involves higher training costs. I noticed the authors provide the training time comparisons with different methods in their supplementary material, which shows that their LFME requires 1.5 times more training time than ERM, and it is also one of the most time-consuming methods compared. While I agree with the authors that requiring more training time is inevitable for methods designed based on KD, the authors should at least list this as a major drawback in their limitation section. As in recent times, Vision Transformers (ViTs) have demonstrated substantial improvements in the classification and semantic segmentation tasks. Although the authors conducted experiments using different ResNet models, they should also consider adopting a ViT backbone for their experiments. This could provide valuable insights into the performance of their method when applied to more advanced architectures, potentially highlighting further benefits or limitations. Including such experiments would enhance the comprehensiveness of the study and align it with current trends in the field.
Questions
In Section 4.2, the authors claim that their method improves DG by explicitly mining hard samples from the experts. However, Tables 3 and 6 show that specifically mining hard samples from the experts improves performance of ERM only in the TerraInc dataset, but not in the PACS dataset (with an average improvement of only 0.1 pp). Could the authors provide an explanation for this phenomenon?
Rating
8
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Authors have adequately addressed the limitations.
The author's rebuttal allayed some of my concerns, and I chose to raise my rating, considering that the paper still had some sparkle to it.
Summary
This paper focuses on improving domain generalization by utilizing multiple experts. Particularly, a simple yet effective framework is proposed whereby a target (student) model is learned from multiple expert (teacher) models through logit regularization. After learning, the target model can grasp knowledge from multiple source domains and shows advantages in handling hard samples. The proposed approach demonstrates consistent improvements across different evaluation tasks, outperforming existing SOTAs.
Strengths
This paper overall is well written. It provides in-depth theoretical insights on its proposed logit regularization and surrogate ablation studies.
Weaknesses
1: The contribution of the proposed approach is marginal. The performance of the proposed approach does not show much improvement compared to ERM or expert models (Table 5). 2: The baseline models for comparison are not clearly introduced, which creates difficulties in understanding the Tables. 3: The comparison to the approaches that directly learn a foundation model using data from all source domains is not discussed. Hence, the benefits of the proposed approach, which have to use a set of expert models plus a target model, compared to the foundation model are not clear.
Questions
1: What is the difference between the proposed approach and foundation models? Why not directly learn a model using all the source domains? How does the performance of the proposed approach compare to Segment Everything? 2: What’s the difference between the proposed approach and the baseline ERM? In lines 77-78, the authors mention ‘only one hyper-parameter’ without further illustration. The difference is clarified only until line 138. Statements near lines 77-78 need to be improved to avoid confusion. More importantly, it can be misleading to say “only one additional parameter upon ERM” since a target model is also involved. 3: How does the value q_*^E relate to the hardness of samples? What causes the sample to be ‘hard’? Low data frequence or image quality? Without these clarified, the provided theoretical insights are not convincing.
Rating
4
Confidence
3
Soundness
2
Presentation
2
Contribution
2
Limitations
No discussions on limitations and potential negative societal impact of the work are provided. One limitation is that the performance of the proposed approach can be upper bounded by the performance of the expert models. This limitation again raises the question of why not directly learn from all source domains?
Summary
This paper addresses the problem of domain generalization from the perspective of ''learning multiple experts''. In particular, they propose to train multiple experts specialized in different domains, whose output probabilities provide professional guidance by simply regularizing the logit of the target model. The proposed logit regularization provides effects of enabling the target model to harness more information, and mining hard samples from the experts during training. Experiments on standard DG benchmarks demonstrate the effectiveness of the proposed framework.
Strengths
- The overall manuscript is well-organized and easy to follow. - The method is simple and effective. Its impressive results in different tasks demonstrate the strong applicability of this idea to potential other tasks. - The related work section is comprehensive, especially regarding comparisons with Meta-DMoE that also tries to distill knowledge from domain experts. - The idea of using MSE loss between logit and probability for distillation is new and well-explained, supported by both theoretical and empirical evidence. - The free lunch idea introduced in sec. 6.1 is interesting and seems to have the potential for more general applications.
Weaknesses
While adopting the idea of knowledge distillation (KD) and the concept of domain experts for improving DG is not new in the literature, it is appreciated that this method can be implemented in such simple form and be explained with extensive insights. Nevertheless, there are still a certain amount of claims in the paper that requires proper explanations or more justification: - Performance reported in the original Meta-DMoE paper is higher than that in Table 1. For example, Meta-DMoE reports an average of 86.9 in PACS, surpassing the best result listed in the table. The authors should provide an explanation for this discrepancy. Moreover, I noticed that the training time comparisons between these two methods are provided, it would also be beneficial to highlight the test and training resource differences between these two methods in the related work section. - The authors mention that their method introduces only one hyper-parameter, which is within a certain range. It would be beneficial to include an ablation study to explain why this specific range was chosen. - It would be beneficial to evaluate this method with more real-world DG problem, such as using dataset from wilds benchmark. - The method [1] with similar settings should be compared in Table 1. Additionally, some methods, such as MIRO, report improved performance when combined with SWAD. Including these results in Table 1 would provide a more comprehensive comparison. - Minor issues. Formats of ”i.e.” in line 145, ”Tab.” in line 578, and ”Fig.” in line 581 are inconsistent with their usages in other places. The authors should standardize the format of similar abbreviations to maintain uniformity. Reference: [1] Domain Generalization via Rationale Invariance, in ICCV 2023.
Questions
- This method needs to train additional expert per domain, what if there are large amount of domains for the training data, any potential solution for this occasion? - Since multiple expert models are required when training the target model, how would this method be applied to the training process of large foundation models? - The free lunch idea appears to be applicable beyond just the DG setting. How does it perform on ImageNet? Including such experiments could enhance the significance of this concept.
Rating
8
Confidence
5
Soundness
4
Presentation
4
Contribution
4
Limitations
yes
Summary
The computer-vision paper introduces a strategy of learning from multiple experts (LFME), which performs knowledge distillation from models specially trained on data from different domains. In particular, the experts are trained jointly with the target model, and a specific form of logit regularization is chosen for knowledge distillation due to empirical verification. The idea is relatively straightforward to understand with strong empirical results on image classification (by Domainbed) and semantic segmentation.
Strengths
1. The experimental results come in strong when compared to many other baselines. Moreover, the study has extended results for semantic segmentation. 2. The visualizations are helpful to illustrate the ideas and evidence. 3. The empirical study on naive aggregation of domain experts shows that naive ensemble does not necessarily lead to better generalization. Then ablation study shows the chosen form of logit regularization outperforms other knowledge distillation alternatives.
Weaknesses
1. The novelty is relatively limited. The proposed approach is very similar to Meta-DMoE with a few changes such as logit regularization and alternating updates, but these changes are not well justified (see below). The idea of utilizing multiple domains have also been explored by [1]. 2. There is a lack of coherent argument to understand the proposed approach. In Section 4.1, the paper briefly discussed how the additional logit regularization enables learning more information. In addition, in Section 6.2, the proposed approach LFME is compared with Label Smoothing. However, the discussion is not very convincing. From the perspective of using information from other classes, I do not see the fundamental difference between logit regularization and Label Smoothing (LS). Both logit regularization and LS have hyperparameters to be tuned, so it might not be appropriate to claim its advantage as "not involve hand-crafted settings” (line 187) and criticize LS has “potential improper heuristic designs” (189). 3. Moreover, in Section 4.2, the paper justifies the advantage of the proposed approach LFME by “mining hard samples from the experts”. It is a general statement that is true for low-confidence predictions from any model, so it does not constitute a strong argument. Intuitively, the in-domain samples are easier than out-of-domain samples for each expert, and the hard samples aggregated from all experts are not specifically representative of any subpopulations. The argument will be more comprehensive if the paper can show what are these hard samples and why it matters to mine the hard samples from the domain experts (and not an ensemble of experts or some random experts). 4. In Section 6.5, the result for in-domain evaluation is only provided for one dataset. This is not sufficient to justify a strong claim that the target model is an expert on all source domains. 5. Code is submitted but it is unclear how the proposed approach can be combined with SWAD. Does it apply to just the target model or also the domain experts? The hyperparameter configurations to obtain the SoTA results are also unclear for reproducibility. [1] Yao, Huaxiu, Xinyu Yang, Xinyi Pan, Shengchao Liu, Pang Wei Koh, and Chelsea Finn. "Leveraging domain relations for domain generalization." *arXiv preprint arXiv:2302.02609* (2023).
Questions
1. In Algorithm 1, the target model and the experts are trained simultaneously, which is a more parallel and iterative scheme rather than a sequential one with two stages, i.e., training the experts first, then distilling the knowledge to the target model using logit regularization. Why is such a design choice made? Are there better justifications than just “for simplicity”? Please explain with more details. 2. Each expert is generally weak as they are only trained using the data from their own domain. Is it necessary and how to ensure the quality of the experts? 3. In Appendix A, Limitations, the authors discussed that LFME cannot be applied to single-domain tasks. How does the number of domains affect the proposed approach? Will the algorithm benefit from having more domains with less data or more data in fewer domains?
Rating
4
Confidence
4
Soundness
2
Presentation
2
Contribution
2
Limitations
The limitations has been discussed in Appendix A.
Response to further concerns
We appreciate the reviewer's feedback and thoughtful response. We answer the additional questions below. > Q: This is because originally we are doing gradient descent using ERM. When we multiply a negative value in front of it, it becomes gradient ascent instead. A: We want to clarify that our only objetive is to minimize the loss $\mathcal{L}_{all}$ in Eq. (3) throughout the training process. It's important to emphasize that there is no gradient reversal operation in our framework; specifically, we are not performing gradient descent using ERM followed by multiplying a negative value for gradient ascent. > Q: How is $\mathcal{F}$ calculated. A: we want to clarify that we do not need to compute $\mathcal{F}$ during updating, $\mathcal{F}$ is used as an analytical tool to show how much the gradient magtitudes is rescaled w.r.t to the baseline model that without using the logit regularization term, by comparing ratio of gradients from the two models [64]. In our case, gradients of the two models (LFME and ERM) are $q_c - y_c + \alpha (z_c - q_c^E)$ (e.g. $\frac{\partial \mathcal{L}\_{all}}{\partial z_c}$) and $q_c - y_c$ (e.g. $\frac{\partial \mathcal{L}\_{cla}}{ \partial z_c}$), there gradient ratio is thus $\frac{\partial \mathcal{L}\_{all}}{\partial z_c} / \frac{\partial \mathcal{L}\_{cla}}{ \partial z_c}$, which leads to the computing of $\mathcal{F}$ and $\mathcal{F}'$ in Eq. (6) and (8). > Q: In the given example, how the mining the "hard" sample from an expert achieved and its relation to the reduction of the logit value. A: For the given example, where $z_1 = z_2$, it indicates that the model is with equal confidence w.r.t the two samples. Without experts involvements, these two samples will contribute equally to the target model's updating. However, when the expert is involved and $q_{\ast 1}^{E} > q_{\ast 2}^{E}$，Eq. (6) and (8) indicate $|\mathcal{F}_1| < |\mathcal{F}_2|$ and $|\mathcal{F}'_1| < |\mathcal{F}'_2|$: the gradient magtitude from $x_2$ is changing more significantly than that from $x_1$ both on the basis ERM, allowing $x_2$ to have a greater impact on the target model's update. The derivations imply that the expert can expose its hard samples to the target model, which is not applicable in ERM. From your point of view, the objetive in Eq. (3) implicitly encourages the target model to be less confident w.r.t $x_2$. We want to clarify that this confidence reduction does not conflict with mining $x_2$, because lower confidence results in a higher training loss, prompting the target model to explore the corresponding sample more thoroughly, where both the dominant features for classification and other information shared with other classes will be explored, with the dominant features being the primary focus (because both $\mathcal{L}\_{cla}$ and $\mathcal{L}\_{guid}$ in $\mathcal{L}\_{all}$ encourage the gt logit to be the largest, which also indicates that lower confidence in LFME does not equate to lower accuracy—a fact validated by our in-domain results). On the contrary, high confidence indicates that the model may overlook the sample after obtaining its dominant features, making it less important on the updating process. Moreover, penalizing high-confidence outputs has been shown to be effective in improving a variety of learning tasks [a], our objective can thus be viewed as specifically mining hard samples from the experts for penalization. From this perspective, in our case, penalizing confident can be integrated within our explanation of hard sample mining. We sincerely thank you for your invaluable time and the fruitful discussions. We are more than willing to provide any additional justifications or clarifications you may need. [a] Regularizing Neural Networks by Penalizing Confident Output Distributions, in ICLR'17
Reviewer Response
Thank you for the prompt clarifications. Unfortunately, I am still not convinced and couldn't agree with the counterintuitive rationale. > Specifically, we are not performing gradient descent using ERM followed by multiplying a negative value for gradient ascent. I understand LFME is not doing this explicitly, but aren't they mathematically equivalent? If it doesn't align, then how does the analysis in Section 4.2 provide valid justifications? I think when we are trying to investigate the working mechanism of the proposed algorithm, the story needs to be coherent and not mislead the readers, especially when the analysis consists of a major section of the main paper. > Penalizing high-confidence outputs has been shown to be effective in improving a variety of learning tasks As far as I understand, arguing from the confidence perspective might be more reasonable than hard sample mining.
Reply to your concerns
We thank the reviewer for the prompt reply. > Q: Is updating LFME mathematically equivalent to doing gradient descent using ERM followed by multiplying a negative value for gradient ascent? A: We want to clarify that the two gradient parts of $\mathcal{L}\_{all}$ w.r.t $z$ (e.g. $\partial \mathcal{L}\_{cla}$ and $\partial \mathcal{L}\_{guid}$) may better be interpreted as a whole in a updating step, as one updating step requires the combination of the two gradients for a joint classification task. According to your suggestion, if we specifically interpret the updating process by separately considering $\partial \mathcal{L}\_{cla}$ and $\partial \mathcal{L}\_{guid}$, we can see that in both terms, $z\_{\ast}$ correspond to largest labels. That is, $z\_{\ast}$ is encouraged to be close to $y\_{\ast}$ by minimizing $\mathcal{L}\_{cla}$ and close to $q\_{\ast}^E$ by minimizing $\mathcal{L}\_{guid}$. This means the updating process will maintain its discriminative ability, and the model will not forget the sample. The different gradient directions can be interpreted as $\mathcal{L}\_{cla}$ encourages $z\_{\ast}$ to become infinitely large, focusing on dominant features, while $\mathcal{L}\_{guid}$ optimizes $z\_{\ast}$ to the opposite direction to be close to $q\_{\ast}^E$ so that other information can also be involved. Therefore, $\partial \mathcal{L}\_{guid}$ should not be sees as tries to "unlearn" what has been learned by$\partial \mathcal{L}\_{cla}$. Rather, on the basis of not compromizing the target model's discriminative ability, they serve different purposes to encourage the model to focus on different types of features. > Q: Is mining hard samples from experts counterintuitive? A: The primary effect of knowledge distillation (KD)-based methods can be attributed to sample reweighting [b], where the weight of a training sample in the student model is related to the teacher's confidence. Although a different KD scheme is used, we apply the same rescaling analytical tool to demonstrate that one of the working mechanisms of the proposed KD form can still be seen as weighting different samples based on the teacher's confidence. Since different KD forms are utilized, the weighting strategies differ between the proposed LFME and the traditional KD approach where easier samples from teachers correspond to larger weights (comparisons between LFME and traditional KD ideas are presented in Sec. 6.3, where LFME performs better for the task of DG). Meanwhile, since hard samples from domain experts are more likely to be domain ambiguous data, enforcing the target model to learn these data more throughly can help it to explore more domain-invariant features, which are crucial for generalization [17]. In this regard, mining hard samples from domain experts is an intuitive motivation for improving DG. > Q: Arguing from the the confidence perspective. A: We thank the reviewer for the thoughtful suggestion. We want to clarify that the hard sample mining is aligned with the proposed confidence reduction perspective: in the given example, the target model will reduce confidence of $x_2$ more than that of $x_1$ even if they are with the same output logits, consistent with our explanation that hard samples from the experts will affect more on the target model's updating. Therefore, there may be no intrinsic difference between the hard sample mining and the suggested confidence reduction idea. Meanwhile, we provide empirical evidence in Sec. D.3 to show that the classification results of hard samples from corresponding experts are indeed improved in LFME, we thus argue the working mechanism from the hard sample mining perspective. On the other hand, merely analyzing the confidence is similar to our initial analysis in Section 4.1, where the idea of smoothing the output probability of the target model is equivalent to making it less overconfident. We sincerely thank you for your invaluable time and all thoughtful suggestions, which definetly help improve our paper. We wish these clarifications are helpful for easing your concerns. [b] Born-Again Neural Networks, in ICML'18.
Reviewer Response
Thank you for the detailed responses and comprehensive experiments, which have addressed most of my concerns. However, I am still not convinced by the arguments on mining the hard samples and the analysis of why the proposed approach works. In Sec. 4.2, the rescaling factor $\mathcal{F}, \mathcal{F}’$ are shown to be negative in Figure 2(e). While it is true that the magnitude is larger for harder examples, if the gradient direction is reversed, then the model is no longer learning the sample as it's going the opposite of the original ERM gradient. I am not sure if we can still interpret this as "mining" the hard samples because they are somewhat being "forgotten". Please clarify.
Thank you for the feedback
We sincerely thank the reviewer for the prompt reply! We would like to clalrify that if two networks have different gradient directions, it does not mean one network is "forgetting" the samples compared to the other. The negative values of $\mathcal{F}$ and $\mathcal{F}'$ apply for all training samples, but this does not imply that LFME is forgetting all samples in contrast to ERM. Since hard samples from experts correspond to larger gradient magtitude for the target model, they will have more significant impacts on the target model's updating, which can shift the decision boundaries not aligned with easy samples that with small gradient magtitude, potentially causing these easy samples to be overlooked by the target instead. We thus say the target model can mine hard samples from the experts. The argument in Sec. 4.2 can be better interpreted from the following example: assuming two samples $x_1$ and $x_2$ from the same class and domain, with the target model producing identical output logits for both: $z_1 = z_2$. If the corresponding expert has different levels of confidence regarding these samples, such that $q_{\ast 1}^{E} > q_{\ast 2}^{E}$, the analysis in Sec. 4.2 suggests that $x_2$ will have larger impacts on the updating of the target model than $x_1$, and the target model will focus more on $x_2$. We thank again for the reviewer's time, and wish our clarification is helpful.
Reviewer Response
Practically as the algorithm is working, I would like to believe that LFME is not forgetting all samples. However, the math and plot in Section 4.2 say otherwise. This is because originally we are doing gradient descent using ERM. When we multiply a negative value in front of it, it becomes gradient ascent instead. According to the plot, it's happening across most of the iteration steps, which is quite puzzling. I am not sure how is $\mathcal{F}$ calculated practically, but it might be helpful to clarify it. Moreover, if we just look at Eq. (4), if $q_{*1}^{E} > q_{*2}^{E}$, the direct interpretation is to reduce the logit value of $x_2$ more than $x_1$, so that the model should not be overconfident about it. I honestly couldn't see how this is mining the "hard" sample from an "unconfident" expert.
Thank you for the detailed responses. My previous concerns have been well addressed. After carefully reviewing the other reviewers' comments and the authors' replies, I believe the paper has no significant flaws, and therefore, I choose to maintain my score.
Decision
Accept (poster)