Domain Generalization (DG) techniques have emerged as a popular approach to address the challenges of domain shift in Deep Learning (DL), with the goal of generalizing well to the target domain unseen during the training. In recent years, numerous methods have been proposed to address the DG setting, among which one popular approach is the adversarial learning-based methodology. The main idea behind adversarial DG methods is to learn domain-invariant features by minimizing a discrepancy metric. However, most adversarial DG methods use 0-1 loss based $\mathcal{H}Δ\mathcal{H}$ divergence metric. In contrast, the margin loss-based discrepancy metric has the following advantages: more informative, tighter, practical, and efficiently optimizable. To mitigate this gap, this work proposes a novel adversarial learning DG algorithm, MADG, motivated by a margin loss-based discrepancy metric. The proposed MADG model learns domain-invariant features across all source domains and uses adversarial training to generalize well to the unseen target domain. We also provide a theoretical analysis of the proposed MADG model based on the unseen target error bound. Specifically, we construct the link between the source and unseen domains in the real-valued hypothesis space and derive the generalization bound using margin loss and Rademacher complexity. We extensively experiment with the MADG model on popular real-world DG datasets, VLCS, PACS, OfficeHome, DomainNet, and TerraIncognita. We evaluate the proposed algorithm on DomainBed's benchmark and observe consistent performance across all the datasets.
Paper
Similar papers
Peer review
Summary
This paper aims to ease the distribution problem from a theoretical perspective, which uses margin loss and a scoring function to describe the relationship between domains, and the generalization bound in terms of functional class complexity is subsequently analyzed. Based on their theoretical analysis, a margin-based adversarial framework, which is developed upon the classical DANN method, is further proposed. Results conducted in the Domanbed benchmark show their results are competitive among existing arts.
Strengths
1. It is always good to see some theoretical analysis for DG. 2. Experiments will the five datasets are appreciated.
Weaknesses
1. The motivation in the introduction is problematic. The literature is happy to see inspiring theory, but the theory itself is not the purpose. Rather, the paper should emphasize what part of existing work requires a proper theoretical explanation. After all, how to solve domain generalization is the problem. The authors may consider revise the introduction part. 2. The idea of adversarial training for DG (either DANN [77] or MMD [79]), is shown to be less effective than ERM according to different benchmarks [32, a]. However, this work shows significantly better results than ERM, what is the advantage of the proposed MADG compared with them? 3. According to Line 282, and the implementation details in the supplementary material, I think the comparisons are unfair, as the Domainbed uses randomly selected hyperparameters (batch size, learning rate, etc.), which are fixed in their experiments. 4. More effective approaches should be compared, such as SD [b], and Miro [c], and it is suggested to reevaluate their code (at least some of them) on the same device. [a] OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization, in CVPR'21. [b] Gradient Starvation: A Learning Proclivity in Neural Networks, in NeurIPS'21. [c] MIRO: Mutual Information Regularization with Oracle, in ECCV'22.
Questions
See weakness.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
It seems domain labels must be available during training, this should be listed as a limitation as domain labels are not always available.
Summary
The paper proposes a new adversarial learning objective using a margin based approach for domain generalization. The goal of domain generalization broadly is to build classifiers that are trained on one or more source domains, and are expected to generalize to an unseen target domain. The key idea is to leverage Margin disparity discrepancy (MDD) which quantifies the level of disagreement between decision boundaries of classifiers using their margins. MDD is used as a proxy to understand generalizability of classifiers in this context across multiple domains. Next, the paper establishes an upper bound on the generalization error on any unseen target domain that is within the convex hull of the source domains and MDD. In other words, the objective is to get the decision boundaries across different source domains to agree as much as possible, while also minimizing the empirical errors on them simultaneously.
Strengths
* The margin perspective for domain generalization is a fresh perspective to this, and the paper takes an interesting approach at using an adversarial learning strategy to optimizing this problem. * Detailed theoretical setup and formulation, which builds on existing work and setup the formulation for MADG * Results are impressive — on several benchmarks, the proposed method appears to perform competitively. * good ablations and analysis of the proposed method.
Weaknesses
* I found the paper to be hard to read in general, at its core, the paper is proposing to model error on the unseen target using a convex hull of the source domains — which is a standard idea in many generalization papers. The novelty, in my opinion, is to use the MDD as an objective for determining discrepancy between domains, and the adversarial game. This can be clarified significantly to make sections 4, 5 more readable. There is too much of notation and terminology that obfuscates the reader from understanding the key contributions of the paper. I am taking the proofs and theorems at face value, and have not verified them. * As far as i understand, MADG requires to train significantly more models (1 per domain, a feature extractor and the main classifier) in order to compute MDD and perform training. This is vastly more complex than any existing method. It may be that making progress in a hard problem like domain generalization requires this, but i think this trade-off needs to be made more explicit — compute vs generalization performance. Whereas a simple ERM or Mixup have little to no overhead and perform very closely on the metrics considered in the paper.
Questions
See above
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
some what, yes.
Summary
- The paper presents a margin-loss based analysis of domain generalization. - First, the paper derives a bound on margin disparity discrepancy (MDD) of any unseen domain within the convex hull of source domains in terms of the margin loss on source domains, ideal margin loss, and max MDD between any two source domains. - The paper relates this to any unseen domain via projection onto the convex hull of source domains, and an additional factor $\gamma$ - The paper then realizes the bound for the empirical setting using a bound based on the Rademacher complexity of the function class - Based on this bound, the paper proposes an adversarial learning method to regularize training by minimizing MDD in addition to classification loss. - The paper demonstrates the efficacy of the method on a variety of real world datasets in the DomainBed benchmark.
Strengths
1. The paper proposes a principled method for domain generalization, and the empirical method follows neatly from the derived generalization bound. 2. The proposed bound for error on an unseen domain based on margin disparities between source domains is novel to my knowledge. 3. Empirical results show modest improvements over ERM. 4. The paper presents empirical ablations for some design choices used in the algorithm.
Weaknesses
It is not clear if the single optimization step for the adversarial models f’ is sufficient for tightly approximating MDD. The method effectively regularizes a lower bound on the MDD term, which requires the adversarial models to be effective maximizers — this may require many steps for the adversarial models per main-model step. Table 1 presents results using the model selection strategy of [71], which uses out-of-distribution data as the validation set for picking hyperparameters. Using labelled OOD data for hyperparameter selection limits the conclusions we can draw about if the proposed method works in practice, where we do not usually have access to labelled test data. While the AD, GD, and M metrics allay some of this concern, presenting results with some methodology that could be used for model-selection in practice, such as leave-one-out domain validation would make the empirical results stronger.
Questions
Why is only the feature extractor trained to minimize MDD loss and not the classifier (line 261)? Theorem 2 shows that the error of any unseen domain can be bounded via a projection onto the convex hull of the source domain, by adding in an additional factor $\gamma$. It is not clear if this factor is small or very large for domain generalization benchmarks. Is there intuition for “how far” an unseen domain in a benchmark is to the source domains?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
It is not completely clear how significant the empirical results are. The method slightly outperforms ERM on average, but ERM outperforms the method on 2 out of 5 considered datasets, and is competitive on 2 others. Additionally, the reliance on labelled OOD data for model selection limits the conclusions one can draw about the efficacy of the method on the DomainBed benchmark.
Summary
This paper is commendable for its innovative use of a margin-based theoretical framework to solve domain generalization problems, which contrasts with the largely heuristic and empirical approaches adopted by existing methods. By grounding their approach in a theoretical foundation, the authors provide more interpretable solutions that can contribute to the field of domain generalization.
Strengths
- The claim presented in the paper is strongly supported by theoretical analysis, as well as the extensive ablation studies presented in section 7. This speaks to the methodological rigour of the study and strengthens its credibility. - The development of a theoretical framework for solving domain generalization problems is indeed a significant strength of this paper. This approach not only enhances the comprehensibility and reproducibility of the method, but also contributes to the broader understanding of the problem.
Weaknesses
- While the average performance of the proposed method is the highest, the performance gap across different tasks is quite significant. The method shows the best performance on the Office Home dataset, but this alone is not enough to convincingly demonstrate the overall superiority of the method. Therefore, the claimed significance of the proposed method appears to be overstated. - The authors' claim that a theoretical approach is necessary is quite a strong statement that seems to undermine the importance of empirically validated methods in the field. This makes it hard to fully agree with their motivation. - While the authors make a reasonable argument for the use of margin as a metric in generalization, they do not provide a sufficient justification for its relevance in domain generalization problems, particularly where style shifts are involved. This makes it difficult to understand the authors' motivation for their margin-based approach. - Similarly, the reasoning behind the use of adversarial learning (min-max framework) is not clearly explained. The proposed method appears to be similar to the one used in [1], and a clear explanation of the differences between these two methods, apart from the task, would be beneficial. - The paper lacks a comparison with recent Domain Generalization (DG) works [2-4]. This makes it difficult to assess how the proposed method stacks up against the state of the art. [1] Maximum Mean Discrepancy Test is Aware of Adversarial Attacks [2] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution [3] Domain Generalization by Mutual-Information Regularization with Pre-trained Models [4] SIMPLE: Specialized Model-Sample Matching for Domain Generalization
Questions
Please see weakness
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors have well described their limitations.
Summary
This paper investigates domain generalization (DG) using a margin-based theoretical framework. The authors first formulate the generalization upper bound by leveraging the margin disparity discrepancy (MDD). Then, an adversarial learning strategy (MADG) is devised to minimize the empirical MMD between source domains. The experiments on the DomainBed benchmark further demonstrate that the proposed MADG could outperform the current state-of-the-art methods.
Strengths
1. The paper is clearly written, and the method is well-designed. 2. The theory of margin-based generalization bound is interesting and could benefit the further designs of DG methods. 3. The proposed MADG could outperform recent state-of-the-art methods.
Weaknesses
1. The primary concern of the reviewer is in its effectiveness. While MADG outperforms most baselines in the DomainBed benchmark, the improvement is marginal/minor. This made it unclear to me if the method actually works better, or it is just a product of optimizing some hyperparameters. 2. The motivation, “very little work has been done in developing DG algorithms that are well-motivated by theoretical”, lacks soundness. In fact, there have been notable works in the literature that utilize theoretical frameworks to analyze the generalization problem, such as gradient matching [71, 88] and invariant risk minimization [80]. It could be beneficial if the authors could provide a detailed comparison with such theoretical methods. 3. In addition to the accuracy comparison, could the authors provide a more in-depth analysis regarding the training dynamics? This would help shed light on the inner workings of the MADG approach. 4. Many advantages, such as efficient optimization, stated in the abstract are not well verified. Specifically, following Eq. (17), it seems that the estimation process could introduce a significant computational cost, as it iteratively computes for all domains.
Questions
1. Table 1 could be simplified, as many baselines are unnecessary. 2. The common approach to compare DG methods is the training-domain validation. Why do the authors mainly utilize the test-domain selection?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
See above comments.
Thanks for the response
Thanks for the rebuttal, some of my concerns are addressed. But my main concern regarding the hyper-parameter setting, which I think leads to unfair comparisons, remains. In Table 8 (not Table 6) in [32], the parameter for each trial is randomly selected in a uniform distribution, and the performance is averaged from different trials. That being said, the parameters are different for different trials, and simply selecting the best-performing parameter for all trials violates the random selecting rule. Besides, different datasets use the same set of hyper-parameter in [32], which are also reported differently in Table A1 in the supp. material. For this reason, I cannot recommend acceptance for this work.
We thank the reviewer for acknowledging our response, and are glad to know that it helped resolve concerns. On the pending issue, we'd like to clarify any misunderstanding. Firstly, on a minor note, we meant Table 6 in the ICLR version of [32] (the version available on OpenReview), which is Table 8 of its arxiv version [32]. Secondly and more importantly, Sec 5 (Pg 6) in the ICLR version of [32] under the ‘Hyperparameter search’ subsection states that: “**`*All hyperparameters are optimized anew for each algorithm and test domain*`, including hyperparameters like learning rates which are common to multiple algorithms.**” Similarly, Sec 5 (Pg 7) in the arxiv version of [32] under the ‘Hyperparameter search’ subsection states that: "**For each algorithm and test environment, we conduct a random search of 20 trials over the hyperparameter distribution (see Appendix D). We use each model selection method from Section 3 to `*select amongst the 20 models*` from the random search.**” Thus, [32] *DOES NOT* average performance across the 20 trials, but suggests selecting the best hyperparameters among the 20 trials, each of which is obtained from random search. We exactly follow the same procedure in our work too (note that the list of hyperparameters in our rebuttal was randomly selected). Also, in the official code implementation of [32] (code link provided in Sec 4, pg 5, of both ICLR and arXiv versions), note that in Lines 114 and Lines 123-128 of `list_top_hparams.py` file , dataset is passed as an argument, which is then used to list the best hyperparameters for each dataset individually (see Line 141). Line 141 in turn calls `model_selection.py`, where Lines 27-40 clearly states that the hyperparameters are chosen per dataset. Following this, we use the same procedure and report the top hyper-parameters for each dataset in Table A1 in the Appendix. We hope this can clarify our choice. If there is any more information required, please let us know.
Thanks for the clarification
My apology for misunderstanding the part. My main concern is resolved now. Perhaps the author can include a representation saying that the model selection method is "test-domain validation" (given that "training-domain validation" is more common in recent works). I'm raising my rating to 5 (not higher because the introduction and experimental parts may require major revisions).
We thank the reviewer for increasing the rating and are glad to know that our response helped resolve their main concern. We understand the editorial concerns. We will leverage the additional page that NeurIPS provides in the final version to include the clarifications for these editorial concerns, as well as other additional information provided in the rebuttals. In particular, we list below the specific changes we plan to make to the paper: 1. In Sec 1 (L37-41), we will rephrase our claim on motivating our method using a theoretical framework and instead motivate the proposed work as an adversarial DG method based on margin loss because of its advantages as discussed in Sec 4 (L129-144). We will mention our theoretical analysis as an additional dimension of our efforts in this work. 2. In Sec 2 (L65-66 and L73-76), we will describe in detail the existing adversarial methods [38-42] and clearly differentiate the advantage of using the margin loss in our work. In particular, we will highlight the comparison with $H \Delta H$ discrepancy, and how the margin loss allows us to use the classifier/task in our solution. 3. In Sec 6 (L279-280), we will clearly state that we follow [71] in using 'Test-domain validation' for hyperparameter selection. 4. In Sec A2 (L86-87) in the Appendix, we will describe the procedure for hyperparameter tuning followed in [32] and also include a Table that lists the different search range values for each hyperparameter used in our work. 5. We will explicity state in the Limitations section (Sec A6 in the Appendix) regarding the requirement of domain labels during training, similar to recent work such as [71][88]. 6. We will include the additional results included in the rebuttal PDF in the paper. (We also have additional comparisons with SD[b] now, which we could not include in our first rebuttal due to time constraints -- our method outperforms SD, and we will include these too.) We hope the above changes capture the requested clarifications and present our research in the paper appropriately to the community. We also hope this can convince the reviewer on our efforts to clearly address pending issues (which are largely editorial at this time), to improve the paper's presentation and to make it useful. We will be happy to provide any further clarifications if needed.
Thank you to the authors for their comprehensive response and the additional experiments provided. Concerns about 2-4 have been resolved. However, I still have concerns regarding the performance gap evaluations across different tasks and the comparisons with recent studies. From the response provided in the PDF, I am not entirely convinced that MADG surpasses MIRO or even when compared to Fishr. Consequently, due to limited evaluation, I will maintain my score at 5.
We thank the reviewer for acknowledging our response, and are glad to know that it helped resolve concerns. Considering the experimental studies in the main paper, ablation studies, additional results on other experimental settings in the appendix and those in the rebuttal, we humbly submit that our evaluations are fairly comprehensive. We believe that the reviewer is pointing to the performance comparisons here than the amount of experimental studies itself. Regarding the gap in performance evaluation, as stated in our first response, DomainBed is a challenging benchmark and we achieve consistent performance on all its constituent datasets as shown using the AD, GD and M metrics. We initially did not have MIRO in our comparisons, since their experimental setting is different from ours (we follow [71] in the experimental settings). However, our comparisons included in the rebuttal PDF, show that we outperform MIRO on our experimental setting. To go further, we also compared our method, MADG, with another recent method, SD [a, Neurips 2021], as suggested by the reviewers and show results in the tables below. The proposed MADG outperforms SD on all domains across all the four datasets. We will add these to the paper. Regarding performance comparison with Fishr, as seen in Table 1 of our paper, our method outperforms the Fishr model on all three evaluation metrics that reward consistency of performance across domains/datasets: AD, GD, M. We'd like to add here that existing work on DG are designed from different perspectives, such as domain-invariant representation learning, data/feature manipulation, and meta-learning/optimization. We believe each perspective has its specific advantages and can achieve superior performance on specific domains/datasets. Consequently, it may not be easy for a method from one perspective to achieve a large improvement than another perspective across domains/datasets. We show that our method outperforms other methods that use the same experimental setting (following [71]) consistently. Our method has a novel contribution in its margin-based discrepancy and adversarial formulation. The combination of our method with complementary ones, e.g. those based on data/feature augmentation may only provide opportunities for future work and even better results. We hope this can address any pending concerns w.r.t. performance evaluation. We are grateful for the opportunity to engage with the reviewer, which has only helped us improve the paper. If there is any more information required, please let us know. Table 1: Accuracy (%) on OfficeHome dataset |Model|Art|Clipart|Product|Real World| Average| |-------|--------|-------|-------|------|------| |SD[a]|64.8($\pm0.9$)|49.9($\pm1.2$)|75.6($\pm1.3$)|79.1($\pm0.2$)|67.4 |MADG (ours)| 68.6($\pm0.5$)|55.5($\pm 0.2$)|79.6($\pm0.3$)|81.5($\pm0.4$)|71.3| Table 2: Accuracy (%) on PACS dataset |Model|Art_Painting|Cartoon |Photo |Sketch | Average| |-------|--------|-------|------- |-------|------| |SD[a]|82.7($\pm0.9$)|77.4($\pm3$)|97.1($\pm0.7$)|68.5($\pm2.4$)|81.4| |MADG (ours)|87.8($\pm0.5$)|82.2($\pm0.6$)|97.7($\pm0.3$)|78.3($\pm0.4$)|86.5| Table 3: Accuracy (%) on VLCS dataset |Model|PASCAL|CALTECH|LABELME|SUN| Average| |-------|--------|-------|-------|------|------| |SD[a]|72.6($1.7$)|91.4($\pm1.7$)|64.1($\pm1.6$)|63.3($\pm1.2$)|72.3| |MADG (ours)|77.3 ($\pm0.1$)|98.5($\pm0.2$)|65.8($\pm0.3$)|73.1($\pm0.3$)|78.7| Table 4: Accuracy (%) on TerraIncognita dataset |Model|L100|L46|L43|L38| Average| |-------|--------|-------|-------|------|------| |SD[a]|39.5 ($\pm8.7$)|41.4 ($\pm1.2$)|49.2 ($\pm1$)|34.1($\pm6.2$)|41.1| |MADG (ours)|60.0 ($\pm1.2$)|45.6($\pm0.5$)|57.4($\pm0.3$)|51.8($\pm0.2$)|53.7| [a] Pezeshki, Mohammad, et al. "Gradient starvation: A learning proclivity in neural networks." Advances in Neural Information Processing Systems 34 (2021): 1256-1272.
The authors have addressed the concerns, in particular regarding the validations.
We are happy to know that the reviewer's concerns have been addressed. We sincerely appreciate the thoughtful feedback which helped us improve the presentation of our work. We will leverage the additional page that NeurIPS provides in the final version to include the clarifications for the reviewer's concerns, as well as other additional information provided in the rebuttals. We would appreciate if you could kindly consider revising the score accordingly. Once more, we express our gratitude for the time you've dedicated to these thoughtful discussions.
Dear reviewer, Thank you for your great efforts in reviewing our paper and offering insightful and constructive comments. The deadline for the author-reviewer discussion is approaching. We wanted to check if our rebuttal addressed your concerns, and would appreciate the opportunity to discuss further with you. Understanding the demands on your schedule, we express our sincere gratitude for your review. Your suggestions/comments will be significantly beneficial to the improvement of our work.
Dear reviewer, Thank you for your great efforts in reviewing our paper and offering insightful and constructive comments. The deadline for the author-reviewer discussion is approaching. We wanted to check if our rebuttal addressed your concerns, and would appreciate the opportunity to discuss further with you. Understanding the demands on your schedule, we express our sincere gratitude for your review. Your suggestions/comments will be significantly beneficial to the improvement of our work.
response
Thanks for addressing my concerns. I reiterate my observation earlier that this is a new perspective on the domain generalization problem, and the authors have done a good job of empirical validation across a range of challenging benchmarks. As some of the other reviewers noted, there needs to be a significant update to the writing to make the theory more accessible, and writing more clear for a final publication. I also agree with the concerns that the performance w.r.t more recent baselines like MIRO show little to no improvement, but on the overall, i think this paper is a valuable contribution. I will raise my score to reflect this.
We thank the reviewer for increasing the score and for appreciating our contributions in this paper. We understand the editorial concerns. We will *leverage the additional page that NeurIPS provides in the final version* to include the clarifications for these editorial concerns, as well as other additional information provided in the rebuttals. We list below a set of changes we plan to make to the paper: 1. In Sec 1 (L49-53), we will clearly state the novelty of this work. In particular, we will discuss the advantages of using a margin-based discrepancy (MDD) for the DG problem, as elaborated in Sec 4 (L129-144), and discuss the adversarial learning formulation of the proposed MADG algorithm that leverages MDD. 2. In Sec 2 (L65-66 and L73-76), we will describe in detail existing adversarial methods [38-42] and clearly differentiate the advantage of using the margin loss in our work. We will highlight the comparison with $H \Delta H$ discrepancy, and how margin loss allows us to use the classifier/task in our solution. 3. To make it easier for a reader to follow Sections 4 and 5, in Sec 4, after the introductory paragraph to motivate (margin-based) MDD, we will revise L145-151 with a clear description of the organizational structure of Sec 4 and 5. In particular, we will state: "*In this section, we derive a generalization bound for an unseen domain based on the margin-based MDD loss in the DG setting. To this end, we first show an upper bound on the unlabeled source domain error given other labeled source domains (Lemma 1 and Theorem 1). We then leverage this to develop the upper bound for the error on an unseen domain that is not necessarily a source domain (Lemma 2, Theorem 2 and Corollary 1). We subsequently analyze the upper bound from Corollary 1 using the Rademacher complexity framework and develop our final generalization bound for the unseen target domain in the DG setting (Lemma 3 and Theorem 3) using our magin-based loss. In Section 5, we show the formulation of the proposed adversarial learning algorithm, MADG, motivated by the generalization bound in Section 4, that employs MDD to address the DG problem.*" 4. In Sec 7, we will add a 'Computational cost' analysis subsection that explicitly states the tradeoff between the computation and the generalization performance for our model, and empirically show that our computational cost is similar to other DG baselines. We will also include Table 1 from our rebuttal in this discussion. Regarding comparison with recent baselines, as stated in our first response, DomainBed is a challenging benchmark and we achieve consistent performance on all its constituent datasets as shown using the AD, GD and M metrics. We initially did not have MIRO in our comparisons, since their experimental setting is different from ours (we follow [71] in the experimental settings). However, our comparisons included in the rebuttal PDF, show that we outperform MIRO on our experimental setting. To go further, we also compare our method, MADG, with another recent method, SD [a, Neurips 2021], as suggested by the reviewers and show results in the tables below. The proposed MADG outperforms SD on all the domains across all the four datasets. We hope the above changes capture the editorial clarifications and present our research in the paper appropriately to the community. We are grateful for all the reviewer's suggestions, which has only improved the paper and its presentation. We will be happy to provide any further clarifications if needed. Table 1: Accuracy (%) on OfficeHome dataset |Model|Art|Clipart|Product|Real World| Average| |-------|--------|-------|-------|------|------| |SD[a]|64.8($\pm0.9$)|49.9($\pm1.2$)|75.6($\pm1.3$)|79.1($\pm0.2$)|67.4 |MADG (ours)| 68.6($\pm0.5$)|55.5($\pm 0.2$)|79.6($\pm0.3$)|81.5($\pm0.4$)|71.3| Table 2: Accuracy (%) on PACS dataset |Model|Art_Painting|Cartoon |Photo |Sketch | Average| |-------|--------|-------|------- |-------|------| |SD[a]|82.7($\pm0.9$)|77.4($\pm3$)|97.1($\pm0.7$)|68.5($\pm2.4$)|81.4| |MADG (ours)|87.8($\pm0.5$)|82.2($\pm0.6$)|97.7($\pm0.3$)|78.3($\pm0.4$)|86.5| Table 3: Accuracy (%) on VLCS dataset |Model|PASCAL|CALTECH|LABELME|SUN| Average| |-------|--------|-------|-------|------|------| |SD[a]|72.6($1.7$)|91.4($\pm1.7$)|64.1($\pm1.6$)|63.3($\pm1.2$)|72.3| |MADG (ours)|77.3 ($\pm0.1$)|98.5($\pm0.2$)|65.8($\pm0.3$)|73.1($\pm0.3$)|78.7| Table 4: Accuracy (%) on TerraIncognita dataset |Model|L100|L46|L43|L38| Average| |-------|--------|-------|-------|------|------| |SD[a]|39.5 ($\pm8.7$)|41.4 ($\pm1.2$)|49.2 ($\pm1$)|34.1($\pm6.2$)|41.1| |MADG (ours)|60.0 ($\pm1.2$)|45.6($\pm0.5$)|57.4($\pm0.3$)|51.8($\pm0.2$)|53.7| [a] Pezeshki, Mohammad, et al. "Gradient starvation: A learning proclivity in neural networks." Advances in Neural Information Processing Systems 34 (2021): 1256-1272.
Response
I thank the authors for addressing my questions. The new ablation resolves my concern related to adversarial optimization. I am currently inclined to retain my score due to the following remaining concerns: 1. I thank the authors for checking the approximate values of gamma on DomainBed. It appears that the empirically observed values are all close to or above 1. Does this not make the bounds of theorems 2 and 3 very loose for settings such as DomainBed where there may be very few source domains (the error rate can be 1 at most anyway)? If that is the case, it would support the story of the paper to motivate why optimizing the other terms still leads to improved DG. Alternatively, are there empirical settings the authors can identify where gamma is small? 2. I appreciate that DomainBed is a challenging benchmark to consistently outperform ERM on and no DG methods lead to meaningful performance improvements. However, to show that MADG is a useful algorithm for domain generalization, there needs to be an experiment where MADG leads to convincingly better results than ERM. This doesn't have to be DomainBed, it could be a simpler synthetic setting (for example with the unseen domain close to or within the convex hull of source domains) — [71] considers ColoredMNIST and a linear task where Fishr outperforms ERM significantly.
We thank the reviewer for acknowledging our response, and are glad to know that it helped resolve concerns. The reviewer is absolutely correct. The empirically observed values of gamma are all close to or above 1 for a challenging benchmark like DomainBed. Hence, in this work, we focus on optimizing the first two terms in Theorem 3 in our proposed MADG method and demonstrate improved performance on the DomainBed benchmark. Optimizing all the terms in Theorem 3 would be a natural and interesting extension of our work. Based on the reviewer's suggestions, we ran experiments on ColoredMNIST (more discussion below), where we noted the gamma values to be small for all domains (expectedly so since the distributions of different domains in ColoredMNIST are relatively close to each other, when compared to real-world datasets). Regarding the comparision with ERM model, we thank the reviewer for the suggestion. We conducted studies on the ColoredMNIST dataset, as shown in the tables below. As seen in Table 2, with almost no hyperparamter tuning (due to limited time), the proposed MADG algorithm achieves 65.6\% average accuracy, which is significantly higher than the ERM model (57.8\% average accuracy), thus showing that using a margin-based DG algorithm, MADG, learns better domain-invariant features on the ColoredMNIST dataset. Besides, Table 1 shows that gamma is small across domains in this dataset, showcasing the promise of the proposed method when the unseen domain is within the convex hull of source domain. We will leverage the additional page provided in NeurIPS for the final version to include these results and discussions. We hope that this clarifies the reviewer's concerns. We are grateful for the opportunity to engage with the reviewer, and improve our paper. We'd be happy to provide any further information or clarifications if required and permitted. Table 1: Pairwise JS divergence for the ColoredMNIST dataset and the approximated projection value ($\gamma$) of the unseen domain on the convex hull. |OfficeHome|+90\%|+80\%|-90\%| ~$\gamma$| |-----|-----|-----|-----|-----| |+90\%|0|0.019|0.024|0.043| |+80\%|0.019|0|0.029|0.048| |-90\%|0.024|0.029|0|0.054| Table 2: Accuracy (%) on ColoredMNIST dataset |Model|+90\%|+80\%|-90\%|Average| |-------|--------|-------|-------|------| |ERM|71.8 ($\pm0.4$)| 72.9($\pm0.1$)| 28.7 ($\pm0.5$)| 57.8| |MADG| 72.3 ($\pm0.3$)| 74.0 ($\pm0.2$) | 50.5 ($\pm0.4$)|65.6|
Response
Thanks to the authors for the follow up. The new results on Colored-MNIST provide early evidence that MADG can outperform ERM in settings where the unseen domain is close to the convex hull of the source domains. I have raised my score to reflect these results, and encourage the author to revise the paper to include them.
We thank the reviewer for increasing the score. We will leverage the additional page provided in NeurIPS for the final version to include these results and discussions.
Decision
Accept (poster)