Summary
This paper investigates the performance of parameter-efficient finetuning methods (PEFT) and mixture-of-adapters (MoA) for the domain generalization task. They argue that simple empirical risk minimization (ERM) on the source data without regularization performs well on the target domain data, providing a benefit compared to methods that require regularization or custom training schemes. They argue that PEFT methods have a flatter loss landscape, and suggest this as an explanation for the generalization of such methods in this setting.
Strengths
The strength of the method is that running the ERM algorithm with LoRA or KAdaption is easy to implement and doesn't require complicated hyperparameter tuning, and gives compelling performance out-of-the-box. The analysis of the loss landscapes for PEFT methods in this context is a nice idea, and they provide some nice visualizations.
Weaknesses
I have some qualms about the evaluation of the method in the paper. First, there is no comparison with other works that study domain transfer using PEFT approaches, such as [1, 2, 3, 4] which use prompt tuning (a common PEFT approach). The papers [1, 2, 3, 4] evaluate on the CORe50 domain transfer benchmark and [1, 4] evaluate on DomainNet (one of the datasets used in the paper under review). Admittedly these papers are continual learning papers which is a slightly different setting, however since domain-incremental learning (seeing domains sequentially) is a harder task we can take the performance in these papers as a rough lower bound of what one should get in the setting of training on the union of all source domains "source combine", which is the setting studied by the authors in the present submission. We note that when prompting the CLIP trained VIT-B encoder (note the submitted paper uses OpenCLIP which is very similar) the paper [4] achieves 67.0 accuracy on the unseen domains in DomainNet using the cumulative accuracy metric (which is a harder task than training on all domains then transferring to a single domain). Meanwhile the present work gets around 62-63% on this same benchmark in an easier setting. The earlier work [1] achieved 67.78 in the same evaluation setting when prompting both the vision and language encoder for CLIP (admittedly the prompt training of the language encoder comes at an additional training cost, but for inference the embeddings can be cached and thus this has a negligible effect on inference).
My second qualm is this paper compares against baselines that use a different pretraining e.g. SWAG IG3B. The issue with this is the success of domain transfer depends critically on which representations you have from your backbone architecture. If the backbone architecture has been trained on similar domains to the test data then the representations will transfer, however if there is a significant domain gap (e.g. evaluating on FGVC aircrafts using an ImageNet pretraining) then performance is poor. In the case of domain transfer studied in this paper this issue becomes even more crucial, as there is not sufficient training data to cover the target domain.
[1] Yabin Wang, Zhiwu Huang, Xiaopeng Hong. S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning. NeurIPS 2022.
[2] Benjamin Bowman et al. A-la-carte prompt tuning (APT): Combining
distinct data via composable prompting. CVPR 2023
[3] Zifeng Wang et al. Learning to Prompt for Continual Learning. CVPR 2022.
[4] Julien Nicolas et al. MoP-CLIP: A Mixture of Prompt-Tuned CLIP Models for Domain Incremental Learning.
Questions
**Questions**
(Figure 3): You are plotting a spectral density using only 5 eigenvalues? Why is the w/LoRA line flat? What is the x-axis and the y-axis depicting? There are no labels for either axis.
(Figure 2): Which two dimension of the loss landscape are you exploring in the figure?
Note: I like the patch visualizations, however I am not fully convinced of the argument of the routing corresponding to semantic information etc. In my opinion it is hard to discover any visual patterns just by looking at the examples. If there was a segmentation mask for the objects and a corresponding quantitative evaluation I would be convinced, however visually it is hard to make such an argument.
**Typos and Writing Issues**
“domain generalization (DG) algorithm aims” → “domain generalization (DG) algorithms aim”
“One of the most effective method” → “One of the most effective methods” (added s to methods)
“(a.k.a unseen domains)” → “(a.k.a. unseen domains)” (added period after second a in a.k.a.)
“to well predict on the target domain” → “to predict well on the target domain”
“shows that ERM algorithm” → “shows that the ERM algorithm”
“minimizes the loss on each domains” → “minimizes the loss on each domain”
“where K′ denotes the total number of training domains” → “where K’ is the number of target domains”
Add periods after paragraph titles “Effect of fine-tuning in pretrained models” and “Ablation study on routing strategy in Mixture-of-Adapter” for formatting consistency
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.