Subspace Identification for Multi-Source Domain Adaptation

Summary

This paper considered multiple-source domain adaptation problems in the context of causal representation learning. Specifically, this paper considered quite general scenarios in the domain adaptation by figuring four different representations. - Domain specific and label irrelevant - Domain specific and label relevant - Domain invariant and label irrelevant - Domain invariant and label relevant Notably, this paper derived theoretical assumptions via illustrating when we could identify these four representations (or subspace). A VAE based approach in deep learning is further proposed to identify the subspace. Empirical results are further validated in synthetic data, benchmark domain adaptation dataset (including challenging DomainNET dataset)

Strengths

Overall this reviewer feels that this paper has a solid contribution in the theoretical aspects in causal based domain adaptation. Extensive experiments further validated the practical utility. - **Significance & Originality** This paper proposes a very solid theoretical justification in the identifiability issue in multi-source da, which is a fundamental problem. More importantly, this paper studied the general setting (or most difficult settings) in multi-source da. Besides, theoretical assumptions are proposed, which is generally non-trivial. - **Quality** The theoretical assumptions and proof seem valid and reasonable for me. The proposed practical method is also reasonable. Empirical results in toy data and real-data demonstrated that the proposed method could better identify different representations. - **Clarity** In general, this paper is clearly written and well-explained. Since this paper is a bit theoretical, there are still several theoretical points that need to be clarified (see the weakness part).

Weaknesses

Since this paper covers several fundamental proofs, this reviewer feels uncertain about several points within the paper. 1. Theorem 1 about theoretical assumptions. In general, I could follow the proof, where intuitions within these assumptions could be better explained. I just take theorem 1 as an example. - About conditional independence. If I understand correctly, this implies, the representation variable $P(Z|U) = \prod P(z_i|U)$ ? (sort of disentanglement or mean-field approximation assumption)? If we write it as a log probability term, it will be $\log P(Z|U) = \sum \log P(Z_i|U)$, which is consistent with your paper… - About linear independence. This assumption seems to illustrate that the gradient of the probability density over each component should be linearly independent? That implies if we use a gradient based approach to learn each component, each component will converge to a distinct direction. Thus we could possibly identify different independent components? - I have a general question in theory 1 (for example). Is it possible to extend this assumption into the blockwise vectors? i.e, $P(Z|U) = P(Z_1|U)P(Z_2|U)P(Z_3|U)P(Z_4|U)$, where $Z_1$ is a subspace vector rather than a scalar. 2. About explanations in toy data. It seems that the proposed method works quite significantly in toy data. However the experimental details (such as the data generation) could be better clarified. For example, is there some sort of spurious correlation? 3. In Equation (1) Line 3, I would think this holds when we have figure 1 by considering the conditional independence. 4. I like the example in explaining different subspace in line 82-92. It could be better to consider this example into more real-world scenarios such as drug discovery and health. For example, data collected from different hospitals may create different $Z_1$, this could be helpful to better understand the importance of considering a general scenario.

Questions

See comments in the weakness part.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper studies the problem of multi source domain adaptation where we have access to multiple labeled source domains and unlabeled target domain. Authors consider a novel data generation process by modeling them through 4 new latent variables (i.e. combinations of domain specific or domain invariant and label relevant/ label irrelevant). This is done to relax the assumptions made by previous theoretical analyses. The authors provide sufficient theoretical analysis in this context by providing subspace identification guarantees and using that to improve the results. They conduct multiple experiments include synthetic experiments and the results are satisfactory.

Strengths

* The paper is well-presented. * The paper provides new framework through their modeling of data generation process by introducing the 4 latent variables and then considering them to influences from the domain index and the target labels. * Their theoretical analysis is convincing and experiments are impressive.

Weaknesses

* This framework is somewhat similar to an earlier work on domain generalization(https://arxiv.org/pdf/2102.11436.pdf) . I agree that your data generation process further expands on their structural causal model but noting the similarities it should be cited and expand on the differences * Importantly, since we model label relevant attributes separately, should we still make the assumption that source and target contain the same labels. * My question specifically is can you comment on if this framework naturally support open-set/partial MSDAs.

Questions

Please comment on the weakness I raised above and if you could provide some results even on one dataset where there is either a lesser number of classes in target than in source (https://arxiv.org/abs/1808.04205) or like in other open set DA problems that would be great

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors have adequately addressed the limitations.

Summary

In this paper, the authors investigate the problem of multi-source domain adaptation. In detail, the authors first devise a causal generation process, which consider the domain-specific and label-irrelevant, domain-specific and label-relevant, domain-invariant and label-invariant, domain-invariant and label-irrelevant variables. Based on the aforementioned generation process, the authors raise the identification guarantees for the latent variables. The authors also evaluate the proposed methods on several benchmarks and achieve good performance.

Strengths

1. The authors provide an interesting perspective for multi-source domain adaptation, which uses nonlinear ICA to identify the latent variables. Compared with the existing methods, the proposed method relaxes the assumptions. 2. The authors evaluate the proposed methods on several datasets and achieve the ideal results.

Weaknesses

1. According to the theoretical results, the authors give three assumptions which is similar to those of nonlinear ICA literature. Why are these existing assumptions applicable in real-world scenarios? Moreover, according to the subspace identification results, the dimension of latent variables is small and restricted by the number of domains, which is hard to be met in practice. 2. For the real-world scenario, the number of ground-truth latent variables is unknown, is it reasonable to assume that their number is according to the observed domains? 3. The authors equip a cross-attention module to the ResNet101 for the DomainNet dataset, it is suggested the authors provide the experiment results of other datasets, e.g., Office-home, with the proposed cross-attention module. Minors: There are some typos in this paper. For example, the illustrations of Lemma2 in the main and appendix are inconsistent.

Questions

NO

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NON

Summary

This paper proposes a novel framework to tackle the problem of multi-source unsupervised domain adaptation. Existing methods often have stringent requirements, such as a large number of domains and invariant label distributions. To address these limitations, this paper proposes a subspace identification theory that disentangles domain-invariant and domain-specific variables under less restrictive constraints. The proposed Subspace Identification Guarantee (SIG) model is based on variational inference and additionally incorporates class-aware conditional alignment to handle domain shifts. Experimental results demonstrate that the SIG model outperforms existing SOTA techniques on four benchmark datasets (OfficeHome, ImageCLEF, PACS, DomainNet), showcasing its effectiveness in real-world applications.

Strengths

- The paper defines three major drawbacks of [a] – invariant label distribution, requiring a large number of domains and monotonic transformation between latent variables. These assumptions are relaxed by the *Subspace Identification Guarantee* (SIG) model proposed in this paper. Most interestingly, while Kong et al. claimed that for $n$ dimensional latent space, $2n+1$ domains would be necessary, authors here have shown that “n+1” domains are sufficient. - The core ideas on subspace identifiability are theoretically grounded. The paper defines a new data generative process based on a 4-way split of the latent space and builds upon the theoretical results to define a new multi-source unsupervised domain adaptation framework using a VAE-based architecture. - Results are presented on 4 benchmark datasets, with the proposed model outperforming all other SOTA approaches. [a] Kong et. al., Partial Identifiability for Domain Adaptation, ICML 2022

Weaknesses

- In the data generation process, the paper introduces 4 latent variables from the product of two sets {domain-specific, domain-invariant} $\times$ {label-relevant, label-irrelevant}. I am unsure what the intuitive reasoning behind the "domain-invariant label-irrelevant" latent variable $\mathbf{z}_4$ is. - There are no individual constraints on $\mathbf{z}_1$ and $\mathbf{z}_4$ during the learning process. Are they identifiable individually? - While the paper claims that the proposed SIG model can handle all three types of shifts: covariate, target, and conditional, experiments focus primarily on covariate shift (based on the datasets used). It’s unclear how the datasets, which contain mostly covariate shifts, were used for studying the two other shifts. - The clarity could be improved, especially when introducing the SIG model. For example, in Section 4.2, $\hat{\mathbf{z}}_{3,\mathcal{S}}^{(i)}$ is denoted as "..latent variables of i-th class from source". How is this estimated?

Questions

Please see weaknesses. Also, the derivation of the ELBO in Equation 5 should be provided to enhance the completeness of the paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors discuss the limitations adequately.

Answers my questions.

Dear Authors, Thank you for taking the time and running some more experiments. I appreciate the effort and I note the improvements in MSDA settings too. I am happy to increase my score.