Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation

Summary

This paper presents a novel unsupervised domain adaptation method called invariant CONsistency learning (ICON). ICON is very simple; assuming that labeled source samples and clustered target samples are available, ICON uses BCE losses to make the inner product of the features (softmax-normalized) of any two samples in a mini-batch in each domain close to 1 if they are of the same class/cluster, and 0 otherwise. Evaluation experiments on 10 different benchmark datasets (Office-Home, VisDA-2017, WILDS 2.0) show that ICON achieves superior performance to several representative methods on most of them.

Strengths

The proposed method, ICON, is surprisingly simple and easy to use. A theoretical property of ICON is discussed; ICON gives the optimal classifier under certain assumptions. (this is a fairly high-level characteristic, and the realism of the assumptions are somewhat questionable, though.) Various experiments are conducted. Despite its simplicity, ICON achieves sufficiently high accuracy on many datasets. The paper is generally well-written and easy to follow.

Weaknesses

1. The performance of ICON on Office-Home and VisDA-2017 is inferior to that of SoTA. For example, CDTrans [Xu+, ICLR2022] achieves 88.4% on VisDA-2017 and 80.5% on OfficeHome, both higher than ICON. 2. Since the assumptions underlying the theorem appear to be quite strong, it is questionable to what extent they are valid in practice. (this is discussed more or less in the limitation part in the supplementary material, though.) 3. Intuitively, the principle of ICON (i.e., bringing features within the same class/cluster closer together) seems highly similar to that of contrastive learning, which is also a major approach to unsupervised domain adaptation (e.g. [Shen+, ICML2022 ], [Wang+, TMM2023]). Discussing the differences will highlight the property and uniqueness of ICON. 4. While this may be outside the scope of this paper, it would be interesting to discuss the possibility of extending to more advanced domain adaptation problems, such as universal domain adaptation and source-free domain adaptation. Since ICON (perhaps implicitly) assumes that the number of classes in the source and target are the same, and the impact of the number of clusters on accuracy is significant (see Table 2). So the performance of ICON on universal domain adaptation, where the number of classes cannot be assumed to be the same, may not be as promising. Application to source-free domain adaptation is also non-trivial. [Xu+, ICLR2022] CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation [Shen+, ICML2022] Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation [Wang+, TMM2023] Cross-Domain Contrastive Learning for Unsupervised Domain Adaptation

Questions

I have no specific questions that I would like the authors to answer, but it would be great if I could read some discussions on the weaknesses I mentioned above, especially on 1. the comparisons with SoTA.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I found a discussion of that limitation in the supplemental material, and it is very reasonable.

Summary

This paper proposes ICON (Invariant CONsistency learning), a method to utilize the distribution of unlabeled target data in the UDA task. And it obtains stable performance improvements over 8 UDA tasks.

Strengths

1. The idea of the article is simple, but it makes sense that the distribution of unlabeled target data does have an impact on UDA tasks. 2. In terms of learning the distribution of unlabeled data, this paper has some novelty in using pseudo labels from low-dimension clustering. 3. This article provides sufficient experiments, and the effectiveness of ICON has been verified under various UDA settings

Weaknesses

Essentially, I think that the core of the ICON method is a pseudo label-based contrastive learning technology, where the pseudo label is from low-dimension clustering. And the authors provide some experiments about the rationality of the ICON, like Fig. 5,6,8. But none of this fundamentally or theoretically explains why low dimensional pseudo labels are more accurate.

Questions

1. Could you provide more analysis about "why low dimensional pseudo labels are more accurate?" 2. If the low-dimensional pseudo labeling works better, why do you still use high dimensions in the final validation?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The author has provided limitations in their appendix.

Summary

This paper deals with unsupervised domain adaptation problem. This paper focuses on how to exploit the inherent distribution of target domain to improve the adaptation performance. In detail, it trains two classifier: one is on source domain and the other one is on target domain. Each classifier is trained with BCE loss on a pair of source samples or target samples. For source domain, the same ground-truth label of a pair of samples indicates a positive pair; for target domain, the same cluster label (obtained through clustering) of a pair of samples indicates a positive pair. Finally, the intersection of two classifier is exploited to form the optimal classifier. Additional cross-entropy loss and self-training loss are added to the objective to supervise the adaptation process. Experiment results on various benchmarks validate the effectiveness of proposed method.

Strengths

- The idea is interesting and reasonable to an extent. - The paper is generally easy to understand and follow. - The method has been tested on various UDA benchmarks.

Weaknesses

- In one of the claimed contributions, the authors state that their method ICON is orthogonal to previous UDA methods. But I don't see any evidence showing that previous UDA methods combined with ICON should achieve a better adaptation performance. - In my opinion, it is not suitable to claim outperforming all the conventional methods on WILDS 2.0 UDA benchmark without fully evaluating existing previous works. - It is interesting to use REx to realize the goal of the second line of Eq. (2). I just wonder what if we simply share the $f$ between two domains. Will it achieve similar performance? I would like to see such ablations. - For the experiments, the authors should compare with more recent SOTA UDA methods. The current status of comparisons listed in Table 1 is unsatisfactory and may not justify the performance superiority of proposed method. - As shown in Fig. 5, the model seems to be sensitive to the hyper-parameter selection. As we know, it is hard to perform hyper-parameter selection in UDA as we don't have labels of target domain data. I feel concerned about whether this work can be really useful in practice. - The details about how to utilize the self-training loss are not clear. The ablations for self-training terms are missing. Without providing such ablations, we cannot know if the improvement really comes from what the authors claimed.

Questions

See the weakness part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes a new UDA method which strives to produce a consistent classifier for labels in source domain and clusters in target domain. Specifically, this paper introduces an auxiliary task for distinguishing whether the input image pair share the same class/cluster or not. This binary classification task would encourage the features from the same class to be similar and features from different classes to be dissimilar. The groundtruth for this binary classification task on target pair is determined by performing clustering on target features. By combining this binary classification task with supervised learning on source domain and self training on target domain, this method obtains state-of-the-art performance on multiple challenging UDA benchmark.

Strengths

1. This paper utilizes clustering on target features to determine whether the target image pair share the same class. Compared to previous self-training methods, this design better reduces the noise in the pseudo labels and alleviate overfitting on incorrect pseudo labels. 2. The proposed method obtains state-of-the-art performance on many challenging UDA tasks. A sufficient ablation study and analysis clearly depicts the advantage of the proposed method, which would provide great insight for furtehr works.

Weaknesses

Basically I believe the method proposed method greatly meets its motivation and sufficient analysis has proven its effectiveness. The introduction part could be further polished to emphasize the core idea better. For figure 1, the lack of comparison on t-SNE of baselines might make the reader difficult to understand the advantage of ICON.

Questions

1. In line41 there are double commas. 2. The BCE loss is only applied on source-source pair or target-target pair. What if applying it on source-target pair? Does it bring better performance? 3. How the pseudo labels for $L_{st}$ is obtained? A short introduction should be included in the implementation details. 4. Now that the main idea is to utilize the cluster label to group target domain features, is it necessary to add a BCE task for UDA classification? Is it possible to utilize the cluster label as the pseudo label for self-training?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

This paper adequately discussed the limitation of the proposed method and provide an analysis about the possible reasons. No potential negative societal impact is discussed.

Summary

The authors propose an unsupervised domain adaptation method, ICON. The algorithm is similar to self-training on the unlabeled target data, but at the start of each epoch, the unlabeled data are first projected from feature space to a reduced-dimension space and clustered. An auxiliary loss enforces consistent pseudo-labels within clusters. The method attained state-of-the-art on the WILDS unlabeled benchmark [1]. [1] Sagawa, S., Koh, P. W., Lee, T., Gao, I., Xie, S. M., Shen, K., ... & Liang, P. (2021). Extending the WILDS benchmark for unsupervised adaptation. *arXiv preprint arXiv:2112.05090*.

Strengths

- ICON’s strong empirical performance is striking, especially on WILDS; the authors outperform similar self-training-style methods, e.g.. vanilla self-training, FixMatch, and Noisy Student. Notably, ICON attains this performance without using aggressive data augmentations. - The paper evaluates on ten datasets across 3 modalities, and evaluations are conducted over multiple seeds. - The presentation is good, and the experiments are notably well-documented.

Weaknesses

- Given its similarity to other self-training methods, it seems important to analyze, whether through experimental ablations or theory, why ICON outperforms vanilla self-training so strikingly. One unique aspect of ICON is that the unlabeled consistency loss uses clusters computed in a reduced dimension space; this seems important in ablations (Table 2). The authors motivate this by stating that dimensionality reduction acts to suppress environmental features and highlight the causal feature (L138), but this statement seems unsupported to me. Could the authors clarify theoretically why this is? - The theory in Section 4 makes two strong assumptions, to the effect of (1) the classes in T are cleanly separated by clusters, and (2) if the model separates classes correctly in S and clusters in T, then the classes in T are predicted correctly, as prediction uses the invariant feature.

Questions

- The authors present ICON as orthogonal to existing UDA approaches, e.g. consistency regularization using views from data augmentations. Could the authors share how ICON performs when combined with strong data augmentations, e.g. RandAugment? - On page 4, the authors state that any clustering algorithm can be used, so long as the number of clusters should be equal to the number of classes. This is a useful ablation; in general, understanding which representation space and how to cluster seems important. Did the authors have k-means experiments to support this? - I could not find the UMAP hyperparameters in the appendix as the main paper suggested; in particular, what was the output dimension used? - The results of iWildCam2020-WILDS and CivilComments-WILDS are a bit unaligned with the theory, as in these cases, the unlabeled data is *not* drawn from the test distribution. It would be great if the main paper could comment briefly on this discrepancy.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors note that the assumptions made in Section 4 are restrictive.