Summary
The paper proposes an extension of [10] (Domain Adaptation via Prompt Learning Ge et al., 2022) to the multi-source UDA set-up.
(i) Distinct soft prompts are learnt via contrastive loss for each source-target pair; each source-target prompt is composed of class-wise source- and target-prompts. In the target domain, learning is achieved by utilizing pseudo-labels from the pretrained CLIP model, which possesses strong zero-shot ability.
To encourage consistent target outputs across all learned prompts, a consistency L1 loss is applied to the soft outputs of all target samples.
(ii) Prompt reconstruction: to eliminate redundant information that may impede performance, an autoencoder (AE) is learned to "denoise" the acquired prompts.
(iii) Latent Subspace Tuning (LST): efficient adaptation to a new target domain, after learning from the first target domain, is accomplished by optimizing on the latent space of the learned AE using pseudo-labels of the new target.
The proposed framework demonstrates superior results compared to previous State-of-the-Art (SOTA) methods in multi-source UDA.
Strengths
The proposed method leverages the powerful zero-shot capability of the pretrained CLIP model for the multi-source domain adaptation (DA) task. Empirical results demonstrate its effectiveness across various benchmarking setups. The presentation is good, enabling easy comprehension of the method.
Weaknesses
The main concern of this work is the lack of technical novelty and a more rigorous evaluation. The proposed framework is a straightforward extension of the work by Get et al. (2022) [10] to the multi-source setting. In terms of method evaluation, it is expected to compare against stronger CLIP-based baselines, both in the multi-source UDA experiments (Tables 1 and 2) and in the LST experiment (Table 3). Further details are provided in the following section.
Questions
Given that CLIP alone produces results close to the state-of-the-art (SOTA) as shown in Table 1, and even outperforms the SOTA as shown in Table 2, my main concern is whether the authors have made adequate efforts to create decent baselines using CLIP-pretrained models. I appreciate their effort in Section 4.3 when they ``swap MFSAN’s ResNet50 backbone pre-trained on ImageNet to CLIP’s image encoder''. However, I would love to know more details to know as if careful considerations were taken into account for MFSAN + CLIP or simply the authors swap the backbone's weight. For example, did the authors preserve the text classifier and the contrastive los of CLIP ? As the proposed framework is based on self-training with pseudo-labels, I wonder if how the CLIP + self-training and Single Prompt + self-training baselines perform.
Furthermore, I am curious to know if CLIP with fixed prompts has been pushed to its maximum extent. In other words, how far can this naïve baseline reach with better prompt engineering instead of using the simple prompt "a photo of [CLS]"? One can imagine incorporating domain-specific information into the prompt or defining a set of templates rather than relying on just one template.
In the LST experiment, how good is MPA (on the first domain) + self-training? For efficiency, one can try test time prompt tuning (TPT).
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
This work lacks significant technical novelty, and the experiment could be improved by using stronger baselines. Based on the current state of the submission, my recommendation leans towards the negative side.
---- After rebuttal ----
The rebuttal is convincing and has helped clarify most of my technical concerns. I believe that this work is indeed interesting and help advocate the usage of prompts in domain adaptation. While the novelty limitation still pertains, the original set of experiments, along with the new ones provided during the rebuttal, is sufficient. I believe the paper after revision would certainly pass the NeurIPS' bar.