Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization

Summary

This paper tackles the task of test-time prompt tuning. It proposes to align the out-of-distribution test sample statistics to those of the source data. The ImageNet is used as an effective proxy source dataset. Evaluation results show that the proposed method obtains better performances on domain generalization and cross-dataset generalization tasks.

Strengths

1. This paper studies test-time adaptation for foundation models. Given the popularity of V-L models and prompt tuning, the paper may be interesting to many researches. 2. Exploring distribution alignment in V-L models at test time is new. The proposed method is easy to implement and can be generalized to different model architectures. 3. The paper is well structured and easy to follow. Experiments are extensive enough. PromptAlign obtains consistently better accuracies than previous methods in most cases.

Weaknesses

1. The method highly relies on the ImageNet statistics. In the supplementary, the authors show that using ImageNet statistics gives better performance than using other small datasets. It is unclear if this also applies to V-L models other than CLIP. 2. The alignment loss, in my understanding, is a regularization term on the visual prompts to avoid large devation during adaptation. A simple manner is to regularize the difference between initial prompt values (or statistics from the test image) and adapted values. Such comparison is missing.

Questions

1. In Tab.4, is PromptAlign$^\dagger$ equivalent to MaPLe+alignment loss? If the alignment loss alone does not improve test-time adaptation, how to validate that narrowing the distribution gap real matters when adapting with entropy loss? 2. How does the proposed method perform on different CLIP backbone architectures?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have included possible limitations in the end of paper. The discussions are constructive.

Summary

Building on top of the prompting method and architecture from MaPLE and test-time adaptation strategy of TPT, the work ads an additional alignment penalty, based on pre-computed statistics. The statistics are computed offline on a target source dataset. The method is evaluated then for domain generalisation on a set of datasets.

Strengths

- The paper reads well and its easy to follow. - Table 1 reports gains compared with the TPT method baseline. - The conept of test-time adaptation is interesting.

Weaknesses

- Somewhat low novelty: The method combines 2 existing works: takes the architecture and prompting strategy from MaPLE and the adaptation mechanism from TPT. The only addition is an alignment loss based on pre-computed statistics. - Can this approach be combined with other methods (ie. CoOp, CoCoOp etc), similarly as its the case for TPT. If so, what is its performance? Further evaluations are needed in this direction. - Similarly, the comparison from Table 2 and 3 are unfair. The proposed method should be compared with a similar baseline, i.e. TPT+MaPLE, as done in Table 1. Currently, especially for Table 3 its unclear what are the gains, if any. - Missing comparisons with current state-of-the-art methods (e.g. Neural Prompt Search, LASP, ProDA etc). - No evaluation on the standard settings from CoOp (i.e. without ImageNet pretraining) and CoCoOp( base-to-new setting). - Missing analysis of certain ablation. For example, why is L1 offering the best results in this case? What makes it more suitable? - What is the impact of the quality of the statistics on the overall performance? - No FLOPs/compute analysis: What is the inference time and FLOPs (i.e. including adaptation time) per sample? How does this compare with TPT? How much extra time is spent on pre-computing the statistics? It looks than in some case ity may be simply faster to train and eval CoOp or a similar method, especially as the size of the dataset and/or the number of classes increases. - L154, sum from 1 to K, where is the index i used on the denominator? It's a bit ambigous currently which f_t abd f_v are used.

Questions

See the above section

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Adequate limitation statement is present.

Summary

This paper introduces a distribution alignment strategy, termed PromptAlign, for V-L models to improve test-time adaptation. To be specific, they propose a distribution alignment loss that utilizes offline computed source data statistics to encourage the test sample token distributions to be aligned with the source data token distributions. Since CLIP-pre-training data is not publicly released, they directly use ImageNet as a possible candidate for the source distribution. Experiments on multiple benchmark datasets reveal the zero-shot generalization ability of the proposed PromptAlign during test-time adaptation.

Strengths

- The paper is well-written with a clear structure. The research on prompting for large Vision-Language models holds practical value. - The proposed PromptAlign, with its straightforward implementation, has achieved good results on a variety of test benchmarks.

Weaknesses

- *Proxy source dataset*. Although using an off-the-shelf large-scale dataset for distribution alignment is intuitive. It lacks theoretical justifications or insights regarding the choice, making the whole process somewhat heuristic. Moreover, as the statistics of the source dataset are computed offline, how can the method be applied to some dynamically changing environments that may significantly differ from the source data distribution? - *Distribution alignment*. Using mean and variance for distribution alignment still lacks theoretical or more in-depth analyses. This appears to be the foundation of the paper, yet I haven't observed a particularly robust theoretical explanation. Could the authors provide more insights regarding this point? - *Experimental results*. Based on the reported experimental results, the proposed method does not demonstrate a significant performance improvement compared to TPT-based methods.

Questions

Please refer to the weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

In this paper, the authors proposed a novel approach for enhancing test-time prompting with distribution alignment of Vision-Language (V-L) models for zero-shot generalization. A loss between the means and variances of the test sample and the proxy source dataset statistics was proposed to align the distributions during testing. The authors conducted several experiments on multiple datasets and showed improved performance over several baseline methods.

Strengths

1. Combining Test-Time Prompting and Distribution Alignment for Zero-Shot Generalization makes sense to me and the proposed alignment loss is also technically sound. 2. The authors conducted extensive experiments and ablation studies to sufficiently validate the proposed approach over multiple baseline methods on several benchmarking datasets. 3. Writing is good and easy to follow. The tables and figures are also easy to understand.

Weaknesses

1. I am not sure the mean and variance of a single test example would be sufficient for distribution alignment even though multiple augmented views are used. Could this lead to over-fitting? Would there be further gain with more test examples? 2. I am not sure how the proxy source dataset would affect the performance. Since LAION400M is more closer to the training data of CLIP, it could also be used as a proxy source dataset. It would be interesting to ablate the effect of ImageNet versus LAION400M as proxy source dataset. 3. The authors use means and variances of tokens to align the distribution. I am wondering would these be sufficient or more sophisticated statistics or losses are needed for better alignment. I would expect more insights on why these two was used.

Questions

Please refer to the weaknesses section for details.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Thanks for the responses

I appreciate the authors' responses. My major concerns have been well addressed. After carefully reviewing the feedback from other reviewers, the authors' replies, and re-reading the original manuscript, I've decided to maintain a relatively positive score.