WATT: Weight Average Test-Time Adaptation of CLIP

Summary

The authors introduce a Test-Time Adaptation technique named Weight Average Test-Time Adaptation (WATT) for Vision-Language Models (VLMs) such as CLIP. WATT is the proposed method that improves test-time adaptation by borrowing ideas from different text prompt templates to develop pseudo-labels for model updates and also carrying out weight averaging so as to consolidate learned information from those generated labels. The approach is tested across different datasets, revealing its capacity to boost performance without any further metamorphosis of the model or additional trainable modules.

Strengths

+ The novelty of the study is that it proposes a novel test-time approach. It involves averaging weights while using varying text prompts which deviates from the conventional TTA techniques— making it a high-quality extension to existing TTA methods. + It ensures quality by using rigorous experimental setup: involving thorough evaluations on various datasets, with good performance demonstrated by improvements over the current leading techniques. + It maintains clarity in its presentation: both of methodology and results. This has been made possible by visual aids and detailed elaborations that ensure better understanding, thus though we still need to work more on this document to achieve perfection. + The importance of the proposed method lies in its ability to adapt using a single image without requiring any further changes to the model; this level of adaptability is highly significant for real-world applications across different fields.

Weaknesses

- Limited Scope of Evaluation: State-of-the-art Methods [1, 2] published in recent CVPR/ICCV are missing for comparing in Experimental part and discussing in the Related Work section. [1] Diverse data augmentation with diffusions for effective test-time prompt tuning. ICCV, 2023. [2] Efficient Test-Time Adaptation of Vision-Language Models. CVPR, 2024. - Computational Cost: The paper mentions the efficiency of WATT but does not provide a detailed comparison of computational costs with other TTA methods, which could be crucial for practical deployment. - Generalization to Other Tasks: The paper focuses on image classification tasks. Extending the approach to other vision tasks like segmentation or detection and evaluating its performance there could provide a more comprehensive understanding of its applicability.

Questions

1. Could you provide a more detailed comparison of the computational cost of WATT compared to other TTA methods? 2. Have you considered evaluating WATT on real-world datasets with more complex domain shifts? 3. How would WATT perform on other vision tasks such as segmentation or object detection?

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have addressed the limitations and potential negative societal impacts adequately. They highlight the focus on image classification and suggest future work extending the method to other tasks. However, they could discuss more on the computational resource requirements and scalability in practical deployments.

Summary

This paper proposes a test-time adaptation (TTA) method for CLIP by integrating various textual prompt templates into Weight Average (WA) methods (Ref [11], [22]). Experiments on multiple types of domain shifts show the effectiveness of the proposed method.

Strengths

- The proposed method is effective yet simple. - The experiments span different domain shift scenarios and demonstrate better performance of the method.

Weaknesses

- The second contribution at the end of Section 1 lacks sufficient evidence. Table 4 only shows that the proposed method can handle small batch sizes without comparison with others. Additionally, there are existing TTA works studying small batch size scenarios. (SAR [Towards Stable Test-Time Adaptation in Dynamic Wild World, 2023], MEMO [MEMO: Test Time Robustness via Adaptation and Augmentation, 2022], etc.). Recent diffusion model-based TTA methods (such as DDA [Back to the Source: Diffusion-Driven Test-Time Adaptation, 2023]) also show robustness for small batch sizes. - The technical contribution seems not that significant. The core idea is introducing multiple prompt templates into two previous WA methods, to my understanding.

Questions

Is the evaluation done online (like the setting in the TENT paper where inference and adaptation happen on each batch in data streams), or does the model adapt with a single batch and then perform inference on the whole test set?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors mention limitations in the checklist.

Summary

This paper proposes a method called Weight Average Test-Time Adaptation (WATT) to improve test-time adaptation (TTA) for the CLIP model. The core idea is to use different text templates to construct multiple text prompts and adapt the model weights using these different prompts. During the evaluation stage, the prompts provide predictions based on the adapted weights and text embedding averaging across multiple prompts. The authors conduct comprehensive experiments that demonstrate the generalization and strong performance of the WATT method.

Strengths

* The idea is simple but works effectively. * Comprehensive experiments support the effectiveness of this method in the following aspects: Robustness, as reflected in Table 6. Efficient adaptation, as reflected in Table 4. Generalization and state-of-the-art comparison, as shown in Table 7.

Weaknesses

There are no clear weaknesses, but I do have some questions and potential limitations that do not affect my rating.

Questions

* Is it possible to combine text prompt augmentation (this method) with TTA strategies that use image augmentation to achieve better results? Some templates may be incorrect. For example, as shown in Table 2, prompt T0: "a photo of a {class k}" and prompt T2: "a bad photo of the {class k}" refer to the same image, but why is this photo considered bad? If you add distortion or corruption, then it might be appropriate to call it bad. Also, for T6: "art of the {class k}," if you use diffusion to generate an art image of the original image, then the prompt would match. * Can you measure the similarity score of these text templates in the CLIP text encoder space?

Rating

7

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

Can the authors test whether this method can also improve the latest VLM, such as SigLIP?

Reply to authors

Thank you for your rebuttal. All my concerns have been addressed. The results reflected in SigLip are very positive, so I have increased my score.