Adaptation of pretrained vision-language models such as CLIP to various downstream tasks have raised great interest in recent researches. Previous works have proposed a variety of test-time adaptation (TTA) methods to achieve strong generalization without any knowledge of the target domain. However, existing training-required TTA approaches like TPT necessitate entropy minimization that involves large computational overhead, while training-free methods like TDA overlook the potential for information mining from the test samples themselves. In this paper, we break down the design of existing popular training-required and training-free TTA methods and bridge the gap between them within our framework. Specifically, we maintain a light-weight key-value memory for feature retrieval from instance-agnostic historical samples and instance-aware boosting samples. The historical samples are filtered from the testing data stream and serve to extract useful information from the target distribution, while the boosting samples are drawn from regional bootstrapping and capture the knowledge of the test sample itself. We theoretically justify the rationality behind our method and empirically verify its effectiveness on both the out-of-distribution and the cross-domain datasets, showcasing its applicability in real-world situations.
Paper
Similar papers
Peer review
Summary
This paper proposes a training-free test-time adaptation approach for vision-language models. It combines the idea of entropy minimization with a training-free adaptor to enhance adaptation performance. The experimental results generally demonstrate the effectiveness of the proposed method.
Strengths
1. The paper effectively combines entropy minimization for storing sample information and a training-free adaptor for test-time adaptation on VLMs. This approach proves to be effective in most cases. 2. The figures presented in the paper are clear and easy to follow. The performance figure, in particular, is very intuitive. 3. The method is theoretically supported, which adds credibility to the proposed approach. 4. The experiments are well-conducted and cover a broad range of considerations.
Weaknesses
1. While the proposed method is effective, it lacks significant novel ideas or insights. The contributions could be seen as incremental rather than groundbreaking. 2. **Figure 2 (b):** The double-arrow directions in Figure 2 (b) should be colored in green and orange for clarity. This part is confusing. 3. **Figure 3:** The term "Boosting Cache" in Figure 3, which appears to store both boosting samples and historical samples, is not explained or used elsewhere in the manuscript, leading to confusion. 4. Several typographical errors need to be corrected. 5. The paper misses parameter study such as the threshold $\tau$. 6. In Figure 4b, the setup for how the model performs entropy minimization, updates learnable prompts or updates the LN/FULL model is missing. This information is crucial for understanding the ablation study. 7. The "Hand-crafted Prompt" is used in this paper, but it is unclear what it entails. Additionally, it is important to investigate if the method maintains stable performance across different prompts. 8. I am also wondering if the method could be categorized into TTA, as it maintains low-entropy samples per class in the cache. Even with such extra information for one or two, it cannot surpass the zero-shot clip. (Figure 4.c)
Questions
1. What unique insights or techniques does your approach introduce that differentiate it from existing methods? 2. Why the proposed BoostAdaptor cannot perform as well as clip when the total shot capacity is less than 3? Is this a general phenomenon for all datasets? 3. Could you explain why the independent cache is good for some datasets as shown in Table 7?
Rating
6
Confidence
3
Soundness
3
Presentation
3
Contribution
2
Limitations
1. While the method is effective, the paper lacks significant novel insights or groundbreaking ideas. The combination of entropy minimization and a training-free adaptor, though practical, may be viewed as an incremental improvement rather than a major innovation. 2. Refer to the weakness and question part.
Summary
This paper studies the problem of test-time vision-language model adaptation. The authors devise training-free method by maintaining a key-value memory for feature retrieval from both historical and boosting samples. The boosting samples are drawn from regional bootstrapping and capture the knowledge of the test sample it self. Experiments demonstrate the effectiveness of the proposed method.
Strengths
The studied training-free adaptation setting is practical, broadening the application scope of test-time adaptation in real-world scenarios. The authors propose incorporating augmentations of each sample into the cache to enable the cache-based classifier to perform better on more fine-grained classifications. This approach is both interesting and technologically sound. The authors also provide theoretical analyses to establish connections between training-required and training-free TTA methods.
Weaknesses
It would be beneficial for the authors to discuss the detailed technical differences from TDA more thoroughly. From my perspective, the key difference appears to be the introduction of additional augmented views of the same sample into the Boosting Cache for intra-sample iteractions. If this is the case, the technical contribution of this work might be a bit limited. The proposed method relies on multiple augmentations, requiring multiple forward passes to achieve better performance than TDA, which sacrifices efficiency. For Figure 4(a), could the authors provide results on more datasets (both OOD domains and cross domains) to demonstrate the sensitivity of the proposed method to this hyper-parameter? This would help verify whether the proposed method can achieve good performance with fewer augmentations across multiple datasets. The improvement on Cross-Domain Benchmarks with RN-50 backbone is a bit marginal.
Questions
Could the authors provide some computational complexity analyses and comparisons for the proposed method, including wall-clock time and GPU memory consumption? It would also be much better to conduct the analysis in Figure 4(c) on more datasets. Is the proposed method applicable to, or have the authors tested it on, pure CNN/ViT models? If not, I recommend including ‘vision-language model adaptation’ in the title. The authors claim that *“prior methods like TDA only consider inter-sample interactions and may fail to generalize well when the downstream tasks require fine-grained knowledge or there is 109 insufficient similarity across samples.”* Although I acknowledge the technological soundness of the proposed method can perform better on more fine-grained classification, are there any empirical evidences to further justify this? In the Boosting Cache, do the authors store the original samples or their corresponding features? Storing images may pose privacy and additional computation issues. It would be helpful to indicate this in Figure 3. [minor] How about the performance of the proposed on corruption datasets such as ImageNet-C?
Rating
4
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
the potential limitation is the computational efficiency of the proposed method, which can be alleviated if the method works well on multiple datasets with a small number of augmentations
Dear reviewer, We would like to thank you for your insightful feedback. We hope that your concerns are addressed with our rebuttal. As we are getting really close to the deadline of the discussion phase, please let us know if there are any further questions that need clarification. Many thanks, Authors
Follow up from reviewer
Thanks for the authors’ response. Regarding the computational efficiency, could the authors provide more details of the experimental setup, including but not limited to the GPU, and batch size? For FPS, I am confused about why BoostAdapter (64 forward passes of image encoder) achieves 11.23 fps and CLIP only achieves 82.3 fps which only needs 1 forward propagation. Meanwhile, the memory consumption of BoostAdapter and CLIP is the same. Do you test Inference Speed (fps) using batch size 64 (64 views also equal to batch size 64), and test Memory using a different batch size? Moreover, the performance could also be directly reported in Table 1 of the PDF.
**Q1. More details** We follow the setting in TDA and deal with the test samples from the data stream one by one. Therefore, we cannot increase the batch size to handle multiple different test samples simultaneously, but only for different views of the same test sample. Thus, we augment 64 views of the test image as TDA does and use 64 as the batch size to obtain the corresponding features. In BoostAdapter, we utilize a simpler augmentation (random crop and random horizontal flip) than AugMix in TDA to save time. Additionally, we set num_workers=8 in the dataloader to leverage multiprocessing for acceleration. Furthermore, we perform feature retrieval over the stored features instead of images. The additional retrieval time compared to TDA comes from the operation of updating the cache with boosted samples. All our experiments are conducted with a 64-core Nvidia 3090 24GB GPU. **Q2. FPS and memory.** For a better view of the time consumption, we provide the average wall-clock time consumption of each component for 1000 samples in TableA. The total time can be mainly divided into three parts: data augmentation, model forwarding, and feature retrieval. - Regarding augmentation time, note that we set num_workers = 8 for the dataloader, so the augmentation takes up a small percentage of the total time. TDA and BoostAdapter take a similar amount of time since they both utilize 64 views of augmentations. - The model forwarding of BoostAdapter utilizes approximately 8 times more time than CLIP. This is reasonable since we use parallel forward propagation for the 64 views instead of sequential forward propagation, so the difference in comsumption overhead will not be as large as 64 times. The only difference is the batch size of the model input (1 for CLIP and 64 for BoostAdapter). - We build a new cache from the historical cache and update it with boosting samples so the feature retrieval time of BoostAdapter takes slightly longer than TDA. Overall, the time consumption in feature retrieval is much smaller than model forwarding. **TableA. Computation cost of each component.** | | Data Augmentation | Model Forwarding | Feature Retrieval | Total | | ------------ | ---------------------- | ----------------------- | ---------------------- | ---------------------- | | CLIP | - | 0.01208 seconds (100%) | - | 0.01208 seconds (100%) | | TDA | 0.00116 seconds (1.4%) | 0.08065 seconds (96.6%) | 0.00167 seconds (2.0%) | 0.08348 seconds (100%) | | BoostAdapter | 0.00098 seconds (1.1%) | 0.07989 seconds (91.5%) | 0.00644 seconds (7.4%) | 0.08731 seconds (100%) | The memory consumption of BoostAdapter is similar to TDA because both utilize 64 views of augmentation, and the model takes 64 views of the image (batch size 64) as input. This model forwarding part accounts for most of the memory usage. During feature retrieval, we store the features in the cache without significant memory overhead, even with the integration of boosting samples' features. In the online test-time adaptation setting, we deal with test samples from the data stream one by one, so we cannot increase the batch size by parallel modeling of different test samples. We only perform parallel forward propagation for different views of the same test sample. Therefore, changing the batch size here is not appropriate, as it corresponds to the number of augmented views for the current test sample. **Q3. Efficiency analysis table.** Thanks for your insightful advice, and we add the performance into the efficiency analysis in the TableB. **TableB. Efficiency analysis with performance results.** | | Augmentation | Views | Inference Speed (fps) | Memory (GB) | OOD Benchmarks Results | Cross-Domain Benchmarks Results | | ------------ | ------------------------------ | ----- | --------------------- | ----------- | ---------------------- | ------------------------------- | | CLIP | - | - | 82.3 | 1.2 | 57.20 | 63.58 | | TPT | Augmix | 64 | 0.29 | 4.5 | 60.81 | 65.10 | | DiffTPT | Diffusion | 64 | 0.10 | 14.4 | 60.52 | 66.92 | | TDA | Augmix | 64 | 11.89 | 1.2 | 63.89 | 67.53 | | BoostAdapter | Rand. Crop & Rand. Horiz. Flip | 64 | 11.23 | 1.2 | 65.15 | 68.52 |
Thanks for the clarifications. It gets clearer now. > "We only perform parallel forward propagation for different views of the same test sample." In this implementation, a further question concerning memory usage is: why do BoostAdapter and CLIP have the same reported memory consumption? Shouldn’t the parallel forward passes of 64 views in BoostAdapter require more memory than a single view forward propagation in CLIP?
Sorry for the confusion about the memory usage of CLIP. It is reported by mistake when we compare the memory consumption of BoostAdapter with TDA. Actually, the memory consumption of CLIP is just 0.7 GB, which is lower than the 1.2 GB of BoostAdapter and TDA. We hope that your questions are addressed.
Summary
The paper focuses on gradient-free test time adaptation of CLIP model with ViT-B/16 and ResNet-50 backbones on out-of-distribution datasets. The authors take inspiration of augmentations from gradient-based test time methods and incorporate this concept of augmentations in gradient-free and memory (cache) based test time methods. Previous work considers only historic samples to be in cache, whereas this work also includes the augmentations of test samples with lower entropy into the cache. Results on OOD benchmark and cross-domain benchmark show improved average performance compared to prior works.
Strengths
1. Proposed a simple but effective approach to include low entropy augmentations into the memory. 2. Established theoretical bounds to justify the inclusion of augmentations into memory and its relation to minimization of empirical risk. 3. Proposed approach brings significant improvements on ImageNet-A, Aircraft and EuroSAT datasets.
Weaknesses
I don’t have major concerns on the proposed approach, as the method is simple and straightforward. My concern lies in increase of computation overhead and extended run time due to running CLIP model on multiple augmentations during test time (as acknowledged by the authors). In addition, it can be noticed that results are comparable on almost all datasets except ImageNet-A, Aircraft and EuroSAT datasets. It would be interesting to provide rationale for the proposed approach to work much better on these datasets. Results are shown on single ViT architecture, however providing results on multiple CLIP based ViT backbones would be interesting as the method targets for test time performance. It would be helpful to evaluate the method on more OOD distributions like corruptions (ImageNet-C) to understand it better.
Questions
Please refer to Weakness above.
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
Limitations are discussed in the work.
Dear reviewer, We would like to thank you for your insightful feedback. We hope that your concerns are addressed with our rebuttal. As we are getting really close to the deadline of the discussion phase, please let us know if there are any further questions that need clarification. Many thanks, Authors
Dear authors, I thank you for the responses, and additional experiments provided in the rebuttal. I have read fellow reviewers comments and authors responses. I find that the method is slightly better than TDA consistently across all benchmarks, particularly helpful for ImageNet-A. However, my major concern on limited novelty still exists and hence I tend to keep a borderline rating. I will discuss with my fellow reviewers for a final decision.
Thank you for your reply. I appreciate your additional experiments and explanation. Actually, after check out your rebuttal, I still feel confused about the Q1 and answer in your global reply. I've checked TDA and the idea of this paper is quite aligned with them. While discussing entropy minimization in TTA (and as you claimed ``Our main contribution lies in the theoretical and experimental connection between training-required and training-free methods.''), for me, it seems like combining effective training-free and training-required techniques. In this case, I would like to explain this paper in the way: 1. Using a Tip-adapter-like (also TDA-like) training-free adapter. 2. Using a memory bank to save reliable (historical and ) and diverse (augmented) test samples, using clip prediction as evidence. 3. Apply this stored information to ``correct'' the clip logits for final prediction. Please correct me if I have any misunderstanding. I am also confused about the relationship between historical and boosting samples. Will the boosting samples constructed by augmented filtered historical test samples? If this is the case, I saw so many similar ideas in the online test-time adaptation on pages 12-13 [R1]. Could you explain further about the insights of your strategy and why it is better than others? How to differentiate your contribution among them? It will be better to make the whole sample saving or caching process a bit clearer. In the current version, it is hidden among the equations. Another question is, could you specify why using EATA as the TTA baseline? EATA has carefully designed the FIM module and used the source sample information. It is also better for continual TTA. Why don't use other TTA methods? Thank you. Reference: [R1]. Wang, Z., Luo, Y., Zheng, L., Chen, Z., Wang, S., & Huang, Z. (2023). In search of lost online test-time adaptation: A survey. arXiv preprint arXiv:2310.20199.
We thank the reviewer for the insightful comments, and we are happy to discuss some implementation details. **Q1. The implementation steps of BoostAdapter.** Your description of the implementation steps is correct. We would like to add some noteworthy points. - The boosting cache (memory bank) in step 2 is instance-adaptive. We create a copy of the historical cache (referred to as the boosting cache) and construct boosting samples for current test image to update the this cache for feature retrieval. - These boosting samples will be discarded and will not be used by other images after step 3. It is rational because the boosting samples are close to the current test image rather than others, ensuring the bound of empirical risks in Proposition 3. **Q2. The relationship between historical and boosting samples** For the current test image, the boosting samples will be derived only from itself rather than from historical samples. We check the survey and find that all the relevent methods perform techniques like augmentation and clustering over **only historical samples** in the memory bank, without considering any information of the current test image. We would like to point out that these methods may show poor generalization performance especially in downstream tasks that require fine-grained knowledge or when historical samples share insufficient similarity. So we construct the instance-aware boosting cache to perform information mining over the current test sample and incorporate this knowledge with historical samples during feature retrieval. The survey provides a clear description of memory-bank-based methods, so we will mention it and rewrite the corresponding section for a detailed discussion with these methods in the revision. **Q3. Using EATA as additional training-required methods.** The online test time adaptation setting discussed in BoostAdapter can be seen as a special case of continual TTA since we deal with samples from the test data stream. Therefore, we previously use EATA [1] simply due to its importance in test time adaptation and its applicability to our tasks. In practice, we follow the idea of a diversity-based selective strategy proposed by EATA to construct boosting samples in the cache. We further incorporate techniques from more training-required methods, including the Pseudo-Label Probability Difference metric (PLPD) from DEYO [2] and the consistency filter from TSD [3]. Specifically, in the BoostAdapter+DEYO variant, we filter out augmented views with a PLPD lower than 0.2. For the BoostAdapter TSD variant, we discard augmented views that have different cache predictions and CLIP predictions to ensure consistency of the boosting samples. The results are provided in Table A and we can observe performance improvements with the help of different training-required methods, demonstrating the versatility of BoostAdapter. **TableA: Unification of more training-required methods.** | | -V | -S | -A | -R | Average | | --------------------- | ----- | ----- | ----- | ----- | ------- | | CLIP-ViT-B/16 | 60.86 | 46.09 | 47.87 | 73.98 | 57.20 | | TDA | 64.67 | 50.54 | 60.11 | 80.24 | 63.89 | | BoostAdapter | 65.03 | 50.66 | 64.27 | 80.64 | 65.15 | | BoostAdapter+EATA [1] | 65.27 | 50.82 | 64.83 | 81.15 | 65.52 | | BoostAdapter+DEYO [2] | 65.51 | 51.01 | 64.57 | 81.11 | 65.55 | | BoostAdapter+TSD [3] | 65.49 | 51.50 | 64.37 | 81.15 | 65.63 | **Reference** [1] Niu, Shuaicheng, et al. "Efficient test-time model adaptation without forgetting." International conference on machine learning. PMLR, 2022. [2] Lee, Jonghyun, et al. "Entropy is not enough for test-time adaptation: From the perspective of disentangled factors." The International Conference on Learning Representations. ICLR (2024). [3] Wang, Shuai, et al. "Feature alignment and uniformity for test time adaptation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
Thank you for your reply and your additional experiments. Although I feel like the novelty of this work is still kind of limited, I would like to increase my score due to your prompt and detialed reply.
Please engage in discussion
Dear Reviewers, The authors have provided responses to the reviews. Please take a look at the responses (and other reviews) and engage in a discussion. Thanks for your service to NeurIPS 2024. Best, AC
Decision
Accept (poster)