Vision models are often vulnerable to out-of-distribution (OOD) samples without adapting. While visual prompts offer a lightweight method of input-space adaptation for large-scale vision models, they rely on a high-dimensional additive vector and labeled data. This leads to overfitting when adapting models in a self-supervised test-time setting without labels. We introduce convolutional visual prompts (CVP) for label-free test-time adaptation for robust visual perception. The structured nature of CVP demands fewer trainable parameters, less than 1\% compared to standard visual prompts, combating overfitting. Extensive experiments and analysis on a wide variety of OOD visual perception tasks show that our approach is effective, improving robustness by up to 5.87% over several large-scale models.
Paper
References (68)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The paper addresses test time adaptation by employing learnable convolution operation named CVP on out-of-distribution (OOD) images. The motivation is to take advantage of the inductive bias imparted by convolution operations with the added advantage of learning fewer parameters compared to existing approaches. CVP in conjunction with other sota approaches improve the performance on benchmark datasets.
Strengths
1. Using small convolution operations at the start for OOD test time adaptation to take advantage of inherent inductive bias seems to be new. 2. The combination of selfsupervision and convolution makes it both parameter as well as label efficient. 3. Main and ablation experiments are largescale
Weaknesses
1. Eqn. (1): When $y_{i,j}^s$ is 0, $i$ and $j$ are from different samples (negative pair). So, in the loss calculation negative pairs are not considered. Then how is it a contrastive formulation? 2. Line 111: It is said that $\mathcal{X}_s$ is the source domain data for training. If source domain data is used then it is not a test time adaptation framework and comparing it with such works would not be fair. 3. Line 150: One of the important assumptions made in this paper tells the distribution shift in images is often visually structured. Why is this assumption right? What tells that the distribution shift is 'visually structured'? 4. Line 180+: When describing the proposed model, it is told that the logits before the fully connected layer of the backbone model are extracted for training the SSL model. This is making me confused. Does it mean the conv operation is applied on the logits and not on the image? I think having a figure describing the proposed approach would have been useful here. I don’t see such a figure and thus I have to presume things. Also, it is said that a MLP gets trained in addition. Is it used during inference? Also, is this taken into account in the learnable parameter count, especially when compared with other related approaches like TENT or other traditional prompting methods? In this regard, the figure 6 in the supplementary is incomplete. What is the unit in x-axis? Millions, billions, K? 5. Line 184: When extending the proposed approach to CLIP, why only vision encoder is used for prompting? Why not text encoder only? Is there any ablation that is done? 6. Section 4.1: FT and PFT Baseline: Why is this restoring to initial weights required? Is it standard in literature? Any reference? 7. Line 206: What is a sharpness kernel? 8. Section 4.2, CVP Complements Other Test-Time Adaptation Methods: More details are required on what is meant by 'combining' CVP with these methods. Also, the results of this experiment (Table 4) are not convincing. Improvement is very marginal compared to TENT, in all three imagenet variants. Compared to standalone CVP, CVP+x provides huge error rate drop - then this means the major workforce behind the performance is x and not CVP. 9. Ablation study – Low rank structure: More details are needed on how the low rank prompts are created? Are they directly applied on the input? According to the original LoRA paper [24], low rank adaptation is shown to be effective in transformer architectures. How is it adapted to convolutional architectures? As two low rank matrices are involved (I'm supposing so, as details are missing), this evidently involves more learnable parameters and thus susceptible to overfitting. So, the comparison to conv kernels is not fair. The corresponding figure in Appendix (Fig. 9), does not have any descriptive captions. It’s better not to give such figures if everything about the figure has to be inferred by the reader. 10. Line 266-271 and Table 7: What is 'standard' SSL task? Is it employing eqn. (1)? Its difference with 'contrastive' variation needs to be detailed. While rotation or MAE were shown to perform worse compared to the standard SSL task (in Table 7), there is no detail of these two other SSL approaches. How many variations of rotation prediction were used? How much percentage of the images were masked in MAE? Is there any study by varying the mask percentage? How long are these other SSL tasks trained? Generally Contrastive and MAE tasks require a long training for better learning. Was this good practice followed?
Questions
Questions are all detailed with the associated weaknesses raised above (in ‘weaknesses’ section). Here, in addition, some minor typos are mentioned. - Line 156: Between ‘shifts’ and ‘In’, a stop is missing. - Line 174: ‘… method, We …’ -> ‘method, we’ - Line 175: Appendix number is missing - Line 178: ‘contains’ -> ‘containing’ - Line 200: ‘More baseline detailed are shown’ -> ‘More details about the baselines are provided’ - Line 204-205: ‘random initialize the kernel size k’ -> ‘randomly initialize the kernel of size k’
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
Limitations are not explicitly described. It may be useful to discuss the scope of the claims regarding the convolution operation in CVP handling structural bias in OOD.
Summary
This paper proposes a novel label-free approach CVP for test-time adaptation on out-of-distribution data. The main idea is to use a convolutional kernal as the visual prompt. It captures the structure of data distribution shift, and reduces the trainable parameters. Experiments show that CVP improves model robustness and complements existing weight-adaptation methods.
Strengths
1. Prompt tuning has become an important technique to adapt large visual models. This paper provides a simple solution and may be useful in various applications. 2. The paper is well motivated. Using convolutional structure to inject the inductive bias is simple yet effective. It reduces the required parameters, and shows significant better performances than low-rank prompts. 3. The method does not require labeled data. 4. The method can be combined with other weight adaptation methods. It can also be generalized to multiple self-supervised objectives. The authors provide corresponding experiments with encouraging results. 5. The paper is well structured and nicely written. Empirical results are quite extensive. The attention visualization provides a clear insight.
Weaknesses
1. The corruption types considered in this paper seem to be restricted to low-level transformations. This explains why the convolutional kernel is suitable. I wonder if CVP would still work well for high-level distribution shift, such as style changes? Typo: Reference missing in Ln 175.
Questions
1. In Ln 206, "starting from a sharpness kernel is effective". Could the authors elaborate on what 'sharpness kernel' is and why it is effective? 2. The convolutional kernel applies on the whole image. Sometimes, the corruption may happen in some local regions, e.g., motion blur on foreground objects only. How do the authors comment on such situations?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
The authors have not discussed limitations. But I feel that the method is simple, and experimental analyze are extensive enough. It is helpful to discuss the effectiveness under other OOD types like open-class data, high-level style changes.
Summary
.The paper proposes a new method for test-time adaptation (TTA) to structured distribution shifts. The proposed method learns convolutional visual prompts to prevent the model from overfitting to SSL objectives due to high dimensional prompts. The results show that the proposed approach consistently results in performance improvements for TTA to OOD variants of CIFAR-10 and ImageNet.
Strengths
- The idea of using convolutional prompts to avoid SSL overfitting is novel and well-motivated (Table 5 shows how SSL results in overfitting and Figure 3 shows the need for some form of inductive bias). - The results indicate consistent results when CVP are added to baselines and other prior TTA works. - The method tunes very few parameters. The parameter efficiency allows it to be adopted in edge computing devices working with low memory and compute.
Weaknesses
1. The paper compares their method with only two types of visual prompts (patch and padding based [40]). The paper does report results with CLIP ViT-B (Table 3), but does not compare to other prompts used for adapting ViTs [26] and text-only prompts for CLIP-pretrained ViTs. [A] 2. The paper mentions on L63: “The only one that does not update the model (for Test time adaptation) is proposed by [40]”. However the following uncited works do not update the model as well [A, B]. 3. The paper does not show results with unimodal ViT architectures (eg. pretrained models from TIMM). Similar gains over the baselines (and [26]) in the context of ViTs can boost the applicability of the approach. 4. The paper sets different hyperparameters for different datasets (range of $\lambda$, kernel sizes), without providing guidelines on how these are to be determined for a new dataset (or corruption type). 5. Missing details on evaluation and implementation (covered under Questions). Minor writing: $\lambda$ is introduced much later in text, while being introduced in 3.3. [A] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models. Shu et al. [B] Test-Time Training with Masked Autoencoders. Yossi et al. [C] Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model. Xing et al.
Questions
I am happy to update my rating if the concerns around comparisons to prior work and hyperparameter selection are adequately addressed. But I request the authors to provide the missing details. 1. In Table 8, multiple choices are provided for kernel size and update iters, how are the chosen hyperparameters validated? 2. What are the hyperparameters used in Table 4/5/7, where the kernel size is not specified? 3. How is CVP applied to CLIP-ViTs? Is the image first convolved and then tokenized? 4. Which CLIP-ViT/32 model and can was used for the results in Table 3? [A] does report numbers on CLIP-ViT-base/16. 5. What architecture is used for the results in Table 7? 6. For the MAE baseline in Table 7: is the contrastive objective of CVP directly replaced by MAE objective? But this introduces additional decoder parameters as well right? It would be good to see a comparison of the number of parameters that are optimized in different SSL methods. 7. L9: “1% when compared to standard visual prompts”. How does this compare to the shallow prompts used in [26]. 8. Tables 9 and 12 show that CVP does not outperform baselines for fog/frost/snow corruptions. Also, Figure 3 shows CVP does not outperform low-rank prompts for snow corruption. Any intuition on why CVP does not perform well when weather conditions are changed? 9. In Table 15, are the reported losses on samples that were used to tune prompts? Reporting the loss on unseen samples may give a better idea of the extent of overfitting. 10. Since [A] is a prior TTA work for CLIP, would there be any benefits of dual text-visual prompt tuning (like [C]) with: CVP (your method) + [A] (NeurIPS’22)? [A] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models. Shu et al. (NeurIPS'22) [B] Test-Time Training with Masked Autoencoders. Yossi et al. (NeurIPS'22) [C] Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model. Xing et al.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
4 excellent
Limitations
The paper may need to add a limitation (and future work) section if the above concerns (applicability to uni-modal ViTs and hyperparameter tuning) can not be addressed in the current submission.
Summary
The paper proposes a variant of Visual Prompt Tuning (VPT) where the prompt is applied to a penultimate layer of the encoder network and is a result of a convolution with a 3x3 or 5x5 kernel (essentially it is an added residual block or residual adapter if you will). The added adapter is trained at test time by applying a contrastive loss on a test batch (so, in fact, it seems to be a variant of transductive learning, requiring a batch of test images - to form a negative set for the contrastive loss, so not truly online and does not seem to be able to operate on a single test sample). The authors test their approach on a collection of OOD benchmarks, some synthetic, and some real - these ones are in fact UDA / UDG benchmarks, however, UDG baselines are not compared. Small improvements are observed over naive baselines, though prior methods like [53] and [60] seem to deliver much better performance (Table 4) and the authors only show the benefit of their approach via ensembling / combining with the other methods.
Strengths
- test time adaptation, VPT, and handling domain shifts are important topics - some improvement is observed over simple baselines
Weaknesses
- novelty: adding a convolutional residual adapter + contrastive loss for transductive learning (test time adaptation on a batch) is hardly novel - comparisons to prior works: * table 4 shows [53] and [60] attain significantly better results than the proposed approach * relevant baselines / benchmarks seem to be missing (e.g. https://arxiv.org/pdf/2210.04831.pdf reports better results on ImageNet-C + DomainNet benchmark could be added) * I would expect comparison to / on top of UDG baselines that solve the same problem * combining methods as in table 4 is fine, but other methods could also be combined, would it yield even higher results? - Supervised variant of the approach explored in the ablation needs to be compared to few-shot and in particular transductive few-shot methods, expecting much higher results there. - what about the online test-time tuning (from one sample) seems not possible due to the use of the contrastive loss?
Questions
- PromptTuning is a weaker variant of the broader family of PEFT methods, usually applied on the input level (penultimate layer in this case) it does not have enough representation power to model more complex deviations, as opposed to more popular LoRA or prefix tuning methods. Would the authors' approach be applicable to the more powerful PEFT methods?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
1 poor
Limitations
limitations not discussed
Summary
- The paper introduces Convolutional Visual Prompts (CVP), a novel methodology designed to increase the model robustness when faced with Out-of-Distribution (OOD) data during test time. - The primary innovation of CVP is the use of convolutional structures as inductive biases for adapting to visual OOD instances during test time. - The experimental results show that these convolutional structures are efficient and effective at dealing with OOD corruptions.
Strengths
- *Originality*: The paper proposes a novel method, the use of convolutional structures as inductive biases for visual prompt tuning to OOD data. This work would interests the visual prompt community. - *Quality*: Extensive experiments have been conducted across a variety of OOD recognition tasks, providing empirical support for the effectiveness of CVP. - *Clarity*: The paper is well-written and comprehensible, offering clear explanations of the problem, the proposed solution, and the obtained results.
Weaknesses
The major concern of this work is its Motivation behind the choice of convolutional structures: The paper does not provide a clear motivation or justification for choosing convolutional structures as the basis for the Convolutional Visual Prompts (CVP). While the paper mentions the effectiveness of convolutional operations for handling structured data with local motifs, a more explicit link between this principle and the task of OOD adaptation would be beneficial. For instance, the authors could hypothesize that the learned convolutional kernel might act as the de-corruption operator for specific corruptions, offering a possible reason for their choice (the point was raised by the visual evidence like Figure 11, which seems indicate the reason behind why CVP works).
Questions
- What motivated the choice of convolutional structures over other possible structures for the CVP? - Could you discuss the wider usage of CVP when applied to settings other than test-time adaptation? - Could you also discuss the extendability of CVP model, like the possibility/effectiveness of adding more kernels? I am not saying to develop a new variant during the rebuttal, but to indicate how the followers can use the method to the best extend would enhance the significance of the work.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The paper seems not discuss the limitations, but when the authors try to answer the weakness and questions above, they might come up with more discussion on the limitation of the work.
Dear reviewer uZKT, We appreciate your reviews and comments. We hope our responses address your concerns. Please let us know if you have further question after reading our rebuttal. We hope to address all the potential issues during the discussion period. Thank you.
Post-Rebuttal Comment
Overall, the authors have addressed the primary concerns and clarified the underlying motivations and potential extensions of their work. The broader applications of CVP, as highlighted in the rebuttal, make this work even more compelling. So I keep my original score.
Dear reviewer zs9s, We appreciate your reviews and comments. We hope our responses address your concerns. Please let us know if you have further question after reading our rebuttal. We hope to address all the potential issues during the discussion period. Thank you.
Dear reviewer HEpT, We appreciate your reviews and comments. We hope our responses address your concerns. In our rebuttal, we have added all the missing details and added the experiment, including ViT results, shallow prompts baselines on CLIP, and the loss analysis for unseen samples (in our PDF file). Please let us know if you have further question after reading our rebuttal. We hope to address all the potential issues during the discussion period. Thank you.
Thanks for the rebuttal
I thank the authors for the additional experiments and clarifications provided. I appreciate the comparison to shallow prompts, clarification of low performance on weather perturbations, and the loss analysis on unseen samples. I am generally satisfied with the responses, which suggest that CVP is a better-prompting strategy compared to prior prompting strategies when dealing with structured (eg. blur) and style perturbations (eg. sketch, renditions). But after carefully going through all reviews and rebuttals, I have a few remaining concerns and encourage authors to provide clarifications on these: > The shallow prompt doesn't improve under our setting because it requires more learnable parameters in the prompt and leads to overfitting for adaptation. Are the improvements with CVP (here, and in general) only due to much less learnable parameters? Is there a direct comparison with prompts having exactly the same number of parameters as CVP (eg. partially learnable prompts) that shows the importance of using convolutional structures? I thank reviewer QKSZ for pointing out that the low-rank prompts, although lightweight, actually have a higher number of parameters than CVP. > We feel that combining text prompts and CVP is an interesting idea. We will add it in future work. It would have been nice to show CVP leads to improvements when combined with text prompts as well, but leaving it to future work sounds good to me. In the existing comparisons that show improvements over prior work - do the authors use the SSL objective during source training for the MEMO, BN and TENT methods? Do the MEMO and MEMO + CVP rows start with the same source model? If not, is it possible that the SSL pretraining on source gives an unfair advantage to MEMO+CVP compared to MEMO?
Thank the reviewer HEpT for carefully reading our rebuttal and giving us valuable feedback. We would like to provide more clarification here. 1. Regarding the improvements of CVP, it is not just less learnable parameters but also the convolutional operator that helps to avoid overfitting. Without the convolutional kernel, the structure can not be captured well during the adaption. In Tables 1 and 2, we show the results on SVP (patch), which is the baseline without convolution structure and only adds the $\ell_p$-norm perturbations as the same shape of input. We agree that the results compared with the same number of parameters as CVP are necessary, and we thank both reviewers HEpT and QKSZ for pointing out this. Therefore, we add another experiment to compare the CVP with SVP (patch) under different learnable numbers of parameters. We show robust accuracy on the CIFAR-10-C, including average of 15 types of corruption with five severities. The model backbone for Standard results is WideResNet18. - | | S1 | S2 | S3 | S4 | S5 | Avg. Acc. | |------------------|--------|--------|--------|--------|--------|-----------| | Standard | 59.68% | 47.88% | 40.31% | 32.75% | 28.20% | 41.76% | | SVP (patch 3*3) | 59.71% | 47.90% | 40.34% | 32.76% | 28.23% | 41.79% | | **CVP** | **68.07%** | **54.73%** | **45.96%** | **37.79%** | **31.61%** | **47.63%** | - Here, we set up the patch size as 3x3. In the third row of the table, the SVP (patch 3x3) has the same learnable number of parameters when CVP uses the 3x3 convolutional kernel. When with the same number of parameters, CVP has improved the performance by 5.9 points which indicates the importance of using convolutional kernels. 2. Regarding the existing comparisons that show improvements over prior work, do the authors use the SSL objective during source training for the MEMO, BN, and TENT methods? - For MEMO, BN, and TENT methods, we do not use the SSL objective during source training. The source model is frozen all the time. Besides, the SSL model that we train for the purpose of combining with MEMO, BN, and TENT will not affect the source model. 3. Do the MEMO and MEMO + CVP rows start with the same source model? If not, is it possible that the SSL pretraining on source gives an unfair advantage to MEMO+CVP compared to MEMO? - Yes, MEMO and MEMO + CVP indeed start from the same source model. As our flow figure show in the PDF, we don't need to retrain the source model and only require to train an auxiliary SSL model based on the frozen source model. The SSL model on the source will not affect the source model or change any pretrained weight in the source model. During the test time, for every adaptation step, the MEMO + CVP first optimizes the convolutional kernel with SSL objective to generate a new sample and then use the new sample for adapting the model weight. Therefore, the comparison is fair. We have tried our best to answer all the questions, and sincerely hope the reviewer can fully understand. We look forward to receiving other feedback and we deeply appreciate the time and effort you've dedicated to reviewing our work.
Thanks for the clarifications
I thank the authors for the clarifications. I am happy to increase my rating because the authors have addressed all my concerns. I strongly encourage the authors to update the paper with the new results and figures.
Dear Reviewer HEpT, Thanks for your valuable comments on our paper and for deciding to raise the rating. We will update the paper with all implementation details, new results, and figures in the upcoming version. Thanks a lot for your time again!
Dear reviewer XDjm, We appreciate your reviews and comments. We hope our responses address your concerns. Please let us know if you have further question after reading our rebuttal. We hope to address all the potential issues during the discussion period. Thank you.
Dear reviewer QKSZ, We appreciate your reviews and comments. We hope our responses address your concerns. In our rebuttal, we have added all the missing details and drew the flow for our proposed method (in general response PDF file). Please let us know if you have further question after reading our rebuttal. We hope to address all the potential issues during the discussion period. Thank you.
Post rebuttal comments
Thanks for the detailed response. While some of the responses are helpful (clarification for contrastive formulation, what is a sharpness kernel etc.), some increased the confusion. For example, Figure 1 in general responses PDF. It provides the flow of the proposed method. It can be seen that during training source domain images (in-distribution) are used in the Training phase (left hand side of the figure). Then how can the proposed method be compared with testtime adaptation based methods like TENT. Table 1 in the TENT paper [60] tells that fully test time adaptation does not use source data (even if unlabeled). This is related to my earlier question in point 2 of the “weaknesses”. I also agree with reviewer zs9s and am equally concerned about the incremental nature of the work along with its performance in comparison to the approaches in table 4 (I also mentioned it in point 8 of the “weaknesses”). The response says that the proposed approach is orthogonal. However, “orthogonality” is not described enough. As I said previously, it seems these methods are less privileged than the proposed approach as they are not using source data. Now, going ahead with the argument that the proposed approach is different, the fact that combining it with the existing approaches results in only incremental performance improvement, does not convincingly say that the ‘differences’ are useful. The same table has the performance by ‘only’ the proposed approach (without combining it with any other existing approach) and this performance is pretty low compared to the existing approaches. So, whether combining the proposed approach with the existing ones is useful or combining the existing one with the proposed approach does a better job, is still debatable.
Thank you for replying. As mentioned above, we do not compare directly with those test-time adaptation-based methods like TENT, BN, or MEMO due to the different adaptation settings. Table 4. which we show here, wants to convince the reader that our method can also complement those TTA methods. As our main goal considers a more challenging scenario than other TTA; that is, the model weight can not be adjusted during the test time. If under our setting, then most of the test-time adaption methods can not work because all of them need to change the model weight for every step. Therefore, we want to claim that ours are orthogonal to those methods and have benefits while the model weights are frozen. We already show our method has the SOTA performance in Table 1,2, and 3, compared to other visual prompts baselines, which only change the input weight and are more related to our settings. Besides, we want to clarify that those TTA methods also use the source data during training as they always need to have a pre-trained model that already learns the pre-knowledge from the source domain so that adaptation is allowable. For example, the TENT, BN, and MEMO, if they want to make a model adapt to the ImageNet-C, then they need a model that is well "pre-trained" on clean ImageNet, which is indeed the source domain data.
Dear Reviewer QKSZ, We have tried our best to answer all the questions, and we sincerely hope you can fully understand. We look forward to receiving other feedback, and we deeply appreciate the time and effort you've dedicated to reviewing our work. Thank you
Dear Authors, Thanks for the response. I have carefully read that and other reviews. The response generally solves my previous questions on high-level shift and sharpness kernel. The proposed method works on ImageNet-R/S/A according to the experiments, though I agree with Reviewer uZKT that the motivation of using convolution should be made clearer, especially in the case of high-level shift. E.g., whether 'de-corruption operator' works for style-shift? Whether 'those shifts are often locally structured'?
We thank reviewer XDjm for the responses. We want to provide more clarification and discussion. Whether 'those shifts are often locally structured'? / Whether 'de-corruption operator' works for style shift data? As some literature [1, 2] suggests that part of the style shift is the change in the local information, such as textures, and part of it is the change in the global, such as sketches where object shapes are preserved yet texture cues are missing. Our prior experimental results show that CVP improves on several style-shift OOD benchmarks, such as ImageNet-Rendition, Sketch, and Adversarial. For the ImageNet-Rendition, the data contains several styles, such as art, cartoon and are more related to the local (texture) change. For the ImageNet-Sketch, the data contains all sketches that only preserve object shapes and are more related to global change. Therefore, the de-corruption operator like CVP can work on both local / global change of style shifts. [1] Geirhos, Robert, et al. "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness." arXiv preprint arXiv:1811.12231 (2018). [2] Wang, Haohan, et al. "Learning robust global representations by penalizing local predictive power." Advances in Neural Information Processing Systems 32 (2019).
post rebuttal
thank you for your response. Reading the rebuttal, I still have concerns: 1. regarding comparison with Gao et al. 2022 - stating the difference does not change the fact it is a higher performance baseline. If prompt pre-training is seen as an advantage, I urge the authors to show their method could be used to improve on top of Gao et al. result by using prompt pre-training etc. Or adding CVP to the Gao et al. method and showing further improvement. 2. Regarding CVP only improving over other methods when combined with them - my concern was - other methods can also be combined, and also standard VPT can be added to other methods - all those are additional baselines the authors need to show gains over to establish the importance of CVP design. Also, in most cases, the combination gains of CVP with other methods seem rather small (Table 4) and I wonder if it is significant enough. 3. for UDG baselines - many UDG methods could be employed with single-domain pre-training. In fact, some are evaluated in source-target pairs as a standard benchmark - I think a comparison to some of those could be shown, but perhaps concern 2 above is more important 4. for single sample vs transductive - if a single sample method can perform better than (transductive) contrastive approach - it should have been featured in all other experiments besides Table 7, if not - transductive baselines should have been used - from my experience, transductive inference (offline prediction for several test samples at once allowing them to learn from each other without any label knowledge) always adds a significant (could be even 5% or more) improvement to the results, and I think it is covering the improvement range observed in the current paper. In light of the above, I prefer to keep my original rating.
Thanks for your replying 1. Regarding comparison with Gao et al. 2022 - We thank the reviewer for pointing out this amazing work. However, we also realize this work is still unpublished and under review. Therefore, we will consider comparing with them in future work. 2. Regarding CVP only improving over other methods when combined with them - We are currently running the experiment of combining standard VPT with other TTA methods. We will report the numbers as soon as possible.
Dear Reviewer zs9s, thank you for taking the time to review our paper. We want to provide more clarification and experimental results for question 2. - Regarding CVP only improving over other methods when combined with them, we ran another experiment to combine the standard VPT with other TTA methods, including BN, TENT, and MEMO. Our evaluation is on the CIFAR-10-C average on 15 corruption types and five severities. The results show that as below, in general, CVP + others have a lower error rate when on top of those three TTA baselines compare to VP + others. | | Avg. Error Rate | |-------------|:-----------------| | source-only | 58.84 | | BN | 38.51% | | BN + VP | 38.50% | |**BN + CVP** | **38.39%** | | TENT | 38.52% | | TENT + VP | 37.65% | | **TENT + CVP** | **36.69%** | | MEMO | 56.13% | | MEMO + VP | 55.43% | | **MEMO + CVP** | **54.85%** |
Dear Reviewer zs9s, We have tried our best to answer all the questions, and we sincerely hope you can fully understand. We look forward to receiving other feedback, and we deeply appreciate the time and effort you've dedicated to reviewing our work. Thank you
Decision
Accept (poster)