Neural Priming for Sample-Efficient Adaptation

Summary

This paper introduces neural priming, a retrieval based approach for adapting large pretrained models to downstream tasks. Neural priming first collects a priming pool of pretraining data relevant to the downstream task using natural language descriptions, then at test-time the priming pool is further filtered by keeping the top-k most similar pretraining samples to the test samples w.r.t. cosine similarity of the CLIP embeddings. From this filtered pool, a classification head is set to the nearest-class mean and averaged with the CLIP ZS text embeddings (similar to WiSE). Across several classification datasets, neural priming is shown to improve over existing prompt-tuning and test time adaptation methods in the zero-shot and few-shot settings, and is complementary to prompt-tuning methods.

Strengths

The problem of fitting pretrained classifiers to downstream tasks effectively, specifically models like CLIP, is of great interest to the ML community and the retrieval based approach of neural priming is both simple and a pleasant departure from current robust finetuning methods. Overall the paper was well-written, easy to follow, and the evaluation shows consistent improvements over existing prompt tuning and retrieval-based transfer learning methods. I also enjoyed the open questions posed in the discussion section on why retrieval methods work well for transfer learning which I could see sparking future work in this area. While the method is simple and there are some eval limitations that I would like addressed, I think this paper is of value to the community and is above the acceptance threshold.

Weaknesses

Given the simplicity of the method and the vast number of related works, my critiques mainly lie in differentiating from related works more and adding in some simple yet relevant baselines. **Related works** My main critique with the related works section is the lack of differentiation from test-time training/adaptation and transductive learning in section 2.3, as well as few-shot learning in section 2.4. Not a critique and more of a suggestion, but a recent paper[4] explores how removing the least relevant items in a pretraining dataset can improve transfer learning accuracy and I believe would be good motivation for neural priming. There are also some robust CLIP finetuning methods[1] and prompt tuning methods[2,3] that should be mentioned in related works. [1] [Using Language to Extend to Unseen Domains](https://arxiv.org/abs/2210.09520) \ [2] [Visual Prompt Tuning for Test-time Domain Adaptation](https://arxiv.org/abs/2210.04831) \ [3] [Conditional Prompt Learning for Vision-Language Models](https://arxiv.org/abs/2203.05557) \ [4] [A Data-Based Perspective on Transfer Learning](https://arxiv.org/abs/2207.05739) **Evaluation** Here the main thing that is lacking is a description of the baselines. There is some brief descriptions of CoOp (without a citation) but I found it hard to interoperate exactly what these baselines were. \ Why not compare to CLIP LP or WiSE for the few shot setting? It seems like that would be a natural baseline (even just setting the ZS weights to the few shot examples, from what I understand the NCM baseline uses the priming examples).

Questions

Does the CLIP ZS baseline in Table 1 just include the class name or does it also include context (i.e. 'a photo of a {} flower' instead of 'a photo of a {}'")?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

One big limitation that I did not see mentioned was the time it takes to retrieve the priming pool from the original pretraining data, especially for large datasets like LAION 2B.

Summary

The paper proposes a technique for adapting large pre-trained models to distribution shifts and downstream tasks given few or no labeled examples named Neural Priming. The method can be used at test time on very large scale datasets. Experiments on various datasets show the effectiveness of Neural Priming.

Strengths

1. The proposed Neural Scaling method is novel, with a good motivation (diversity of pretraining datasets introduces competing objectives) and clear presentation. 2. The technical details of Neural Scaling is simple yet effective: Classical ideas of information retrieval, clustering and prototype calculation were used. This may possibly open a line of research on advancing the proposed techniques. 3. Neural Scaling has high efficiency on saving the time of doing linear probing.

Weaknesses

1. In line 182, what is P and \sigma? The main text should include their meanings rather than putting them in the appendix. Also, ablation studies/discussions can be included for the choice of \sigma. 2. I am not familiar with the performance of the compared baselines e.g. VLM (whether these methods are strong and competitive), thus I cannot determine to which extent Neural Priming outperforms other relative methods, especially SOTA ones. While this do not effect the paper's advantage in novelty, it does draw questions on the "absolute" performance of Neural Priming.

Questions

The questions here are for open discussion. 1. I wonder if all the filtered priming pools for a specific downstream task are integrated into one dataset, will it be a better pretraining/finetuning dataset (excluding noises)? 2. Is there a possibility that some samples are harmful (e.g. low quality or wrongly labeled), but selected into the priming pools or even the filtered priming pool?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are discussed in terms of the method itself. But issues on broader aspects such as model safety, data leakage are not included.

Summary

This paper presents a novel approach to zero-shot learning in vision-language models. The authors propose a new tunning schema using prior knowledge in the CLIP model. First, the authors construct a priming pool, which consists of clusters of retrieval images by the CLIP model. Each cluster collects a specific class by searching all text-image pair owning that class name in the text. The authors then tuned this priming pool using the test set. For each image in the test set, the 10 best matching images are selected and de-duplicated to make up the final priming pool. The authors claim that such an operation can adapt the priming pool to the test domain. At last, the authors tune the CLIP model by adding a linear head at the end of the model. The linear head is the weighted sum of a zero-shot trained head and NCM of all retrieved examples per category. Experiments show a 2.45% improvement on ImageNet and a 3.81% improvement on average across some traditional transfer learning benchmarks. Overall, this paper provides valuable insights and promising directions for future research in this field.

Strengths

# Novelty: This paper provides a novel perspective of aligning images by the class name directly. Previous methods are often trapped in playing with feature representations and embeddings. None has looked into whether using strings directly, is a good practice. This work fills this unseen point with good improvements. The method can leverage massive training data, to help the model adapt better to specific tasks. # Clarity: This paper is extremely well-written and easy to follow.

Weaknesses

# Novelty: The key innovation of this work can be split into three parts, 1) **class name retrieval to construct priming pooling**, 2) **filtering the priming pooling using cosine similarity with the test set**, and 3) **tuning the CLIP model by adding a linear head using NCM of the priming pooling**. So I will give my opinion to them separately as follows. 1. **Retrieval**: a) **Retrieval is not a new idea in transfer and zero-shot learning.** I think using retrieval is not new in both domains of transfer learning or few-shot\zero-shot learning, like in NeurIPS2021,2017 paper "Re-rankingforimageretrievalandtransductive few-shot classification", "Few-ShotLearningThroughanInformation RetrievalLens", there may be too many to list. b) **String retrieval I think is not powerful enough.** So the new stuff is using the class name as the retrieval feature. This is somewhat too direct for me. Is there a possibility that CLIP representations combining the string retrieval perform better? According to my experience, this is usually the case. The authors do not discuss the possibilities of using other tools, instead, they come out with a belief that string retrieval is the fastest one so it is superior to other methods, which I may respectfully disagree. First, time efficiency should not be a major problem in constructing the priming pool, after all, it is not required to be online learned or real-time computed. In your pipeline, you first get the naive priming pool from the training data. Then you prune it to preserve the top 10 similar images with each image of the test data. This step won't consume more time if you use semantics as you need to go through all the data in the priming set either way. Then at inference time you only need to compute the mean of each class, there is no need for exact string matching so it also won't consume more time if you use semantics or other stuff. So using semantics or not won't influence the inference time at all, I can't understand why only string is used. Perhaps some experiments comparing those retrieval methods can be helpful to demonstrate the advantages of this work. c) **Potential conflicts.** Another problem of only using string retrieval is that it only conflicts with each other. Like the class "guinea pig" is not a pig at all, "wolf spider" is not a wolf, and "sea snake" may not contain sea in the image. I am interested in how this method can tell the "guinea pig" category (339 in ImageNet) and the "pig" category (342 in ImageNet)? On the hand, combining string with semantics can handle this case, as the embedding of guinea pigs will be more similar to the mouse's. Also, I don't understand why the results of string retrieval will be more relevant, can the authors explain why? 2. **Filtering the priming set.** I think the authors can try combining CLIP similarity with Pixel similarity in ranking, inducing more metrics to cooperate. In transfer learning and few-shot learning, such methodology is usually adopted. 3. **NCM head.** This step uses the final outputs of the language model to predict the mean. Why not also use the output of the Image encoder? Or how about the intermediate representations of the CLIP model? I think the authors can add an explanation of why the output of the language model is the best. Also, tuning a new head is not a new thing. It is popular in incremental learning, transfer, and zero-shot learning. But it is not necessary to limit your imagination in here. Many works have discovered some other parts of the CLIP model, like attention layers, and convolutional kernels, MLPs have rich semantics and an important role in making predictions. If the authors can discuss why they choose to add a linear head instead of tuning other parts, I think this section can be more convincing. For example, the authors can compute the gradients of each parameter when input new examples of test sets, and find out that the terminal linear layer has the most significant gradients thus influencing the results on new examples the most, so tuning them is vital and more efficient than tuning other parts. Overall, the three parts are a bit split and there seem no strong connections among them, at least not provided by the authors. They seem like A+B+C. If the authors can reorganize them, and engage them together with one purpose of enhancing transfer learning capability, I think this paper can be more fluent. # Experiments: 1. **No errors and error bars.** While it is recommended by NeurIPS author guidelines, papers in this domain seem seldom report error bars. I think it could enhance the paper as some figures are close, and errors can help us understand the true advantage of the proposed method. As in Tab. 1, in ImageNet, FGVC, Food101, and SUN397, the differences are smaller than 0.5. We may suspect it within the variance of measures. Also in Fig.3, performance is close, error bars can further tell the differences between methods. 2. **Why no LAION-5B?** I see many relevant papers conducted on the LAION-5B dataset. LAION-2B is a new one and includes it very well. However, omitting LAION-5B may be hard to align the results of this paper to previous work. 3. **Ablation is not enough.** Like in the novelty part, the superiority of the chosen components to their alternatives need to be clarified by the experiments. This will make this submission stronger. 4. ""Fair comparason.** As you add a head to the model, you may control the equivalence of all parameters and training steps of all methods. This is important when performance is close. # Minor issue: 1. Line 143 "However,143 withneuralpriming,prioritizingprecisionoverrecalliscrucial,consideringthesize,diver", better to add references. 2. Line 164, the last "for" is redundant. 3. It is not very clear whether Neural Priming is another name of this method or the task this method tries to handle. Better clarify and unify it throughout the paper.

Questions

1. Will the method improve finetuning efficiency of other prompt tuning methods? I mean after applying it, how many training times or steps can be saved for prompt-tuning? 2. Is the method stable among different tasks? 3. How is the influence of the filtering step? If remove it, how much accuracy will lose? Also, is there any ablation on the choice of $\alpha$? The following are suggestions rather than comments, the authors may take them advisedly. 1. **Notations.** I am in favor of notations used in SimCLR, which are much clear to read and easy to follow as they use different font styles for different concepts. I would suggest a slightly modified version as follows: sets, using \mathcal command like $\mathcal{S}$; neural networks, using bold font with parameter subscript, like $f_{\theta}$, number field, like the real number set, using $\mathbb{R}^d$, integers using $\mathbb{N}$; loss function, using \mathcal command with subscript the \mathrm environment, like $\mathcal{L}_{\mathrm{CLIP}}$; scalars using plain font and vectors using bold fond to distinguish. Currently, the notations carry only limited meaning, make readers need to look back to their definitions several times to remember them. For example, Line 130 uses $\mathcal{R}^d$ for real numbers while Line 178 uses $\mathbb{R}^{d\times n}$, V,P,B,L stand for model, data pool (while training data using font $\mathcal{D}$), data clusters, and model but using the same font style. I also recommend not to use L as a notation for the model, it can be confused with the loss function. 2. **Fontsize of figures.** Currently fontsizes of figures are set carelessly. Fig. 1 using a giant fontsize, much larger than the context, seems not necessary, while some words in Fig.2 are too small to tell. I also recommend you to use some images to tell readers the meaning of each component in Fig. 1, like putting some training images above the pretraining data, and putting some images from the priming pool beneze the priming set, using a neural network model to replace the CLIP block and color those parameters that need to be tuned. 3. **The hyphen.** In the title you use **Data-Efficient**, words connected by the hyphen are in capital. While in **Zero-shot**, **Nearest-class-Mean**, they are not.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

1. The method needs to know all test set to filter the priming set, online learning may exceed the scope of it. 2. Filtering step may consume considerable time and computation resources, not affordable for ordinary researchers.

Summary

This work proposed a new method, named Neural Priming, which uses retrieval to adapt vision-language foundation models to downstream tasks given few or no labeled examples in the test time. Specifically, Neural Priming first uses the set of class names in a new task to retrieve relevant samples from the pre-training dataset to form the priming pool and also uses the test examples to narrow the priming pool. Then, the CLIP model can be attuned using the priming pool by constructing the linear prediction head. The experiments are performed in three settings: zero-shot, few-shot and transductive.

Strengths

1. The idea of using retrieval to adapt large vision-language models to downstream tasks is interesting and intuitive. Compared with similar approaches, such as REACT, this work does not need to add new trainable modules. 2. Experiments on several settings (zero-shot, few-shot and transductive) show the effectiveness and broad applicability of the proposed method. Ablation studies on the number of samples per class in the priming pool and the model capacity are interesting.

Weaknesses

1. Regarding the retrieval idea for test-time adaptation, the major difference between Neural Priming and REACT is that REACT adds new training modules while Neural Priming only constructs a linear prediction head. It is not clear to me how significant this difference is and how original the proposed method is. Also, there is no direct comparison between these two approaches in experiments. Since Neural Priming and REACT are used in similar settings, it is not clear to me if the proposed method can largely improve over REACT. 2. It is not clear how sensitive the proposed approach is to the retrieval database. From experiments, we see the improvement will be less significant if the target dataset is well presented in the pre-training dataset (see lines 237-241). On the other hand, the performance will also drop if there is a large gap between the target dataset and the retrieval dataset. Another question is whether the retrieval database has to be the pre-training dataset? Or we can arbitrarily construct a large-scale dataset as retrieval database. 3. In the few-shot setting, what is the comparison with CoOp+TPT (which is better than CoOp according to [44])? 4. In the transductive setting, it is unfair to compare TPT as TPT only relies on a single test sample while the proposed method takes all samples in the test set into account. I think another baseline called TENT (https://openreview.net/forum?id=uXl3bZLkr3c) should be considered. 5. Writing can be improved. There are many repeated sentences across the whole paper. For example, the quantitative results have been mentioned in both introduction and experiment sections repeatedly. 6. Not sure what “3-shot CLIP model” refers to in line 214.

Questions

My major concerns and questions are in how the experiments support the effectiveness and significance of the proposed method. Please see the weaknesses part for more details. I’m willing to increase my rating if my concerns can be well addressed.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Response to Additional Reviewer Comments

We appreciate your thorough review. Below we've responded to the remaining review comments that did not fit into our initial rebuttal. We are happy to engage in further discussion and are interested in any further comments you may have. **Ablation on the choice of alpha?** We’ve included an ablation on the choice of $\sigma$ which determines the value of $\alpha$ according to $\alpha = e^{-|P|^2/\sigma}$ where $|P|$ is the cardinality of the priming pool. This formulation can be viewed as a Gaussian prior over the language embedding with posterior updates from the priming pool. Below we've included a table with an ablation over $\sigma$ and added these details to the paper. Ablation over choice of $\sigma$. Zero-shot top-1 accuracy is reported on various downstream tasks. | $\sigma$ | ImageNet | Stanford Cars | FGVC-Aircraft | Flowers102 | |--------|----------|---------------|---------------|------------| | 1 | 68.9 | 87.54 | 25.9 | 71.94 | | 10 | 69.1 | 88.98 | 29.75 | 74.97 | | 100 | 70.75 | 89.3 | 33.03 | 79.81 | | 1000 | 70.64 | 89.11 | 31.32 | 78.81 | **Questions:** **Will the method improve fine tuning efficiency of other prompt tuning methods? I mean after applying it, how many training times or steps can be saved for prompt-tuning?** Neural Priming does not improve training efficiency compared to the base CLIP model. The method improves sample-efficiency. In other words, a neural primed model can achieve the same accuracy as the base model with fewer labeled examples from the target distribution. The Neural Priming does improve training efficiency compared to other retrieval augmented methods such as REACT [7]. **Why not use LAION-5B?** LAION-2B is the english subset of LAION-5B commonly used by other works for pre-training and benchmarking[6,7]. For instance, OpenCLIP models are trained on LAION-2B and not LAION-5B. LAION-5B is comprised of LAION-2B, LAION-multilingual, and LAION aesthetics. We will add this clarification to the paper. **Is the method stable among different tasks?** Across all of the datasets we benchmarked, Neural Priming provided consistent accuracy improvements. If the question is about training stability, our method is deterministic and therefore stable from an optimization perspective. If the question is whether Neural Priming works for tasks such as segmentation, that is an interesting future direction. **Limitations:** **The method needs to know all test set to filter the priming set, online learning may exceed the scope of it.** Fair point, though Neural Priming would likely work in online settings. From our few-shot experiments, the transductive filtering approach works even with only a few samples from the target distribution. **The following are suggestions rather than comments, the authors may take them advisedly.** Thanks for the suggestions regarding notation, font size of figures, and use of capitalization with hyphens. We have updated the paper accordingly to reflect your comments and improve the readability of the work.

Update

I want to thank the authors for clearing up my concerns related to the baselines and differentiating from prior work, and believe that with these clarifications this paper provides an effective and easy to implement retrieval method. Therefore, I have raised my score from a weak accept to an accept.

Thank you for your response

The authors have resolved my questions. I am raising my score.

Reply to author rebuttal

Thanks for providing very detailed responses to my concerns. Since most of my concerns have been addressed, I'm happy to raise my rating to weak accept. I hope the authors can incorporate all the changes and new results into the revised version of the paper.