Your representations are in the network: composable and parallel adaptation for large scale models
We propose InCA, a lightweight method for transfer learning that cross-attends to any activation layer of a pre-trained model. During training, InCA uses a single forward pass to extract multiple activations, which are passed to external cross-attention adapters, trained anew and combined or selected for downstream tasks. We show that, even when selecting a single top-scoring adapter, InCA achieves performance comparable to full fine-tuning, at a cost comparable to fine-tuning just the last layer. For example, with a cross-attention probe 1.3% the size of a pre-trained ViT-L/16 model, we achieve performance within 0.2% of the full fine-tuning paragon at a computational training cost of 51% of the baseline, on average across 11 downstream classification. Unlike other forms of efficient adaptation, InCA does not require backpropagating through the pre-trained model, thus leaving its execution unaltered at both training and inference. The versatility of InCA is best illustrated in fine-grained tasks, which may require accessing information absent in the last layer but accessible in intermediate layer activations. Since the backbone is fixed, InCA allows parallel ensembling as well as parallel execution of multiple tasks. InCA achieves state-of-the-art performance in the ImageNet-to-Sketch multi-task benchmark.
Paper
Similar papers
Peer review
Summary
The paper presents a study on the benefits of training a small cross-attention based adapter instead of performing full fine-tuning of a large VIT model. The authors propose a cross-attention layer they dub InCA that has trainable queries to cross attend to intermediate layers in the large pretrained VIT model and then extract the information by average pooling, layer-norm and Linear. They claim a single query is enough in most cases but also propose a multiple query version they call OpenInCA.
Strengths
* There are strong benefits of having a methodical study of adaptation and I think this paper fits the bill. * The method is simple, well explained and a priori easy to reproduce and expand. * The experiments are broad enough to be interesting to the community. * There are clear computational benefits from the two stage training.
Weaknesses
* The claim in L177 is a bit of a red flag for me. If only one query is enough it may be that the way this is achieved is when the query is very far from the "latent" data and thus the attention weights become flat ie performing average pooling. The nice thing is that it would be an easy and i think useful experiment to check this. Redoing figure 3 with avg pooling but with all the extra bits like the layer-norm that is in the model. I presume the LP doesn't have it ? * I haven't found any experiments to validate that the class-incremental learning benefits of the Open InCA architecture which is claimed in line 203.
Questions
* Have you tried multiple self-attention layers instead of just one? This would be equivalent to running a small perceiver model [30] and then either average the resulting latent tokens. * The average pooling has to be done to make sure that the cross attention IS the right architecture here and not a simpler one. It would also add a bit of depth to the paper and answer an obvious question. I think this is required to recommend acceptance. * Why do you include both tables 1 and 3 ?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
I think there is no obvious negative impact.
Summary
- This paper proposes a new way to adapt a pretrained deep neural network for downstream tasks called InCA. InCA does not modify the intermediary representations of the pretrained network and thus doesn’t require backproping through it, which makes it memory- and compute-efficient. To use InCA, one first heuristically identifies a few layers, the activations of which are processed through cross-attention with trainable query tokens. - The cross-attention setup is more expressive than linear maps, which is supported by empirical and theoretical arguments. It also allows the addition of new classes through the injection of new query tokens. - Finally, because the forward pass of the original network is not at all modified, one can cache the relevant activations of the entire dataset once and apply InCA without calling the original model at all. It is also possible to adapt to many tasks in parallel for the same reason. - Experiments on many image datasets show that InCA performs competitively with finetuning and other efficient adaptation methods.
Strengths
- This work includes a broad range of baselines in their experiments. - The proposal is simple yet appears to be effective. The stated benefits are technically sound.
Weaknesses
- The paper “[presents] a framework for transfer learning that efficiently adapts a large base model by learning lightweight cross-attention modules attached to its intermediate activations.” However, the experiments are limited to mostly computer vision classification tasks. Some of the baseline approaches, such as LoRA and BitFit, are extensively used for language understanding, language generation, and image diffusion. The result of this paper will be much more convincing if experiments in these domains are included, especially if InCA can outperform existing baselines. - A significant portion of the benefit of InCA comes from using a subset of the layer activations. However, this important choice (as described in B.2) is done heuristically according to Section 3. The paper can benefit from more clarity on how to choose such layers. For example, how are these layers chosen for the experiments? - The clarity of the writing can be improved. For example, it was not clear during my first pass what the dimension of z is and what T is. The description for diag-head is confusing: it appears to be a simple matrix multiplication between [a_1, …, a_c] and W. Section 3 overall can be better organized by heuristics, methodology, and benefits. - The “signature” phenomenon is interesting, but the given analysis doesn’t provide much insight. It would be great to hear more about what we can learn from these “signatures,” especially given that they are presented as one of the key advantages of InCA in the conclusion.
Questions
- How competitive is InCA in other domains where many of the compared baselines, e.g., LoRA and BitFit, dominate? E.g., language understanding, language generation, diffusion, etc. - How is the subset of layers chosen for the experiments? Could a similar selection process benefit other baselines?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
N/A
Summary
This paper proposes an efficient fine-tuning method that works parallel to the pre-trained network. Based on cross-attention between intermediate activations, InCA can generalize to various classification tasks from different domains. Additionally, the framework inherently supports class incremental learning, multi-task learning and continual learning objectives. The overall objective is cross-layer feature merging using cross-attention between them while freezing the base model.
Strengths
1. There are various parameter-efficient fine-tuning methods in the literature; the major strength of this work is described in the last part of Sec. 3 - using cached activations to train the model in mere seconds. 2. The overall formulation of cross-attention is simpler and elegantly extends to multi-task and continual learning paradigms. 3. Experiments are shown on various vision classification datasets and one multi-task learning benchmark.
Weaknesses
1. The formulation of Open-InCA is not completely clear and can be better presented; However, I understood the complete idea, and I had to re-read it a couple of times for deeper understanding. 2. The field of efficient fine-tuning/transfer learning is rapidly moving, and using standard benchmarks helps understand the performance gains more. The authors can give results on VTAB-1k, few-shot learning experiments, etc, as shown in SSF[1], VPT[2], FacT[3] 3. Also, I found comparison with existing methods a major weakness, as only LoRA, BiTFit and AdaLN are shown. [1] Scaling & shifting your features: A new baseline for efficient model tuning. [2] Visual Prompt Tuning [3] FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer
Questions
See the weakness sections. 1. Standard benchmarks and a thorough comparison with existing works will make this work a lot stronger 2. A major emphasis on peak training peak memory, inference latency and parameter overhead needs to be provided.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes
Summary
The paper presents a method termed InCA (Introspective Cross-Attention) to learn compact adapter modules for large vision models that can be used for various downstream tasks (image classification domains). The proposed approach has an advantage over the entire model finetuning due to its parameter efficiency resulting in a smaller GPU memory footprint required for training. In addition, the proposed introspective cross-attention module is architecture-agnostic, enabling simple implementation for different intermediate network representations and network types.
Strengths
The proposed approach (InCA) demonstrates stable performance when applied across different model types (ViT, SWIN, CNN). Unlike previous architecture-specific adaptors, the method can be implemented without modifications with different base model architectures. The experiments demonstrate that the method can reach performance comparable to the entire model finetuning. At the same time, the number of trainable parameters constitutes only a tiny fraction (1-2%) of the backbone model weights. Because the training process doesn’t require backpropagation through the backbone, it is GPU memory efficient, making the large pre-trained transformer-based backbones reusable for different downstream tasks. Compared to the linear probing, the proposed cross-attention architecture of the adaptor module has a significantly larger “extraction capacity” that leads to an improved classification accuracy of the adapted model. When applied in parallel to different activation maps, the method produces a network “signature” - w.r.t. to the downstream tasks. All this highlights the fact that the internal representations of the pre-trained large models have sufficient representation power for many downstream domains.
Weaknesses
The paper builds on the large body of literature on efficient transfer learning. While exploring the utility of cross-attention as a choice for adaptor architecture, the authors follow the steps of many prior works, such as Perceiver [30] or [A]. On the other hand, the current study combines empirical results for different backbone architectures (e.g., both CNNs and Transformers) and suggests applying cross-attention modules in parallel to different activation maps to identify the most relevant features for the downstream tasks systematically. Thus, despite being a comprehensive study, the novelty is limited. As discussed in the Appendix, InCA, as an efficient and modular model adaptation framework, can be helpful for domain practitioners (e.g., medical imaging domain) to bridge the gap between cutting-edge research in visual representation learning and real-world applications. [A] Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation, in EMNLP 2021.
Questions
Could you highlight the most interesting (maybe counterintuitive?) findings? “The second best adaptation approach is LoRa … at additional training costs” invites the comparison along axes other than the error rate. Could you compare the training costs directly? Table 2: “\dagger indicates full FT was avoided due to prohibitive computational costs” – could you elaborate on what costs were considered prohibitive and why?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
n/a
Summary
This work presents a method called Introspective Cross Attention (InCA), which aims to identify high-performing adapter models for handling downstream tasks using large-scale pre-trained models. InCA achieves competitive performance compared to the well-established baseline of full fine-tuning, while also enabling parallel multi-task inference. The experimental results effectively demonstrate the effectiveness of this method in terms of both performance and efficiency.
Strengths
1. The paper is well-written and easily comprehensible. The Introduction section, in particular, effectively establishes the context for the entire work. 2. The exploration of introducing adapters for large-scale models in the context of vision transformers is still relatively less explored. 3. The promising results on well-known transfer learning datasets at a smaller scale indicate the potential of the proposed method.
Weaknesses
1. The datasets employed in this study may not be the most appropriate test-bed for evaluating large-scale pre-trained models. Previous transfer learning works [1, 2, 3, 4] have achieved comparable accuracies using lightweight CNN models that demand significantly fewer FLOPs and parameters. To gain a comprehensive understanding, it would be valuable to compare the FLOPs and parameter counts. 2. While I acknowledge that the authors have utilized commonly used datasets for transfer learning tasks, it is worth noting that these datasets may not provide sufficient challenges when employing models such as ViT-G/14 and other large variants. References 1. [Co-Tuning for Transfer Learning](https://proceedings.neurips.cc/paper/2020/hash/c8067ad1937f728f51288b3eb986afaa-Abstract.html) 2. [Stochastic Normalization](https://proceedings.neurips.cc/paper/2020/hash/bc573864331a9e42e4511de6f678aa83-Abstract.html) 3. [Bi-tuning of Pre-trained Representations](https://arxiv.org/pdf/2011.06182) 4. [$\Delta$-Networks for Efficient Model Patching](https://arxiv.org/pdf/2303.14772) -- Table-7
Questions
1. Could you please provide (or comment) a comparison in terms of FLOPs? 2. In terms of memory and compute how does CNN based transfer learning approaches compare with ViT based approaches? 3. This approach might suit very well for tasks like model patching, see [PAINT](https://model-patching.github.io/) and [Delta-Networks](https://arxiv.org/abs/2303.14772)
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Discussed in the appendix
Summary
Summary: Firstly, I would like to kindly point out that this paper proposes an "adaptation" method; however, I believe there may be some serious concerns in multiple aspects. Firstly, the paper appears to lack innovation in its methodology. Secondly, the experimental design also seems to have some significant shortcomings, such as lacking adequacy and detailed experimental evidence. Additionally, I would like to express my concerns about the writing aspect of the paper, where some details seem to be missing and where some over-claims are made.
Strengths
(Positive) Although this paper possesses some flaws, including the lack of innovation and rigorous experimental design, I must acknowledge that the approach in this paper appears reasonable to some extent. Additionally, it is commendable that the authors have provided a significant amount of supplementary material. However, I believe these aspects may not fully compensate for the shortcomings observed in other areas of the paper. I genuinely hope that in future research, the authors will earnestly consider these critical points and endeavor to enhance the quality of their work.
Weaknesses
(Negative) Allow me to further inquire about some of the claims made in the paper. In the first sentence of the article, the authors claim that in natural language tasks, the data and hypothesis space are shared, which appears quite astonishing. Considering the diversity of tasks in natural language, can this assumption be readily applicable to every task? I believe this general statement may lack sufficient basis and in-depth investigation. (Negative) Furthermore, the authors assert that their model can enhance "robustness," but it is important to note that "robustness" is an extremely specialized concept. Therefore, I hope they can provide more detailed experimental evidence to support this claim. A simple assertion may not be sufficient to substantiate such an important proposition. (Negative) I must emphasize that the approach presented in this paper does not seem to demonstrate significant differences compared to existing methods, including but not limited to LoRA. This apparent lack of innovation in the technical aspects is somewhat disappointing. (Negative) Moreover, it appears that the authors' method has not undergone sufficiently large-scale model validation to demonstrate its feasibility in terms of large model transferability. Additionally, the model they have chosen may not entirely qualify as a so-called "foundation model." This lack of rigor in the experimental design raises doubts about the viability of this method. (Negative) Additionally, the authors seem to have not thoroughly validated the capabilities of their method on various diverse tasks. The scope of their experiments appears to be rather narrow, omitting coverage of multiple tasks such as generative models and discriminative models simultaneously. For instance, in generative models, the training cost of the so-called "stable diffusion model" may be high, but its adaptation is crucial. Unfortunately, the authors' in-depth research and contributions in this area seem to be limited. (Negative) What is also concerning is that the authors of this paper appear to have a somewhat limited perspective, primarily focusing on a few tasks they studied, while neglecting the introduction of task details. Moreover, they may hold the belief that ImageNet pretraining is the default choice worldwide, a viewpoint that is challenging to comprehend. In reality, a foundation model should not be confined to ImageNet pretraining but should consider larger benchmarks to enhance its generalization ability. I would like to strongly urge the authors to be mindful of this and clearly indicate in the paper where their model was actually pretrained.
Questions
See *Weaknesses
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
No. See *Weaknesses
The revised writing will make things much clear. However, a major concern is a limited comparison with existing works which is still not presented in the rebuttal. This makes the work rather weak. The authors need to address this issue during the discussion period.
Response to reviewer Part 1
We disagree with the premise that we do not compare with existing work. In fact we already extensively compare with the suggested work of VPT that the reviewer ask that we compare with (already in the manuscript, e.g. see Tab. 1, Tab. 7). Below we summarize the results of VPT on ViT L/16 as compared with InCA. | Method | Full fine-tuning | InCA | VPT Deep | |------------------------|------------------|------|----------| | Average | 10 | 9.8 | 12.3 | | Maximum gap to full FT | 0 | 2.3 | 6.8 | We observe that InCA has a smaller maximum gap to full-fine tuning, 2.3 vs 6.8 of VPT. In addition, in Appendix C we provide a detailed comparison with the **computational efficiency** of VPT and InCA applied to larger models. We observe that per-run, InCA is 290% faster than VPT on ViT-L/16 (with VPT efficiency issues exacerbated for bigger models, cf. line 692, Fig 1 Right). The 290% speed-up is for a single training run and the gap widens if we compare *per-dataset* due to the extensive hyper-parameter search required for good results with VPT (see the original work of VPT paper, Tab. 6). For the other works of SSF, and FaCT, we added and cite both of them as additional interesting work in the related work section for parameter efficient approaches. Note each of SSF, FaCT and VPT require back-propagation through the entire model as we discuss (lines 75-80) this makes them expensive for fine-tuning of large-scale models akin to the computational costs of running full-fine-tuning. On the other hand InCA does not require back-propagation through the pre-trained model which makes the method much more scalable to larger models (see Fig. 1, Right for comparison of model scaling for InCA and VPT, Full-FT). Further as we discuss (lines 55 - 58) the modification of the backbone execution limits each of the approaches (SSF, FaCT, VPT) in more flexible learning and inference scenarios such as multi-task and class incremental learning, whereas InCA’s modular adaptation has a “one to many” property allowing for computation sharing of the backbone computation. In "Response to reviewer Part 2" below we compare and discuss FaCT, SSF in detail and compare them with InCA. As we note above both FaCT and SSF are parameter efficient but lack some of the other key benefits of InCA.
Response to reviewer Part 2: Discussion of FaCT and SSF in comparison to InCA
**FaCT: FacT: Factor-Tuning** is a method enabling extreme *parameter storage* efficiency in adaptation, yet we do not believe comparing directly with FaCT is necessary for our manuscript since FaCT can be viewed akin to full-fine-tuning. The method of FaCT comprises of 2 steps: In the first step full-fine-tuning is applied to optimize the model. In the second step, the fine-tuned parameters difference is computed (Delta W), rearranged as a 3D tensor and then efficiently decomposed (e.g. via Tucker tensor decomposition). This approaches leads to immense parameter efficiency (final storage in KBs). The authors utilize the homogenous ViT architecture for creating a uniform 3D tensor and we note that it unclear how to use FaCT in the context of CNNs which have heterogeneous layer layout. In contrast InCA automatically applies to CNNs (see. Tab. 2). In the context of InCA, since in the first step of FaCT exactly applies full fine-tuning we may utilize the full-fine tuning results as a sense for the performance axis. For computational efficiency, as noted, since FaCT already contains a full fine-tuning step, it is prohibitively expensive for large-scale model (see Fig. 1 Right). Beyond the parameter efficiency front, FaCT suffers from all other issues of full-fine-tuning. As we discuss in the paper (lines 290-292) full-fine-tuning does not allow for fast multi-task inference presented in Tab. 4, nor class-incremental learning as shown in OpenInCA. Lastly FaCT’s adaptation results in an opaque decomposition of the parameters as compared with InCA’s performance signatures (Sec. 5, Appendix A). Regarding parameter efficiency, InCA uses 1.3% of the trainable parameters of a ViT-L/16 which amounts to about 4M parameters, whereas FaCT reports as low parameter count as 8K final parameters. However in the context of visual tasks - the image itself requires disk storage in the order of Megabytes hence for most applications there are diminishing returns from such extreme parameter efficiency for storage (e.g. when considering networking, IO, and storage bottlenecks). Without any storage optimizations, a single InCA adapter disk storage size is already comparable to the disk storage size of a single image. Post-training storage optimizations as presented in FaCT are complementary yet orthogonal’s to InCAs approach which can enable future storage gains if extreme storage efficiency is warranted for an application. **SSF:** Is an adaptation method which inserts learnable affine parameters (think LN’s affine gamma, beta) after linear operations in the network (e.g. linear operations in LN, MSHA, FF). Post-training, those affine parameters can be merged with the linear operation that preceded them since two linear operations can be composed. Like FaCT, SSF requires back-propagation through each of the network’s activations which makes it computationally expensive and non-scalable for large models, albeit the number of affine parameters is small which leads to parameter efficiency. Similar to FaCT, it is also *not* modular and flexible in terms of class-incremental learning, multi-task efficiency and we would expect it to have comparable computational efficiency as a method like LoRA which is 65.3% more GPU memory intensive than InCA for a ViT-H/14 model. We conclude that we have cited FaCT and SSF as additional context, already extensively compared with VPT in the original manuscript and provide detailed descriptions and comparisons with both FaCT and SSF above. In summary each of the above methods requires back-propagation through all of the model’s activations which makes them computationally expensive for large models, and each of the methods do not exhibit the modularity of the InCA adaptation limiting their application in the settings which we demonstrate in detail for InCA.
These comments have justified the chosen methods to some extent. Still, in the case of single vision tasks, why did the authors choose the datasets in Table 1. why not the standard VTAB-1k or FGVC dataset? This is my only concern; it will make the evaluation more fair. Also, even if SSF, and FacT require backprop through the whole network, it is worth comparing them for single vision tasks and giving the training/inference memory/throughput gain to justify the claims. VTAB-1k/FGVC will make this much easier as the numbers can be directly picked from the existing works, and the authors only have to experiment with their work. A standardization of datasets/tasks is the need of the hour for efficient learning methodologies.
Response to Reviewer
We thank the reviewer for going over our response and are glad that our clarification to the questions justify the method of InCA in the eyes of the reviewer. With regards to training memory, we report detailed training memory comparison of different methods at different scales in Figure 1 (Right). Throughput gains are reported in Tab. 4 for multi-task inference. For inference memory, the additional required inference memory of running InCA is dwarfed compared with the memory needed for the pre-trained network inference and without any optimizations, the additional inference memory overhead of InCA is < 4%. With regards to the datasets, we would like to point out that in our paper we used 11 standard and official datasets which are high quality, challenging, and diverse. Each of the datasets we use is widely researched and used in the literature. The term FGVC is popularized by the CVPR FGVC workshop and we in fact consider many such FGVC datasets (see below). Compared with SSF for example we already use 5/6 of the datasets which SSF labels as FGVC. In particular, in Table 1 (and rest of the experiments) we already include 1) FGVC Aircrafts, 2) FGVC CUB-200, 3) FGVC Flowers (oxford flowers), 4) FGVC Cars (Stanford Cars), and 5) FGVC Dogs (Stanford Dogs). The only dataset in the FGVC consideration of SSF which is not in Tab. 1 is the FGVC NABirds dataset which considers fine-grained bird classification. Nonetheless, NABirds is correlated with the CUB-200 dataset and they both operate on the same task of fine-grained bird classification with overlap with CUB-200. Further in our experiments we consider additional challenging high quality datasets which are not used in SSF. For example this includes the FGVC Herbarium dataset (presented in FGVC-2020) which is considered a modern and challenging fine-grained dataset. Other high quality and diverse datasets we consider include MIT-67, Flood Depth, Describable textures dataset (DTD), EuroSAT, and Oxford Pets. Each of the datasets we use is an official dataset that was directly downloaded from the original source which presented the dataset (e.g. the FGVC workshops dataset website). This ensures a single source of truth at the dataset level. As compared with a dataset set like VTAB-1K, which was produced by post-processing and sub-sampling of original sourced datasets, poses challenges in reproducing index splits among other variations. For example with VTAB-1K, see this open issue [“Dataset split ids? #7”](https://github.com/google-research/task_adaptation/issues/7) concerned with not having access to consistent and direct index splits of the original datasets repository. For a dataset as small as 1000 samples, variation in which sample indices are used and the class composition lead to significant performance difference and result in additional variations and barriers for easy evaluation.
Thanks for the detailed response
My concerns are addressed. Please include the clarification in the revised manuscript. I've raised the score accordingly.
Response to reviewer
We are glad the rebuttal addressed the main concerns raised by the reviewer. We would like to thank the reviewer for taking the time to review the contents of our rebuttal.
comment
I thank the authors for their additional work and comments. I think this makes the paper more easy to justify in my mind. i will thus change my ranting to borderline accept.
Response to reviewer
We thank the reviewer for taking the time to consider our rebuttal response and change their rating accordingly.
I thank the authors for their detailed response.
I have carefully reviewed the initial submission, the authors' response, and the feedback from other reviewers. I appreciate the effort that has been invested in addressing the concerns raised, and the responses have addressed my concerns to a reasonable extent, however, I remain aligned with my initial assessment.
response to authors
-- ImageNet pretrained models are not foundation models. In this paper, only Table 2 reports a few results regarding CLIP, ViT-H/G, and ResNext-101. The primary focus of this paper is on ImageNet pretraining. Could this potentially be seen as an overclaim? -- If the authors are not acquainted with NLP, kindly avoid employing unscientific overclaims. -- If the authors are not familiar with certain scientific terms, such as "robustness," please refrain from using them without providing a precise definition. In summary, this paper seems to include some overclaiming, which might diminish its scientific quality. It would be advisable to consider toning down the language.
Decision
Accept (poster)