HyperCLIP: Adapting Vision-Language models with Hypernetworks

> **Can you explain why do you think your experiments are conducted in zero-shot setting?** This is certainly a fair point, but we really want to emphasize that this is exactly the terminology used by existing benchmarks and CLIP training pipelines in general. Universally, CLIP training datasets use the ImageNet (training) data in their collection of images and often filter based upon these keywords. Indeed, the setup we use here is not from our own model, but exactly follows the methodology of the DataComp paper [1], in order to maximally fit within standardized training setups. In this setup, a large pool of data in the form of images and captions is collected (often from the web), and the models are trained on this pool. Such data pools may be filtered to improve the downstream performance of the trained models. The evaluation is ultimately done on test sets of standard image-classification datasets, e.g., ImageNet, CIFAR100, etc., that are not included in the training dataset. Specifically, the filtering process in our paper involves using the ImageNet-21k class names as a filter of the captions (again introduced as a baseline in [1]). Indeed, the filter does not involve using any of the ImageNet images, and using just the words from ImageNet-21k to filter the captions is a very weak form of bias. More importantly, the *aim* of this paper is to contrast HyperCLIP and CLIP under the *exact same* setup (i.e., both zero-shot and fine-tuning) for (classification, image retrieval) and on diverse datasets (including fairness benchmarks). While the filtering process is important, it doesn't give our method any systematic advantage, and we evaluate on other datasets besides ImageNet. There is ongoing research into the extent (if any) of data leakage that might be included in these data pools, i.e., if the test sets of standard image datasets have found their way into large training data pools, but that is outside the scope of our work and, as mentioned, does not invalidate our architectural contribution. [1] DataComp: In search of the next generation of multimodal datasets NeurIPS 2023. --- > **The approach doesn't generalize to broader VLMs, especially the larger models. CLIP models are generally the smaller models among recent VLMs. I don't think the proposed approach generalizes to the larger VLM models such as LLaVa.** This is really a separate point, because broader VLMs and CLIP models are quite different in their nature: broader VLMs integrate vision tokens in LLMs to let the system produce text pertaining to images, while CLIP is fundamentally an image/text encoder that produces well-aligned representations. Perhaps most notably, VLMs like LLaVA *use* CLIP as their underlying tokenization of the images and benefit hugely from these pre-trained representations. Thus, the domains are really orthogonal (but also nicely related), and we suggest not discounting work on CLIP-based methods simply because they are a “small” component of larger VLMs. --- > **HyperCLIP reduces CLIP applications to only image classification.** This is not entirely correct. We also evaluate HyperCLIP on retrieval tasks, which is perhaps closer to another more popular use of CLIP models. In this setting, we measure the recall when the text embedding (of the CLIP text encoder) is used to retrieve an image from a set of images, and similarly when the image embedding (of the CLIP image encoder) is used to retrieve a caption from a set of captions. But more broadly, this approach is fundamentally about producing a highly compact, text-specific image encoder in CLIP, which certainly could find application to other domains like VLMs, etc. (as mentioned above, CLIP is the image encoder used by many open-source VLMs). While we believe this to be orthogonal to the current paper and likely a question for future work, we absolutely feel that improvements to CLIP such as this are valuable contributions to the field as a whole, even if CLIP zero/few-shot image classification is only one potential application.

> Will you plan to release the code ? Yes, we will publish the code and model weights after the review process.

We thank the reviewer for their thorough review. ### Re: Questions > **Why the text transformer (a width of 768, 8 heads, feed-forward dimension of 2560) is not a small model, since the proposed HyperCLIP is targeted for compute-restricted scenarios.** As discussed in the paper, the cost of inference when deploying a CLIP model, e.g., for classification, predominantly comes from the image encoder. This is because we pre-specify the prompts for each class and then obtain the corresponding text embedding using the text encoder. During inference, we don’t need another pass through the text encoder; however, we always require a forward pass through the image encoder. Consequently, the main bottleneck with respect to size (for inference) is the image encoder, not the text encoder. > **Is the HyperCLIP trained from scratch or initialized from SigLIP?** HyperCLIP is trained from scratch in all our experiments, as mentioned in the paper (line 215). > **Which part of the experiments are zero-shot results, and which part are linear probing results?** The last two columns of Table 2 refer to the results of fine-tuning a linear layer (i.e., linear probing) on ImageNet-1K and Cifar100. The remaining results in Table 2, as well as additional results in Appendix A1, refer to zero-shot performance. --- ### Re: Weakness > **The experiments show performance improvement over a baseline without a hyper-network but fail to address the efficiency problem. In other words, the hyper-network enhances performance only in small-scale models rather than improving CLIP's efficiency directly.** We believe there may be a misunderstanding here. We are explicitly developing a model to improve the test-time deployment of a zero-shot classifier, which directly corresponds to using a smaller image encoder. Our notion of efficiency is the model size (number of active parameters during inference). In practice, model size is the dominant factor when deploying models in edge applications, as it directly impacts latency and is often constrained by memory. This means that if we can reduce the size of a CLIP model (in particular, the image encoder) while maintaining its performance, we achieve a more efficient CLIP model. In this paper, we show that we can achieve this goal, and our experiments demonstrate that HyperCLIP models with *less* parameters perform better than CLIP. For example, HyperCLIP-adapted EfficientNetB1 outperforms EfficientNetB2 by 1% (on ImageNet-1K), despite an additional 1.2M parameters in EfficientNetB2. While we acknowledge that model size is an imperfect notion of efficiency, EfficientNet *is* explicitly an architecture designed for inference efficiency, and thus seems a reasonable choice when focusing on test-time performance. > **Poor Writing and Difficult to Follow** Thank you for highlighting specific areas for improvement. We will revise these passages for clarity, remove ambiguities, and simplify complex sentences. > **Limited References and Literature Review** Methods like LoRA are orthogonal to our work, as they focus on introducing new parameters that are then fine-tuned for specific capabilities. We will include a discussion of these methods in a revision. Could the reviewer clarify why diffusion-transformers, i.e., [b], are relevant to our paper, which focuses on the efficiency of CLIP models in resource-constrained deployment?

We have uploaded a revised version of the paper, highlighting edits made to improve the readability of the introduction and to include additional references. Additionally, we have expanded the discussion in the related work section to address the connections between HyperCLIP and LoRA. We hope these revisions address your feedback, and we would be happy to discuss any remaining concerns.