Bayesian-guided Label Mapping for Visual Reprogramming

Summary

This paper focuses on a technique called label mapping (LM) which is used in visual reprogramming. It finds the relation between pre-trained labels and downstream labels. In conventional fine-tuning, LM is just the backpropogation using the loss function. However, this paper and previous similar papers aim to develop a gradient-free way to find the weights in the last layer in the conventional fine-tuning. The whole paper is based on Bayes' rules (using it on the output of a pre-trained model, which is interesting), which is solid and theoretically justified.

Strengths

1. Good paper flow. The flow of this paper is good and appreciated. Directly analyzing the loss using the Bayes' rule is clear. 2. Extensive experiments. Except for the good performance, the authors also try to understand the good performance, which is interesting. Especially on the visualization of top relationship between pre-trained labels and downstream labels. The result is convincing, as the selected top pre-trained labels are indeed related to downstream tasks. 3. Timing topic. I can see some applications which might need this kind of techniques. Keeping the integrity of a pre-trained model has many advantages, e.g., not lose the generalization of the pre-trained model. 4. Solid analysis. Two theoretical contributions are included in this paper. One is how to embed a pre-trained model into p(y^T|x^T), another one is how BLM has a higher accuracy.

Weaknesses

1. Benchmark datasets follow the previous papers, which is good. However, these tasks seem not the most relevant to what you propose in this paper. Are there some scenarios where the fine-tuning the last layer is not feasible? 2. The paragraph between 253-266 is confusing, which can be moved to other section. Appendix perhaps. 3. One major drawback of the presentation is the algorithm tables. The equations are still not the best way for practitioners. Main algorithm table should be moved in Section 4. After looking at the algorithms, it is easy to find that the proposed method is easy to be implemented. However, I cannot feel that after I read the section 4. 4. How to choose Padding or Watermarking in practice? 5. Experiments to increase n are not necessary. Could the authors explain why it matters? As long as the method is valid, the accuracy will be improved when n increase.

Questions

See the weaknesses above.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

No concerns for this paper.

Summary

Visual reprogramming is an interesting way to reuse a pre-trained classifier or an VLM. In previous methods, the way to change the output interface is basically gradient-free and one-on-one mapping. In this paper, the authors found that the previous way is suboptimal and ignores information. Then, from a theoretical perspective, a new objective is derived using Bayes' theorem. By optimizing this objective, the performance of VR methods is significantly improved, which is demonstrated via extensive experiments. A theoretical study is provided as well.

Strengths

1. The research topic is of significance. VR is useful in practice. This paper understands and extends the label mapping from a mathematical perspective, which steps much further compared to previous methods. 2. The derivation regarding w is could generally cover previous methods (which is probably the first). The drawback of previous methods is manifested based on the derivation. 3. Math expressions are easy to follow. 4. Abundant experiments are provided, including large datasets, large models, and VLM. From the experimental results, we can see the improvement is universal, justifying the general effectiveness of the proposed method.

Weaknesses

1. In Eq. (1), what is the real gap between right and left terms? It seems not easy to get this gap. I suggest using right one directly, which is easy to be acceptable. 2. Above the Eq. (6), one might misunderstand the way to calculate the frequency. Using () might be better. 3. Figure 2 is confusing. It looks like step 3 follows step 1, rather than step 2 following step 1. What is the actual sequence between step 1~3? 4. In section 4.2, the calculation of accuracy is confusing. $x$ is missing. 5. What does $\bar{r}$ mean in l.262? 6. Training time is potentially a concern. How many training epochs are needed for BLM?

Questions

Please refer to "Weaknesses".

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

No further concerns regarding limitations.

Summary

Another type of transfer learning approach is considered in this paper: model reprogramming. Different from classical transfer learning approaches, model reprogramming only changes models via changing the input space and output space, which is more efficient to fit a pretrained model to some downstream tasks. Specifically, this paper focuses on the output space, label mapping. The motivation is strong as it is clear that single map does not work well. Then, new methods are proposed based on Bayes' theorem. Experiments are enough and solid, in terms of model size or dataset size.

Strengths

1. Transfer learning is even more important in the current era. This paper focuses on an interesting direction, which is significant to the field. 2. The main result of this paper looks general, compared to previous label mapping methods. It is novel to first consider a pretrained model's output into the P(y|x) and then analyze what we should do. 3. I enjoy reading the experiment section, which is quite comprehensive. I did not see some necessary experiment missing.

Weaknesses

1. Transfer learning literature is missing in this paper. Although model reprogramming is not considered as a transfer learning method in the literature, it is one of transfer learning techniques. This review can help position this paper better. 2. Can we transfer a pretrained model to any possible task? Are there standards to help choose which model should be used for a specific downstream task? 3. For example, if we can use a model trained with cars to help recognize images of animals? It looks impossible. How to tell when we can use a pretrained model? 4. Computing w in BLM also needs time. Is there efficient advantage of VR anymore? What is the performance of finetuning? The gap should be reported in this paper. 5. Main algorithm table should be moved to the main paper. It is not very easy to implement BLM or BLM+ based on the formula, but the algorithms are very helpful. 6. It would be better to explain meaning of some conditional probabilities.

Questions

1. Transfer learning literature is missing in this paper. Although model reprogramming is not considered as a transfer learning method in the literature, it is one of transfer learning techniques. This review can help position this paper better. 2. Can we transfer a pretrained model to any possible task? Are there standards to help choose which model should be used for a specific downstream task? 3. For example, if we can use a model trained with cars to help recognize images of animals? It looks impossible. How to tell when we can use a pretrained model? 4. Computing w in BLM also needs time. Is there efficient advantage of VR anymore? What is the performance of finetuning? The gap should be reported in this paper.

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

No concerns for this paper.

Summary

Pretrained models play a crucial role in current machine learning and computer vision tasks, and effectively leveraging them in downstream tasks has become increasingly important. This paper explores the research area of visual reprogramming (VR), which diverges from traditional fine-tuning by adjusting the input space rather than the parameter space of pretrained models. Additionally, VR necessitates establishing a mapping from pretrained labels to downstream labels, which is the primary focus of this paper. The proposed method for mapping pretrained labels to downstream labels is well-motivated and convincing. The experimental results are robust and demonstrate the efficacy of the approach.

Strengths

1. The experiments are solid and impressive, covering larger models compared to previous works in this field. 2. The motivation is very clear, adhering to basic probability rules. 3. Two methods are proposed based on different estimation approaches for the key component, demonstrating sufficient technical contribution. 4. Some experiments provide insightful explanations on why the mapping learned from the proposed method is effective, which I found particularly interesting.

Weaknesses

1. Although the motivation is clear, it feels somewhat dry. More detailed explanations are needed for Section 4.1. Specifically, some of the conditional probabilities require further clarification (e.g., Eq. (6)). 2. It is unclear how Section 4.2 can be extended to the multi-class case. The accuracy metric (Acc) used here appears to be suited only for binary cases. 3. While it is commendable that a universal hyperparameter performs well in your experiments, it raises the question of whether these hyperparameters can be tuned using a validation set. How do all methods perform when evaluated on the same validation set? 4. I am not entirely convinced by Figure 4. Why is pineapple selected as well? This needs further explanation. 5. The absence of algorithm tables in the main content is a significant omission and should be addressed. Similar to the first weakness, the current method presentation feels somewhat dry. 6. Is BLM+ without Top-K and Bayes the second column in Table 3? If so, additional descriptions should be included for clarity. 7. What are the fine-tuning results in this context? It is essential to establish a standard to determine when VR is needed. Additionally, what is the total time cost compared to fine-tuning?

Questions

See Weaknesses.

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors have adequately discussed the limitations of the work.

Thanks for your support!

Dear Reviewer zxuh, Many thanks for your support and increasing your score to 8! We will merge all comments into the updated version. Best regards, Authors of Submission7993