Robust Fine-tuning of Zero-shot Models via Variance Reduction

Summary

This paper addresses the ID-OOD trade-off of the ensemble-based method under the pretrain-finetune regime. The authors propose a simple yet effective adaptive ensemble method that determines the ensemble coefficient based on the feature distance between a test sample and the zero-shot failure set. The proposed method has a theoretical foundation, and it consistently improves the OOD generalization performance compared to the vanilla ensemble baseline in diverse situations.

Strengths

* The paper is well-written overall and easy to follow. * The research problem this work focuses on is important in the era of foundation models, and the proposed method to address this is very straightforward and reasonable. * The proposed method shows consistent performance improvement in numerous distribution shift setups, and the author provides an intuitive theoretical justification for the proposed method. * Besides the empirical success in terms of improved ID-OOD performance trade-off, the authors present extensive qualitative and quantitative analysis results that give numerous insights into this field.

Weaknesses

* The method requires 1) accessibility to the entire training dataset and 2) distance computation between each test sample and the entire train sample during inference time. These limit its application to resource-constrained (in terms of accessibility, memory storage, and runtime) situations. * Limited implication of the theoretical result * While the current theoretical analysis provides a justification for the optimal strategy for determining the ensemble coefficients, which is realized by the authors, it does not say anything about the relative generalization error between the proposed method and other non-ensemble-based fine-tuning methods. * Lack of possible comparison with more advanced fine-tuning [4,5,6,7] or ensemble [1,2,3] methods. * While the proposed approach has its unique implication compared to the vanilla ensemble, missing comparison with advanced baselines raises concern about the practical utility of the proposal. --- > Reference * [1] SIMPLE: Specialized Model-Sample Matching for Domain Generalization, Li et al. 2023 * [2] Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models, Zhu et al. 2023 * [3] Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization, Mavromatis et al. 2024 * [4] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, Kumar et al. 2022 * [5] Finetune like you pretrain: Improved finetuning of zero-shot vision models, Goyal et al. 2023 * [6] Trainable Projected Gradient Method for Robust Fine-tuning, Tian et al. 2023 * [7] Towards Calibrated Robust Fine-Tuning of Vision-Language Models, Oh et al. 2024

Questions

* Could the proposed method also be applied to the weight-space ensemble rather than the output-space ensemble? * How sensitive is the proposed method against varying numbers of training samples that determine the quality of the zero-shot failure set?

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors adequately stated the limitation in the paper.

Summary

This study examines the robust fine-tuning of CLIP models. The proposed method, Variance Reduction Fine-tuning (VRF), employs a sample-wise ensembling technique to enhance both ID and OOD accuracy, reducing trade-offs between them. Experimental results on ImageNet and associated distribution shifts empirically validate the effectiveness of this approach.

Strengths

- The proposed approach is straightforward, combining outputs from zero-shot and fine-tuned models through output-space ensembling. Unlike conventional methods that use uniform or predefined weighting coefficients, the notable aspect here is the use of different weighting coefficients for each individual instance. - It is clear that adjusting these weighting coefficients according to whether zero-shot or fine-tuned models are suitable for specific data can improve the final performance. Consequently, the primary challenge lies in devising an effective method to determine such instance-specific weighting coefficients. The authors propose a method that makes use of the Zero-Shot Failure (ZSF) set, consisting of training examples where the zero-shot model fails to predict accurately but the fine-tuned model succeeds. Experimental validation confirms the efficacy of the proposed approach. - I appreciate Section 5.3 for further presenting an empirical analysis of the proposed method. Essentially, the approach is a variation of OSE, determining how much the zero-shot model and the fine-tuned model are used on a per-instance basis, and VRF is not the only strategy to achieve this. It is gratifying to see that VRF outperforms other alternatives and that the analysis examines how different VRF designs produce varying outcomes.

Weaknesses

- Although there has been considerable research on robust fine-tuning for CLIP (as noted by the authors referencing various studies), the baselines analyzed in Tables 1 and 2 are quite limited. A key advantage of the VRF method is its applicability to any (zero-shot, fine-tuned) pair, both retrospectively and prospectively. Therefore, applying VRF to other robust fine-tuning methods beyond FT and LP-FT would further demonstrate the effectiveness of the proposed VRF approach. - The proposed method has a drawback of needing more than double the inference cost compared to WSE and other robust fine-tuning methods that yield a single fine-tuned model, since it requires performing a forward pass through the entire network twice (once for zero-shot and once for fine-tuned models) and calculating the ZSF-based weighting coefficient. However, an analysis of such inference costs has not yet been performed. It is important to note that Wortsman et al. (2022) favored WSE over OSE for combining zero-shot and fine-tuned models, highlighting this choice not only for its improved performance but also for its lack of additional costs. - In the context of robust fine-tuning, we can only observe performance on in-distribution data. Therefore, the parameters $a$ and $b$ in the proposed Eq. (6) for sample-wise mixing coefficients should be determined based on the in-distribution validation set, as mentioned by the authors in lines 141 and 198-200. From Appendix C.1, it is understood that $\\alpha \\in \\{1.4, 1.5, 1.6\\}$ and $b \\in \\{0.5, 0.6, ..., 1.0\\}$ were considered. However, it seems that the range of values for $a$ and $b$ is too narrow. Based on the in-distribution results in Table 8, larger values for $a$ should be explored, but the authors did not do so. What happens if the value of $a$ is increased further? Would it still outperform OSE and WSE baselines?

Questions

1. In combining zero-shot and fine-tuned models, WSE offers the advantage of requiring only half the inference cost compared to VRF by maintaining inference costs equivalent to that of a single model through weight averaging. When combining zero-shot and linear-probed models, since the image encoder part, which carries most of the inference cost, only needs to be processed once, it appears that VRF's concerns regarding cost could be mitigated. In this context, could the proposed VRF method also extend to "Linear classifier" models, alongside "E2E-FT" and "LP-FT"? It is worth noting that Wortsman et al. (2021) explores ensembling not only between zero-shot models and end-to-end fine-tuned models ("E2E-FT" in this work) but also between zero-shot models and linear classifier fine-tuned models (referred to as "Linear classifier" here). 2. In Figures 3, does "Ensembling with varying coefficient $\alpha$" refer to OSE? It would be helpful to also provide the curve for WSE. 3. Table 7 shows the results of exploring $\alpha$ values for OSE. It would be beneficial to provide the same results for WSE as well. 4. The proposed VRF approach also produces multiple points on trade-off plots (e.g., Figure 3(a.1)) depending on the values of $a$ and $b$ (especially $a$). It would be advantageous to visualize a scatter plot across a broad range of $a$ and $b$ values, allowing us to observe the trade-off curve similar to OSE and WSE, which interpolate between zero-shot and fine-tuned models.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors addressed the limitations and potential negative societal impact in Section 6.

Summary

This paper studies the trade-off between in-distribution (ID) and out-of-distribution (OOD) performance of pre-trained models before and after fine-tuning. The authors observed that the sample distance is inversely proportional to $\frac{Acc_{ft}}{Acc_{zs}}$. After modeling the residual error of the model, they found that for a training sample $x$, the optimal ensemble weight is proportional to the sample's accuracy. Therefore, the authors set different ensemble weights for different samples based on the distance between the samples and the erroneous samples in the training set. Experimental results demonstrate the effectiveness of the proposed method.

Strengths

1. This paper effectively improves the model's ID-OOD performance by setting different ensemble weights for different sample. 2. Experiments show that this method can be effectively applied to different fine-tuning techniques and can significantly enhance performance.

Weaknesses

1. VRF requires identifying and saving the zs classification error samples for subsequent use, which presents certain limitations. 2. The description of Step 2 is unclear: is it calculating the distance to the k-th nearest sample in $V$, or is it clustering the representations in $V$ first and then calculating the distance to the $k$-th cluster center? 3. Is the representation $ v $ of $ x $ calculated by ft or zs? 4. For each sample, recalculating and sorting the distance to $V$ incurs additional inference costs. How much extra computational cost does this introduce? 5. Since line 278 indicates that different values of $k$ have a minor impact on the model, why not use the nearest sample to calculate the distance? What is the reason behind choosing the $k$-th sample for distance calculation?

Questions

See Weaknesses.

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes

Summary

This paper aims to tackle the ID-OOD trade-off in the fine-tuning of pre-trained models with zero-shot abilities like CLIP. The proposed Variance Reduction Fine-tuning (VRF) is a sample-wise ensembling method concerning the zero-shot and fine-tuned models. The ensemble weights are determined by the distance from the test sample to the training samples that are incorrectly predicted by the zero-shot model. Theoretical analysis and experimental results justify the effectiveness of VRF.

Strengths

1. The proposed VRF can achieve better performance on both ID and OOD data compared with existing ensemble-based methods. 2. Unlike previous methods, the ensemble weights in VRF does not require tuning. 3. The experiments are thorough, and the results are clearly presented in the tables and figures.

Weaknesses

1. A major downside of VRF compared to previous ensemble methods is that the determination of the ensemble weights requires the storage of ZSF features at test time and computation of the distance from the test samples to each sample in ZSF set. This may bring a significant budget when the ZSF set is large. There should be a discussion on the theoretical and practical computational complexity regarding space and time. 2. The proposed distance calculation using the k-th nearest neighbor is borrowed from the previous work on OOD detection [24], and there is a lack of explanation for the specific choice of this metric. It seems that other metrics like the averaged k nearest neighbor distance discussed in [24] may also work in VRF. 3. It is claimed that the proposed method “can simultaneously attain the best ID and OOD accuracy without the trade-offs” (Line 9). However, this is not sufficiently justified. Specifically, it is unclear whether the “best ID and OOD accuracy” refers to the comparison with other existing methods or only considers the performance of the proposed method under different hyperparameter settings. Judging from Figure 3 (a.1), VRF does not always achieve the “best ID and OOD accuracy” compared with the peak ID/OOD accuracy of the existing ensembling method.

Questions

1. To what extent is VRF less efficient than existing ensembling methods? 2. Why is k-th nearest neighbor distance selected for the distance calculation step of VRF? 3. What is the meaning of solving the ID-OOD trade-offs?

Rating

5

Confidence

4

Soundness

2

Presentation

4

Contribution

2

Limitations

The authors have addressed some limitations and potential impacts of the work. I suggest additional discussions on the efficiency of the proposed method in the limitation part.

Summary

This paper addresses the issue of ID-OOD (In-Distribution vs. Out-of-Distribution) in the context of robust fine-tuning techniques commonly used in ensemble methods. The authors propose a sample-wise mixing approach to resolve this problem. The method involves creating a Zero-Shot Failure set, which contains samples that fail in zero-shot models but succeed in fine-tuned models, and mixing the predictions of the zero-shot and fine-tuned models based on the distance between test samples and the Zero-Shot Failure set. This approach demonstrates superior performance over traditional weight-ensemble and prediction ensemble methods on the ImageNet variants benchmark.

Strengths

- The paper is well-written, clearly explaining the proposed method and its motivation. The intuitive explanation, especially the use of Figure 2 to show the monotonic relationship between fine-tuned/zero-shot accuracy ratio and distance, strongly supports the proposed approach. - The paper makes a novel contribution by addressing the ID-OOD problem, which is not fully addressed by the existing ensemble-based robust fine-tuning. - The proposed method is simple and easy to implement, yet it achieves better performance compared to existing ensemble methods.

Weaknesses

- The main drawback of the proposed method is the reliance on multiple hyperparameters (a, b, p), making tuning more complex compared to traditional ensemble methods that typically involve only one hyperparameter. Finding optimal values for these hyperparameters is challenging as they can vary significantly across different datasets, potentially affecting performance. - To demonstrate the robustness of the proposed method, evaluations should be extended to more datasets beyond the ImageNet variants, similar to the experiments in the WiSE-FT paper, which include datasets like iWILDCam and FMoW. Differences in hyperparameter sweep ranges across datasets could highlight potential issues with the method. - There is a lack of performance comparison with other robust fine-tuning methods, such as FLYP[1] and MaskFill[2], which limits the assessment of the proposed method’s effectiveness. In the ImageNet classificaiton experiments, the proposed method is inferior to [1,2] in performance. - The paper need to report the hyperparameters found in experiments with different datasets and architectures to analyze their variability and sensitivity, providing deeper insights into the robustness of the proposed method. [1] Finetune like you pretrain: Improved finetuning of zero-shot vision models, Goyal et. al., CVPR 23 [2] Masked images are counterfactual samples for robust fine-tuning, Xiao et. al., CVPR 23

Questions

I am curious about the performance when finding hyperparameters for each test set as an oracle result.

Rating

5

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed some limitations of their work. However, they have not sufficiently discussed the critical issue of hyperparameter sensitivity and its impact on performance. There are no negative societal impacts.