OTTER: Effortless Label Distribution Adaptation of Zero-shot Models

Summary

This paper introduces a novel approach to address the issue of label distribution mismatch in zero-shot models, which is a common problem due to the imbalance in the pretraining datasets. The proposed method uses optimal transport to adjust the predictions of pretrained models based on an estimated downstream label distribution, without requiring additional training or access to labeled downstream data.

Strengths

1. The use of optimal transport to handle label distribution mismatch in zero-shot models is a novel and elegant solution. 2. The paper provides a solid theoretical foundation for the proposed method, including characterizations of the improvement under mild conditions and error bounds for misspecification. 3. Extensive empirical validation across a wide range of image and text classification tasks demonstrates the effectiveness of OTTER.

Weaknesses

1. dependence on estimated label distribution: the method relies on an accurate estimate of the downstream label distribution. In practice, obtaining a reliable estimate may be challenging, and errors in this estimate could impact the performance of OTTER. 2. evaluation on diverse datasets: while the paper includes a variety of datasets for validation, a more detailed analysis of performance across different types of datasets (e.g., varying in size, complexity, and imbalance levels) would provide a deeper understanding of the method's strengths and limitations. 3. comparison with recent methods: the paper compares OTTER with some existing baselines, but it could benefit from a more comprehensive comparison with the latest state-of-the-art methods in label distribution adaptation and zero-shot learning. There are many training-free or test-time adaption zero-shot baselines[a][b][c]. How does OTTER perform compared to these methods? 4. scalability: although described as lightweight, the paper does not thoroughly discuss the scalability of OTTER for very large-scale datasets. [a] Mirza, Muhammad Jehanzeb, et al. "Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections." NIPS 2024. [b] Zhao, Shuai, et al. "Test-time adaptation with clip reward for zero-shot generalization in vision-language models." arXiv preprint arXiv:2305.18010. [c] Abdul Samadh, Jameel, et al. "Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization." NIPS 2024.

Questions

The paper presents a compelling and well-validated approach to address label distribution mismatch in zero-shot models using optimal transport. Its innovative method, strong theoretical foundation, and significant empirical improvements make it a valuable contribution to the field. However, further exploration of its limitations, more comparisons with recent methods, and practical implementation details would enhance its impact and utility.

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors do not claim limitations.

Summary

Zero shot classification suffers from label distribution mismatch. This paper suggest to adjust pretrained model prediction via optimal transport. By showing the optimal transport prediction is equivalent to the bayes optimal classifier's output theoretically and good model performances on various experimental settings empirically, the paper shows the validity of its proposed method.

Strengths

- The problem is well formulated - Various experiment

Weaknesses

- Novelty is limited. This paper suggests applying OT for managing label distribution shift. The class imbalance for pretrained network prediction is already well known, and OT has been already applied for various distribution shift cases. Therefore, naively applying OT for the biased pretrained model prediction is imaginable. - The proposal saying that Bayes optimal classifier can be derived through optimal transport is inadequate and too strong since it requires a true cost matrix. If $P_t(Y|X)$ is accessible, we don't even need to use the optimal transport. - Please link theorems of the main paper and proofs in the appendices for readability. - I don't think figure 2 shows OTTER remains unaffected. The acc is deteriorated from 90 to 7X, meaning about 15%p decrease.

Questions

- Considering each cell of the cost matrix $C$, it is a loss (or the classifier score) calculated from the pretrained network. It means the cost matrix is biased to the source data distribution. In that sense, how the optimal transport can be considered as assigning true label? - How can the true label distribution of the target, $\nu*$, be accessible? - Can the author show more details for the equation of section 4.1? I cannot understand how to remove $P_s(X)$ and $P_t(X)$ - In table 1, what happened to Caltech256 Prior matching? Why the performance is far worse than zero-shot? - In figure 2, Why OTTER shows acc increase when the total variation distance between the source and the target increases from 0.4 to 0.8? With larger distribution shift, acc is supposed to deteriorate. If not, the synthetic experiment may be poorly modeled. - How long does it take to apply OTTER? I suppose it will take quadratically proportional to the dataset size.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

1

Limitations

.

Summary

Zero-shot models such as CLIP suffer from imbalanced predictions inherited from the uncurated pre-training data. This paper proposes adjusting the predictions of zero-shot models via optimal transport. This method requires only the downstream data and the label prior of the target distribution. Under the assumption that the conditional likelihood is unchanged between the pre-training data and the downstream data, the authors prove that the induced predictions are Bayes Optimal. The authors validate the effectiveness of the proposed method on several benchmark tests.

Strengths

The paper is well-organized and the presentation is clear. The idea of using optimal transport to adjust zero-shot predictions on downstream data is novel to me. Besides, the authors prove that the resulting predictions are Bayes optimal and justify the cost matrix using posteriors of zero-shot models. I have coarsely checked the proofs and find them convincing.

Weaknesses

Major: (1) Related works are missing. Some papers have addresses the issue of removing the label bias in zero-shot models. For instance, [1][2] use a subset of the pre-training data to mitigate the bias. [3] focuses on only using the labeled downstream data to address imbalanced predictions and proposes two methods. The authors should consider including these prior works. [1] A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models. [2] The neglected tails of vision-language models. [3] Generalized Logit Adjustment: Improved Fine-tuning by Mitigating Label Bias in Foundation Models. (2) The proposed method is transductive and requires the entire test dataset to perform optimal transport. I am wondering if OTTER can estimate the label distribution of pre-training data $\pi$ from unlabeled downstream training data and then the debiasing procedure can be performed by subtracting the logarithm $\ln \pi$ from the zero-shot predictions, as described in [3] (https://arxiv.org/pdf/2310.08106), without needing to operate on the entire test dataset Minor: (1) Duplicate citations: [29] and [30]. (2) Line 120: should reference Algorithm 1.

Questions

Please refer to the weaknesses, and I am willing to raise the score if the major weaknesses are addressed. Additionally, I have a question to discuss with the authors. Both [3] and OTTER use downstream data to generate Bayes optimal predictions based on the assumption that $P(x|y)$ remains unchanged between the pre-training distribution and the downstream distribution. In practice, this assumption might not hold. For instance, the pre-training data might mostly consist of real photos, while some test benchmarks might be sketches, e.g., ImageNet-Sketch. Under this situation, is your method theoretically sound in mitigating the pre-training bias?

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations have been discussed and no negative societal impact has been identified.