Summary
Based on the observation that CLIP undercalibration will affect the existing prompt-tuning-based method's OOD regularization, i.e. samples with uncertain True-Class Probability (referred to as ID uncertainty in this paper) may provide false OOD features and harm to negative training used in the existing methods; therefore, the author proposes a simple training strategy Self-Calibrated Tuning (SCT) weighted with the ID uncertainty, which can help improve FPR95 through experimental verification.
Weaknesses
My main concerns about this work are that the work is relatively incremental and empirical; there is insufficient discussion on the pros and cons of field-related methods (including other paradigms); the experiments are not sufficient, rigorous, and analyzed; and the method's effect on improving common benchmarks is rather one-sided, etc. The details are as follows:
[Method]
- (Inaccurate motivation verifications) If I understand correctly, the misclassified ratio on the horizontal axis in the right panel of Fig. 2 refers to $p(pred\neq gt), pred=\arg\max_yp(y|x)$ while your ID uncertainty refers to sth like True-Class Probability, marginal softmax $p(y=gt|x)$. These are not two identical things, though there is a certain correlation: only within some ranges, TCP could indicate accuracy [1]. Therefore, I think the author may have a biased understanding here, and the experimental results cannot fully reflect the motivation of the work: when "uncertain" ID samples are used as OOD regularization, some ID data are misdetected as OOD (FPR). If I have misunderstood, could the author clarify? Or maybe additional correct experiments are needed?
- (No calibration verifications) The claim and results in the work show that SCT helps with CLIP calibration, but there is no visualization of the calibration after training to illustrate the point (e.g. could add the before and after calibration comparison like in Fig. 2).
- (Lack of discussion of weighted training; modulating function rationale) The idea of weighted loss is very direct and easy to think of. Previous work should also be mentioned and discussed. For example, [2] is based on the feature representation paradigm and uses activation scale as the ID-ness (similar to ID uncertainty in the context) indicator for weighted training to improve OOD detection. In comparison, I do not quite understand why $\phi$ must be monotonically decreasing w.r.t. $p$ (e.g. $1-p$) and not monotonically increasing (e.g. $p$), because the weighting method in [2] is monotonically increasing, and the result is also improved. Could the authors elaborate on this?
- (How about post-hoc CLIP calibration?) Usually, calibration is divided into two types: training and post-hoc [3] (calibration related works are lacked in the paper). The former is used in this paper. The latter may be explored in OOD feature extraction methods, e.g. changing the rank operation (Eq. (6) & Fig. 3(d)). The author may lack discussion in this aspect.
[Experiments]
- (No much AUROC improvement) I understand that the method in this work is mainly to improve FPR95, but unilaterally only improving FPR95 does not seem to be comprehensive enough, because AUROC is also an equally important indicator and methods need to be proposed to improve it.
- (Lack of CIFAR results) Although the comparison method LoCoOp has not been experimented on CIFAR, CIFAR is indeed another important benchmarks in the field of OOD detection, and I think it is necessary to supplement it.
- (Discussions with simpler yet more effective pre-trained features + post-hoc?) I just would like to know what the author thinks about the (potential) advantages of the prompt-tuning-based method studied in the paper compared to the post-hoc method. After all, post-hoc does not require additional training and uses the basic ResNet backbone; the FPR95 and AUROC on the main task of Tab. 1 on ImageNet-1k have reached **20.05** and **95.71** respectively, which are much better than the results reported in the paper (26.47, 93.37).
- Could the authors clarify on what validation set are the hyperparameters tuned?
- (Interpretations of the ablations.) Figure 3(b) shows that the results of selecting other regularization functions are very different, and the paper (L294-298) does not provide any analysis. I am curious about how the author could try to interprete these ablation study experiment results. Similarly, the quality of OOD features extracted by different extraction methods also varies greatly, which seems very empirical (Fig. 3(d)).
- Table 1 is suggested to include results of combining more newer post-hoc methods (e.g. ASH (Djurisic et al., 2022), Scale [2]) and fine-tuned methods, which will give readers a more comprehensive sense.
[Presentation]
- The paragraph introducing the OOD features (L189) should be moved forward, or at least before refer to Fig. 1, which will give readers a clearer background.
- Why is the left panel in Fig. 2 not arranged in ascending order of softmax output? The arrangement of 0.02, 0.89, 0.04, and 0.67 affects reading. What is it trying to say? It would be better to display the classes and images together for clarity.
References:
[1] Corbière, Charles, et al. "Addressing failure prediction by learning model confidence." NeurIPS, 2019.
[2] Xu, Kai, et al. "Scaling for Training-Time and Post-hoc Out-of-distribution Detection Enhancement." ICLR, 2024.
[3] Guo, Chuan, et al. "On calibration of modern neural networks." ICML, 2017.