On the Importance of Feature Separability in Predicting Out-Of-Distribution Error

Summary

This paper proposes an easy-to-use dataset-level method for predicting OOD scores. The authors analyze two desiderata in representation learning: high inter-class dispersion and high intra-class compactness. Through some experiments on CIFAR and TinyImageNet, the authors reveal that the inter-class dispersion is strongly correlated with the OOD performance while intra-class compactness does not really correlate with the OOD accuracy.

Strengths

1. The proposed method is well-motivated and explained. The authors first claim that MMD and Fr\'echet distance is not good surrogates for OOD error prediction and suggest using dispersion score instead. Figure 3 indicates that the dispersion score is indeed better than conventionally used distance. 2. The proposed method is easy-of-use and training-free. It is also flexible to different OOD data in sample size and class distributions. 3. The method outperforms previous approaches in most benchmarks.

Weaknesses

1. The experiments are all conducted in elementary and simple experiments settings, i.e., the OOD dataset is set to be some augmentation and corruptions applied on the ID set. This is not really the real-world OOD benchmark setting as the corrupted OOD dataset still has some class-overlapping information with the ID set. Can it apply to the standard OOD benchmark in CIFAR and ImageNet? For example, the authors could use ImageNet-1k as ID and iNaturalist, Places, Textures, and SUN as OOD. With the current experimental setting with simple data augmentation, it is really hard to judge whether the method can be useful for real-world usage. 2. It would be much more interesting to have non-overlapping ID and OOD dataset in experiments. For example, the authors can train model on CIFAR100 and use CIFAR10C as OOD or vice verse (CIFAR10 as ID and CIFAR100 as OOD). It also meets more real-world setting as CIFAR10/CIFAR100 is a commonly used OOD benchmark. 3. Though the intra-class compactness alone is not useful. Would it be better if the intra-class compactness is combined with the dispersion? Would it be an interesting ablation study?

Questions

Please see weaknesses and limitations.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

My main concern is still the simple experiment setting. In my opinion, it would make the proposed method useful only if the OOD benchmark is replaced with real-world ones. I suggest the authors verify the proposed approach in standard CIFAR and ImageNet-1k benchmarks as done in [1,2,3]. Of course, it is not necessary to plot the curve of accuracy versus distance as done in Figure 1 and Figure 3 (a single correlation value would be sufficient), but it would be important to show the method of predicting OOD error is useful in practice especially given that the method does not rely on the class distribution. [1] React: Out-of-distribution detection with rectified activations. NeurIPS21 [2] On the importance of gradients for detecting distributional shifts in the wild, NeurIPS21 [3] RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection. NeurIPS22

Summary

In this paper, the authors focus on the task of predicting model performance on unseen/shifted datasets, without support of annotations. To do so, they first show the connection between feature separability and test accuracy, with an intuitive example and theoretical explanation. Based on the analysis, they propose a novel metric, Dispersion score, which measures the inter-class divergence from feature representations. They also reveal that intra-class compactness does not reflect the generalization performance, while inter-class dispersion works as a good indicator. They conduct experiments on CIFAR-C and TinyImageNet-C datasets to show the efficiency and effectiveness of the proposed metric. Furthermore, they also show the advantages of their method in some extreme cases (limited data, partial label set, class imbalance).

Strengths

1.The motivation of this work is reasonable. Intuitively, the prediction performance should be tightly tied with the quality of the learned feature. High inter-class dispersion is one of the goals of self-supervised learning for learning a good representation. The authors also provide an interesting explanation from a theoretical perspective, which is one of the highlights of this work. 2.The analysis of those methods with distribution distance is thought-provoking. In my view, the shifted distance might not be necessarily connected to the test performance, as the shifted features might not be important for the final prediction. For example, an image classifier with sufficient generalization ability would not change its predictions when the background is changed a lot. 3.The proposed method is novel and interesting. To the best of my knowledge, this is the first work to exploit the feature properties of test instances for predicting OOD error. It does not require access to the training data and only uses a forward propagation, which is much faster than existing SOTA methods. 4.The empirical results are extensive and convincing. The authors not only show the simple method outperforms existing training-free methods, but also compare to ProjNorm on the computational efficiency (575 second v.s. 11 second). The most exciting part for me is the analysis on the flexibility in OOD data, which considers some real-world settings, like class imbalance and limited data. The authors also present that a high intra-class compactness is not necessary for good prediction performance, which may provide a new insight for representation learning.

Weaknesses

1.The presentation of some Figures can be improved. For example, in Figure 1b, the magnified box seems not match the original area. The four images of Figure 2 are a little small, which might be improved by reducing the blanks between images. 2.Why intra-class compactness does not work is not clear. Although the authors show that intra-class compactness is not an effective indicator empirically, it could be better if the authors can provide an intuitive explanation.

Questions

see weaknesses.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The authors discussed the limitations of the proposed score in the adversarial setting, where the feature quality is also broken by adversarial attacks. The authors also provide analysis to show the underlying reason, which is also an interesting contribution.

Summary

This research focuses on predicting test accuracy on shifted datasets without access to ground-truth labels. The authors begin by analyzing the potential issue of the existing methods which were based on the shift distance and point out that the previous distributional distances were not always correlated highly to the out-of-distribution (OOD) error. Then, they proceed with an intuitive example and theoretical explanation, showing the connection between feature separability and test accuracy. Based on this, they propose a novel metric that measures the inter-class dispersion, which is demonstrated as an effective factor for the OOD error estimation. They conducted experiments on CIFAR-C and TinyImageNet-C to validate the advantages of the proposed method. The authors further demonstrate the robustness of the proposed method against class imbalance and data shortage.

Strengths

1. The task studied in this paper is practically important. In some real-world applications, it is necessary to assess the model performance on a given unlabelled dataset. Under those scenarios, OOD error estimation becomes inevitable and valuable. 2. The proposed method is supported by both empirical observations and theoretical analysis. The motivation is clear. 3. The proposed method is novel, effective, and efficient. Previous SOTA methods like ProjNorm need to update the model which is computationally expensive (It can be unachievable with some large models). They also show that intra-class compactness cannot work well, which motivates deeper exploitation of the properties within the features distribution rather than the coarse-grained property on the whole dataset adopted in AutoEval. 4. It is also interesting to see that this paper further investigated the performance under some scenarios with imperfect data, e.g., class imbalance, smaller sample size, and partial OOD error prediction. To the best of my knowledge, this is the first work providing such a complete analysis. Compared with the previous studies, the proposed approach achieved comparable performance even under these extreme cases. 5. This paper is well-written and easy-to-understand. The analysis and figures provided by the authors are clear and informative. I believe readers can easily get the core idea and implement it.

Weaknesses

1. Some potential typos should be corrected in the next version. See the Questions part for more details. 2. The discussion about the pseudo labels used for cluster centroid determination can be further extended. See Questions for more details.

Questions

- I noticed that when computing the dispersion score, the pseudo labels were used to determine the cluster centroid. In Appendix E, the author compared the pseudo-labeling with another clustering method, K-means. However, I am interested in if it can be further improved by improving the quality of these pseudo labels. For example, filtering out some low-confidence predictions, adopting soft pseudo labels instead of hard ones, or considering wrong pseudo labels as noisy annotations? - I also noticed that the previous work, ProjNorm, also adopted pseudo labels for the OOD error prediction. Could you please summarize the differences in how to adopt the pseudo labels between the two methods? - About the evaluation times of Dispersion score on TinyImageNet (Table 2). The time of ResNet50 should be longer than that of ResNet18. Is there a typo that records the times in the wrong order? - From the analysis in 4.4, does it mean that we can focus more on the inter-class dispersion instead of intra-class compactness in representation learning? - Some potential typos: - Line 221, Spearson -> Pearson? - Line 240, Table 6 -> Table 1?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

I did not see any severe limitations in this paper.

Summary

This work aims to predict classifier accuracy on unlabeled test samples. To achieve this goal, this paper proposes a feature separability-based dataset-level score to check whether features have high inter-class scatter. This feature separability score is calculated by measuring how far the centroids of features that share the same pseudo-label predicted by the model deviate from the center of all features on average. The experiments show that such inter-class dispersion is strongly correlated with model accuracy.

Strengths

+ [***Good clarity***] This work is well-written and easy to follow. The method is well presented, the visualizations are helpful, and the experimental settings are clearly introduced. + [***Measuring feature separability is well-motived***] Under distribution shifts, the features of the source and target can be scattered differently. Using such information to reflect model accuracy seems reasonable.

Weaknesses

- [***More results to illustrate the relationship between distribution gap and accuracy***] In the accuracy prediction, there are two metrics are proposed to measure the distribution gap. Please show their results in Figure 1 to well illustrate the motivation regarding the potential limitation of the distribution gap for accuracy estimation. - [***The definition of dispersion score is not sound***] The class cluster is based on the classifier's prediction. What if the classifier gives bias predictions on test sets? For example, in adversarial examples, the classifier maintains class-class separability but totally misclassified data. Moreover, why using the gound-truth label to define class clusters does not give stronger correlation strength (Section D). Moreover, whether the proposed method can handle the cases where some classes do not appear or some unseen classes appear. For example, ProjNorm discusses the label shift where some classes are missing. Under such a scenario, ProjNorm is less effective than other methods. - [***The experimental setting is somewhat limited***] This work only provides the experiments on small-scale datasets (e.g., CIFAR-10/100 and TinyImageNet). Considering the literature, the results on iWILDS and ImageNet should be included. For example, DoC and ATC report the results on ImageNet datasets with several natural distributions, such as ImageNet-V2, ImageNet-R and ObjectNet. Without the results on such realistic datasets, it is hard to conclude the robustness and effectiveness of the proposed method.

Questions

- Why using the gound-truth label to define class clusters does not give stronger correlation strength (Section D). - Under an adversarial attack, the features might still have high dispersion but the classifier archives low accuracy. Please comment on this and discuss the potential solution to alleviate this. - Class imbalance results (Table 3) are not sufficient. How about other existing methods (e.g., DoC and ATC) under such class imbalance?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

This work reports results for some special cases, such as class imbalance and adversarial attacks. Another potential limitation is the open set problem, where some unseen classes arise. Also, some classes may be missing during testing. It would be better to mention and discuss both cases, as the proposed scatter score can be significantly affected. ***[Post-rebuttal]*** > I would recommend excluding the open-set results from the main paper due to the evaluation metric's limitation in assessing only the seen classes. Moreover, ATC reports the results on some real-world and large-scale datasets like iWILDS and ImageNet, including such datasets would make the submission solid. This addition would enhance the robustness of your research and highlight its practical applicability. Given that you've already showcased results on domain adaptation datasets such as PACS, Office-31, and Office-Home, I am inclined to think that the current evaluation is sufficient. > [Additional suggestions which do not impact the rating of this paper] You might consider referring to two relevant works: "Unsupervised Accuracy Estimation of Deep Visual Models using Domain-Adaptive Adversarial Perturbation without Source Samples," which also features results on domain adaptation datasets, and "Characterizing Out-of-Distribution Error via Optimal Transport." This latter work, which assumes a consistent marginal label distribution between training and test sets, stands in contrast to your method and may help highlight the merits of your approach.

To response

Thank you for the detailed response. My concerns have been addressed. I also read the other reviews and your responses. You have done a good job and I believe those discussion can further enhance this work.

Follow-Up Discussion

Dear Authors, Thanks for the response! - I am interested in the provided "The performance of Dispersion Score when the test set includes unseen classes". With new classes, how to evaluate classifiers? Viewing the unseen classes as one outliner class or just evaluating on original seen classes? - "Why using the true labels cannot outperform our method with pesudo labels?" This part is still not clear to me, especially "Dispersion score with ground-truth labels only measures the quality of the learned features, regardless of the last linear layer". Best, Reviewer 1A8L

Response to Author Rebuttal

Thanks for the response from the authors. My concerns have been addressed. In addition, I attached some further comments w.r.t. the response from the authors as follows: Firstly, it is great to see that the authors provided additional experiments by adopting different confidence levels w.r.t. the pseudo label. The results indicated that filtering out samples with a confidence threshold is even detrimental to the final performance. According to my understanding, the model obtained on the training set can be poorly calibrated on the OOD dataset (usually we do not apply a calibration process in OOD error prediction), and discarding low-confident samples with a threshold can be considered as introducing more over-confidence in disguise during calculating the dispersion score. Is this a reasonable explanation? Secondly, it is also interesting to see the results on the open-set OOD prediction setting (Table. 2 of pdf attachment). Although the current OOD error prediction task does not consider the existence of label space change, it is still interesting to see that the proposed method outperforms the baselines under the constructed open-set setting.

Thanks for the response!

Thanks for the response! Most of my questions have been solved. I would like to increase the score by one level. I sincerely suggest the authors include a discussion about the difference with OOD detection and add some works of literature. This would help readers to better understand the work.