LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies

Summary

This paper focuses on addressing the challenge of assessing model generalization under out-of-distribution conditions. They reintroduce the Least Common Ancestor (LCA) distance. Particularly, they utilize the LCA to measure the taxonomic distance between labels and predictions, presenting it as a benchmark for model generalization. In the experiments, the proposed method is evaluated on multiple datasets.

Strengths

It is interesting to address the challenge of assessing model generalization under out-of-distribution conditions.

Weaknesses

1. The Introduction Section is not clear. The authors indicate that most methods involve modeling correlations with in-domain accuracy or agreement. And many studies evaluate generalization on OOD datasets that feature limited visual shifts. These interpretations are very unclear. I am not clear the concrete meaning. I recommend the authors modify their paper carefully. 2. The authors indicate that to address the analyzed issues, they introduce a method to benchmark model generalization, i.e., using the taxonomy loss. Firstly, the authors do not interpret whether the research is meaningful clearly. Secondly, the authors do not sufficiently introduce the advantages of the taxonomy loss. I recommend the authors draw a figure to clearly describe the motivation. 3. In Table 1, the evaluated methods are somewhat old. The authors should verify the effectiveness of the proposed method on more state-of-the-art methods, e.g., the works from CVPR 2023, ICLR 2023. Meanwhile, the experiments are somewhat unclear. The authors only evaluate the classification performance. I recommend the authors evaluate the proposed method on other tasks, e.g., object detection and semantic segmentation. Finally, for Fig. 1, the authors should give more interpretations.

Questions

See Weakness.

Rating

3: reject, not good enough

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Summary

In this paper, the Least Common Ancestor (LCA) distance via the WordNet hierarchy is employed to measure the taxonomic distance between labels and predictions, utilizing it as a benchmark for model generalization. Extensive experiments are performed on model evaluation, including vision-only and vision-language models on natural distribution shift datasets. A strong linear correlation is observed between in-domain ImageNet LCA scores and out-of-domain (OOD) Top1 performance across many variants of ImageNet.

Strengths

A thorough experimental analysis of the relationship between Out-of-Distribution (OOD) detection performance and Least Common Ancestor (LCA) distance for image classifiers is provided in this paper. The following are key strengths of the work: (1) Experiments are conducted on a diverse set of neural network architectures, including ResNet, VGG, EfficientNet, Vision Transformer (ViT), and Vision-Language Models (VLMs), enabling conclusions that potentially generalize across problem domains. The rigor and comprehensiveness of the methodology are underscored by the scale of experiments, involving up to 75 network variants. (2) A major point about this submission is the analysis that demonstrates a correlation between OOD detection performance and LCA distances in the classifier. For instance, it is quantitatively shown how images from OOD datasets with more distant LCA relationships to the training data tend to be easier to detect as anomalies. The intuitive justification provided is that greater separation between the semantics of the origin dataset and OOD dataset in the hierarchical LCA structure leads to more separable distributions.

Weaknesses

Major: (1) The paper does not evaluate multiple OOD scoring methods like energy scores[1], ODIN[2], Mahalanobis distance[3], and ReAct[4], which would have provided insights into the validity of the key conclusions across different anomaly scoring approaches. The interaction between the choice of scoring method and LCA distance remains unclear. Understanding how these scoring methods affect OOD performance from the perspective of the LCA distance is important. It should be noted that the calculation of this LCA distance is limited to variants of ImageNet considering the WordNet hierarchy used. (2) The claim is made that the findings offer "invaluable insights and actionable techniques" to enhance robustness and generalization. However, no concrete solutions leveraging the LCA distance are proposed or analyzed. Further theoretical or empirical analysis that establishes a connection between these insights and improved generalization would be beneficial. Minor: “Given two classes, y (the ground truth class) and y′, we define the LCA distance according to (Bertinetto et al., 2020) as lcad(y′, y) := f(y) − f(lca(y, y′), where f(y) ≥ f(lca(y, y′) and lca((y′, y) denotes…” All formulations lose the right brackets. “As highlighted in Fig 1 (indicated in red), when adhering to ’accuracy on the line’,” Wrong quotes. In summary, the rigorous experiments and novel analysis of OOD detection vs. LCA distance are valuable contributions, but additional evaluation of alternative scoring methods and practical applications of the findings could further strengthen the work. [1] Liu, Weitang, et al. "Energy-based out-of-distribution detection." Advances in neural information processing systems 33 (2020): 21464-21475. [2] Liang, Shiyu, Yixuan Li, and Rayadurgam Srikant. "Enhancing the reliability of out-of-distribution image detection in neural networks." ICLR 2018. [3] Ren, Jie, et al. "A simple fix to mahalanobis distance for improving near-ood detection." arXiv preprint arXiv:2106.09022 (2021). [4] Sun, Yiyou, Chuan Guo, and Yixuan Li. "React: Out-of-distribution detection with rectified activations." Advances in Neural Information Processing Systems 34 (2021): 144-157.

Questions

Please see the weaknesses above.

Rating

5: marginally below the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Summary

This paper focuses on evaluating the Out-of-Distribution (OOD) generalization by using Least Common Ancestor (LCA) distance based on the WordNet hierarchy. LCA measures the taxonomic distance between labels and predictions, and the paper shows that LCA is a better measure of OOD generalization than top-1 accuracy (when both LCA and top-1 accuracy are computed on in-domain data). Intuitively, LCA is able to better evaluate how well a model has learned semantic knowledge of the classes (since lower LCA indicates that the model’s wrong predictions are semantically closer to the true class). This enables LCA to be a better measure of OOD performance compared to top-1 accuracy which only considers whether the prediction is correct or not. Specifically, they test 75 different models (including both vision models and vision-language models) and find a linear correlation between ImageNet LCA and OOD accuracy across 4 standard ImageNet-OOD datasets. They also use the pairwise LCA between classes as soft labels for linear probing over pretrained models, and find that OOD performance can be improved at the cost of in-domain performance.

Strengths

* The idea of using LCA for evaluating OOD generalization is simple, intuitive, well-motivated, and effective. * The paper is fairly well-written (except some typos which can be fixed). * The experimental analyses are quite extensive. The contributions of this work will likely be quite significant for industry applications where predicting OOD performance ahead of deployment tends to be important.

Weaknesses

* Explanation of LCA is complicated * A visual illustration of LCA computation with a small part of the WordNet hierarchy and 2-3 example pairs of classes would help readers to quickly and better understand the LCA distance measure. * Maybe a figure in the main paper like Fig. 3 (in Suppl.) but with some actual classes and example LCA values for a few pairs of classes. * Fig. 1 is very difficult to read and understand * The legend is too small. It would be better to show only top-1 here (first row) with larger font sizes (match caption size, roughly) and show the full figure in supplementary. * Implementation of inferred class taxonomy is difficult to understand * It is unclear what "establishing the cluster level where both classes share the same cluster as the height of LCA" means. Please clarify and try to simplify it. * A figure illustrating the method would be ideal to help readers understand it better.

Questions

* Please see the weaknesses section. * Minor comments * Abstract (second paragraph) has a typo: “Beside” → “Besides”. * Introduction (second paragraph): “effective robustness(Taori et al., 2020)” space needed between robustness and the citation. * Above the contributions list, typo: “in measure model’s semantic awareness” → “in measuring a model’s semantic awareness”. * Sec. 2 (second paragraph): extra or less brackets in three equations in this paragraph. * In many places in the paper, quotes are used incorrectly in LaTeX. Please use ` ' in LaTeX (i.e. backtick and quote instead of both quotes). * Paragraph below Table 3 has a typo: “As illustrated in Table3” → “As illustrated in Table 3”, i.e. add space. * Sec. 3.3 (last paragraph) has a typo: “natural image(ImageNet)” → “natural image (ImageNet)”, i.e. add space.

Rating

8: accept, good paper

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

2 fair

Contribution

4 excellent

Looking forward to hear from reviewer!

We highly appreciate the feedback from reviewer on our work, and we have go over each point carefully to provide a reply. Please let us know if the reply address the concern. We would love to incorporate your suggestions into our refined version of the work! **Below please find attach a comprehensive summary of our work. Hope this is useful for AC and reviewer!** In this study, we focus on evaluating how well models generalize to unseen, out-of-distribution (OOD) datasets. Specifically, we aim to predict a model's OOD performance, based on its performance in a familiar, in-domain setting. Previous research in this area has primarily focused on measuring model performance on OOD datasets with limited visual shifts. Or have not adequately compared vision models trained on datasets like ImageNet, with vision-language models trained on datasets like LAION. To address these gaps, we introduce 'LCA-on-the-line.' This is the first approach to uniformly measure model robustness across different model modalities and OOD datasets with significant distribution shifts. Our approach is grounded in the understanding of how models learn. During training, models learn to classify by creating decision boundaries and extracting distinguishing features between classes. A model's ability to generalize to OOD datasets depends on the transferability of these learned features during training. This brings us to the concept of spurious correlations. For example, a model might mistakenly associate grass with ostriches if they frequently appear together in training, and expecting the same association during testing. We hypothesize that only semantic features that align with human understanding of object definitions are universally transferable to any OOD dataset. For example, a model that learns to associate ostriches with features like 'long legs' and 'long neck', which are more transferable to OOD datasets, will likely predict classes semantically closer to the ground truth, such as flamingos. In contrast, a model influenced by spurious correlations by falsely associate ostrich with grass, might predict a semantically distant class, like an Jaguars, which are also appear often on grass. Our method involves measuring a model's generalization based on its in-domain semantic severity of mistake. We use the 'Least Common Ancestor' (LCA) distance, which is the distance between the model's prediction and the ground truth class in a predefined taxonomy hierarchy, like WordNet. We tested our hypothesis using 75 models with varying settings, including vision models trained on ImageNet and vision-language models trained on LAION. We used ImageNet as the in-domain dataset and four OOD datasets with significant distribution shifts. Our results show that the in-domain LCA distance on the ImageNet test set correlates robustly and linearly with accuracy across all OOD test sets. This finding challenges previous studies using in-domain accuracy that failed to establish a unified measurement for models trained on different datasets. For dataset without a predefined hierarchy, we construct a latent hierarchy by apply K-mean to group classes into different node at each level of the tree, achieved robust performance similar to WordNet hierarchy. Additionally, our work explores how aligning model predictions with class hierarchies during training can improve generalization. Lastly, our research challenges the long-held belief that LCA distance offers no additional insights beyond top-1 accuracy since ImageNet dataset. We provide a compelling alternative to understand why vision-language models with lower in-domain accuracy might generalize better to OOD datasets than vision models.

Significant update to submission by updating 7 plots.

Dear reviewer, please refer to our updated submission for reference. We have updated 7 plots as suggested in your valuable review to improve readability. We have also address other minor issue in our update. Please take a look and see if the updated plot address your concern. We are looking forward to hear from you! Reviewer 5fCB: - (Figure 2) We have include a plot visualization LCA score with actual class in our main paper. - (Figure 7) We have also include a plot illustrating K-mean clustering. Reviewer r8af: - (Figure 1) We have updated an teaser figure in introduction to address the problem we are targeting. Reviewer CuPJ: - (Figure 8) We have updated a plot in appendix illustrate our setting compare to prior work. - (Figure 5,6) We have also include 2 plots in section 3.3 to explain our motivation and significant of the work. - (Figure 4) We have also include one plot beside Fig. 1 to give more interpretation.