ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets

Summary

This work performs an empirical study on the correlation between ID and OOD performances of models in vision and NLP. While some previous work claimed that there is usually a positive correlation between ID and OOD, this work argues that there are real datasets on which ID and OOD are negatively correlated, so ID should not be used as a proxy for OOD. The study was originally mainly done on Camelyon17, and this work also includes other datasets in Section 5, as well as similar observations in other papers. Section 6 provides a toy example where there is a trade-off between ID and OOD. Section 7 shows that the positive correlation occurs mostly in cases where the shift is mild, and with a larger shift the correlation could turn into a negative one. Section 8 points out several fallacies in previous work, including suggesting using the ID performance as a single metric, incorrect evaluation methodologies, etc.

Strengths

- As someone very familiar with this area, I am well aware of the "accuracy-on-the-line" folklore that ID and OOD are positively correlated, and it has been reported in a lot of previous work. So to me, this submission is a breath of fresh air. A lot of work on OOD draws conclusions based on pure empirical observations. This is understandable because the OOD problem is an ill-posed problem by nature. However, the caveat is that lots of follow-up work takes these conclusions for granted, proposes new methods based on them, and claims to achieve "state of the art" while their evaluation is usually flawed. Such an unscientific methodology has haunted the OOD community for quite a long time, and I believe that this work conveys a strong message that this needs to be changed. - This work is very clean and well written. The main argument of this work is by no means novel. In fact, in Section 5.1 the authors list previous papers that made similar arguments. Nevertheless, to my knowledge, this work is the first to convey this important message in such a clean way, and I believe that it can influence the perception and methodology of many researchers in the community.

Weaknesses

- Section 6 is not very useful, and in my opinion, could be replaced with more experiments. It is very easy to construct a toy example where ID and OOD are negatively correlated. For example, if ID is a binary classification task, and OOD simply flips the labels, then of course they are negatively correlated. The reason why this "accuracy-on-the-line" has been there for so long is not because it is theoretically correct, but because it has been observed in many reports by different people. Everyone sufficiently familiar with OOD knows that there is no theoretical correlation between ID and OOD. I think this paper should be posed as a large-scale empirical study that shows that the correlation between ID and OOD on real datasets can be anything, so Section 6 is unnecessary. - Section 5 is too short and the experimental results are not well presented. I suggest the authors remove Section 6 and put more experimental results in Section 5.

Questions

- In Section 7, it is demonstrated that when the distribution shift is mild, the correlation between ID and OOD is usually positive. Are there any real datasets where the shift is mild but there is still a negative correlation between ID and OOD? In many real applications, it is safe to assume that the shift from train to test is mild. And if under this assumption ID and OOD is usually positively correlated, then one could argue that ID could still be used as a proxy for these applications. Note that I am not asking whether it is *possible* for there to be a negative correlation - It is easy to construct such a toy example. I am asking whether the correlation is always observed to be positive on real datasets with mild shift. **Post rebuttal note:** I have read the rebuttal and had a discussion with the authors. I prefer to keep my score and vote for accepting this submission.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are discussed in Appendix A.

Summary

The manuscript observe a phenomenon that ID and OOD performance can be inversely correlated because spurious correlations under the partially informative invariant feature (PIIF) assumption. The authors analyze the reasons of this missing phenomenon in previous studies, and conduct dataset-wise and shift-wise experiments. The paper provide a theoretical analysis and an OOD experiment guidance.

Strengths

- The introduction of the ID/OOD inverse correlation is reasonable and easy-to-follow. - The experiments are comprehensive. - Theoretical analysis is equipped to provide insights. - (Update) The empirical results coherently reflect previous theoretical findings, which is worth spreading in the OOD community.

Weaknesses

~~The most obvious weakness of this paper is that the analyses are mostly experiment-centric, which leads to farraginous explanations and insights. This organization makes readers hard to follow the logics and insights of this paper. The authors may strive to highlight the inverse correlation is "surprising" instead of trying to explain this phenomenon more natural and intuitive.~~ The overall idea can be clearer and more insightful if the authors use causality as the backbone. For example, according to section 6, the observed phenomenon is **natural** with the structural causal model analysis under the partially informative invariant feature (PIIF) assumption. Other weaknesses: - Line 193: The structural causal equation definition is not formal. Use ":=" instead of "=". - Line 102: Link is not valid. ~~Although I lean to accept, I'm not sure if this version of paper is enough beneficial for the research community.~~ The authors should revise their paper according to their proposal in the discussion period.

Questions

I suggest extending section 6 and reorganizing the paper based on it.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

No limitations, broader impacts, and licenses included.

Summary

The paper presents an empirical examination of ID vs. OOD performance on several real-world datasets showing inverse correlation patterns can indeed arise in many datasets and experimental settings. This conflicts with past papers mainly observing a positive dependence structure. The paper further discusses why past studies missed such patterns and theoretically analyzes how inverse correlation patterns can arise.

Strengths

- The paper highlights and important OOD generalization misconception, namely that increased model performance on in-distribution data is enough to achieve good OOD generalization. As the paper points out, it should be self-evident that this should not hold in general. Still, many works suggest that improving ID generalization helps OOD generalization. Model diversity seems to be a key factor that previous studies failed to account for. This is ensured via a training dynamics argument by examining model performance during a training run. I find this treatment convincing. - The provided experimental evidence shows that inverse correlations are possible under starker distribution shifts and that previous results on positive correlations were mainly obtained by a restrictive experimental setting. It makes sense that an inverse trend should arise at some point under misspecification. - The paper points out specific cases where the positive ID-OOD generalization insight has been observed but also where a negative relationship has been observed. This paper still seems to add value as in contextualizes all of these occurrences while explaining why a positive dependence structure might have appeared. - I also like that the paper contains a theoretical section showing that an inverse correlation is expected under a spurious feature. At the same time, I am not sure it adds that much value as it mainly relies on a formalization of the effect of spurious correlations which have been previously theoretically analyzed in the domain generalization literature.

Weaknesses

- Although the paper does provide meaningful value in analyzing the ID-OOD tradeoff, the paper sometimes mentions this idea as a new insight. I am not sure this is necessarily true. I would make sure not to oversell the "discovery of inverse ID-OOD tradeoffs" but not lose focus on explaining how previous papers could have overlooked this possibility (which the paper already does for the most part but not everywhere). - I found point 4 in the description of Figure 2 confusing. The difference in the ID axis as compared with panel 2 does not appear to be large. 2 seems to show a more consistent variance while 4 seems to show a growing variance. Would it be appropriate to refer to theses as homoscedastic / heteroscedastic noise models, respectively? - Last point in 3.1 (lines 182-183) is missing references; it would be good to back up this claim. ### Post-rebuttal I have read the author's rebuttal and found that my concerns were adequately addressed. I maintain my positive score.

Questions

- Why was the diversification method not used in the second set of experiments?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- Although limitations of the approach are not explicitly discussed, current shortcomings and future work are discussed in section 8 and 9.

Summary

Through extensive experiments, the authors showed that ID and OOD performances can inversely be correlated. The key ingredient is to track the performances throughout the training dynamics, instead of looking only at the end point.

Strengths

By tracking the training process, instead of the final point, the authors gave a more comprehensive view on the matter. This methodology should be more widely adopted, in my opinion.

Weaknesses

While Theorem 1 demonstrates this diverse performance phenomenon can happen even to simple linear models, how NN does so was not analyzed. I'd like to see the authors analyze a shallow neural net. There could be broader and deeper theoretical insights.

Questions

Has the authors conducted an experiment for two or more OOD data performances, for a single set of models trained on an ID data, where the OOD data can arguably be ordered in a way that becomes gradually more OOD to the ID training data? For example, one can train on CIFAR 10, and track its OOD performances on some selected OOD classes from CIFAR 100, and do the same on SVHN as a further away OOD dataset. I'm interested in the interplay between training dynamics and the more OOD axis.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper examines the previously proposed idea that in-distribution (ID) and out-of-distribution (OOD) performance of models are positively correlated. Through various experiments, the authors demonstrate that ID and OOD performance can in fact be inversely correlated. They also discuss how the previous work have missed this observation due to selection bias, and criticize the conclusions made by them. Moreover, a theoretical analysis on the linear case supplements the empirical findings regarding inverse correlation of ID and OOD performance.

Strengths

Disclaimer: The paper is merely experimental as it is evident from its title. Before reading this report, I have not seen such activity in the OOD generalization literature; Note that benchmarks such as DomainBed [1] are fundamentally different as they do not provide suggestions for OOD generalization based on mere experiments. To understand OOD generalization, I believe the theoretical analysis must be the foundation, and ideally the results need to be backed with experiments. However, similar work have been approved in the literature; for example Wenzel et al. [2] which has motivated the empirical study of ID and OOD performance was published in NeurIPS 2022 despite making the false claims that are elaborated in this work. Therefore, I see the relevance and necessity of a rebuttal such as this work to correct the mistakes made by the previous work, and complement the existing literature. Only for this reason, I value the efforts made by the authors, and I recommend accepting the paper. - The paper addresses important shortcomings of the previous work in methods and evaluation. - The paper is well-written, and easy to understand for the audience who might be unfamiliar with the literature. The Reviewers’ FAQ in the appendix is very helpful. - The empirical evaluation is comprehensive. [1] I. Gulrajani and D. Lopez-Paz, “In search of lost domain generalization,” 2020, arXiv:2007.01434. [2] Florian Wenzel, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-of-distribution generalization in transfer learning. arXiv preprint arXiv:2207.09239, 2022.

Weaknesses

- The premise of section 6 is to explain the inverse correlation witnessed by the experiments on the real-world datasets. To this end, the authors chose a linear model, and proved (Theorem 1) that adding a spurious feature decreases the OOD performance while it boosts the ID performance. It is unclear how this result corresponds to learning diverse models (section 3). In other words, how do the models learned in the experiments are related to adding a spurious feature to the feature set of a regression model? Such correspondence is necessary for section 6 to be part of a cohesive report. - The work is missing experiments on synthetic data to connect its theory and the real-world experiments. It would have been ideal if the authors used the same experimental method to sample from ID/OOD performance of models fitted to the data generated by the linear model described in section 6. Ideally, the plots in Figure 7 must be reproducible with the toy model and synthetic data for different instantiations of the linear model.

Questions

- To confidently recommend acceptance of this paper, I need to see the relevance of the theoretical analysis (section 6) to the experiments. In the previous section, I elaborated my recommendations to make this connection happen, and I am asking the authors to address my concern. - Is there a solid conclusion/suggestion made by the paper apart from the presence of inverse correlations between ID and OOD performance? Apparently, the paper suggests that contrary to the conclusions made by previous work, ID performance can be irrelevant to OOD performance, however, the authors do not provide an alternative guideline for practitioners to make claims about OOD performance. Such discussion would be very helpful for the audience.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Mentioned in the previous parts.

Rebuttal Response

I thank the authors for their response to my comments. As a result, my concerns have been mostly addressed. > We added this new set of experiments to show that diversity could be induced from simple, standard variations in hyperparameters and training methods. It would have been great if the authors had included these experiments in the rebuttal PDF. --- I leave my score unchanged.

Rebuttal Response

Thank you for your rebuttal and clarification. >The theoretical possibility of an inverse ID/OOD correlation is quite direct as noted by reviewer zdij. It indeed is. However, the theoretical analysis that is provided in Section 6 is neither complementing the experiments, nor expands our theoretical understanding. I agree with reviewer zdij that an experimental work does not necessarily need to have a theoretical contribution. However, if theory and experiments are present in one paper, they have to correspond to one another, which is not the case in this manuscript as I elaborated in my review. I suggest removing the theoretical contribution or taking it to the appendix in order to preserve cohesion. > (...) The key strength of this paper is therefore in the observations on actual data, and simulations on synthetic data would only distract from this main message. Conducting synthetic experiments corresponding to the toy model presented in section 6 was a suggestion to bridge the gap between section 6 and the rest of the work. If you view such synthetic experiments as distracting from the main message of the paper, then by the same argument section 6 is also distracting from the main message of the paper. Had the previous work been grounded theoretically, such false claims wouldn't have been made about positive correlation between ID and OOD performances. The main reason that I weakly suggest accepting this merely experimental work in OOD generalization, is the mistake in previous work that must be corrected. **Thus, I leave my score unchanged.**

Not addressing the weakness

> The claims of this paper are much more fundamental: they are not restricted nor dependent of particularities of neural networks. The connection with training dynamics of NNs [1] would be interesting for future work. A relevant analysis of a shallow neural net appeared recently [2]. I'm not suggesting the paper's claims are restricted. I merely mean that Theorem 1 is, under the linear regression setting. This creates a bigger discrepancy between Theorem 1 and your experiments which are based on neural nets. Your paper's primary contribution is the experimental studies. I find Theorem 1 peculiar in the paper flow. I'd recommend either move it to the Appendix or replace it with a neural net version, which are the models used in your experiments. > The challenge is to obtain multiple test sets where the distribution shift can be quantified. The suggested ones would be problematic because of the different classes. Will ID (SVHN) and OOD (gradually binarized SVHN, e.g. replace the [0, 1, ..., 255] scale to [0, 5, 10, 255] ) do it? It doesn't sound difficult as long as we look for it.