Spuriosity Rankings: Sorting Data to Measure and Mitigate Biases

Summary

Shorcut learning has received increasing attention from the community recently. The paper proposes measuring and ranking data by "spuriosity," or the degree to which relevant spurious cues are present, as a way to detect and mitigate biases in deep models that arise from their tendency to rely on spurious correlations in the data. The authors use an interpretable model to identify neural features relevant to a class and then select the spurious ones based on limited human supervision. Ranking data by the activation of these spurious features yields many benefits, like revealing less biased subsets, quantifying model bias via "spurious gaps" in accuracy between high and low spuriosity data, and finetuning models on less spurious data to obtain robust performance. Analyzing spurious gaps for 89 models reveals that all underperform on less spurious data, indicating that bias stems more from what data models are trained on rather than how they are trained, and that spuriosity rankings provide an efficient and interpretable method to complement model-centric bias mitigation approaches.

Strengths

1. This paper works on a timely and important problem. The paper is overall well written (except its dependence on a related work [42]) and I enjoy reading it. 2. It provides a scalable method to discover and rank data by how much spurious features they contain, i.e., spuriosity, which can help detect and mitigate biases in models. 3. This paper proposes to finetune models on low spuriosity data to improve performance on less biased instances while maintaining overall accuracy, resulting in more robust and more stable models. 4. Experiments results are insightful. The analysis of spurious gaps for many models indicates that biases stem more from the data than how models are trained, suggesting that data-centric approaches like this paper's can complement standard model-centric bias mitigation techniques.

Weaknesses

1. One major concern is that this work is heavily based on a related work [42], which seems like from the same team of this submission. This makes the writing of this paper less friendly for readers not familiar with [42]. The authors are encouraged to revise the writing to make it more self-contained. 2. In line 176-177, the authors claim that “we are the first to uncover this racial bias in the Celeb-A benchmark”. Actually, this racial bias issue in Celeb-A has been well studied in various literature.

Questions

None

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes, the authors adequately addressed the limitations.

Summary

This paper propsed a framework to measure model biases by ranking images within their class based on the strength of spurious cues and evaluate the gap in accuracy on the highest and lowest ranked images. The analysis is comprehensive for a very large number of models.

Strengths

This paper propsed a simple method to measure model biases by ranking images within their class based on the strength of spurious cues. The proposed method is simple but novel and unprecedented and seems reasonable. In particular, the most strong point is that the analysis is comprehensive for a very large number of models.

Weaknesses

It seems more compatible with a computer vision conference than a machine learning conference. The definition of spuriosity in Sec 4.1 is a bit ad hoc. The ending without discussion or conclusion is a bit of a dead end in terms of the paper structure.

Questions

The definition of spuriosity in Sec 4.1 appears to implicitly assume the normality with respect to r. For example, if the distribution of r is long-tailed, the variance would be too large and this definition is undesirable. This may not be a problem if the features are after batch normalization, but for some models, this calculation may b undesirable, and would that affect the measurements?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The validity of the interpretation method for deep learning models itself is directly related to the validity of this method. The significance and meaningfulness of the proposed method is entirely dependent on the interpretation model. Because several papers have shown that explanations using heat maps are sometimes fragile, the usefulness of this technique is influenced by the goodness of other techniques.

Summary

The paper proposes a method to rank images in a given dataset in order of their "spuriosity". The proposal uses methodology from previous work [42] which examines most-active neural features of a trained model at per-class level, and hand-labels some sample images w.r.t. whether those features are “core” or “spurious”. The authors 1) scale out the labeling to imagenet-1k, 2) use these sample labels to rank instances according to "spuriosity", 3) present a range of qualitative and quantitative results arguing that they can, e.g., reveal minority subpopulations, measure model bias & mitigate it, enable study of model stability by distributional perturbations, identify noisy labels, etc.

Strengths

1. The authors give a data centric view to the problem of spurious correlation – i.e., instance rankings instead of feature / attribute labels or descriptions. This paper extends the study of [42] for all 1000 classes of imagenet. 2. The authors present some useful analyses e.g., showing ubiquity and correlation of bias across models and training methods, demonstrating “auto-”discovery of novel dimensions of bias using the new supervision, and the class-specificity of certain biased features. In particular, it makes sense, as the authors argue, to consider the “spuriosity” of a feature at class granularity rather than dataset granularity. 3. The proposal of data rankings opens up the possibility of trying out many mitigation methods and metrics to evaluate bias. The authors demonstrate one simple mechanism, and also suggest that their rankings can help flag mislabelled examples.

Weaknesses

1. There is no quantified measure to evaluate the quality of rankings proposed. Is it possible to look at synthetic or small data to get a quantitative sense of ranking quality? 2. The bias mitigation approach is not compared with the very wide existing literature on bias / spurious feature mitigation – this limits a thorough assessment of the value of the proposed rankings in mitigation. 3. The claim about identifying incorrect labels is largely qualitative – some quantification would strengthen the case. 4. The process of identifying spurious correlations requires human supervision–this can limit the applicability of the approach, both in terms of only considering or covering some aspects of bias, and in terms of labeling costs which are higher than typical category labeling.

Questions

1. Overall, if many of the qualitative / exemplar applications could be quantified with more rigorous comparisons, this would strengthen the paper. E.g., recovery of “known” rankings, human-rating or other quantification of ranking quality, comparison against at least some simple bias mitigation methods, quantitative success at identifying label noise, etc. 2. Are the rankings / human labeling data to be released publicly? Are the authors claiming a “dataset contribution” as part of their submission?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The process of identifying spurious correlations depends on a) neural activations of trained models, and b) human supervision–this can limit the applicability of the approach, both in terms of only considering or covering some aspects of bias, and in terms of labeling costs which are higher than typical category labeling (although, of course, the entire dataset does not need to be labeled).

Summary

- This work proposed Spuriosity, a quantity for determining the spuriosity ranking of data. The framework builds upon [42], which identifies spurious and core neural features by analyzing the neural activation map of an adversarially trained model. Spuriosity is defined based on these spurious neural features. The author also proposed the concept of ‘spurious gap,’ which measures the accuracy drop between the top-k highest and lowest validation images. By re-training the classifier head from a subsampled dataset using spuriosity, the authors demonstrate a reduction of around 10-20% in the spuriosity gap at the cost of 1-3% reduction in validation accuracy. - While I acknowledge the challenging nature of the addressed problem and the novelty of this work, I believe that the experiment in this paper primarily focuses on the use case of the proposed quantities rather than proving comprehensive validation of them.

Strengths

- This paper addresses the challenging problem: figuring out the spuriosity of large-scale datasets and investigates its impact on model training. I highly appreciate these contributions, as the field often focuses on improving the performance of simplistic benchmark datasets such as Waterbirds. - The paper introduced a novel approach that leverages the neural features to distinguish minority samples, which is algorithmically distinct from the previous works such as [1, 2, 3], which rely on classification results of auxiliary models. - Their method can be effectively combined with balanced re-training methods, such as DFR. [1] Liu, Evan Z., et al. "Just train twice: Improving group robustness without training group information." International Conference on Machine Learning. PMLR, 2021. [2] Kim, Nayeong, et al. "Learning debiased classifier with biased committee." Advances in Neural Information Processing Systems 35 (2022): 18403-18415. [3] Hendrycks, Dan, et al. "Natural adversarial examples." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

Weaknesses

- Regarding the Spuriosity ranking, I have concerns about the lack of quantitative metrics and comparison with baselines, making it hard to estimate how reliable the suggested quantity is and how much better compared to the other methods. I suggest comparing the performance of detecting minority samples with [1, 2] on the group annotated benchmark datasets such as WaterBrids and CelebA. - Another concern is the lack of a baseline and metrics of Spurious Gap. Say, someone proposed a Spurious Gap ver.2, then how to know which one is better? - The proposed method for detecting spurious correlation relies on human effort. - Minor correction for misinterpreted terminologies - Interpretable model: Consider using an ‘adversarially trained model,’ as normally trained models are also interpretable. - [L239] How it is trained: This expression is too broad. It seems to encompass algorithmic approaches like GroupDRO, etc.

Questions

- [Figure 8] How about comparing the reduction in the spurious gap with a subsampled dataset constructed using misclassified data from an auxiliary model? - Is there any idea for establishing the quantitative metrics and baselines of Spuriosity ranking and Spurious gap?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

- Their method heavily relies on interpretation methods and human judgment, which is costly. - The utilized neural activation map cannot distinguish certain features that are not discernible based on spatial information alone, such as the color and texture of the objects. - Most of the framework originates from previous work [42], which diminishes the contribution of this paper.

response to authors

Thank you for your detailed response. I remain convinced that the work is broadly along directions useful to image understanding and machine learning in general. I am comfortable with my assessment / rating of "weak accept".

Response to authors

I appreciate your detailed response. I believe the contributions of this paper are good enough to be accepted. However, there are remaining concerns. - As y13p mentioned, revising the paper to be self-contained would be important. - I agree that the result presented in Appendix F.1.2 is convincing for validation. Nonetheless, I believe there are still remaining tasks required to fully validate the Spuriosity ranking. For example, within this paper (https://arxiv.org/abs/2206.10843), Figure 4 presents the ability to rank the spuriosity of the training dataset. Also, one could establish the naive ranking based on learning speed (sorting samples based on training loss). Although these methods might be affected by label noise, employing them as baselines and checking their performance through the human study in Appendix F.1.2 could benefit this community. Many of my original concerns are alleviated now, so I increased my review score.