A Closer Look at AUROC and AUPRC under Class Imbalance

Summary

A widespread claim in machine learning that AUPRC is a superior metric than AUCROC for tasks with class imbalance is not strictly true. Based on this statement, this paper challenges this claim from two perspectives. On the one hand, the authors theoretically characterize the behavior of AUROC and AUPRC in the presence of model mistakes, showing that optimizing for AUROC equates to minimizing the model’s FPR in an unweighted manner, whereas optimizing for AUPRC equates to minimizing the FPR in a weighted manner. On the other hand, experiments on both semi-synthetic and real-world fairness datasets support their theory.

Strengths

- The proposed ideas are novel and innovative. Theoretically, the authors explore the relationship between AUROC and AUPRC, revealing their key differences. Specifically, the authors show that while optimizing for AUROC equates to minimizing the model’s FPR in an unbiased manner over positive sample scores, optimizing for AUPRC equates to reducing the specifically for regions where the model outputs higher scores relative to lower scores. In addition, the authors propose that AUPRC is explicitly discriminatory in favor of high-scoring subpopulations. - The authors validate the proposed theoretical findings with the help of numerical results. To rigorously confirm the findings of differences between AUROC and AUPRC, the authors conduct many synthetic experiments and real-world validation on popular public fairness datasets. - Guidance on how to choose AUROC and AUPRC is provided. In section 4, the authors offer detailed instructions for use in different scenarios. For context-independent model evaluation, deployment scenarios with elevated false negative cost, and ethical resource distribution among diverse populations, AUCROC is a more proper metric. However, for reducing false positives in high-cost, single-group intervention prioritization or information retrieval settings, AUPRC is a better choice.

Weaknesses

- The presentation needs to improve. For example, the term “high prevalence subgroup” is introduced without explanation, hindering my understanding of the theorem. A more detailed explanation of the high and low-prevalence subgroups should be provided. Besides, In theorem 3, the authors say that there exists a prevalence disparity sufficiently severe, but I can't see it directly from this theorem and suggest that the authors provide a clearer explanation. - I find this paper (Exploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability, NeurIPS 2022) also has a similar form (equation 2) and conclusion about AUCPRC. Can you explain the difference between your method and this one? The corresponding citation is also necessary. - The caption of Figure 1 is too long. It is recommended to make it more concise.

Questions

- Theorem 3 is based on two subgroups but real-world datasets generally have more than subgroups. Is recommended that the authors provide analysis based on multiple subgroups.

Rating

5

Confidence

4

Soundness

4

Presentation

2

Contribution

3

Limitations

Yes

Summary

The paper challenges the claim that area under the precision-recall curve (AUPRC) is a better metric for model comparison to the area under the receiver operating characteristic (AUROC) when it comes to tasks with class imbalance. The paper offers three formal results, proving i) a characterization of the two metrics in terms of how they relate to FPR, ii) a description of how the two metrics optimize for the correction of mistakes and iii) in fairness-sensitive settings, AUPRC introduces biases in favor of the highest prevalence subpopulations wrt AUROC. These findings are supported by synthetic and real-world experiments corroborating the findings. They conclude that the aforementioned claim is unfounded and AUROC is preferable in several settings. Moreover AUPRC is potentially harmful in fairness-sensitive scenarios due to its bias towards the correction of mistakes in the highest prevalence subpopulations. Finally, the paper contains a review on the literature generating the claim and some guidance on when using which of the two metrics.

Strengths

1. The paper combines theoretical results with empirical investigations on both synthetic and real-world data. 2. The paper does important work in challenging a claim that is widespread in the prediction model literature. 3. The practical implications of the theoretical findings, especially when it comes to fairness, are thoroughly investigated. The authors also research the origin of the claim and provide guidance on when to use which metric. 4. The text overall reads well and makes clear points. There are however some things to fix urgently in the presentation (which is why the presentation score is low and the overall score cannot be higher).

Weaknesses

1. Section 4 has a fair amount of repetition wrt Figure 1. I suggest compressing one or the other. 2. Figure 2 is corrupted and the results cannot be assessed. This needs to be fixed. 3. The explanation is line 129-133 is not clear, I suggest re-writing it. 4. A minor point, but I would suggest the authors to not use both boldface and italic at the same time (e.g. in the introduction). It feels unnecessarily intense. 5. Another minor point, but I would discourage the authors from using self-praising expressions such as “our analyses are thorough and compelling”. Let the reader be the judge of that. 6. In the caption of Figure 1, repeating the definition of mistake is not really necessary I believe. 7. The theoretical results have some strong assumptions, e.g. perfect calibration. Typos: line 290: significnat -> significant line 310: These -> these

Questions

1. For the relaxation of perfect calibration in Theorem 3, have the authors tried to consider different models with ascending levels of calibration, to see to what extent the property expressed by Theorem 3 holds?

Rating

7

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The limitations seem to be properly discussed.

Summary

The paper proves that AUPRC weights mistakes in higher score ranges higher, while AUROC weights all mistakes uniformly. This property of AUPRC can be underable in many real-life settings. It goes against the widespread belief that AUPRC is somehow “better” than AUROC in low-prevalence domains, which is a common belief both in academia and industry. The paper also proves a theorem that states that AUPRC is be discriminatory against low-scoring subpopulations, and it backs up this theoretical finding by a series of experiments that confirm that these discriminatory effects occur in practice when selecting models based on AUPRC

Strengths

It has long been well-established in the literature that the AUROC is prevalence-invariant (or as they put it in concept drift-focused ML subcommunities: it is invariant to prior probability shift), while the AUPRC is not [1,2,3,4]. In the light of this, the argument that AUPRC would somehow be “better” than AUROC in low-prevalence domains has always appeared paradoxical to me (how could a prevalence-invariant metric be unsuitable in low-prevalent domains?). This paper does an excellent job in theoretical analysis of AUPRC and AUROC, an engaging read, and is clear on the practical implications. Figure 1 provides a compelling overview of the main results and manages to convey the implications clearly in one glance. I really enjoyed this read and consider this to be an important paper.

Weaknesses

A minor critique goes to the experimental section. In both the synthetic and the real-world evaluation the confidence intervals appear rather wide. While the Figures 2 and 3 do appear to support the message that the authors claim that it does, it could be more convincing. For example, Is it really true that the AUROC for the low-prevalent sub-group decreases in Figure 2d, or is this noise? The AUROC of the high-prevalence group in Figure 2b appears to trend up while increasing steps while the one of the low-prevalence group appears to stay flat, as the theory would suggest. However, the confidence intervals do still overlap everywhere. Most of the CIs for the correlations in Figure 3 include 0.0. This appears to simply be a matter of sample size: 20 randomly sampled datasets in the synthetic evaluation and relatively small tabular datasets for the real-life data evaluations. It appears that Figure 2 may become more convincing simply by simulating more randomly sampled datasets (reducing the standard error of the mean), while for Figure 3 we may obtain a more convincing plot by simply including a larger real-life dataset. That said, I consider this to be a minor point. The main contributions of the paper are theoretical, and despite my critique, the current figures do support the theory. Just not as convincingly as could have been, and it appears easily solvable.

Questions

- Line 199/200: “We also evaluate the test set AUROC gap and AUPRC gap between groups”. I believe this may be a mistake, as Figures 3, 7, and 8 only appear to report AUROC gaps and not AUPRC gaps. I believe it is the right choice to refrain from reporting AUPRC gaps: the fact that AUPRC is known to be prevalence-dependent would imply that prevalence-gaps by themselves may already explain AUPRC-gaps, even without existence of underlying fairness issues. The experimental setup based on AUROC gaps appears correct. - Line 154: "such that $AUROC(D1) \approx AUROC(D2) \approx AUROC(D1\cup D2) = 0.85$. What is the algorithmic procedure that is used to obtain scores at the target AUROC, and how precisely does this achieve the target AUROC? - Line 163: "Next, we profile an optimization procedure that randomly permutes all the (sorted) model scores up to 3 positions." This description seems imprecise, what is the precise reordering procedure that is applied?

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors adequately addressed the limitations and potential negative societal impact of their work.

Summary

The paper considers a common in literature claim that AUPRC is “better” to use than AUROC for class imbalance datasets and attempts to prove it wrong based on theoretical results, empirical observations, and real-world experiments. The major focus of the paper is on the fairness gap that AUPRC exhibits. The authors provide guidance on when each of the metrics should be used and advocate for careful metric selection.

Strengths

1. The paper provides a thorough review of the literature, aiming to get to the root cause of Claim 1, and advocates for more responsible practices in choosing evaluation metrics. 2. The authors attempt to examine the problem from different perspectives, including theoretical analysis, synthetic, and empirical experiments. 3. The paper includes extensive details on notation, definitions, experimental details, and figures.

Weaknesses

1. It is unusual to see in the synthetic example in Section 3 that AUPRC is being optimized for the classification setting (as it is more common to use AUPRC as an evaluation metric). Could you please elaborate on when it can potentially be useful? 2. For the synthetic dataset example, since everything is controllable, including the number of mistakes, does it make sense to learn boosted trees or any other model of the authors' choice for different levels of mistakes and report AUROC/AUPRC instead of the proposed optimized procedure? 3. Since AUPRC focuses on the positive class and does not use true negatives, it is "expected" by definition of this metric to increase existing fairness gaps in the positive (minority) class. Could you please elaborate on what Theorem 3 adds beyond the definition? 4. The incorrectly ranked adjacent pair mistake model seems to favor AUROC over AUPRC. By definition, AUROC measures the ability of the model to distinguish between positives and negatives across all possible thresholds, so fixing this mistake improves AUROC uniformly. By definition of AUPRC, correcting such a mistake can have a significant or minor impact depending on the threshold. 5. While p-values and checking for for statistical significance is important, in addition for the experimental results on real-world datasets, could you please provide actual values of AUROC and AUPRC (mean and variance), for example in a bar plot, similar to what is typically reported in fairness literature? 6. The broader idea of the paper—to advocate for the careful selection of metrics—is highly welcomed. AUROC and AUPRC are different metrics with different goals, just as ROC and PR curves are. They can provide different insights when evaluating data with class imbalance. Wouldn’t it make sense to encourage readers to explore both AUROC and AUPRC, as well as ROC and PR curves, under class imbalance, given the authors' findings and reference [83] instead of proposing to use one metric as in Section 4? Minor: Some figure legends seem to be cut off in the appendix figures.

Questions

1. What causes the variance to be so high in Figure 2? 2. Could you please elaborate more on what 'Prevalence (Higher)' and 'Prevalence (Lower)' measure in Table 1? If class imbalance, shouldn’t they sum up to 1?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors discuss the limitations of the paper.

Response to rebuttal

I thank the authors for the rebuttal, which addresses my points. Concerning Figure 2: this is visualized as corrupted when the file is opened on Safari; I see this was not a problem for the other reviewers but I am nonetheless surprised since this never happened with other papers on OpenReview. I recommend the authors double-check the formatting of the image in their iteration on the paper.