Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes

Summary

This paper studies how particular examples in different benchmark datasets perform across multiple models using the HAPI and DDI datasets. It finds first that many examples are “systemic failures,” meaning that they are misclassified by all of the commercial models evaluated. Then, it also finds that model improvements over time do not tend to improve the rate of systemic failures. Even though some examples are no longer failing on all models, other examples can become systemic failure and there is overall nearly no net change. Finally, in the DDI dataset, the authors find differing rates of error by skin tone in human annotations, but show that the model does not have the same disparity. Overall, the paper is one of the first to study performance from this cross-model, cross-task “ecosystem-level” view.

Strengths

This paper’s primary strengths are its originality, quality, and clarity. First, this paper is an original addition to the emerging homogeneous outcome literature. To my knowledge, there has yet to be a study that understands whether individual model improvements actually improve outcomes for people for whom other models have failed. The finding here that they do not is a valuable contribution to the field and important supporting evidence for the dangers of algorithmic monoculture. Second, the authors do an excellent job of clearly defining the technical limitations and strengths of their work, in particular in section 6. I found no major technical errors, and the authors make great use of the two datasets. For example, this seems like exactly the kind of study that HAPI was collected for. Finally, the paper is exceptionally clearly written.

Weaknesses

This paper’s primary weakness, in my view, is specifically the model put forth in section 3.2 to compute the expected rates of failure. Though the authors do a good job in section 6 of addressing some of these concerns, I am not sure that assuming independence of the ML models is the correct probability model for this case. Because each model is tested on the same dataset, their failures by definition cannot be independent of one another. The authors’ test in section 6 with a subsample of harder examples and subsample of easier examples does offer an alternative failure model, but I’m not sure that fully captures the correlation of having the same example passed through each of the models. At the same time, I don’t necessarily feel that an “expected rate” of failure is really essential for the overall message of this paper. One option would be to remove section 3 and the corresponding results (or at least the part about expected failure rates). Another would be to just replace the model in section 3 with the slightly more detailed model of section 6, since that is at least capturing some of the correlation. I still feel that some more treatment of correlation of model performance across individual instances would be necessary though. I also recognize that this correlation is really what the paper is trying to measure empirically, which is again why I feel that a computation of expected failures is unnecessary, and most models of that expected rate are inherently unrealistic.

Questions

When the paper refers to “negative outcomes”, is it referring to a misclassification or, more generally, a harmful event to a person (like being rejected for a job)? In the introduction, it wasn’t clear to me and it might be worth explicitly defining what it is for the purposes of this paper. In section 4, could you also include information about successes that turn into failures? For example, you observe that examples that are improved on are usually ones that were already passing for another model. Is there a similar effect that successes that become failures are also examples that were already failing for another model? In lines 242 to 243, there is a statement that systemic failure rate is monotonically correlated with annotator disagreement. I think this is a significant point towards potentially explaining the nature of systemic failures. Would it be possible to include a figure or more details on this result?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors do a great job of acknowledging their limitations in section 6. My only comment is that some of that material might work better at the ends of the specific sections they address. There were a couple of cases where I found myself with questions at the end of a results section that ended up getting answered in section 6.

Summary

The paper empirically investigates ecosystems consisting of many machine learning models. Rather than analyze each model in isolation, the paper instead focuses on the extent to which all models in an ecosystem incorrectly classify an individual (the systemic failure rate), building upon recent work on *algorithmic monoculture* (Kleinberg and Raghavan, 2021) and *outcome homogenization* (Bommasani et al., 2022). The paper draws upon a large-scale audit (HAPI, Chen et al. 2022) that consists of 3 commercial systems per modality and 3 different modalities. They show that model improvements do not significantly reduce systemic failures: in particular, model improvements often occur on instances that are already classified by the other 2 systems. The paper then investigates two ML models (ModelDerm and DeepDerm) and two dermatologists on the dermatology dataset (DDI, Daneshjou et al., 2022). They show that the humans are more polarized (i.e. humans all agree or humans all disagree) than models. However, models (unlike humans) are more polarized for darkest skin tones than for lighter skin tones. The paper concludes by proposing several data-centric and model-centric explanations for the observed trends.

Strengths

The paper provides an interesting empirical investigation into how interactions between models within an ecosystem can affect outcomes for individuals. The results are a compelling extension of previous work on *algorithmic monoculture* (Kleinberg and Raghavan, 2021) and *outcome homogenization* (Bommasani et al., 2022). Systemic failures is a natural metric to study, and the empirical analysis of systemic failures in model ecosystems is rich and fairly thorough. Altogether, the analysis of systemic failures on the HAPI dataset and DDI datasets provide a fairly convincing argument that a model ecosystem can exhibit different behavior a single model in isolation. The paper is well-written and the key ideas are clearly presented.

Weaknesses

There seems to be a gap between the results in Finding 4 and the implications for fairness. While Figure 6 focuses on polarization, it seems that the metric of interest for fairness should be systemic failures. This raises the question of how the systematic failure rates compare for darker skin tones and lighter skin tones. See question below. The commentary section (Section 6) is somewhat high-level and does not provide significant insight into which factors may be driving systemic failures in a given ecosystem. As a result, it is not entirely clear what factors drive polarization and whether one should expect to see polarization in different contexts. Understanding what types of ecosystems are more susceptible to systemic failures seems like an important question for policymakers and researchers. See question below.

Questions

Questions: - How do the systemic failure rates compare for darker skin tones and lighter skin tones for the models? What about for the humans? How much larger is the gap between systemic failures that the gap between failure rates for these two groups on an individual model/human? - Please comment on which of the factors in Section 6 are likely to most responsible for systemic failures in different contexts. What factors of a given ecosystem might influence the level of systemic failure observed?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors adequately addressed the limitations of their work.

Summary

The paper introduces ecosystem-level analysis where instead of a single model, a collection of models will be evaluated together and their common failures/successes will be identified. Across three modalities, the paper shows a pattern of polarization where an instance is either more likely to be misclassified by all the models or correctly classified by all of them. Similar patterns are observed in medical imaging. The paper suggests potential mechanisms responsible for polarization and concludes with a discussion of implications for researchers and practitioners.

Strengths

- The framework appears promising to me because it centers people in the analysis and highlights how repeated failures of machine learning models for an individual can result in harm. - It provides concrete findings for previously mostly conceptualized homogenous outcomes. - It discovers a new form of racial parity that couldn't be identified without ecosystem analysis and is not present in human judgments.

Weaknesses

1. It could be more interesting if systematic failures were studied by linking modalities rather than each modality in isolation. But I understand the available dataset does not support this. 2. As discussed in Sec 6.1, the baseline of independent models is very unrealistic, for example, difficulty may confound the outcomes. In the current baseline, the majority vote of different models significantly improves their accuracy which is not at all expected. 3. Sometimes it's not clear what level of polarization is significant. For example, Fig 5a looks as no polarization to me. 4. I was expecting more models to make a conclusion about polarization. In many places, only two models are used to draw conclusions. 5. Since the number of models is always small, I have no good sense of how the effect scales with more models. 6. To address the last two points, authors may want to increase the number of models by evaluating some models on the same datasets themselves. This will be also helpful in experimenting some conjectures about the polarization mechanisms. 7. The new form of disparity (Finding 4) is interesting, but the framing seems to obscure the fact that humans are still more polarized on all skin tones. I'd recommend emphasizing this. 8. As a general comment, I'd like to see an expanded discussion on the causes of polarization and its implications. Typos: line 19: remove do, table 1: $h^1 \rightarrow h_1$

Questions

No specific questions. Feel free to address the weaknesses I mentioned if you think they are not the case.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

N/A

Summary

This paper introduces the concept of "ecosystem-level analysis," which evaluates the performance and impact of a collection of machine learning models in a given context rather than analyzing individual models in isolation. This approach could significantly contribute to understanding the broader impacts of deployed machine learning models on society. The authors identify a new phenomenon called "systemic failure" in machine learning models. This happens when a user is consistently misclassified by all available models in the ecosystem, pointing towards potential systemic biases in machine learning deployments.

Strengths

This paper provides empirical evidence of these systemic failures across three different modalities (text, images, speech) and 11 datasets, establishing a broader trend that transcends specific applications or domains.

Weaknesses

The study under review makes an ambitious attempt to analyze the societal impact of machine learning (ML) from an ecosystem-level perspective. Although the authors should be commended for their unique approach, the paper has notable gaps that prevent its acceptance, particularly for a high-standard conference such as NeurIPS. The paper's theoretical framework lacks rigor and the theoretical grounding is insufficient, making it more suitable for a venue like FAccT, which is more focused on ethical and societal aspects of ML. The authors propose the idea of systemic failure in deployed machine learning, which is indeed intriguing. However, the theory behind this proposition appears to be weak and not well-articulated. The paper lacks robust theoretical to support this claim, as the analysis seems to be limited in terms of the ML models considered. For example, what are the factors that could contribute to system failure? Does these factors differ in terms of features distribution and machine learning algorithm used? Furthermore, the implications of such systemic failures on society, an aspect of paramount importance, is not adequately addressed, making it difficult to comprehend the real-world significance of these findings. The authors conducted a large-scale audit covering three modalities and 11 datasets. While this might sound impressive initially, the appropriateness of the methodology is questionable. There is a lack of clarity about the selection criteria for these modalities and datasets, and their representativeness of the ML ecosystem as a whole. This omission undermines the credibility of the results. This is again because of the lack of theoretical analysis, that may make the result questionable -- does the result only hold on the datasets at hand? Had another ML method, data distribution changed, would there still be system failure? The claim that improvements in individual models do not significantly reduce systemic failures is an interesting observation. However, the paper fails to provide a detailed theoretical examination and mechanism analysis of this phenomenon. A stronger theoretical grounding for this observation would help the NeurIPS community to better appreciate and understand this claim. The study on medical imaging, especially in dermatology, opens up a promising area of research. However, the introduction of a new form of racial disparity unseen in human predictions is a serious claim that requires more in-depth explanation and rigorous validation across a broader range of ML methods. Without comprehensive exploration, the validity of this claim remains dubious. I actually think that the paper can focus on this unique dataset, provide more mechanism analysis using one single dataset. And then validate the conjecture on other datasets.

Questions

See weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See weaknesses.

Update following discussion period

We thank all four reviewers for engaging with our rebuttals. Overall, we believe the updated scores of 5, 5, 6, 8 provide strong evidence to back accepting this paper on the basis of strong empirical work, conceptual clarity, and clear exposition/presentation. We agree with multiple reviewers that a stronger theoretical basis would be an important contribution, one that we see as appropriate for future work to address. To summarize the changes we will make, including how we plan to use the additional page granted upon acceptance: 1. **Framing.** One area which the author response helped to clarify is confusion around our use of "expected" to describe the comparison we make to the observed phenomena, namely the ecosystem-level behavior we would observe if model failures were independent of each other. While we chose "expected" to contrast with "observed", it seems clear that a term like "baseline" would better describe what we intend. 2. **Importance of systemic failures.** Due to page limit, we abridged our discussion of the societal significance of systemic failures, largely deferring this to past works that have conceptualized the matter. With that said, with the additional space, we plan to bring more of this discussion into the main paper. Also, more specific to our datasets, we provide evidence of the market-level impacts of MLaaS in our rebuttals that we can also include if reviewers/ACs feel it is helpful context. 3. **Discussion of Finding 4 on racial disparities in medical imaging.** As currently presented, we identify and highlight the importance of a racial disparity in ecosystem-level outcomes for models that does not manifest for humans. We believe this is important, especially since it is not something currently being considered in the literature, but as Reviewer HXE5 highlights, we can do a better job of explaining why, while also highlighting the higher systemic failure rates for all races/skin tones for humans than models. 4. **Encouraging future work that strengthens the theoretical foundations.** As multiple reviewers highlight, there is room for stronger theoretical understanding, including establishing various (potentially causal) explanations for why/how homogenization arises. Having thought about this extensively ourselves, we will try to provide guidance for future work to address this important matter. Finally, we will do additional passes over the writing and also address the typos identified by reviewers. Overall, we are glad to see the reviewers recognizing the importance of this work on improving our ability to characterize the societal impact of ML, especially in concrete and widespread deployments.