Summary
This paper presents a new approach for operating machine learning systems in settings where results are personalized in some fashion to individuals or group (i.e., some categorical feature that “personalizes” predictions). The goal of this new approach is to make the process of personalizing ML results more participatory for data subjects / data contributors, moving beyond the existing practices that are lacking when it comes to facilitating informed consent of data contributors. Specifically, the authors propose that by adopting the presented “participatory systems” approach, individuals who are subject to ML predictions can make decisions about sharing personal information at prediction time and better understand how information sharing impacts ML outputs. Further, the paper proposes an approach to help individuals navigate the opt in / opt out tradeoff.
The basic idea underlying the paper is to limit data use where personalization does not improve performance. The authors use clinical prediction models as a running use-case. Motivating examples include personalization based on features that are protected groups, self-reported by patients, or costly to acquire (e.g., requires doctor to label).
The paper first provides some relevant definitions that are mathematical in nature. An algorithm for “learning participatory systems”, i.e., selecting from a pool of personalized models and pruning away models that violate incentive compatibility, is presented. Then, the paper presents some experiments using the clinical prediction setting.
Strengths
Overall, I found this paper to paint a compelling vision of an alternate paradigm for ML. The overall quality was high, and I believe this paper can make a large contribution to the ML research community.
The originality is high. I think this draft does a great job describing problems with the status quo around personalized modeling. While the work certainly draws on existing scholarship and practical work in privacy, human-computer interaction, and FAccT, the specific formalization presented here is original. See below for several lines of work that may be particularly relevant (though I understand if the authors see this as out of scope).
The paper makes a compelling case for the importance of this line of work. Motivation was strong in general and could attract more interest in the broad class of participatory systems.
Overall clarity of the paper is high. The primary task being studied here is described well. On the whole, assumptions are stated explicitly and the claims in the paper are precisely stated.
The presentation of ideas and results is also strong. Figure 2 and Table 1 are particularly helpful.
Weaknesses
Overall, the current draft makes a strong contribution. A high-level weakness of the paper is that this is very much a user-centric / human-centric paper (the core idea is help data subjects) but no evaluation with actual users/subjects is included. Nonetheless, I think this is still a reasonable starting point for advancing this line of work.
Several other areas for potential improvement:
- The proposed approach / framework could be easier to follow with a summary of assumptions about the behavior of relevant agents, especially ML operators. I understood the basic assumption about groups wanting to maximizing accuracy for their group, but I think readers will want more on how different ML operator incentives matter here.
- There’s a lot of related work in human-computer interaction, privacy, and data economics that could be discussed more heavily. This may be out of scope, however.
- The paper could more explicit about how the resulting training datasets different when using this approach. The intended user experience for data subjects is very clear (i.e. I understand how a patient might interact with participatory systems), but it wasn’t totally clear if ML operators who use this approach will actually have substantially different looking training sets. Will ML operators eventually lose the ability to evaluate models for certain groups if those groups choose not to participate? In other words, what does this look for the perspective of a data engineer?
- Relatedly, while the motivation is strong, I’d like to see a bit more discussion of how ML operators would adopt this approach. It seems the actual interface design could be a major barrier to use.
Questions
The authors may wish to engage with some recent work on data minimization.
- Biega, A.J., Potash, P., Daumé, H., Diaz, F. and Finck, M., 2020, July. Operationalizing the legal principle of data minimization for personalization. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval (pp. 399-408).
- Shanmugam, D., Diaz, F., Shabanian, S., Finck, M. and Biega, A., 2022, June. Learning to limit data collection via scaling laws: A computational interpretation for the legal principle of data minimization. In 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 839-849).
Similarly, some recent work on collective action by data contributors may be useful for this paper (or future work, as this discussion may be out of scope). This work has great potential to scaffold and support data-related action by groups, perhaps to enable bargaining between data creators and data users.
- Vincent, N., Li, H., Tilly, N., Chancellor, S. and Hecht, B., 2021, March. Data leverage: A framework for empowering the public in its relationship with technology companies. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 215-227).
- Das, S., Edwards, W.K., Kennedy-Mayo, D., Swire, P. and Wu, Y., 2021. Privacy for the people? exploring collective action as a mechanism to shift power to consumers in end-user privacy. IEEE Security & Privacy, 19(5), pp.66-70.
In the Introduction, a distinction between online platforms (that do use informed consent) and personalized models (that do not) is made. I think this distinction is not quite as clear as the current draft implies; online platforms and models are highly intertwined in many contexts. This might be downstream of the focus on medical personalization, but I think there are many other domains which much larger “grey area” between platforms data collection and data collection for modeling.
Minor
- Duplicate refs: [4] and [5]
- Line 57 missing bracket
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The work is heavily focused on societal impacts of ML. On this front, I think the paper is very strong.
On the methods limitations front, it might help readers to talk a bit more about the limitations of evaluating participatory systems with offline evaluation techniques, as opposed to user studies.