The Impact of Algorithmic Risk Assessments on Human Predictions and its Analysis via Crowdsourcing Studies

As algorithmic risk assessment instruments (RAIs) are increasingly adopted to\nassist decision makers, their predictive performance and potential to promote\ninequity have come under scrutiny. However, while most studies examine these\ntools in isolation, researchers have come to recognize that assessing their\nimpact requires understanding the behavior of their human interactants. In this\npaper, building off of several recent crowdsourcing works focused on criminal\njustice, we conduct a vignette study in which laypersons are tasked with\npredicting future re-arrests. Our key findings are as follows: (1) Participants\noften predict that an offender will be rearrested even when they deem the\nlikelihood of re-arrest to be well below 50%; (2) Participants do not anchor on\nthe RAI's predictions; (3) The time spent on the survey varies widely across\nparticipants and most cases are assessed in less than 10 seconds; (4) Judicial\ndecisions, unlike participants' predictions, depend in part on factors that are\northogonal to the likelihood of re-arrest. These results highlight the\ninfluence of several crucial but often overlooked design decisions and concerns\naround generalizability when constructing crowdsourcing studies to analyze the\nimpacts of RAIs.\n