Algorithm Auditing Policies Rest on Flawed Assumptions About Public Sector Systems

Abstract

Acknowledging that automated decision-making systems can generate significant societal harms, policymakers have proposed regulations that require audits of algorithms. However, although these policies are popular, scholars have warned that algorithm audits can provide a veneer of accountability without meaningful impact. In this paper, we evaluate the emerging landscape of algorithm auditing policies. First, we analyze 25 United States policies that require audits of public sector algorithms to distill their core assumptions. Second, we test those assumptions against a representative case of public sector algorithms gone wrong: the Michigan Integrated Data Automated System (MiDAS). We find that auditing policies rely on flawed assumptions that limit their ability to identify and address algorithmic harms. While audit policies conceptualize harm as discriminatory bias, government algorithms can produce a much broader range of issues, including misallocation of public resources and violations of due process. While audit policies investigate algorithms as static and isolated pieces of software, these systems are dynamic and embedded in complex relationships with other technical systems and human practices. While audit policies envision that problems uncovered by audits will be corrected, many public agencies lack the capacity to fix or replace automated systems. In light of these flawed assumptions, we provide recommendations to improve audit policies and the broader governance of public sector algorithms.