Artificial Intelligence in Document Review: Statistically Defending the Process and Results of E-Discovery Workflows

The use of artificial intelligence (AI) for document review allows legal teams to process and produce electronic documents on a scale, speed, and cost that would have been unheard of only a few years ago. But as legal teams become increasingly dependent on AI to locate and prioritize responsive documents, courts and opposing parties rightly demand heightened verification that the workflow is well designed and that the evidentiary output is sufficiently accurate to ensure a full and fair adjudication of the facts at trial. This Article explores the processes and statistical tools available to litigants to respond to related objections and concerns, focusing on the methods available to measure the completeness and accuracy of an e-discovery workflow. Drawing on principles of information retrieval and machine learning, this Article examines common metrics and explains their application within the framework of a defensible e-discovery workflow. This Article also addresses statistical sampling techniques, including random and stratified sampling, and discusses how confidence intervals and margins of error can be used to assess the reliability of the AI-driven output. It further examines the validation of technology-assisted review protocols, including continuous learning, simple learning, and other iterative workflows that adapt to human input and subsequent machine analysis. By examining both quantitative approaches and qualitative safeguards and by drawing on interdisciplinary literature from academia, industry, and government, this Article evaluates each technique’s strengths, limitations, and situational suitability, thereby providing courts and practitioners with an actionable guide for selecting defensible validation strategies across diverse review contexts. This Article provides relevant stakeholders with a better understanding of how to evaluate and measure the performance of an AI-driven workflow and verify whether it meets the requirements of the Federal Rules of Civil Procedure. This Article is particularly helpful because it provides a roadmap for litigants and judges who must confront AI-related issues at critical junctures in the litigation stream. It is particularly timely because continued advances in AI, particularly generative AI, will bring this tool to a broader range of litigants who will use it and judges who must evaluate its merits.