Thank you for your thoughtful review.
> Limited Novelty in Contribution: Although the paper effectively demonstrates the value of task-specific fine-tuning, the concept itself is not novel. Prior work has already shown that fine-tuned, specialized models outperform general-purpose LLMs [1, 2, 3].
Frontier LLMs are state-of-the-art in many domains, such as mathematical reasoning (e.g., MATH), code (e.g., HumanEval), reasoning over text (e.g., DROP), and graduate-level question answering (e.g., GPQA). Prior to our work, it was not at all evident that specialized open-weights models could match the performance of frontier models on legal annotation, let alone substantially outperform them. It is precisely for this reason that recent works that use LLMs for legal annotation rely heavily on frontier models (e.g, [1, 2, 3, 4]).
Please note that the references provided focus on more “classical” NLP tasks such as named entity recognition, natural language inference, or sentiment analysis. We will make sure to cite and discuss the provided references in the related work.
[1] Saromir Savelka and Kevin D Ashley. The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts. Frontiers in Artificial Intelligence, 6, 2023.
[2] Michael A Livermore, Felix Herron, and Daniel Rockmore. Language model interpretability and empirical legal studies. Virginia Public Law and Legal Theory Research Paper, (2023-69), 2023.
[3] Jens Frankenreiter and Eric L Talley. Sticky charters? the surprisingly tepid embrace of officer-protecting waivers in delaware. European Corporate Governance Institute-Law Working Paper, (762), 2024
[4] Morgan A Gray, Jaromir Savelka, Wesley M Oliver, and Kevin D Ashley. Empirical legal analysis simplified: reducing complexity through automatic identification and evaluation of legally relevant factors. Philosophical Transactions of the Royal Society A, 382(2270):20230155, 2024.
> Exploration of Advanced Prompting Techniques
We have now evaluated the models using zero-shot chain of thought (CoT). We follow the standard methodology of eliciting CoT by appending to the prompt “Let’s think step by step.” Since CoT requires two orders of magnitude more compute for evaluation than the standard QA approach, we only evaluate Llama 3 8B Instruct and Llama 3 70B Instruct. This required over 500 H100 GPU hours. We observe that CoT leads to modest improvements of performance for both the 8B and 70B model, on average of 2 to 3 accuracy points, see Figure 17 in Appendix E.5. Nonetheless, Lawma 8B still strongly outperforms Llama 3 70B, by over 20 accuracy points.
> Narrow Focus on Text Classification
We study precisely the narrow specialization of models for legal annotation. This is a feature of work. We show that specialized models strongly outperform generalist models such as GPT-4. Such narrow specialized models have rich scientific applications for empirical legal studies and the broader “law as data” research paradigm. Entire subfields of legal scholarship, political science, economics and sociology build on law as data. Other NLP tasks (i.e. text generation) are very much of secondary importance for this domain of research.
> What are some examples of typical errors Lawma makes, especially on complex tasks?
The types of errors made by Lawma are highly task dependent, and thus it is difficult to draw broad conclusions. Let us give one illustrative example of a failure case. For the Supreme Court issue area classification, Lawma tends to miss classify habeas corpus cases as “Criminal Procedure” rather than “Civil rights” cases, since the language of habeas corpus cases tends to be more similar to that of criminal cases. More broadly, we find that Lawma tends to excel in tasks for which only surface correlations (e.g., the “language” of the case) suffice for accurate prediction, but struggles for tasks that require deeper understanding of the substantive aspects of the case. Lawma also tends to perform worse for tasks that have fewer training data and larger number of classes.