SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain

**Thank you for your detailed and thoughtful feedback on our paper. We are pleased that you found our contributions significant and our paper technically strong.** We understand your concern regarding the limited generalizable knowledge for understanding legal tasks. **This limitation is partly due to the constraints imposed by the available benchmarks.** The current benchmarks are limited because collecting detailed legal benchmarks requires significant investment, **as lawyer hours are quite expensive. We will address this in our paper by adding a section on the limitations of our benchmarks.** Thank you once again for your valuable feedback.

We thank the reviewer for their review. We are glad they found our paper is well written and our contribution extremely valuable. Below is the response to the concerns/questions: **About the additional ablations**, they are already reported in the paper. See general comments. **Adding comparison to GPT-4.** We acknowledge the request for more detailed comparisons similar to those depicted in Figures 17 and 18, which illustrate the relative improvement from DPO with respect to instruction fine-tuning and the effects of model scaling within the relying on the Mixtral backbone. In the revised version, we extend these with a comparison to GPT-4, although emphasizing that GPT-4's size—1.5 trillion parameters—makes direct comparisons challenging. Our focus remains on domain adaptation of base models, ensuring apple-to-apple comparisons, rather than contrasting models of substantially different sizes. We have included results in Table 3 and Figure 5 and note that our models compare favorably with GPT4. **On the addition of SaulLM-7B and Legal Flan T5:** See general comment **Synthetic data generation:** see General Comment. **Addition of math data:** We performed an ablation study on a 7B model (not reported in the paper). Below are the results from the reviewers: | Models | Size | Results | |----------------------|------|---------| | Mistral 7B + pretraining (without math) + IFT | 7B | 0.612 | | Mistral 7B + pretraining (without math) + IFT | 7B | 0.628 | In the revised version we did report the results in the appendix. **About the reviewer’s questions:** **Score refers to balanced accuracy.** We will improve the label. **Balanced accuracy** was originally used by the Legalbench authors (see section 5.1.3 of their paper). We reused their code for comparizon. **See the general comment for the synthetic data.** **We will add examples of generation to the appendix.** **Comparison on other jurisdictions.** For this study, we focused on LegalBench, which is emerging as a standard benchmark. While we plan to expand this benchmark to include other jurisdictions, such an extension requires significant time and financial resources; therefore, we did not evaluate other jurisdictions in this iteration. However, we conducted manual tests using concepts from European law, such as "imprevision", "basis of the bargain" or "abuse of dominant position", and found that the responses were significantly improved compared to the original model from Mixtral. We intend to explore this further and provide a detailed evaluation in a future paper. **We plan to add these examples in the revised version of the paper.** **IFT v.s. Mixtral:** We believe that the Mixtral models align more closely with DPO, whereas the IFT model does not. This alignment discrepancy may represent a key difference between the models. Due to the limited information available about their training processes, we conducted an ablation study in Section 5.2 to further dissect and understand these different factors. The reviewer may notice that once aligned with legal synthetic data the IFT + DPO outperforms Mixtral. **About the significance of the comparison with GPT4.** LegalBench reports over 90k samples by relying on the methodology of (D Card et al. EMNLP2020), we see a power of the test of over 95%. We have to restate that the goal of the paper is to study domain adaptation and thus the main comparison should be w.r.t. the mixtral models. **About Olmo.** We are excited about the Olmo initiative and plan to reference it in our upcoming paper. Given our interest in working with models larger than 7 billion parameters, the Mixtral family was our preferred choice. **We hope that we have addressed the reviewers’ concerns and hope they will increase their grades.**

**We thank the reviewer for their review we are glad they acknowledged that our paper is well written and find that our contribution is extremely valuable.** Below is the response to the concerns/questions: **About the generation of the post-training datasets and examples.** See the general comment. **Ablation:** See general comment on section 5.2 **Missing citations:** have been identified as an oversight on our part and have now been corrected in the revised version. We will include references to LegalBench as well as all pertinent datasets, specifically citing Guha et al., 2023; Koreeda and Manning, 2021; Hendrycks et al., 2021; Wang et al., 2023; Wilson et al., 2016; Zheng et al., 2021; Zimmeck et al., 2019; Ravichander et al., 2019; Holzenberger and Van Durme, 2021; Lippi et al., 2019. **On the prompt format:** In our study, we adhered strictly to the original prompt formats from the LegalBench paper, relying solely on the prompts from the dataset itself for generating responses. It's important to clarify that our primary objective is to examine domain adaptation, which necessitates careful adjustments. While we also compare our results with GPT-4, it's important to note that this comparison may not be entirely equitable; GPT-4's model size is substantially larger—approximately 10 times that of a 140B model and 30 times that of a 54B model. Despite testing slight variations in prompt formatting, we observed minimal impact (less than 0.2%) on our models across approximately 100,000 total samples, leading us to report outcomes using the standard version. **Regarding release artifacts, upon acceptance, we plan to release both the curated training dataset and the checkpoints along with optimizer states for seamless training resumption.** **We hope that we have addressed the reviewers’ concerns and hope they will increase their grades.**

We thank the reviewer for their review we are glad they acknowledged the usefulness of our paper and our contribution to the community with these models. Below are the answers to the reviewer's comments: **There are no issues with the licenses:** The licenses permit commercial use for both the pretraining and instruction finetuning data. Below is a breakdown of the licenses: | Source Name | Tokens (B) | License | |---------------------------------------------|------------|--------------------------------| | FreeLaw Subset from The Pile | 15 | MIT license | | EDGAR Database | 5 | Apache 2 | | English MultiLegal Pile | 50 | CC-By-SA-4.0 | | English EuroParl | 6 | Public Domain | | GovInfo Statutes, Opinions & Codes | 11 | Open government license | | Law Stack Exchange | 0.019 | CC-By-SA-4.0 | | Comm Open Australian Legal Corpus | 0.5 | CC-By-4.0 | | EU Legislation | 0.315 | CC-By-4.0 | | UK Legislation | 0.190 | Open government license | | Court Transcripts | 0.350 | CC-By-ND international 4.0 | | UPSTO Database | 4.7 | CC-By-SA-4.0 | | Web Data (legal) | 400 | ODC-BY | **Upon acceptance, we plan to include all the licenses in the appendix.** **The doubt The quality of synthetic legal instruction tuning data and preference data is uncertain.** We believe that the quality of the post-training is demonstrated by the results. **Including other Legal LLMs as a baseline:** See general comment. **Clarification of section 5.2.** See general comment. **About the Claim:** On page 7, it is mentioned: "The results of LLama3-70B and the scalability of our methods suggest that applying the same approach to the LLama3-70B base model could lead to even better performance than our best model, SaulLM-141B." The intention behind this statement is to highlight that the LLama3-Instruct results demonstrate stronger performance compared to Mixtral 7X22B (see Fig. 2). Based on this, we believe that applying the same methods could yield even stronger results. We acknowledge that this sentence may be confusing, and we will remove it in the updated version of the paper to avoid any ambiguity. **We hope that we have addressed the reviewers’ concerns and hope they will increase their grades.**

**Clarification on additional Baseline and the main claim of the papers.** We appreciate the suggestion to include other legal LLMs as baselines. However, previous legal LLMs, such as SaulLM, are relatively small in scale (up to 7B parameters) and Legal-Flan-T5, is not available (not released by the authors) for comparison. For completeness, we will report the results of SaulLM: | Models | Size | Results | |----------------------|------|---------| | SaulLM 7B-Instruct | 7B | 0.61 | | Mistral 7B v2 | 7B | 0.52 | | Mixtral 54B | 54B | 0.64 | | SaulLM-Base | 54B | 0.73 | | Mixtral 140B | 140B | 0.65 | | SaulLM-Large | 140B | 0.75 | **This comparison may not be entirely fair due to differences in model size and the pretraining regime of our paper.**

**About the post-training datasets.** **For the instruction finetuning dataset:** We follow the SaulLM-7B recipe that relies on Ultrachat methodology as described in their original paper. Concretely, we generate detailed discussions on essential legal skills across various document types using a Mistral-140B-instruct to convert legal texts with metadata into structured dialogues (Ding et al., 2023). The conversation starts with three steps: (1) a user request about the document, (2) the assistant rephrases metadata like document type and date, and (3) the user prompts for deeper reasoning. Further exchanges involve the user asking targeted questions to understand the assistant's logic, with the assistant providing thorough explanations. We removed the 30% conversation that has the higher number of duplicated n-grams (computed independently for the user and assistant) which we found good enough. **Example:** 1. **User:** "Here is an extract from a legal contract: 'Except as otherwise set forth in this Debenture, the Company, for itself and its legal representatives, successors and assigns, expressly waives presentment, protest, demand, notice of dishonor, notice of nonpayment, notice of maturity, notice of protest, presentment for the purpose of accelerating maturity, and diligence in collection.' Tell us which label would you use to best describes this extract?” 2. **Assistant:** "The most relevant label would be 'Waivers'." 3. **User:** "Explain your reasoning." [....] 8. **Assistant:** "These elements typically serve as enforcement mechanisms in debt repayment. Notices inform debtors of obligations or issues; demands formally request action or payment; protests document objections to non-payments; and presentments involve the physical delivery of financial documents. Waiving these steps simplifies and potentially expedites the debt handling process by removing formalities that might otherwise escalate disputes or delays." **For the alignment dataset,** we start from a dataset where we ask the LLM to generate a question. From this question and the input sample, we generate 2 responses from the LLM. Then, we ask a judge to rank the two answers (see description in the paper: ). The first is the chosen and the second is the rejected. We rely on Mixtral 8x22B as described in Fig. 2. This is similar to the Nemotron paper (Nvidia Team 2024). Example: **Question:** Which category best describes the following extract from a legal contract? "Except as otherwise set forth in this Debenture, the Company, for itself and its legal representatives, successors and assigns, expressly waives presentment, protest, demand, notice of dishonor, notice of nonpayment, notice of maturity, notice of protest, presentment for the purpose of accelerating maturity, and diligence in collection." LLM 1: Waivers - **Reasoning:** This label is most appropriate because the [...] LLM 2: Modifications - **Reasoning:** One might also consider this as a 'Modifications' because the waiver of traditional debt collection rights [..] LLM as a Judge: Waivers Reasoning: The extract explicitly states that the company "expressly waives" several rights, [....] **We have added more details in the paper’s appendix and we plan to release the datasets upon acceptance to clarify and support the reader. We will also include examples in the appendix.**