From-Scratch Small Language Models for Legal and Financial Text: A Staged Pretraining Recipe and an Honest Account of Their Limits

We build two decoder-only language models from scratch — 125.8M and 528.5M parameters — pretrained on US case law, SEC filings, and educational web text, then fine-tuned for legal and financial question answering. We report three findings we believe are broadly useful beyond this specific project. First, naively mixing five heterogeneous pretraining sources at their natural size proportions measurably hurts held-out quality on the original, smaller sources (fineweb-edu perplexity worsens from 20.73 to 22.82 despite more total training data); a staged Warmup-Stable-Decay schedule, adapted from MiniCPM and SmolLM2, recovers this and improves perplexity on every shared domain over the smaller model. Second, a subtle supervised fine-tuning failure mode — two datasets sharing clause-category names under different task framings (extraction vs. classification) — causes the model to collapse to bare label output regardless of what is asked; we document the bug, its 100% reproduction rate, and the fix. Third, and most centrally, we run both models through five externally accepted benchmark categories via lm-evaluation-harness and find a sharp, consistent split: open-book contract-clause tasks are a real, usable capability, while closed-book legal knowledge and reasoning sit at or below random chance for both model sizes. We further report a methodological case study on legal-domain contamination checking: a naive n-gram overlap check flags an implausible 8,712 lines as "contaminated," which on inspection are uniformly generic legal boilerplate rather than evidence of benchmark leakage; a stricter 40-word contiguous-run threshold combined with per-hit provenance attribution narrows this to 5 of 1,853 documents (0.27%), each independently explainable and none affecting the reported results. We release both models, the fine-tuning data recipe, an in-browser ONNX deployment, and all evaluation code. Code, model weights, and the full evaluation codebase: https://github.com/DeependraVerma/legal-slm-125M. Model cards: https://huggingface.co/DeependraVerma.