Efficient Continual Pre-training of LLMs for Low-resource Languages

Open-source Large Language models (OsLLMs) propel the democratization of\nnatural language research by giving the flexibility to augment or update model\nparameters for performance improvement. Nevertheless, like proprietary LLMs,\nOs-LLMs offer poorer performance on low-resource languages (LRLs) than\nhigh-resource languages (HRLs), owing to smaller amounts of training data and\nunderrepresented vocabulary. On the other hand, continual pre-training (CPT)\nwith large amounts of language-specific data is a costly proposition in terms\nof data acquisition and computational resources. Our goal is to drastically\nreduce CPT cost. To that end, we first develop a new algorithm to select a\nsubset of texts from a larger corpus. We show the effectiveness of our\ntechnique using very little CPT data. In search of further improvement, we\ndesign a new algorithm to select tokens to include in the LLM vocabulary. We\nexperiment with the recent Llama-3 model and nine Indian languages with diverse\nscripts and extent of resource availability. For evaluation, we use\nIndicGenBench, a generation task benchmark dataset for Indic languages. We\nexperiment with various CPT corpora and augmented vocabulary size and offer\ninsights across language families.\n