Tower: An Open Multilingual Large Language Model for Translation-Related Tasks

Summary

The authors describe the process of adapting large language models to translation-related tasks (e.g., translation, pre-editing, post-editing) in great detail. They evaluate the resulting models (TOWER-BASE and TOWER-INSTRUCT) against open and closed models on a variety of tasks, showing clear performance improvements (outperforming open models and approaching or on par with large closed models). The authors pledge to release their models, code, and data sets to the community.

Rating

9

Confidence

4

Ethics flag

1

Reasons to accept

This work contains two types of contributions that can significantly benefit the field: - Methods, results, and observations related to effectively customizing existing LLMs to a class of tasks. These can be directly used or extended in ongoing work. - Actual models, code, and data. These can be deployed to benefit real-world users as well as used as starting points for further exploration. On a more detailed level: - The authors thoroughly describe the processes used to construct data sets, train models, and run evaluations. - The authors include detailed analyses (including ablation experiments) to explore where and how different techniques are effective. - The authors include a variety of sound baselines that make the results credible.

Reasons to reject

Since the authors' models are based on Llama 2, they inherit Llama's usage restrictions (cannot be used to improve non-Llama models and cannot be used by some companies: https://llama.meta.com/llama-downloads/). This could make it challenging for some members of the research community to build on this work.

Questions to authors

Appendix C.2 states that the parallel data used in continued pre-training is prompted with "<SRC LANG>: <SRC>\n<TGT LANG>: <TGT>". This is an important enough design decision to move to the main paper (e.g., that parallel data is labeled/prompted instead of simply concatenated).

Summary

This paper sets out to create a large language model specifically designed for translation and translation-related tasks. This is done by continual pre-training of LLaMA-2 on monolingual and parallel data in multiple languages, then instruction-tuning on a curated and filtered labeled dataset of translation-related tasks (translation, post-editing, named entity recognition). The resulting model, TowerInstruct, achieves impressive results on translation, named entity recognition, automatic post-editing, and grammatical error correction (with GEC being a held-out task unseen during instruction-tuning), performing on par with much larger models. Furthermore, the authors publicly release the model, dataset, and evaluations. This is an important contribution that will be of interest to the industry. My main quibble is that I wish that the set of translation tasks explored were more extensive (e.g. including different domains or modalities).

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

1. The approach to training a translation-focused LLM is useful. Furthermore, it is effective, as shown by several extensive evaluations. 2. In addition to the paper, the authors will release their LLM, dataset, and evaluation framework. This will foster future research in the area. 3. The analysis in section 4 is interesting and clearly shows the impact of each component of the training "recipe".

Reasons to reject

1. I am a bit perplexed by the choice of "translation-related" tasks in the paper: translation, automatic post-editing, named entity recognition, and grammatical error correction. I don't think named entity recognition or grammatical error correction are specifically translation-related (although they may be useful for translation, like several other tasks), while more related tasks are omitted (e.g., metric development, quality estimation, parallel data curation, bilingual lexicon induction). In addition, I think it would be useful to see more comprehensive coverage of machine translation -- including subtasks like domain-specific MT, low-resource or dialectal translation, or multimodal MT. 2. In some cases, it is unclear how experimental settings are selected. Examples: L100 "We train for 4 epochs using a low learning rate and a large batch size", L121 "We prompt TOWER and closed models in a 0-shot fashion and others with 5 examples randomly selected from the development set." It would be good to know how these design decisions were made (optimization on a development set?).

Questions to authors

1. Not a question, but a suggestion: "automatic post-edition" should be "automatic post-editing" (in a couple of places). 2. Why did you select those 10 languages? (EN, DE, FR, NL, IT, ES, PT, KO, RU, ZH) 3. L63: Why do you need language identification?

Summary

This paper discusses a multilingual recipe for large language models (LLMs) to have the LLMs acquire multilingual capability in the translation task and multilingual relevant tasks. In the recipe, the authors takes LLaMa-2 models as their open LLM models, perform continued pre-training in the LLM on a multilingual mixture of monolingual and parallel data, creating TOWERBASE, and do finetuning on instructions for translation processes, creating TOWERINSTRUCT. The experiments in the translation-related tasks show that the proposed model outperforms open alternative on the tasks and achieves competitive performance wit closed models such as GPT-4.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

The paper is well organized and easy to follow each section. Most parts are clearly described. Each result table is carefully designed, as those model ranking report helps understand. The authors conducted a variety of experiments in the translation-related tasks. Their experimental results are technically sound, showing that the multilingual capability is well acquired by the proposed approach. The proposed approaches work well at different LLM model scale (7B->13B).

Reasons to reject

- Baselines seems weak as they evaluate (non-multilingual) models in multilingual tasks. Also, the system comparison against the previous work (Xu et al., 2024b) seems misleading and needs more clarification. The ALMA models are same LLaMa-based models but trained on different data sets, regardless of its different finetuning recipe. It is not clear to me whether those improvements come from either data set and/or the proposed TOWER recipe. - The proposed approach is only applied to LLaMa-2 7B and 13B. We would love to know how much performance we would get from other open LLMs like Mistral-7B-Instruct.

Questions to authors

- The authors described the parallel-data training as "During training, loss is calculated on both source and target." in the 67th line. Can you clarify this by referring the prompt template (reported in C.2)? Did you calculate the loss over the whole template including <SRC-LANG> and/or <TGT_LANG>? - The column of GEC in Table 3 seems wrong. Since it reports the error rate, I guess that lower score is better, hence the GPT-3.5-turbo should be best-ranked with the error rate of 15.06. - Why did you use COMET-kiwi for data cleaning? - What kind of prompts did you use in each of LLM baselines? - Did you do full fine-tuning in all the experiments? Have you tried parameter efficient model finetuning like LoRA finetuning [1]? [1] "LoRA: Low-Rank Adaptation of Large Language Models", Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. arXiv:2106.09685 (cs).

Summary

This paper discusses a method for improving the performance of LLMs in translation workflows by tailoring them to handle multiple tasks. The authors introduce TowerBase, created through continued pretraining on a multilingual mixture of monolingual and parallel data, and TowerInstruct, obtained by finetuning on instructions pertinent to translation processes. Their approach results in models that outperform other open LLM alternatives on various tasks and are competitive with general-purpose closed LLMs.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

The paper explores continued pretraining and fine-tuning specifically for translation tasks. The methodology is clear and easy to follow, demonstrating a well-structured approach to enhancing translation performance in large language models. The specialization dataset and evaluation framework will be useful for the community.

Reasons to reject

The approach of continued pretraining combined with instruction fine-tuning (as depicted in Figure 1) is quite standard in current LLM development. The paper lacks clarity on the specific technical contributions it offers beyond established practices.

Re: Rebuttal by Authors

Thank you for adding this information. The first point is strong enough to mention in the abstract and introduction: beyond training and releasing specific models, you share a recipe that can create Tower-style models from newer and stronger open models as they are available.

Thank you for the response.

The response clarified my questions. The final ver will be more technically sound with those additional results. Hence, I increased the rating to "7: Good paper, accept".