A Benchmark for Learning to Translate a New Language from One Grammar Book

Summary

This paper investigates how effectively LLMs adapt to a task which is guaranteed to have no overlap with LLM training data. The task is translation between English and Kalamang, an endangered language with little to no online presence. The authors introduce a new dataset, MTOB (Machine Translation from One Book), which contains (1) a linguistic analysis of the Kalamang language, (2) a bilingual dictionary, and (3) a small English-Kalamang parallel corpus. They benchmark several LLMs on the translation task, experimenting with different in-context learning settings. They find that the bilingual dictionary and parallel corpus enables some translation capabilities, and a large context size learning from the grammar book leads to notable performance gains. However, no LLMs outperform a human baseline.

Strengths

This work is a solid scientific investigation into an unexplored topic. The paper is excellently written - clear, well organised, and engaging. **-- 1. Novel benchmark --** MTOB is a unique benchmark that enables interesting experiments. It would undoubtedly be a useful resource for future work, as it offers an alternative to the current paradigm of raw text training and highly structured fine-tuning. The idea of using it to mimic second language learning is interesting. **-- 2. New experimental ideas --** The paper introduces a few new ideas in its experimental framework with the aim of testing generalization capabilities beyond the training set. 1. Using content guaranteed to not be on the web. 2. Testing model knowledge on the task *before* training to check for potential train-test overlap. 3. The motivation of testing crystalised intelligence, as opposed to fluid intelligence. These are all ideas that could find use in other contexts/domains. **-- 3. Interesting findings --** The experiments reported reveal some insightful findings. For example, the failure of traditional finetuning in this setting is interesting, and so is the difficulty of incorporating grammar knowledge into smaller context LLMs. **-- 4. Handling of ethical considerations --** The authors actively engage with ethical considerations around working with an endangered language. Their approach in working with the Kalamang language community is a model for how other NLP researchers should proceed.

Weaknesses

While the topic of interest and proposed dataset are novel contributions, my main concern with the paper is the lack of innovation in terms of modelling and evaluation. Furthermore, I believe more should be done to prove that this task is truly distinct from standard extremely low-resource translation. This could be shown empirically through experiments comparing Kalamang translation to translation involving other (somewhat online) extremely low-resource languages. **-- 1. Narrow modelling comparisons --** The experiments would be improved by comparing other types of models besides recent LLMs. Sequence-to-sequence PLMs like mT5 have been shown to perform well on low-resource MT (https://aclanthology.org/2022.naacl-main.223.pdf) . Furthermore, the nature of MTOB could be leveraged by more specialised neural architectures, such as neural MT models that incorporate bilingual dictionaries (https://aclanthology.org/2021.acl-long.382/, https://aclanthology.org/2020.acl-main.143.pdf). An analysis spanning different models would reveal more about the true difficulty of the task. There is a growing literature evaluating LLMs for low-resource MT (e.g. https://arxiv.org/pdf/2309.07423v1.pdf) and so far it seems that they fall short of massively multilingual NMT models, which could just be because they are trained/tuned for very different types of tasks. **-- 2. Insufficient interpretable evaluation--** While some qualitative examples are provided in the appendix, the paper would be strengthened by more such analysis. Could error types across models / in-context settings be quantified to some extent? More generally, since MTOB is framed as a unique benchmark, I would expect some interesting findings from a more nuanced evaluation framework (along the lines of page 7, paragraph 3 “In contrast, the grammar book…”). Such analysis could help motivate why MTOB is unique (e.g. if it is found that LLMs make certain types of errors on MTOB that they do not make on other tasks). **-- 3. Lack of comparison to extremely low-resource MT--** While any knowledge of the Kalamang language is new to the LLMs, the task itself (translation) is well known to LLMs given the wide availability of parallel corpora online and the popularity of machine translation as a task. The authors discuss this distinction themselves (fluid vs crystalised intelligence). However, there is some doubt as to how different English to Kalamang translation is in terms of task difficulty, compared to translation involving other extremely low-resource languages that have a limited online presence. This would call into question the value of MTOB as an NLP resource, as it is currently being claimed in the paper. For example, it seems that for some extremely low-resource languages LLMs have basically no translation capabilities (see some of the zero-shot experiments here https://arxiv.org/pdf/2309.07423v1.pdf), even thought these languages are included in publicly available test sets. The paper would be improved through some comparison of the MTOB task with extremely low-resource MT e.g. qualitative differences in model performance or proof of data contamination even for extremely low-resource languages.

Questions

1. Was any part of the translation train/test set previously released along with the Kalamang grammar book? 2. Did you test any other type of baselines (e.g. sequence-to-sequence models) on the translation task? 3. Have you considered using chrF++ as your primary metric since it (arguably) enhances automatic evaluation?

Rating

6: marginally above the acceptance threshold

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Summary

The work targets the OOD problem and the difficulty to assess it during training, especially when pre-trained models have seen very large swathes of open internet. Picking a domain where we can find an example scenarios that is rare on the web helps and they selected a low resource language with limited web presence. They create a benchmark which validate a model which has been trained by very few samples - in this case grammar content which is very high quality to demonstrate its value.

Strengths

The paper is very well presented and the core hypothesis is both clear and verified. Additionally, the paper does a good job of calling out the limitations of the work and future enhancements. The work itself feels motivated by a sincere desire to help communities who are disadvantaged due to their language being marginalized.

Weaknesses

As the authors themselves list in limitations, though a specific dataset was chosen to enhance the model, at least part of the information might have leaked into the pre-training dataset, especially since there are other languages close to the source language which may have a larger presence. The work would benefit from going beyond one sample to another language, especially since, as the authors state, there is a very high number of low resource languages. This would help the hypothesis and solution more convincing. The work otherwise comes across as a focused effort to provide more access to a disadvantaged group and then an attempt at generalization.

Questions

How confident are you that the solution would generalize to other low resource scenarios? There has been recent work (e.g. from Microsoft Research - phi models) on using high quality but smaller sample set to train a high quality model. How does your work compare to that?

Rating

8: accept, good paper

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Summary

The authors propose an interesting approach to translation: having an LLM directly learn a language (Kalamang) from raw linguistics reference materials. They feed these documents to various large language models, in a variety of different ways (via additional pretraining, by providing extra context in the prompt, etc.) and evaluate the model’s ability to translate to/from Kalamang. An additional baseline is also provided in the form of a human who learnt the language from the same reference materials. The approach is interesting in that it constitutes a modern take on rule-based MT, where instead of providing a structured grammar to a heavily feature-engineered model, the hope is that the language model will learn itself to interpret the human-readable descriptions. It is very valuable as a way of evaluating LLM performance on a clearly measurable, task-oriented benchmark which has direct real-world applications (translation of under-resourced languages.)

Strengths

* The authors propose a novel, clearly measurable and well-defined evaluation benchmark of LLM translation capabilities. * The experimental setup is solid, and there is an extensive selection of baselines. The addition of a human baseline is particularly appreciated as it helps put numbers in context. * The authors recognise the risk of the reference materials leaking into LLM training datasets, and take active steps to prevent it. * The work is conducted with the involvement and consent of the language community it concerns.

Weaknesses

* A minor wish would have been to see some additional baselines involving more standard MT approaches. If we're saying that this work has the potential of helping with the translation of under-resourced languages, it might be worth trying, amongst others: standard neural MT, trained on the little parallel data that’s available + parallel data from related languages; traditional rule-base MT; neural MT trained on synthetic data generated from templates. I recognise that these approaches would likely be more involved than simply feeding data to an LLM, but they are all reasonable approaches that a researcher interested in building MT for Kalamang might try.

Questions

* How do you expect this approach to stack up against traditional MT (e.g. neural sequence-to-sequence MT with e.g. a transformer encoder/decoder architecture), when using techniques such as cross-lingual transfer from related languages, backtranslation, data mining, synthetic data augmentation?

Rating

8: accept, good paper

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Response to All Reviewers (1/3)

Dear All Reviewers, Thank you very much for your positive reviews and insightful comments. We have responded to each of your comments individually and have posted a general response below. The one additional baseline that both Reviewer M62r and Reviewer tEGh asked us to run was a comparison to standard neural machine translation using seq2seq models. We agree that this is an important baseline and we provide it here. We finetuned a set of traditional machine translation models on our collection of 1134 parallel Kalamang-English sentences and compared their performance to our existing (in context learning) baselines. Since Reviewer tEGh mentioned the mT5 model specifically, we evaluated both the mT5 and FLAN-T5 series of models. In addition, for the sake of comparison, we evaluated the FLAN-T5 model using our in-context approach (note that this is only possible with FLAN-T5 because only FLAN-T5 is instruction-tuned). We show the results of finetuning compared to in-context learning in the table here: _(this response had to be split into 3 due to the OpenReview character limit)_

Response to All Reviewers (3/3)

We see that finetuning on the small number of available parallel sentences yields results which are reasonable, but not nearly as strong as those obtained from in-context learning. This is consistent with results observed in previous work, such as Adelani et al. 2022 [1], which Reviewer tEGh cited in their response. In Adelani et al. 2022, the least-spoken language has 13K parallel sentences (more than 8x Kalamang) and 1M speakers (more than 5000x Kalamang), and yet the method reaches only 28.5 chrF on Ghomálá → French translation. Of course, chrF scores are not exactly comparable across languages, but this low score still demonstrates the need for large amounts of data in traditional machine translation. In our case, with 1134 parallel sentences, finetuning reaches 35.2 chrF kgv-to-eng and 30.8 chrF eng-to-kgv. By contrast, we reach 44.7 and 45.8 chrF by performing in-context learning on 100K tokens from the grammar book, and we show that at least 52.6 and 57.4 chrF are possible by a human. We will integrate these baseline results into the paper. Once again, thank you for your helpful suggestion. [1] A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation (Adelani et al., NAACL 2022). https://aclanthology.org/2022.naacl-main.223.