The manuscript is now much improved. We will consider vanilla distillation and ablation with the best performing Gemma model in the final manuscript.
Thanks for the positive and detailed feedback.
1. Unfortunately the sizes of PaLM2 models are not public. Barring the differences in model size, however, the gap between PaLM2 and Gemma is also driven by differences in their pre-training recipes (e.g., PaLM2 are highly multilingual LLMs, Gemma models are more monolingual); as well as the fact that PaLM2 are relatively older models than Gemma. We speculate PaLM2 XXS to be comparable to NLLB 1.3B due to their similarity in compute requirements. To roughly align these models nonetheless, we suggest comparing their baseline performance in Table 2.
2. Admittedly, mufu-finetuned models have substantially higher latency than NLLB, which are encoder-decoder models trained specifically for translation. Low-resource languages are however notoriously difficult for LLMs [Robinson et al., 2023; Zhu et al., 2024] due to the huge disparity in resource levels between languages. We now further elaborate this point in lines 500 and 535.
3. Thanks for pointing out the possibility of overfitting in the other models. Despite seemingly superior performance of mufu20lora based on mean chrF, we find lora to be worse than full finetuning in translations to very-low-resource languages (Figure 2a). In Table 2, PaLM2 XS and PaLM2 S finetuned with the baseline method overfit and perform worse than PaLM2 XXS. This is not the case for Mufu finetuned models, as we see improvement with increasing model capacity. It is also possible that the model overfits to translations in HRLs, but not in LRLs — in which case, a reasonable approach might be to terminate HRL training early (i.e., a form of curriculum learning). We leave this experiment to future work, and now address the point in Section 4.3, line 426.
4. We conducted most experiments with PaLM2 XXS–NTL as it was the best performing model. Should the work be accepted we will include the corresponding ablation results for the best performing Gemma model in the Appendix.
5. Thanks for suggesting to compare the distilled mufu20 model against a vanilla distillation setup. We note, however, that the current teacher model (PaLM2 S) is poor in translations to low-resource languages. We are nonetheless happy to consider other teacher models, and include this result in the Appendix of the final manuscript if required.
6. Thanks for raising that there might be accidental translations in the wrong language. We manually inspected a number of Indonesian and Chinese languages, and found no absolute incorrect language in the translations. It is however difficult to assess the accuracy systematically as many of these languages share the same scripts, have similar vocabulary, and borrow words from one another (see for example, Table 4). For other language pairs with <20 chrF (e.g., Fon, Tamasheq, Tigrinya), we could only perform sanity checks for accidental translation in the wrong script, as we are unfamiliar with these languages.
7. Win rate is the percentage of language pairs where the model outperforms a benchmark. For example, NLLB 54B MoE outperforms the teacher in 113/201 ≈ 56.2% language pairs based on chrF; whereas PaLM2 S finetuned with mufu20lora outscores the teacher in 199/201 ≈ 99% language pairs. We now clarify this in Section 4, line 197.
8. Thanks for raising that lines 207-211 could be further improved. In Mufu we finetune the models against the gold-standard translations, and expect improvement from the postediting targets. This is effective for LRLs with low-quality postediting candidates. However, improving high-quality translations in HRLs is harder and requires the student model to also learn the subtle differences between model- and human-generated output. It is also possible that the LLM teacher surpasses human for some translations in HRLs, in which case, learning from the human output could be detrimental. We now elaborate this cause of decline in performance relative to teacher in Section 4, line 263.