Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts

Response from authors

We thank the reviewer for your thorough response. Before we clarify you concerns in the below section. We want to inform that, with new experimental results presented below, our method also works for language-understanding and pure knowledge-based question answering with XQUAD and no-context TydiQA benchmarks. We use LDP with En,Vi,Zh for XQUAD and En,Id,Ko for TydiQA. |XQUAD/chatgpt/F1 | Arabic | Hindi | Thai | --- | --- | --- | --- | | 0-shot | 52.91 | 45.96 | 26.34 | 3-shot | 69.96 | 69.34 | 53.80 | 0-shot-LDP(EnViZh) | 69.80 | 68.94 | 53.98 | TydiQA/chatgpt/F1 | Arabic | Bengali | Finnish | Russian | --- | --- | --- | --- | --- | | 0shot | 19.17 | 5.72 | 21.79 | 12.31 | 3-shot | 27.78 | 20.21 | 34.72 | 16.82 | 0-shot-LDP(EnIdKo) | 23.21 | 18.90 | 32.60 | 16.97 Let us clarify your concerns: 1. **Our paper does not, in absolutely anyway, claim to invent back-translation!** We cited numerous back-translation papers (Edunov et al., 2018; Lample et al., 2018; Conneau & Lample, 2019; Liu et al., 2020; Nguyen et al., 2022b). (Sennrich et al., 2016) was the first, and has been improved and refined in various works we cited above. Each of them is better than (Sennrich et al.) or complement it in other settings. **We thank the reviewer for bringing up this well-known work, and we will cite (Sennrich et al., 2016) in the paper!** But we would like to emphasize that we initially skipped its citation in the same way many papers nowadays have also stop citing (Vaswani, et al, 2017) Transformer paper due to its utmost popularity, but instead cite other improved versions, and not because we are not aware of it. We sincerely plead that missing an amendable citation is not punishable with rejection. 2. Our work tackles the questions of **(1) How to do back-translation with LLMs (2) How to do back-translation without any training because some LLMs are closed-source (3) How to do it with zero or just 3 unlabelled data samples from the target languages. (4) How to do it when the pretraining data for a target language is possibly less than 1MB.** 3. Training on synthetic back-translation data is a well-established idea and this is not our novelty claim. But how to create those data depends on settings (unsupervised or supervised), constraints (data size), tasks (summarization or translation), or target languages (high or low-resource, Latin or non-Latin), and have been explored in the many works we cited. Our work explore BT in the conditions explained above. 4. **Ablation study**: **Table 4a** examines the different types of prompts, language tags and with/without synthetic backtranslation. **Table 4b** explored different language choices, from the number of languages to use, which groups of languages (distant or close), and repeating the same languages. We also conduct other ablation studies in **figure 3, figure 4, table 5, figure 5, and figure 6**. 5. **Figure 1b vs 1c:** **Table 4a** compares Fig. 1b (without BT) vs Fig. 1c (with BT) and shows that Fig. 1c outperforms Fig. 1b greatly for En-->X direction, but observes no difference for X-->En direction. This is consistent with the explanations in figures 1a, 1b, and 1c. Please note that Fig. 1b is **not** standard supervised few-shot prompting, but our LDP method **without** synthetic intra-lingual backtranslation data. 6. **Why BLOOM?** Because LDP is a fully unsupervised prompt technique, as we explain in page 6, we do not want the examined models to be contaminated with supervised data and especially test sets. BLOOMz was trained on test sets! InstructGPT experiments are to show that our method works on instruction-tuned models. We also tested on GPT-3 base and observe similar results, but its performance was much worse than BLOOM in absolute terms for these low-resource languages across different settings. We explain GPTs' weaknesses in figure 6. 7. **Finetuning data size:** sizes are very small! **<500-1000** samples for extremely low-resource languages like Tsonga, and **~50K** samples for more popular ones like Hindi. We understand BT needs millions of samples in data size, but it was difficult to obtain such data for low-resource languages. We will put these details in paper. Given our explanations and additional experimental results, we sincerely hope the reviewer will reconsider your evaluation. If you have other questions or requests for additional experiments, we will try our best to accommodate and perform such experiments.

New results on instruction-following and a humble request for re-evaluation

We thanks the reviewer for the score change and response! We deeply appreciate it. However, in our all-out effort to **win your acceptance rating**, we provide **2** more updates in our humble hope that you would consider our work if there is no critical issue. 1. We add new experiments showing that our LDP prompting method can also perform **instruction following** (like ChatGPT) with a base pre-trained model **without any supervised fine-tuning**. This contributes to the completeness of our paper (in addition to question answering, translation and summarization). Please see the details below as well as the general comment above. 2. **We follow your advice to discuss extensively about back-translation** in the camera-ready updated paper once accepted. Particularly, we will rewrite and/or add the following: **In the introduction:** Our LDP prompting technique is a simple application of back-translation (Sennrich et al., 2016) for the translation tasks under the extreme constraint where we can only use a general LLM without substantial amounts of parallel or unlabeled data for the low-resource languages, and especially where we cannot train or fine-tune the LLM if it is closed-source (OpenAI's GPTs) or too large (~175B params). **In the related work:** Our LDP prompting technique applies the concept of back-translation (Sennrich et al., 2016) in a novel situation of low-resource unsupervised translation, where a general pre-trained language model is used on low-resource languages without any further training. There are major distinctions between our method and standard back-translation (Sennrich et al., 2016). The first one is that standard back-translation requires two backward and forward supervised translation models to generate a huge amounts synthetic data from unlabeled target-language data. This renders the method inapplicable in our setup because only one generic language-model is available and no parallel or huge unlabeled data is available. The second difference is that our LDP method does not require any training to work well and use less than 8 unlabeled data samples. The third one is that our LDP method only use one-way back-translation (X->En) for both En->X and X->En translation directions, in order to avoid poor generative performance in low-resource languages. **New results for instruction-following tasks** We show that our LDP prompts can also do **instruction-following** (like ChatGPT) using the base pre-trained model **WITHOUT any supervised fine-tuning or RLHF**. Along with other tasks like **question answering, translation and summarization** that we have added in the paper and author-responses, we show that our LDP method is universally effective for low-resource languages in many kinds of tasks. For this experiment, we obtain ~200 instructions across different task-types and categories (task solving, natural questions) and translate them into Vietnamese and Indonesian and test them with Llama-2 chat and base models. We use GPT-4 to evaluate the chat responses following [MT-bench framework (Zheng, et al., 2023)](https://arxiv.org/abs/2306.05685). We select 4 random instructions from En, Zh, Fr, Ru as LDP prompts. Scores results are shown in the attached anonymized image. We also display the results in numeric terms in the table below. [https://i.postimg.cc/xT4jwXS3/fig-ldp-bench-category-en.png](https://i.postimg.cc/xT4jwXS3/fig-ldp-bench-category-en.png) | Model & Prompting | Vi/task solving | Vi/math reasoning | Vi/instruction following | Vi/natural QA | Id/task solving | Id/math reasoning | Id/instruction following | Id/natural QA | --- | --- | --- | --- | --- | --- | --- | --- | --- | | ChatGPT(3.5) | 7.47| 7.70| 9.42| 9.05| 7.85| 7.40| 9.80| 9.45 | Llama2-13B-Chat | 6.45| 3.50| 6.15| 4.95| 5.45| 4.25| 7.67| 5.65 | Llama2-13B-**Base** w/ *LDP* | 3.87 | 1.94 | 4.65 | 4.80 | 2.61 | 1.75 | 7.05 | 6.10 | Llama2-13B-Chat w/ *LDP* | 3.83 | 2.78 | 8.57 | 4.94 | 6.54 | 3.06 | 8.72 | 6.85 As shown, our method can invoke relatively good instruction following capability in Vi and Id even with a **BASE** model. With Llama-2-chat, which has been trained with SFT and RLHF, our method can further improve the performance in various benchmarks.

Response from authors

We thank the reviewer for your thorough response. Before we clarify you concerns in the below section. We want to inform that, with new experimental results presented below, our method also works for language-understanding and pure knowledge-based question answering with XQUAD and no-context TydiQA benchmarks. We use LDP with En,Vi,Zh for XQUAD and En,Id,Ko for TydiQA. |XQUAD/chatgpt/F1 | Arabic | Hindi | Thai | --- | --- | --- | --- | | 0-shot | 52.91 | 45.96 | 26.34 | 3-shot | 69.96 | 69.34 | 53.80 | 0-shot-LDP(EnViZh) | 69.80 | 68.94 | 53.98 | TydiQA/chatgpt/F1 | Arabic | Bengali | Finnish | Russian | --- | --- | --- | --- | --- | | 0shot | 19.17 | 5.72 | 21.79 | 12.31 | 3-shot | 27.78 | 20.21 | 34.72 | 16.82 | 0-shot-LDP(EnIdKo) | 23.21 | 18.90 | 32.60 | 16.97 Let us clarify your questions: 1. **Results are reproducible!** We stated that in the paper **"For each of the 68 language pairs, we sample randomly and evaluate 200 sentences from each test set with the same seed to limit the cost of API calls"** . The seed is **0**! We also used seed **0** to collect the prompts for supervised prompting (A.2). In short, **every test set in the paper is the same across all experiments and baselines**. 2. **Linguistic diversity**: As explained in the paper, the name "linguistically diverse" simply means we use prompts from different languages and distant from the target language, and not going any deeper or further into sophisticated aspects of linguistics, such as phylogeny. 3. *Selecting the optimal languages* is a combinatorial search problem with search space expands in factorial order! The paper's main point is to use prompts from diverse languages for many downstream tasks, while leaving the optimal search problem for future work. Thus, we only conduct experiments with languages choices based on common knowledge, such as "English is different from Chinese". We will amend the paper to reflect this point. 4. **Table 4b explains the guidelines for language selection process empirically**. Our method work best if all languages in prompts are *different from each other* and **should be distant from the target language** we want to solve tasks in. 5. Other than that, **as we have tried different combinations of language choices**, there is no significant difference in terms of performances. The choice of Ar,Zh,Vi,Fr is totally a random decision by the authors at the beginning. As we tried other language sets and experiments in Table 4b, we believe will get roughly the same results as long as the guidelines above are followed. For questions: 1. LDP itself is **truly a fully and completely unsupervised** method, both from prompting or data collection perspectives. The only thing that makes the entire process not *fully unsupervised* is which underlying model is used. InstructGPT is not an unsupervised model and we do not know which data it was trained on. Still, we demonstrate that our method can improve InstructGPT zero-shot performances for the target language where we do not have any supervised data at hand! 2. Which language is high or low resource is subjective. In our paper, we assume language with < 0.1% pretraining data or <1GB data size from the CC100 corpora as very low-resource. 3. **Democratizing**: our method is a simple strategy that can boost LLM's performance for low-resource languages that we would not have been able to achieve with naive zero-shot prompting, and without supervised data from those low-resource languages. For example, practitioners can build a simple summarization app for Swahili-speaking community, without ever needing to collect any supervised Swahili data, which would be hard to collect. In this way, under-represented communities can benefits from existing LLMs without needing to invest in human annotations, training computations.

Response from authors

We thank the reviewer for your thorough response. Before we clarify you concerns in the below section. We want to inform that, with new experimental results presented below, our method also works for language-understanding and pure knowledge-based question answering with XQUAD and no-context TydiQA benchmarks. We use LDP with En,Vi,Zh for XQUAD and En,Id,Ko for TydiQA. |XQUAD/chatgpt/F1 | Arabic | Hindi | Thai | --- | --- | --- | --- | | 0-shot | 52.91 | 45.96 | 26.34 | 3-shot | 69.96 | 69.34 | 53.80 | 0-shot-LDP(EnViZh) | 69.80 | 68.94 | 53.98 | TydiQA/chatgpt/F1 | Arabic | Bengali | Finnish | Russian | --- | --- | --- | --- | --- | | 0shot | 19.17 | 5.72 | 21.79 | 12.31 | 3-shot | 27.78 | 20.21 | 34.72 | 16.82 | 0-shot-LDP(EnIdKo) | 23.21 | 18.90 | 32.60 | 16.97 Let us clarify your questions: 1. As demonstrated in **table 4b**, our method work best if all languages in prompts are different from each other and **must be very distant from the target language** you want to solve. Other than that, **as we have tried different combinations of language choices**, there is no significant difference in terms of performances. 2. Number of languages or number of examples are limit by the context length, which is around 6-8. **table 4b** also shows that there is no difference if we put 3 (Fr,Es,Pt) or 7 (Ar,Fr,Es,Pt,Vi,Zh,Id) languages in the prompt. Meanwhile, repeating the same languages (Zh,Zh,Zh,Zh) may hurt performance. 3. Yes, we use the **same set of prompts for every test instance, across all languages**. For selecting back-translation data, we also use the same **seed 0** to randomly select data from the train/valid set. For test set, we also use seed 0 to select 200 samples to ensure **all 200 samples being tested are exactly same!** 4. *Diverse vs Synthetic intra-lingual Prompts*: **table 4a** explains the requested results, indicated as LDP *without* vs *with* intralingual back-translation prompts. In short, there is no difference between them for X-->En direction, but ones with synthetic back-translation significantly outperforms the other and match supervised prompting for En-->X direction. 5. Prompt selection: We randomly pick unlabeled sentences from CC25 dataset for the diverse languages, and used the same ones through the entire paper. We have to tried different random selections and found results are on average the same. As indicated in the paper that the setting is about low-resource languages, so we assume no optimal prompt selection technique is used and **random selection** is the only default choice.