Order Matters in the Presence of Dataset Imbalance for Multilingual Learning

Summary

This paper presents a simple and effective multi-task learning strategy of a joint pretraining followed by fine-tuning, where pretraining is on the high-resource task and fine-tuning is on a mixture of high and low-resource tasks. This significantly improves performance on the low-resource tasks, while performing at par or even better sometimes on high-resource tasks. The authors show results on a machine translation task consisting of two high-resource and two low-resource language pairs and a language modeling task trained on five different languages.

Strengths

- The proposed idea is attractive in its simplicity, and offers good validation loss reductions on translation and language modeling tasks. - The authors provide a fairly extensive empirical analysis of the proposed approach, and offer good baselines to compare against. - The paper is written clearly and the overall narrative flows well.

Weaknesses

While the proposed approach has been empirically validated with multiple experiments, there remain unanswered questions about the initial pretraining on a high-resource language: - How important is the pretraining objective? - What is the influence of the chosen high-resource language on final performance on low-resource languages? - Why does pretraining longer improve the loss for high-resource NMT tasks but degrade the loss for high-resource language modeling tasks (c.f., Figure 9(a) and 9(b)).

Questions

- In Figure 9a, pretraining longer on English seems to benefit Gujarati but not Hindi. Any thoughts on why this might be? - From line 260, "Longer pre-training helps more in terms of performance than exposure to the tokens of the low-resource task" -- This claim demands further probing. Is this dependent on the choice of high-resource language for pretraining? This claim holds for Gujarati but not Hindi. Are there features of the low-resource task (e.g., type-token ratio in the unlabeled text for the low-resource tasks) that makes transfer from the high-resource task more effective? - In the many-task setting, as in Section 3.2, would it be beneficial to introduce an intermediate pretraining stage between pretraining and finetuning? Such strategies have been shown to be effective in prior work (e.g., Phang et al., " Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks", 2018). Also, such works on intermediate labeled-data tasks should be cited in the related work section. - The authors should move up the BLEU results from the Appendix to the main draft. For NMT, it is important to show that validation loss reductions do translate to BLEU score improvements (especially for English->Hindi). - A suggestion: Since pretraining is usually associated with self-supervised objectives, it would be useful to explicitly clarify that the pretraining and finetuning objectives are both cross-entropy losses for the NMT task.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations have been adequately addressed.

Summary

The paper proposes a method called pretraining and joint-finetuning, which pretrains a model on a high-resource task than finetunes the model with joint high- and low-resource tasks, benefiting from both static sampling method and naïve transfer learning. The method is verified on multilingual translation and multilingual language modelling. Experiments show improvement on the two scenarios in term of validation loss.

Strengths

1. The proposed method is simple and easy to implement/reproduce. 2. The method can achieve lower validation loss compared to static sampling and naïve transfer learning. 3. The method verified on multilingual (> 2) setting and extremely resource-imbalanced (English is 16745 times larger than the smallest language) setting.

Weaknesses

1. The algorithm is sensitive to the sampling proportion, resulting a need to search proportion for every model trained. In addition to searching the proportion of dataset size, there is a need for grid search of N1 and N2 (N1 + N2 = N). The hyperparameters (proportion = 0.4 for two tasks, tau = 3.33 for > 2 tasks, and joint-finetuning steps = 50k) are not verified among different tasks. 2. Experiments are done only on multilingual setting, whose scope is a bit different from the title “multitask” learning. Is the proposed method still effective on multitask like NER + POS + sentiment analysis? 3. The improvement on performance is not significant when the evaluation metric is BLUE (from Fig. 26).

Questions

1. Definition of Pareto optimal points. According to the paper, “θ is Pareto optimal if it is not dominated by any other point”. However, for a parameter family for a certain architecture, it is impossible to test all parameters in the whole space. In this case, is it rigorous to say that a parameter θ is not dominated by any other parameters? 2. Inadequate references. There are plenty of papers on the catastrophic forgetting phenomenon in multilingual/multitask learning. I think the claims in this paper might been seen from others. For example, the “order matters” claim has been found in “Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation”. The authors should check previous literature more carefully. 3. Scalarization implementation. In Line 72 of the paper, the static sampling is implementing by adding weights to the losses from different tasks (languages). Will the models trained with different proportions be fed different data? For example, suppose the En->Ro data is fixed, will a model with 0.1 En-Ro proportion be fed more En data than a model with 0.9 En-Ro proportion? If the answer is no (i.e., the data is the same, the only difference is the weights on losses), does the method have a same effect with sampling data in each batch? 4. Why the Figures do not always show all results of the ten proportions (e.g., Fig. 2 Left)? Minor suggestions (mainly for presentation): - I think having a separate section about the proposed method will be better. - For figures of the loss that need a comparison (e.g., Fig. 3 and 8), the illustration will be clearer if the scale of the two subfigures is consistent. After Author Response: I strongly encourage the authors to address Questions 1 and 2, specifically the clarification of Pareto dominance and a more comprehensive discussion with previous literature. These enhancements would undoubtedly contribute to the rigor and solidity of the paper.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors consider an interesting possibility of multiple high-resource languages setting as a future work, which can be seen as a partition of tasks. However, they do not provide a set of hyperparameters effective across tasks, e.g., at least how many steps of joint-finetuning should be taken? Which sampling proportion should we use when training with new data/tasks?

Summary

The paper presents a multi-task training approach that involves pre-training on high resource tasks, followed by joint fine-tuning on the full set of tasks. The intuition is that high-resource tasks need a larger number of steps to reach convergence, whereas low-resource tasks will risk over-fitting if trained for that same number of steps. The authors show that the proposed approach improves the performance for neural machine translation of low-resource languages (en-ro and en-hi, pretrained on en-fr) and similarly improves performance for multilingual language modelling for the case of low-resource languages.

Strengths

* The authors show that _pretraining joint-finetuning_ is a simple approach to boost low-resource task performance in multi-task training. The experiments, which focus on language modelling and machine translation, are clearly designed and presented, with reasonable baselines. * In this proposed training approach, the pre-training phase ends up taking up most of the training time. Since pre-training is conducted on the subset of high-resource tasks, this makes it an easier optimisation problem compared to the standard approach where all tasks are trained. The potentially trickier joint finetuning step requires overall fewer steps. Hyperparameter sweeps could plausibly be limited just to this final step, making them cheaper and faster compared to the baseline joint training approach. * Between the paper and the appendices, enough details are given so that reproducing these run should present no issues.

Weaknesses

* I would have loved to see some more realistic MT experimental settings. Pretraining on en-fr for either en-zh or en-hi (or viceversa) feels very unrealistic. There are much closer language pairs that could have been used, which would have made the results more convincing. * I am concerned about the results of the experiments of section 3.1.1. What's the effect on BLEU scores here? It would have been good to include these in Appendix A5, as was done for the other two language pairs. Minor: * Line 61: _if_ is attached to $\forall$ * Line 171: shows -> show * Nitpick: In the abstract, the wording "experimental settings which range from neural machine translation (NMT) to multi-lingual language modeling" seems a bit out of place. These are exactly the two settings on which the proposed approach is being evaluated, there aren't any other experimental settings that fall within these two.

Questions

* Line 128 refers the reader to Figures 21 and 22 in the appendix to show that the proposed approach does not hurt performance in the high-high case. It's not clear to me how the said figures show this. The clearest evidence for me would have been a version of Figures 26 and 27 for the English+Chinese case. Would you be able to provide such a plot?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors do a good job of listing and addressing limitations.

Summary

This paper presents an analysis of multi-task training with the existence of low-resource tasks (in the context of the paper, they focused on multi languages). The paper argues that it is better to first "pre-train" on a high resource task and then "fine-tune" on a joint of high and low resource tasks. Experiments were conducted in neural machine translation and multilingual language modeling to support the hypothesis.

Strengths

The proposed hypothesis intuitively makes sense and the experiments have been conducted in a careful setting. The proposed solution is simple, but has its effectiveness verified in NMT and language modeling. The paper has a smooth flow and is easy to understand.

Weaknesses

1. While the proposed solution is effective, the benefits of having a multilingual model (or in general a multi-task model) is that a single model can be used in various situations, without the need of (continue) training on different data splits and producing several models. I feel at the current stage the application of this finding might be limited.

Questions

The paper focuses on a specific multi-task setup of multilinguality, however, the finding/hypothesis doesn't seem to have strong ties with multilingualityThe paper presented some limitations on scaling the tasks and settings to be analyzed., and is therefore not necessarily confined to this setting. I am curious if the authors conducted any experiments on the the general multi-task setup.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The paper presented some limitations on scaling the tasks and settings to be analyzed.

Summary

This paper presents an empirical finding: in the context of data imbalance in multi-task learning, pre-training on high-resource tasks then fine-tuning on a mixture of high/low-resource tasks can achieves superior results, compared to standard weighted sampling training. Authors applied the proposed training method *pre-training joint fine-tuning* to neural machine translation (NMT) and multi-lingual language modeling. Both experimental results shows that the proposed method allows the high-resource task to converge faster while preventing overfitting in low-resource tasks.

Strengths

- Achieving a good balance between low-resource and high-resource languages during multitask training is a critical research area, particularly in the context of large-scale language modeling. The proposed approach of pre-training followed by joint fine-tuning is a timely solution that addresses this challenge. - The results obtained from the empirical study demonstrate the superiority of the pre-training joint fine-tuning approach over the conventional method of weighted sampling during training. The model trained using this approach consistently outperforms existing methods, showcasing its effectiveness in addressing the challenges posed by data imbalance in multi-task learning. - An additional ablation study showed that the pre-training joint fine-tuning has a regularization effect (to avoid overfiting to low resource language). This improvement cannot be achieved by simply increasing drop-out ratio.

Weaknesses

- In the main paper, authors only reported perplexity metrics in the evaluation results. For multi-lingual language modeling task, I would like to see the evaluation on downstream tasks as in the mT5 paper. It would be helpful to demonstrate the impact of the proposed method if authors can show the improvement in downstream tasks on XQuAD or XTREME benchmark.

Questions

Why temperature based sampling is only applied in the language modeling task not the NMT task? What is the advantage of temperature based sampling compared to scalarization in the large scale dataset?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

I am curious to see what if the authors applied this method in a more realistic setting: training LM on the whole mC4 dataset, and then evaluate the performance (low-resource language vs high-resource language) on different downstream tasks as in mT5.