Improving Language Plasticity via Pretraining with Active Forgetting

Summary

This paper proposes active forgetting, a rather straightforward method that resets the embedding layer every K updates during pretraining, to quickly adapt PLMs to new languages. Through experiments on different language pairs with RoBERTa, the authors claim that the proposed method can induce faster convergence and better performance when the languages are distant from English.

Strengths

- This paper is well-written in general, with a clear motivation and some novelty. - It is appreciated that the author conduct experiments on many languages from similar languages like German to distant languages like Thai. - The method itself is counter-intuitive, as it resets the embedding layer periodically which intuitively can be bad for PLMs, but effective to improve the cross-lingual transfer. - The authors give in-depth analysis and insights into their proposed method, which I believe is interesting to the community.

Weaknesses

Some figures are inconsistent and visually not good. For example, the upper three subfigures in Figure 4 have a visually bad axis scale. Some values are rounded to integers while some are not, as in Figure 6. The authors should carefully redraw and prettify them in the camera-ready version.

Questions

- In the abstract, the paper claims that it is data and compute inefficient to learn a new embedding layer. But this paper does not address this problem, since a new embedding layer for new languages still has to be learned. Therefore I don't think it is necessary to mention that in the abstract. - The paper claims the method ensures high sample efficiency. How does the experiment support this? - Forgetting is “generally” a bad thing for PLMs, but this work counter-intuitively shows that active forgetting can be beneficial for cross-lingual transfer. The authors are encouraged to give more intuition or explore the reasons behind this phenomenon, in a cross-lingual perspective. - For the experiment w.r.t. RQ3, the authors show that active forgetting is particularly helpful when the new languages are (typological) different from the pretraining language. However, I see also an important possible influencer: the script. The authors are encouraged to also explore if scripts can influence the transfer performance under active forgetting. - The frequency of active forgetting is set to K=1000. As this is an important hyperparameter, I would encourage the authors to justify their choice. - I would be very interested to see a plot of loss over updates during language adapting. I would expect the loss to go down in general but fluctuate a lot every time when active forgetting is used. - It’s natural to think that this active forgetting could be used in other parts of the models instead of merely on embeddings. Therefore it would be very interesting to explore: active forgetting on which part of the models is the most effective?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors include a section of Limitations in the paper, where they mention that this paper only focuses on the simplest forgetting, which is only applied to the embeddings.

Summary

The paper proposed an active forgetting mechanism for PLMs pre-training for cross-lingual transfers and adaptations. The authors propose a multi-stage adaptation framework for better cross-lingual transfer/adaption: 1) First, by resetting embedding layers every K update during pre-training for monolingual RoBERTa model; 2) Second, by adapting parameters separately (embeddings vs backbone) for language-specific knowledge and task-specific knowledge (language+task adaptation). The paper shows the proposed mechanism helps faster convergence during cross-lingual transfers and adaptation as well as achieving better downstream task performances.

Strengths

* The paper shows an interesting empirical finding that resetting partial parameters (in this case embeddings) during pre-training in English leads to better transfers/adaptation in cross-lingual settings. * The paper is clear-written and the proposed method is effective empirically for cross-lingual transfers.

Weaknesses

- Although the proposed method achieved good performances in downstream tasks, this work lacks proper ablation experiments on showing the gain is from resetting during pre-training, vs the gain of adapting/resetting parameters separately during the language+task adaptation stage. I.e. the paper should include the results of the following 4 types of experiments: * Standard pre-training + standard adaptation * Forgetting pre-training + standard adaptation * Standard pre-training + language/task adaption * Forgetting pre-training + language/task adaption - Practically, efficiency (parameters and training) could be an issue with the language adaptation + task adaptation issue.

Questions

* See weaknesses. * Have you tried resetting other parameters during pre-training? What about other PLMs? e.g. Does deBERT style training affect the results? * References: 1. Chen Liu, Jonas Pfeiffer, Anna Korhonen, Ivan Vulić, and Iryna Gurevych. Delving Deeper into Cross-lingual Visual Question Answering. Findings of EACL, 2023 → Partial resetting, and re-initialization of parameters for cross-lingual generalization in VL setting. 2. Vijaya Raghavan T Ramkumar, Elahe Arani, Bahram Zonooz, Learn, Unlearn and Relearn: An Online Learning Paradigm for Deep Neural Networks, TMLR, 2023 → Selective parameters resetting for continual learning (online and few-shot).

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

n/a

Summary

This work introduces a training technique that leverages actively resetting token embedding to improve zero-shot language transfer. Experiments on RoBERTa show consistent improvement in multiple languages, distant languages in particular.

Strengths

1. Simple and innovative approach. 2. Consistent improvement across languages and tasks. 3. Insightful analysis.

Weaknesses

1. Only experimented on one pretrained model.

Questions

How sensitive is this method to the choice of forgetting frequency?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper follows the language adaptation procedure of MonoTrans (Artetxe et al., 2020), and proposes a new pre-training method with active forgetting. By resetting the embedding layer every K updates during training, the language model learns to learn the new embedding fast, similar to a meta-learning effect. The paper conducted experiments on several languages, and evaluate the adapted models on zero-shot XNLI, MLQA and XQuAD. Unlike the experimental setting of MonoTrans, the paper considers a low-resource pre-training setting where an "unseen" language has as few as 5 million tokens for the adaptation step. The results show that the pre-trained models with active forgetting pre-training converge quickly during language adaptation, and outperform the baseline models.

Strengths

The paper studies an important research question, i.e., language adaptation in a low-resource setting. Since the success of language models relies on large-scale pre-training, how to adapt the language models to low-resource languages without large-scale training data is beneficial for low-resource NLP research. The proposed active forgetting is simple and effective for the following language adaptation step. Experiments demonstrate that it outperforms baseline models for zero-shot classification and question answering tasks.

Weaknesses

- I am confused about goals and means: - The goal of language adaptation / MonoTrans (Artetxe et al., 2020) is to transfer existing high-resource language models (English in particular) to unseen languages, and the method is to learn language-specific word embeddings, i.e., language adaptation. - The goal of this paper seems to improve language adaptation results, rather than the goal behind language adaptation. The key is, transferring the currently existing models rather than pre-training new models. I understand the proposed method can benefit language adaptation. However, it greatly limits the scope, only the pre-trained models with the active forgetting can have this effect, i.e., currently existing pre-trained models are out of scope. - Experimental setup. As mentioned in L273, the paper claims that multilingual pre-trained language models have issues that they need large corpora. Since many studies have shown that multilingual models have cross-lingual transferability, training multilingual language models in the low-resource setting should be a baseline. - It would be great to provide the pre-training loss curves, showing how active forgetting affects the pre-training procedure. - (Minor) L192, Page7 footnote, missing ".". Figure 6, "XNLI Accuracy vs Adaptation Steps" is too small.

Questions

- If active forgetting even requires a re-pretraining step, how to adapt currently existing pre-trained models without re-training them?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

N/A

Summary

This paper presents an embedding forgetting mechanism for pre-training, aimed at enhancing robustness in downstream shift embedding fine-tuning. Focusing on the low-resource regime, the study conducts experiments on 10 simulated low-resource languages across three tasks: XNLI, XQUAD, and MLQA.

Strengths

(1) This paper introduces a simple yet effective method, akin to a form of regularization, for improving the multilingual capabilities of a PLM. It demonstrates impressive performance when adapting to downstream X-tasks. (2) The writing is clear, and the illustrative figures are informative.

Weaknesses

(1) The paper is somewhat limited in scope, as it only applies to low-resource multilingual tasks within the framework of Artetxe et al. [2020]. (2) To propose a simple yet effective method, I believe solid experiments or rigorous proof are necessary; however, both are missing. In the experimental section, there is a lack of comprehensive ablation studies (e.g., why set low-resource data at 5M? Is this the real scenario in the multilingual setting? Why not use actual low-resource data instead of simulating one, considering that real low-resource data is different from sampled ones? How does the update frequency affect the results?). (3) The paper lacks comparisons with other established methods, such as multilingual pre-training, multilingual adapters, and multilingual regularization techniques, among others.

Questions

(1) Did you try different hyper-parameters for standard PLM and forgetting PLM when fine-tuning? (2) Did you tie the weights of embedding and LM head? Thanks for authors' detailed rebuttal, which addressed most of my concerns. However, the weaknesses are still existing. I'll keep my score.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have discussed the limitations.