Fast-ELECTRA for Efficient Pre-training

Thank you for the valuable feedback. Please see our detailed response below. **___** __Q. Applicability of ELECTRA-style pretraining__ We note that ELECTRA is widely applied in training encoding models and also among the state-of-the-art. Our method can further extend its applicability as it reduces the training cost and memory consumption of ELECTRA to be comparable to other pretraining schemes such as Mask Language Modeling (MLM). Furthermore, we firmly believe that the ELECTRA-style pretraining, which can achieve huge performance improvement over MLM with the same model size and training data, still has lots of merits that are worth studying, not only for the pretraining of encoding/decoding language models but also for the pretraining of large models in a wider context. **___** __Q. Correlation with continual learning__ We appreciate the reviewer for bringing this work [1] to our attention. We believe there are indeed many correlations between ELECTRA-style pretraining and continal learning. On one side, existing continual learning methods can be great motivations for better designs of ELECTRA-style pretraining. Since strong auxiliary models can often hurt the performance of the main model in ELECTRA-style pretraining, it is necessary to design an auxiliary model training scheme that is aware of the main model learning. This requires the auxiliary model to be able to generate meaningful token replacements while not too difficult to ruin the main model learning. This is highly related to the idea proposed in the work mentioned by the reviewer, namely to adapt existing language models to both preserve necessary general knowledge and incorporate domain-specific knowledge. On the other side, ELECTRA-style pretraining may also be adapted for the continual learning field. For example, one may consider using a small auxiliary model trained on the specific domain to better adapt the main model while preserving necessary general knowledge. We will include the above discussion in the revision. **___** __Reference__ [1] Adapting a Language Model While Preserving its General Knowledge, Ke et al., EMNLP 2022

Thank you for the valuable feedback. Please see our detailed response below. **___** __Q. Larger models__ We believe for encoding models, the BERT-base and BERT-large equivalent model sizes, as covered in this paper, are among the most widely used ones. Existing works have also mostly adopted such model sizes [1, 2]. Nevertheless, we note that our method can extend the applicability of ELECTRA for training potential large-scale models, as it reduces the training cost and memory consumption of ELECTRA, such that it is now comparable to other popular pretraining schemes such as Mask Language Modeling (MLM). **___** __Reference__ [1] RoBERTa: A robustly optimized BERT pretraining approach. Liu et al., 2019. [2] DeBERTaV3: Improving Deberta Using Electra-style Pre-training with Gradient-Disentangled Embedding Sharing. He et al., 2021.

Thank you for the valuable feedback. Please see our detailed response below. **___** __Q. Efficiency improvement over existing methods__ Among Table 1 of our paper, those methods that follow the ELECTRA-style pretraining framework are the most relevant to this paper, including MC-BERT [1], COCO-LM [2], AMOS [3], DeBERTaV3 [4], and METRO [5]. For a brief introduction to these methods, one can refer to the related work (Section 6) in our paper. Nevertheless, note that all these works train the generator jointly with the discriminator. To the best of our knowledge, there is no prior work using a fixed generator, potentially because it will fail to match the performance achieved by joint training, as also shown by Section 5.2 in our paper. Therefore, we believe the major contribution of our proposed method, namely using a fixed generator for ELECTRA-style pretraining without significant performance loss, is **orthogonal** to prior works. Existing ELECTRA-style pretraining methods can be modified with a fixed generator and achieve similar efficiency improvement. Finally, regarding the accuracy-efficiency trade-off, we didn’t observe significant performance loss by using our method to improve the training efficiency. Our method is still among the state-of-the-art in terms of downstream performance, as shown in Table 1 of our paper. Nevertheless, it may be indeed slightly worse compared to the most recent work mentioned by the reviewer, as discussed below. **___** __Q. Comparison to a recent work__ We appreciate the reviewer for bringing up this recent work [6]. We note that our paper is different from [6] in terms of both the motivation and the methodology. In terms of motivation, [6] aims to improve the effectiveness of the ELECTRA-style pretraining, by controlling the training trajectory of the generator for better discriminator learning. In comparison, our work aims to improve the training efficiency of the ELECTRA-style pretraining, by reducing the training cost of the generator. In terms of the methodology, [6] decouples the optimizers of the generator and the discriminator, while the generator is still jointly trained with the discriminator. In contrast, Fast-ELECTRA completely decouples the generator and discriminator training. As a result, the generator can be trained separately, and only needs to be trained once and reused for the discriminator training. Finally, we note that there is indeed a gap between the performance of our method and that of [6]. We agree with the reviewer that it would be tempting if such a gap could be reduced while the auxiliary model is still fixed, such that we can achieve efficiency and performance at the same time. We believe this is possible by designing a better learning curriculum but we would like to leave it as a future work. We will include the above discussion in the revision per the reviewer’s suggestion. **___** **__Q. Training cost of the auxiliary model__** We note that Fast-ELECTRA will reduce the overall training cost even if one pre-trains the auxiliary model themself. First, the practical development of language models often requires multiple rounds of trial and error to locate the best architecture setup and hyperparameter combinations. With the joint training design in the original ELECTRA, the auxiliary model will be trained and discarded repeatedly in each round, resulting in significant resources being wasted. In contrast, with our method, one only needs to train the auxiliary model once and can reuse it for subsequent training rounds of the main model. Furthermore, we note that Fast-ELECTRA can reduce the memory cost of ELECTRA-style pretraining as well, which also improves the training efficiency. In fact, large-scale language model pretraining, calling for larger model sizes and longer sequence lengths, is probably more often bottlenecked by memory consumption [7]. With the joint training design in the original ELECTRA, the auxiliary model will consume significant memory in addition to that of the main model, which limits the applicability of ELECTRA-style pretraining in many memory-constraint scenarios compared to other pretraining frameworks such as Masked Language Modeling (MLM). In contrast, with our method, the auxiliary model’s memory cost during main model training can be significantly reduced or even eliminated (with offline processing), which can potentially make ELECTRA more memory-efficient than MLM, and in turn, can also reduce the training time as one can now fit more training data into the memory.