Efficient transfer learning for NLP with ELECTRA

Clark et al. [2020] claims that the ELECTRA approach is highly efficient in NLP performances relative to computation budget. As such, this reproducibility study focus on this claim, summarized by the following question: Can we use ELECTRA to achieve close to SOTA performances for NLP in low-resource settings, in term of compute cost?

cs.CL cs.AI cs.LG Topic Modeling Natural Language Processing Techniques Multimodal Machine Learning Applications Computer Science

Paper

Similar papers

Fast-ELECTRA for Efficient Pre-trainingChengyu Dong, Liyuan Liu, Hao Cheng et al.2023 · arXiv (Cornell University) · 2 citations In Library

GENERATIVE DISCRIMINATION METHOD FOR EFFECTIVE RETRAINING OF LANGUAGE MODELSAndrei P. Titov2025 · RSUH/RGGU Bulletin Series Information Science Information Security Mathematics In Library

A Deep Dive into Electra: Transfer Learning for Fine-Grained Text Classification on SST-2Muhammad Fikriansyah, Hilal H. Nuha, Muhammad Husni Santriaji2023 In Library

Are ELECTRA's Sentence Embeddings Beyond Repair? The Case of Semantic Textual SimilarityIvan Rep, David Dukić, Jan Šnajder2024 · arXiv (Cornell University) · 3 citations In Library

Maximizing Efficiency of Language Model Pre-training for Learning RepresentationJunmo Kang, Suwon Shin, Jeong-Hwan Kim et al.2021 · arXiv (Cornell University) · 0 citations In Library

ELECTRA is a Zero-Shot Learner, TooShiwen Ni, Kao, Hung-Yu2022 · arXiv (Cornell University) · 10 citations In Library

XLM-E: Cross-lingual Language Model Pre-training via ELECTRAZewen Chi, Shaohan Huang, Dong Li et al.2021 · arXiv (Cornell University) · 139 citations In Library

Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained ModelsMengzhou Xia, Mikel Artetxe, Jingfei Du et al.2022 · arXiv (Cornell University) · 8 citations In Library

Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained ModelsMengzhou Xia, Mikel Artetxe, Jingfei Du et al.2022 In Library

Training ELECTRA Augmented with Multi-word SelectionJiaming Shen, Jialu Liu, Tianqi Liu et al.2021 In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal