Transformer-based language models have achieved remarkable success across diverse natural language processing (NLP) tasks. However, their ability to model sequential dependencies remains limited, particularly in low-resource and morphologically rich languages, such as many of Indonesia's regional languages. This paper introduces a hybrid architecture that integrates Bidirectional Long Short-Term Memory (BiLSTM) and Bidirectional Gated Recurrent Unit (BiGRU) layers on top of the NusaBERT backbone, a Transformer pretrained on Indonesian and regional corpora. We conduct extensive fine-tuning on three benchmark datasets—NusaParagraph, NusaTranslation, and NusaX—targeting multi-class classification tasks. The hybrid variants differ in their pooling strategies (CLS token, last hidden state, mean, and max), and also include a concatenated embedding approach, where the last four hidden layers of NusaBERT are fused to enrich semantic representation, while maintaining identical training configurations. Experimental results demonstrate that the hybrid models outperform strong Transformer baselines, with the NusaBERT+BiGRU with mean pooling variant achieving a macro F1-score of 83.34% on the NusaX sentiment classification task. Further evaluation on previously unseen languages, including Batak Toba, Madurese, and Ngaju, reveals improved generalization and mitigated catastrophic forgetting. These findings highlight the efficacy of combining Transformer-based contextual embeddings with recurrent architectures for robust, low-resource multilingual NLP, and support their potential in fostering language preservation and digital inclusion.
Paper
Full text
Hybrid Transformer-RNN Model for Classification of Indonesian Regional Language
Semantic Scholar · 2025
Abstract
Transformer-based language models have achieved remarkable success across diverse natural language processing (NLP) tasks. However, their ability to model sequential dependencies remains limited, particularly in low-resource and morphologically rich languages, such as many of Indonesia's regional languages. This paper introduces a hybrid architecture that integrates Bidirectional Long Short-Term Memory (BiLSTM) and Bidirectional Gated Recurrent Unit (BiGRU) layers on top of the NusaBERT backbone, a Transformer pretrained on Indonesian and regional corpora. We conduct extensive fine-tuning on three benchmark datasets—NusaParagraph, NusaTranslation, and NusaX—targeting multi-class classification tasks. The hybrid variants differ in their pooling strategies (CLS token, last hidden state, mean, and max), and also include a concatenated embedding approach, where the last four hidden layers of NusaBERT are fused to enrich semantic representation, while maintaining identical training configurations. Experimental results demonstrate that the hybrid models outperform strong Transformer baselines, with the NusaBERT+BiGRU with mean pooling variant achieving a macro F1-score of 83.34% on the NusaX sentiment classification task. Further evaluation on previously unseen languages, including Batak Toba, Madurese, and Ngaju, reveals improved generalization and mitigated catastrophic forgetting. These findings highlight the efficacy of combining Transformer-based contextual embeddings with recurrent architectures for robust, low-resource multilingual NLP, and support their potential in fostering language preservation and digital inclusion.