Streaming End-to-End Bilingual ASR Systems with Joint Language Identification

Multilingual ASR technology simplifies model training and deployment, but its\naccuracy is known to depend on the availability of language information at\nruntime. Since language identity is seldom known beforehand in real-world\nscenarios, it must be inferred on-the-fly with minimum latency. Furthermore, in\nvoice-activated smart assistant systems, language identity is also required for\ndownstream processing of ASR output. In this paper, we introduce streaming,\nend-to-end, bilingual systems that perform both ASR and language identification\n(LID) using the recurrent neural network transducer (RNN-T) architecture. On\nthe input side, embeddings from pretrained acoustic-only LID classifiers are\nused to guide RNN-T training and inference, while on the output side, language\ntargets are jointly modeled with ASR targets. The proposed method is applied to\ntwo language pairs: English-Spanish as spoken in the United States, and\nEnglish-Hindi as spoken in India. Experiments show that for English-Spanish,\nthe bilingual joint ASR-LID architecture matches monolingual ASR and\nacoustic-only LID accuracies. For the more challenging (owing to\nwithin-utterance code switching) case of English-Hindi, English ASR and LID\nmetrics show degradation. Overall, in scenarios where users switch dynamically\nbetween languages, the proposed architecture offers a promising simplification\nover running multiple monolingual ASR models and an LID classifier in parallel.\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC