Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognition

Attention-based end-to-end automatic speech recognition (ASR) systems have\nrecently demonstrated state-of-the-art results for numerous tasks. However, the\napplication of self-attention and attention-based encoder-decoder models\nremains challenging for streaming ASR, where each word must be recognized\nshortly after it was spoken. In this work, we present the dual\ncausal/non-causal self-attention (DCN) architecture, which in contrast to\nrestricted self-attention prevents the overall context to grow beyond the\nlook-ahead of a single layer when used in a deep architecture. DCN is compared\nto chunk-based and restricted self-attention using streaming transformer and\nconformer architectures, showing improved ASR performance over restricted\nself-attention and competitive ASR results compared to chunk-based\nself-attention, while providing the advantage of frame-synchronous processing.\nCombined with triggered attention, the proposed streaming end-to-end ASR\nsystems obtained state-of-the-art results on the LibriSpeech, HKUST, and\nSwitchboard ASR tasks.\n