Diffusion Buffer for Online Generative Speech Enhancement

Online Speech Enhancement (SE) was mainly reserved for predictive models. An advantage of these models is that for an incoming signal frame from an audio stream, the model is called only once for enhancement. In contrast, generative SE models often require multiple calls, resulting in a computational complexity that is too high for many online SE applications. Existing online SE systems are primarily based on predictive models. This work presents the Diffusion Buffer, a generative diffusion-based SE model, which only requires one neural network call per incoming signal frame from an audio stream and performs online enhancement on a consumer-grade GPU. The key idea of the Diffusion Buffer is to align physical time with diffusion time-steps. The approach progressively denoises frames through time, where past frames have more noise removed. Consequently, a delayed enhanced frame is directed to the listener, and that output frame has a corresponding look-ahead. In this work, we extend upon previous work by carefully designing a 2D convolutional UNet architecture that specifically aligns with the Diffusion Buffer’s look-ahead. We observe that the proposed UNet improves performance, particularly when the algorithmic latency is low. Moreover, we show that using a Data Prediction loss instead of Denoising Score Matching loss enables flexible control over the trade-off between algorithmic latency and quality during inference. The extended Diffusion Buffer drastically reduces the algorithmic latency from 320 ms–960 ms to 32 ms–176 ms while performance increases.