Summary
Vocos is a waveform synthesis (vocoder) model operating directly on the frequency scale to estimate Fourier spectral coefficients (magnitude and phase) followed by inverse Fourier transform (iSTFT). Due to the challenges in phase modeling with neural networks, previous non-autoregressive, GAN-based neural vocoders directly generate time-domain waveforms through an upsampling architecture, leading to a slowdown in inference latency (even with the non-autoregressive modeling). Vocos, to the best of the reviewer's knowledge, constitutes the first success in the direct estimation of both the magnitude and phase entirely from the frequency domain, resulting in a highly efficient neural vocoder.
Strengths
As stated in the summary, direct phase estimation has historically been known to be challenging. The major contribution of Vocos is being the first model to succeed in high-quality phase estimation for neural vocoding through the careful design of a sinusoidal activation head that handles phase wrapping. Since the feature propagation is entirely in the spectral domain with the same I/O dimension, Vocos is significantly faster than time-domain models and more flexible in adopting modern isotropic architecture (ConvNeXt), previously proposed in other data domains, for neural vocoding tasks. Keeping the scope of a general-purpose neural vocoder is also a plus, where the proposed method is not dependent on a specific input representation (mel spectrogram) and can also be trained with the latent feature as a neural audio codec.
Weaknesses
While the main focus of the manuscript is realizing an efficient neural vocoder operating in the frequency domain, it would also be interesting to further assess the robustness of the proposed method. Considering that the experimental setup largely follows a previous time-domain neural vocoder (BigVGAN), readers may wonder if Vocos can also achieve similar robustness. The provided samples in the manuscript only contain clean speech samples. To further convince the readers, it would be useful to add non-clean speech and audio samples to the demo (noisy speech, environmental sounds, and music, for example) similar to the results presented in the previous work. The VISQOL score in Table 3 looks promising, so letting the readers form their opinion on the subjective quality by adding such samples will be helpful as well. By adding these samples, we could also confirm that the objective VISQOL scores align with human perception as well.
In my opinion, the statement "ConvNeXt blocks can more effectively model spatially local input patterns" is not fully supported by current results; while it is true that Vocos can easily adopt modern isotropic architectures, the same can also be explored with time-domain neural vocoders. To verify this under scrutiny, adopting ConvNeXt to the time-domain GAN vocoder and measuring the performance gain/loss will be useful (probably with dilation to the depthwise conv in ConvNeXt as done by ResBlock in HiFi-GAN). If the time-domain GAN vocoder shows degradation in performance, it will make the benefits of isotropy in Vocos more convincing.
Questions
Overall, Vocos manifests an exciting step towards enabling a neural vocoder operating entirely in the frequency domain; its fast speed will contribute to accelerating efficient speech synthesis solutions, where the speed of such models has been bottlenecked by the time-domain neural vocoder. But at the same time, I am also interested in the scalability of the method. Specifically, a neural vocoder is also recently viewed as a (de)compression model (similar to VQGAN in the image domain, with EnCodec and DAC[1] as examples) to build the audio generative model in the latent space. For this application, practitioners may want higher quality decoding with increased scale if speed is not a concern. Can we expect that the Vocos (either as a standalone vocoder or the decompression model) can be scaled up further to achieve even better quality? Or, if Vocos exhibits a performance limit at a certain scale, what would be the root cause in the author's opinion?
[1] Kumar, Rithesh, et al. "High-Fidelity Audio Compression with Improved RVQGAN." arXiv preprint arXiv:2306.06546 (2023).
Rating
8: accept, good paper
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.