FINALLY: fast and universal speech enhancement with studio-like quality

Summary

The authors propose FINALLY, a speech enhancement algorithm based on GANs and the Wav LM encoder. They evaluate different feature extractors and then show qualitative and quantitative results on speech enhancement.

Strengths

A major strength is the fidelity of the generated outputs. The results are impressive and are demonstrated across a variety of SNR noise levels, noise types, and accents. I was very impressed by the samples, and the fact that it can be generated in a single pass. The generative nature of the model allows for a greater degree of restoration from low quality signals, compared to discriminative/masking based approaches (e.g. Demucs, Conv-Tasnet, TF-GridNet) that can't restore the speech to this studio like quality. I also liked the rigorous comparison to choose WavLM as the feature extractor in the network. There was proper consideration given to why you chose that encoder. Although a better choice would have been to compare WavLM to using other methods in the ablation study, e.g. WavLM does better on the clustering and SNR rule, but does that actually mean it does better as the network component?

Weaknesses

The main weakness is that the approach is very close to HiFi with only slight modifications. The other contributions stated in the paper are not as major as the authors claim. The first section about sampling from the conditional distribution vs the max seems like it misses the point a bit. The reason people use diffusion models, GANs etc for for conditional sampling is because those models produce high quality unconditional outputs and can often be used without modification Not necessary because there is a desire to sample from the posterior distribution to allow multiple generations for example. Therefore I don't think the analysis in section 2 is such a big contribution. A second major weakness is the experiment section. For speech enhancement, there are some standard datasets and metrics people use to compare results. These include metrics like PSNR, PESQ, STOI, and datasets like VCTK, WHAM, and Librispeech. This would allow comparison against a greater number of recent methods (e.g. TF GridNet, Demucs) which are not included

Questions

Can you provide qualitative comparisons against the baselines like Hifi? The reviewers would appreciate hearing those as well as your generated outputs that you have already provided.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

The paper proposes a GAN based method for universal speech enhancement (SE) demonstrating competitive experimental performance. To justify the use of adversarial learning for SE, the authors provide a theoretical insight regarding the effectiveness of the mode-covering property of LSGAN for the SE task under ideal conditions. However, in practice several additional losses needs to be employed for stable training and better performance. The authors also propose a perceptual loss, LMOS, which combines L1 loss between STFT magnitudes and L2 loss between WavLM convolution encoder features. The choice of using WavLM-conv features for the perceptual loss is driven by two heuristic rules regarding the features, 1) identical (content) speech sounds should be clustered together and 2) adding noise should increase distance from clean cluster centers proportional to SNR. Based on these rules, several models were evaluated and finally the WavLM-conv was chosen. Moreover a Human Feedback (HF) loss is also utilized, which is provided by means of differentiable PESQ and UTMOS predictions. For the model architecture, the authors extend the HiFi++ model architecture, incorporating WavLM features along with SpectralUNet features as input for the Upsampler. Additionally, an upsampling WaveUNet is added which increases the output sampling rate upto 48khz.

Strengths

- Overall the paper is well written. - Strong speech enhancement performance. - Interesting theoretical insight showing the relevance of mode-covering property of LSGAN for speech enhancement task. - The proposed criteria for choosing the SSL feature for perceptual loss is very interesting and the results of the comparison with several SSL models is a very good contribution. - Architectural changes to the HiFi++ model are significant as it enables the model to output 48khz speech and incorporate SSL features.

Weaknesses

**Main weakness is in the experimental evaluations** - Table 2 (main experimental result): - Since the model architecture is based on HiFi++, a direct comparison is probably necessary, though the ablation study covers it somewhat, but comparison with a full scale baseline HiFi++ in the main experiment would be more convincing. - There is no description of the model size/training data scale for the baseline methods, which makes it difficult to contextualize the results. - A self contained description of the evaluation dataset is missing. - Table 3 (ablation study): - The ablation study is missing the results for the 1st stage (pretraining setup, without adversarial loss), while it is mentioned in Appendix E2, but the results are not in the table.

Questions

- From the ablation study, the LMOS loss, model scaling, Upsampler (to 48khz) and HF losses show clear performance benefits, however the advantage of incorporating the WavLM-enc is not clear, as it does not lead to significant performance gain. Is the WavLM-enc necessary? - Since the goal is universal speech enhancement, it would be interesting to see the performance in more focussed evaluation such as bandwidth extension or reverberation individually.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations have been adequately addressed.

Summary

This paper proposes a universal speech enhancement model for real-world recording environments utilizing GAN, referred to as FINALLY. The authors theoretically analyze that using LS-GAN loss leads to finding the point of maximum density within the conditional clean speech distribution. To stabilize the adversarial training process, WavLM-based perceptual loss is integrated into the MS-STFT pipeline.

Strengths

- The paper provides a meaningful analysis of LS-GAN loss in the context of speech enhancement. - The adoption of the WavLM neural network shows performance improvements.

Weaknesses

- Although the paper focuses on real-world scenarios, it still needs results with objective metrics such as PESQ and STOI for a solid evaluation. - The three-stage training process of FINALLY is complex, and the performance gains over HiFi-GAN2 do not justify the increased cost. - Reducing the model size by 90% for ablation studies is not convincing, as it can significantly affect the results.

Questions

- I'm curious about why DNSMOS and other metrics trend differently in Table 3. - Could you clarify why the order of the metrics in Tables 2 and 3 is presented differently?

Rating

5

Confidence

5

Soundness

2

Presentation

2

Contribution

3

Limitations

- Artifacts mentioned in the inference results can degrade perceptual quality. - The paper has a clear contribution, but the presentation and organization are lacking.

Summary

This paper describes a new formulation of GAN-based speech enhancement. It includes an analysis of the convexity in different feature spaces of the distribution of TTS utterances generated from the same inputs, concluding that WavLM's convolutional encoders provide the most convex such space. This representation is then incorporated into the input of HiFi++ and also a separate, unrelated upsampling stage is added at the end. On the VoxCeleb real data validation set and the validation data of UNIVERSE, the proposed approach outperforms other strong baselines in both quality (MOS from a subjective listening test and other objective metrics) as well as in realtime factor. For example, on VoxCeleb, the proposed system is rated at 4.63 MOS compared to the second best system HiFi-Gan-2 at 4.47 while the proposed system is ~15x faster. For systems with comparable RTFs, the difference in MOS is 4.63 vs 3.79 for DEMUCS.

Strengths

Significance: * The argument for the relevance of GANs to speech enhancement is thoughtful, interesting, and convincing. It brings clarity on a point that I did not previously appreciate. * Analysis of the different feature spaces in terms of their convexity is another valuable and interesting contribution. It provides clearly actionable findings that are shown in the experiments to make a meaningful difference to system performance. This analysis can be applied more broadly to compare different SSL representations for various tasks. System performance: * Improving both the performance and the efficiency of a speech enhancement system is quite valuable and these differences appear to be large compared with strong baselines. * Listening to the provided audio examples in the supplementary material shows impressive performance, especially in comparison to the halucinations of UNIVERSE in low-SNR instances, although it is not clear how these particular examples were selected. Clarity: * The paper is well written and easy to follow. The figures are helpful in understanding it. The description of the loss and different training stages is particularly clear and helpful. * The ablations are thorough and informative and show clear benefits to each of the stages of the model/training.

Weaknesses

One point that could be added to the discussion of previous work is that of Maiti and Mandel (2019) and related work, which introduced the idea of speech enhancement by synthesis of a clean utterance that contains the same content as the original utterance. The evaluation was conducted on a crowdsourcing platform without IRB review. This should be reviewed by ethics reviewers. S Maiti and M Mandel (2019). Parametric resynthesis with neural vocoders. Proc. IEEE WASPAA. Minor comments: * Line 187 states, "we report MOS score" but it does not state where this MOS score comes from. Please describe the experiment/measurement that generated the MOS scores. Presumably it was the human listening test described in the appendix, but this is not clear at this point in the manuscript. * Equation (2): please define phi * Line 299 calls losses based on UTMOS and PESQ "human feedback" losses, but since these are algorithms predicting human feedback, I don't think their outputs can be called human feedback itself.

Questions

N/A

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations of the approach are discussed in the appendix, section D.4. The limitation of this approach being non-streaming is mentioned there and is important to highlight.

Response to rebuttal

Thank you for the detailed rebuttal and additional ablation experiments. Most of my concerns have been adequately addressed, however I would still like to see the HiFi++ in the main tables (Tab. 2 and 3) since the proposed work builds on top of it. In the evaluation on the VCTK-DEMAND (in response to reviewer gE54), HiFi++ is included, will that table be included in the main paper? It will add value I think and also adding it to Tab 2 and 3 in the will also make the improvements stand out.

Response to ethics review

Dear Reviewer, We would like to address the points you have raised regarding the use of the LibriTTS-R and DAPS datasets. **1. PII Information and Data Cleaning:** We confirm that no Personally Identifiable Information is included in the LibriTTS-R and DAPS datasets. Both datasets are publicly available and have undergone preprocessing to ensure the exclusion of any PII. Specifically: - **LibriTTS-R:** This dataset is derived from LibriVox recordings, which are public domain audiobooks. The dataset creators ensured that no PII was included by anonymizing any potential identifiers. - **DAPS:** The DAPS dataset consists of recordings from professional voice actors who provided explicit consent for their voices to be used in research. The dataset does not include any PII, and measures were taken to anonymize any potential personal identifiers. **2. Representativeness of the Dataset:** Regarding the representativeness of the datasets: - **LibriTTS-R:** The LibriTTS-R dataset is derived from a diverse collection of public domain audiobooks, which include a variety of speakers in terms of gender, age, and accent. While it provides a broad representation, it may not encompass all demographic and cultural groups. - **DAPS:** The DAPS dataset features recordings from a smaller, controlled group of professional voice actors. While it offers high-quality audio samples, it may not fully represent the diversity of the general population. **3. Licenses for Each Dataset:** The licenses for datasets in our submission: - **LibriTTS-R:** The dataset is distributed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). - **DAPS:** The dataset is released under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). We will include these details in the revised version of our manuscript, along with appropriate citations and acknowledgments for each dataset. We hope this clarifies the points raised in your review. Thank you for your valuable feedback.

Response to rebuttal

I would like to thank the authors for their rebuttal. I have read all of the reviews and the rebuttals and would like to keep my rating as-is. I do think that section 2 is a strong contribution to the literature and our understanding of the problem of speech enhancement and the utility of generative models for solving it.

Response to rebuttal

Thank you for the detailed rebuttal, and particularly for providing the additional results using objective metrics. Rebuttal addressed the most of my concerns. Based on the description in the rebuttal, it seems that no additional training was performed with the VCTK-DEMAND dataset. While the performance on objective metrics falls short of that achieved by HiFi-GAN2, the improvement in MOS scores is indeed Impressive. However, after reviewing the demo samples provided, I was unable to clearly perceive an improvement over HiFi-GAN2. Still, the significant increase in speed of the proposed algorithm compared to HiFi-GAN2 is noteworthy. Moreover, considering that few models have successfully addressed multiple distortions with one pass, I believe this work holds substantial value. Therefore, I will raise my score by one level.

Reproducing FINALLY Speech Enhancement – Clarifications Needed

Dear Authors, We greatly enjoyed your paper, *"FINALLY: Fast and Universal Speech Enhancement with Studio-like Quality"*. Our team at Inverse AI has been developing an open-source implementation of your model, now publicly available on GitHub: [https://github.com/inverse-ai/FINALLY-Speech-Enhancement](https://github.com/inverse-ai/FINALLY-Speech-Enhancement). This follows our previous efforts in open-sourcing models, such as converting PyTorch implementations of Hybrid Transformer Demucs to TensorFlow: [https://github.com/inverseai/Demucs-and-HTDemucs-based-denoiser-with-Tensorflow](https://github.com/inverseai/Demucs-and-HTDemucs-based-denoiser-with-Tensorflow). While implementing FINALLY, we encountered some discrepancies and ambiguities, and we hope you can provide clarifications to help align our implementation with your reported results: 1. **Parameter Mismatch:** The paper reports 454M parameters, including 358M from WavLM. Using WavLM-Large from the provided Hugging Face link, our implementation totals 363M (315M WavLM + 48M model). Model components: SpectralUNet (4.4M), ResBlock (7.1M), Conv (0.8M), HiFi (15.5M), WaveUNet (10.8M), SpectralMaskNet (5.5M), WaveUNet Upsampler (3.9M). Could you clarify this discrepancy? 2. **Training Data Details:** The paper mentions Stages 1 and 2 use LibriTTS-R clean dataset, but it’s unclear whether these stages use identical or different subsets. 3. **Audio File Length:** The typical duration or range of audio samples used for training is not specified. 4. **Batch Size:** The paper mentions a batch size of 32, but it is unclear if this applies to all stages or varies. 5. **Positional Embedding:** The type of positional embedding (sinusoidal or learned) is not specified. 6. **Training Iterations:** The paper mentions 100k, 30k, and 40k iterations for Stages 1, 2, and 3. Our model continues converging beyond these. Did you observe similar behavior, or is there a recommended stopping criterion? 7. **LMOS Loss:** Using mel spectrograms in LMOS loss avoids artifacts in Stages 1 and 2 but causes issues in Stage 3, while STFT introduces artifacts in Stages 1 and 2. Did you encounter similar trade-offs, and could you share insights? **Our setup (for reference):** - Segments: 2s audio files - Batch: 18, 12, 4 for Stages 1, 2, 3 - Dataset: LibriTTS-R clean in Stages 1 & 2, DAPS with DNS noise in Stage 3 (~200 hours) - GPU: 2× NVIDIA 5090 - Positional Embedding: PyTorch 2D learnable - Loss: LMOS (tested with mel-spectrogram and STFT) - Optimizer: AdamW, warmup: 10k steps - Overall MOS: 2.95 on 10% of training data We would greatly appreciate your guidance on these points. Clarifications will help improve our open-source implementation and benefit the broader research community.

WaveUNet Upsampler Implementation Details

We implemented the WaveUNet Upsampler as follows: * **Downsampling:** 4× `Downsample1d` layers with channels `[128→128, 128→128, 128→128, 128→256]`. * **Pre-LSTM convolution:** `Conv1d(256→512, kernel=5, padding=2)` to match the LSTM input dimension. * **LSTM bottleneck:** 2-layer BiLSTM with 512×2 hidden units. * **Post-LSTM convolution:** `Conv1d(1024→256, kernel=5, padding=2)` to match the first Upsample1d input channel. * **Upsampling:** 4× `Upsample1d` layers with channels `[256→128, 128→128, 128→128, 128→128]`. * **Additional upsampling:** `ConvTranspose1d(128→512, kernel=3, stride=3)`. * **Final output convolution:** `Conv1d(512→32, kernel=5, padding=2)`. This configuration yields a total of **15.4M parameters**. Could you confirm if this aligns with your implementation, or if any components, especially after the `ConvTranspose1d`?