UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures

Summary

This paper presents an innovative approach to unsupervised neural speech separation, leveraging the conditions where the number of microphones surpasses the number of speakers. The authors propose a method, named UNSSOR, which transforms an originally ill-posed problem - one that does not have a unique solution - into a well-posed problem - one with a unique solution - thereby facilitating the separation of speakers. Key contributions of the paper include: - The authors establish a linear-filter constraint between each speaker's reverberant images at each microphone pair, which converts the ill-posed problem into a well-posed one, thereby enhancing the separation of speakers. - The authors devise loss functions inspired by the blind deconvolution problem and propose a DNN-based approach to optimize these functions. The speaker images are determined via DNNs, while the linear filters are estimated using a sub-band linear prediction algorithm named FCP, based on the mixture and DNN estimates. - To address the frequency permutation issue that arises when using sub-band FCP, the authors propose a loss term that minimizes a measure known as intra-source magnitude scattering. The authors claim that UNSSOR can be trained to perform under-determined separation, such as monaural unsupervised speech separation, based on over-determined training mixtures.

Strengths

The authors propose a novel unsupervised neural speech separation method. The proposed method, UNSSOR, utilizes the multimicrophone over-determined condition to solve the unsupervised learning problem of speech separation. This is a completely new perspective compared to previous unsupervised speech separation methods. The authors have designed innovative loss functions that guide the unsupervised separation model about the desired sound objects and encourage the separation of speakers. This approach is original and shows a creative combination of existing ideas.

Weaknesses

However, there are several limitations to this paper that undermine its true potential. Please see below for concerns and questions for the authors. 1. The article mentions that MixIT may have two mixtures that are not in the same scene under reverberant conditions, which in turn may result in providing spatial a priori information, or providing a priori information due to differences in the configuration and number of microphones. However, I feel that this statement is a bit too hypothetical. I think these problems are more a result of how the dataset is constructed and not a deficiency of the MixIT method itself. More specifically, these problems should be solved by constructing a room-specific dataset. The SMS-WSJ dataset mentioned in the paper can control the room's parameters, the arrangement of microphones, and the number of microphones. By finely designing and controlling these parameters, the resulting problems can be effectively circumvented, allowing the MixIT method to be evaluated in a more consistent environment. Therefore, I believe the problems mentioned in the paper do not serve as a weakness of the MixIT method. 2. In Section 4.2, I noticed that the authors proposed the MC loss function. However, as far as I understand, the MC loss function is not presented in this paper and has been described in detail in reference [1]. Cite [1] as a reference and clarify whether the MC loss function in the paper is an innovation from [1] and where exactly the innovation is. 3. The authors chose TF-GridNet as the separation DNN structure for the UNSSOR method.TF-GridNet has shown excellent performance in supervised separation tasks. However, the question I would like to raise is: Is the UNSSOR method's effectiveness due to the separation model TF-GridNet's strong performance, or is it due to the contribution of the UNSSOR method itself? I believe that to elaborate the capability of the UNSSOR method better; it is necessary to try to use different separation structures and compare their performance on each structure. If the UNSSOR method performs well on other separation structures, then we can be more confident that the UNSSOR method itself is highly robust and general. Such an analysis is necessary to assess and understand the actual value of the UNSSOR method. 4. I note that the approach in the paper combines traditional methods (Spatial clustering and IVA) with the DNN-based method iRAS. In addition, UNSSOR, a DNN-based scheme, is compared to include only one DNN method, which may not be comprehensive, in my opinion. Of particular note is that MixIT is a popular and widely used method for unsupervised speech separation, as shown in references [2-5]. I was puzzled while reading the paper as to why the authors did not include MixIT in the comparison. 5. The authors set different STFT window sizes and hop lengths in the comparison, which may raise questions about the fairness of the results. For the separation model of DNN, different settings of STFT window sizes and hop lengths can significantly affect the model's performance [6]. ### References [1]. Wisdom S, Hershey J R, Wilson K, et al. Differentiable consistency constraints for improved deep speech enhancement[C]//ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019: 900-904. [2]. Wisdom S, Tzinis E, Erdogan H, et al. Unsupervised sound separation using mixture invariant training[J]. Advances in Neural Information Processing Systems, 2020, 33: 3846-3857. [3]. Tzinis E, Adi Y, Ithapu V K, et al. RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing[J]. IEEE Journal of Selected Topics in Signal Processing, 2022, 16(6): 1329-1341. [4] Tzinis E, Casebeer J, Wang Z, et al. Separate but together: Unsupervised federated learning for speech enhancement from non-iid data[C]//2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2021: 46-50. [5] Zhang J, Zorila C, Doddipatla R, et al. Teacher-student MixIT for unsupervised and semi-supervised speech separation[J]. arXiv preprint arXiv:2106.07843, 2021. [6] Peer T, Gerkmann T. Phase-aware deep speech enhancement: It's all about the frame length[J]. JASA Express Letters, 2022, 2(10): 104802.

Questions

My detailed questions are as described above. (1) Is the MC loss function in this paper an innovation from [1] and where is the innovation marked? (2) Is the effectiveness of the UNSSOR method due to the powerful performance of the separation model TF-GridNet, or is it due to the contribution of the UNSSOR method itself? (3) Why not compare MixIT method? (4) Why not use the same STFT settings, especially compared to the DNN model iRAS.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors discuss the limitation of the proposed method.

Summary

The most popular method for training neural networks for speech separation is by artificially mixing sources, since neural network based training requires supervision. However, this kind of supervised training creates a mismatch since the mixtures seen at test time contain real overlapping speech and noise. This paper formalizes the problem of unsupervised speech separation, and posits that it becomes a well-posed problem for over-determined conditions (i.e., when the number of microphones is more than the number of speakers). The authors propose a method (inspired from signal processing) that uses the images at each microphone as supervision for the neural network training, and a new loss function to address the frequency permutation problem that often happens in blind source separation (BSS). They also propose a method to use these models for monoaural separation.

Strengths

Overall, I believe this is an extremely strong paper with landmark results for unsupervised speech separation. As mentioned above, the majority of literature in the field of speech separation, since permutation-invariant training (PIT) was proposed, has only revolved around better neural network architectures. While the SI-SDR on the synthetic WSJ-Mix benchmark has consequently gone up, significantly less attention has been paid to real overlapping speech. The paradigm of continuous speech separation (CSS) tries to address some of the issues (such as sparse overlaps), but it is also trained with PIT on synthetic mixtures. The result is that there exists no public speech separation models that can perform well on multi-speaker data such as AMI, CHiME-6, AliMeeting, etc. I think UNSSOR is quite promising and should change the face of speech separation research. The paper is strong on many levels, such as the following. 1. The authors correctly identify that the problem with training separation networks has to do with a lack of accurate supervision, and propose that this supervision may come from estimating each speaker’s reverberant image at every microphone. Under the assumption of a linear-filter constraint between a speaker’s images at the microphones, the resulting linear system has a unique solution, as described in Section 3. 2. The authors design loss functions which are inspired from signal processing formulation of the problem. This includes (i) mixture consistency loss for the filtered estimates, (ii) forward convolutive prediction (FCP) for relative RIR estimation, (iii) causal/non-causal stacking to handle time misalignment, and (iv) an intra-source magnitude scattering loss to address the frequency permutation problem. Overall, all these losses have clear motivations and design. 3. The recently proposed TF-GridNet encoder is used as the backbone of the model. This architecture has been shown to obtain state-of-the-art results on PIT-based separation (even surpassing time-domain models), and it is interesting to see that it can be trained well with the unsupervised technique. 4. The authors compare their method with extremely strong baselines, which makes their conclusions stronger. For example, they implemented a novel variant of the recently proposed “Reverberation as Supervision” (RAS) method, called iRAS, which can perform unsupervised separation. This modification could very well be a short paper in itself. 5. The results on SMS-WSJ show that UNSSOR obtains results approaching supervised separation models. For example, for the 6-channel setting, the SDR on the test set is 15.6 dB (compared to 19.4 dB for PIT model). As a comparison, the next unsupervised method is IVA, which obtains 10.6 dB. Moreover, on reducing the number of channels from 6 to 3, UNSSOR’s performance degrades only marginally (to 15.4 dB), whereas PIT degrades to 16.8 dB. 6. The authors even find a way to use models trained with multi-channel inputs for performing mono-aural separation. 7. The code and models for UNSSOR will be released. A side effect of all of the above is that the paper is quite dense and may not be accessible to readers who do not have a source separation background. However, it is a case study in how neural methods can be designed by incorporating the knowledge from other domains (in this case, signal processing).

Weaknesses

1. The main point of concern about the paper is that results are shown just for the SMS-WSJ dataset. Such an evaluation defeats the original motivation for unsupervised training, since SMS-WSJ contains synthetically mixed sources. Clearly, the PIT models outperform UNSSOR on this benchmark, which is expected. The real test of UNSSOR would have been an evaluation on real mixtures, such as AMI, CHiME-6, or AliMeeting, where PIT-based models fail completely. Nevertheless, since the paper proposes a completely novel paradigm of speech separation, I am willing to forgo this point, but at the cost of docking a point from the ratings. 2. Can UNSSOR be trained with data containing different numbers of channels or different array geometries? It seems that the choice of the hyper-parameters $I$ and $J$ may be heavily dependent on the microphone configuration. 3. The authors delegate the “Limitations” section to the appendix. I think it should be put in the main paper, since an important part of presenting our research is talking about its limitations. This would also be useful for other researchers to get ideas for building upon UNSSOR for their own work.

Questions

1. How robust is UNSSOR to change in array configuration or the number of channels for the case of mono-aural separation? 2. In Section 5.2, can you briefly explain what these metrics are, for the unfamiliar reader.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

None, except those presented in Appendix H.

Summary

This paper proposes an unsupervised method for training neural networks to separate speech from multi-microphone recordings. The idea is to separate a reference microphone into separate sources, then use each microphone as a mixture signal that must match filtered versions of the reference microphone separated sources. The linear filters are computed in subbands, which results in a well-known frequency permutation problem, for which an additional "intra-source magnitude" loss term is proposed to ameliorate. Though the method requires "overdetermined" mixtures (i.e. more mics than sources), the method can also be used for underdetermined cases, e.g. 2 speakers with 1 mic, where a single mic is used as input, and multiple mics are used in the loss function. For 3-mic and 6-mic mixtures, the proposed method is shown to achieve better performance in terms of SDR, SI-SDR, PESQ, and eSTOI versus a baseline (the RAS algorithm) that works on a similar principle on the SMS-WSJ dataset. The method does not outperform a supervised PIT method, which is expected.

Strengths

S1) The method is intuitive and is explained well, and the paper provides thorough description of the method and chosen hyperparamters. S2) The proposed method can perform unsupervised training on single mixtures, unlike MixIT which requires combining at least two mixtures into a training examples. S3) The authors clearly describe the differences of the proposed UNSSOR method with the RAS method, particularly why UNSSOR can successfully train unsupervised, while RAS cannot.

Weaknesses

W1) The dataset used, SMS-WSJ, consists entirely of synthetic mixtures that use synthetic simulated RIRs. Thus, there is some concern that the method may not generalize to real-world acoustic scenarios. Some evaluation on real data would further improve the paper. Also, one of the main advantages of unsupervised algorithms is to be able to adapt separation models to unsupervised real-world data, so that models can work better on those data domains. Of course, evaluation on real domains cannot be evaluated by intrusive metrics, as used in this paper, and require subjective human listening tests. I am glad the authors are considering this as a future direction, and I encourage them towards that goal. W2) As acknowledged by the authors, the method assumes stationary sources, which is potentially quite limiting when considering real scenarios where speakers may be moving. Even if speakers are seated, head movement can still produce acoustic effects. There are likely interesting extensions of the method to handle nonstationary spatial scenarios, such as allowing for slowly-varying linear filters. W3) No audio demos are provided. Providing such demos would make it a lot easier to readers to evaluate the quality of the predictions and improve understanding of the method. W4) It would be interesting to see how a single-channel unsupervised method like MixIT compares to the multi-microphone method. Also, since this paper was submitted, a multichannel version of MixIT seems to have been proposed (https://arxiv.org/abs/2305.11151), which seems to operate differently (network with multichannel input and multichannel output, and MixIT applied directly to multichannel outputs using multi-channel mixtures-of-mixtures). It would be interesting to compare both single-channel and multi-channel MixIT to the proposed approach; these MixIT methods will likely be mismatched to test time since they train on mixtures-of-mixtures, but it would be interesting to compare. Minor comments and typos a) "boot-start" -> "bootstrap" ? b) "comptue" -> "compute" c) Would be good to mention what \mathcal{F} is in description of equation (4) d) "doing this would complicates" -> "doing this would complicate" e) "assumed time-variant." -> "assumed time-variance." f) Maybe bold the best numbers in each column for Tables 1 and 2?

Questions

Q1) How much tuning of the number of filter taps was done? This seems like it could have a big effect on the performance of the model. Also, does tuning the number of taps suggest some knowledge about the unsupervised data? I suppose a practitioner could use a blind T60 model to estimate the distribution of T60 across an unsupervised dataset, which could give some insight into optimal filter taps. Besides T60 (i.e. duration of expected RIRs to model), are there other issues that affect the optimal choice of filter taps? Q2) How certain is it that frequency permutation is the primary cause of the drop in performance going from rows 1a->1b and 2a->2b in Tables 1 and 2? The appendix provides an illustrative example, but none of the objective metrics are directly measuring frequency permutation. It seems that an objective metric could be formulated that directly measures the degree of frequency permutation, but perhaps such a metric is not necessary if spot checks of the predictions indicated that frequency permutation was present?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are discussed well.

Summary

In this paper the authors tackle speech source separation, in which multiple fixed speech sources $X(c)$ are recorded by an array of p microphones, resulting in p observable mixtures $Y_p$. The authors propose an STFT domain source separation algorithm called UNSSOR, leveraging a complex-valued deep neural network (TF-GridNet) to estimate the individual sources at a reference microphone (e.g., mic 1). Typically the problem is solved in a supervised fashion using PIT (permutation invariant training) where the clean sources must be known, but the authors propose an unsupervised method, where only the mixtures $Y_p$ are required at training time. The key idea is to relax the under-determined problem, by leveraging physical constraints, i.e. the sources combining at a microphone p are the same sources at the reference microphone convolved with a relative room impulse response (relative RIR). As such the authors estimate virtual sources $Z(c)$ and relative RIRs at each microphone for each source $g_p(c)$. The relative RIRs are computed in closed form using FCP (a least squares optimization problem), leveraging only the mixtures $Y_p$ and the estimates $Z(c)$. Multiplying $g_p(c)$ with $Z(c)$ the authors obtain the FCP-estimated source images, which are summed to obtain mixtures estimates $\hat Y_{p}$ at each microphone, obtaining a mixture consistency loss $L_{MC}$ for training the neural network. The authors notice that solutions obtained with the model trained on $L_{MC}$ only exhibit the frequency permutation problem, where estimates on certain spectral bands are swapped between estimates. To address this problem, the authors regularize the variance of the magnitude of the FCP-estimated source images over frequency bands during training time, with a novel intra-source magnitude scattering loss $L_{ISMS}$. Combining $L_{MC}$ and $L_{ISMS}$ results in improved separation metrics. Finally, the authors try the algorithm in the monoaural unsupervised source separation setting, where they provide only one mixture at training time, but optimize with multiple microphones in the loss.

Strengths

- The idea of reducing the number of equations using physical constraints is natural, leading to a well-structured unsupervised multi-mixture speech source separation algorithm. Computing the RIRs with least squares also is very useful, given that in such a way the mixture consistency loss depends only on the observed mixtures and the estimated virtual sources. The authors present such an idea in a very clear and formal manner, with relevant links to the existing literature. I believe that this approach is way more founded theoretically than the MixIt approach, leading to a better methodological line of research on unsupervised neural source separation. - The intra-source magnitude scattering regularizer ($L_{ISMS}$) seems very interesting because such a loss can be incorporated not only in the presented multi-microphone speech scenario but in every source separation task working on STFT spectrograms (e.g, music, universal). Improving the results for more than 4dB gives strong empirical results to the effectiveness of such a technique. - The method achieves good empirical results with respect to a plethora of metrics (SDR, SI-SDR, PESQ, eSTOI) on SMS-WSJ compared to other unsupervised multi-mic algorithms (Spatial clustering, IVA, RAS) and fares well with respect to a supervised PIT baseline (using the same neural architecture).

Weaknesses

- UNSSOR, while improving the results in the monoaural speech separation setting with respect to iRAS (improves 1.4 dB on the SMS-WSJ test set both with 3 and 6 channels loss), still faces a large gap with respect to the supervised Monaural PIT baseline (~ 4 dB). This is expected since not relying on supervision can negatively impact the separation results. The authors should have included at least a comparison with some other popular unsupervised source separation baseline such as MixIt, in order to prove the superiority of UNSSOR in the unsupervised source separation arena. - As with other unsupervised neural source separators such as MixIt, it is not really clear why one should go unsupervised if the supervised metrics are already better on the same dataset. These types of studies (including MixIt or its regularized versions such as https://arxiv.org/abs/2106.00847), should experimentally showcase that performing unsupervised source separation can benefit over supervised source separation as the dataset size increase. For example in https://arxiv.org/abs/2106.00847, authors cannot beat a supervised baseline trained on FUSS, using an order of magnitude more data (Audioset or YFCC100m) with an unsupervised method (regularized MixIt). Providing a scaling law should be of uttermost importance in papers such as the presented one and in future papers, if not we can continue inventing unsupervised methods that will never surpass supervised or weakly supervised methods (using labels, learning Bayesian priors).

Questions

I will add suggestions for the improvement of the paper and ask relevant questions. - Line 65: The authors should use a term like `metric` instead of `measure`, given the precise meaning of such a term in measure-theory. I know they can be synonyms in an applied field such as audio processing but it is better to be more precise. - Line 120: `Hermitian transpose` instead of `Hermittan` - Line 146: the term mixture consistency is defined here: https://arxiv.org/pdf/1811.08521.pdf, I think it's ok to use the same term but could create a little bit of confusion. - Line 161: Can the authors explain briefly in the text, for better understanding, the rationale of matching the FCP-estimated images with the mixture in Eq. (6)? - Line 167: Regarding the $\xi$ hyperparameter, I don't understand if it multiplies only the max or all the expression - Line 179: I found the causal analysis well developed, but I still do not well understand why if a source sound c reaches the reference microphone earlier it should be processed with future values (non-causal filtering). - Line 193: Given that $Z(c)$ is learned, why it is interpreted like a virtual microphone estimate and not the real dry signal at speaker $c$? - Line 232: In the monoaural speech source separation setting, I don't understand what you match on the different microphones in the mixture consistency loss, when using only one input $Y_p=1$? Do you match the only available $Y_p=1$ at all "virtual" microphones?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors address the limitations of their work in Appendix H, namely that they assume to know the number of sources, they assume they are directional point sources at fixed positions. I do not believe these limitations are much problematic as the main limitation is the unavailability of data required for scaling up the algorithm with respect to the supervised baseline (if it scales).

Summary

The focus of this paper is unsupervised speech separation by exploiting training mixtures with more microphones than speakers (over-determined). The proposed neural separator is using the input mixture to constrain the estimated images of each speaker. It is proposed to also train the system for under-determined scenarios, e.g., for single-channel speech separation, by separating a single-channel signal, while using multichannel loss function.

Strengths

The idea of using multichannel mixtures for unsupervised training is novel and reasonably effective compared to the supervised and unsupervised baselines. The experimental evaluation is sufficient and demonstrates the effectiveness and includes sufficient data to support some of the design choices. The paper is well written.

Weaknesses

It would be useful for the readers less familiar with speech processing to include references to a few papers very relevant to relative IR estimation, mixture consistency and linear prediction.

Questions

Only measurement/modeling noise is included in the model, and background noise or discrete noise sources would complicate things, e.g., since the same RTF could not be applied to the noise component. Is there any initial data on robustness in presence of noise? ll. 119 It is claimed that the relative room impulse response in (2) is typically short. However, it should be noted that this is not necessarily true, even for the microphone array and reverberation times used in the experiments (20 cm diameter, up to 0.5s RT60). A useful datapoint is paper by [Talmon, 2009], Fig. 7 in particular. I think this it would be helpful for many readers at NeurIPS to mention this (or a similar paper) and point out that this assumption does not always hold, and has been investigated in the literature. [Talmon, 2009] Talmot, Cohen, Gannot, Relative Transfer Function Identification Using Convolutive Transfer Function Approximation, IEEE Tr. ASLP, 2009. ll. 128 - Typo in “comptue” Section 4.2 Mixture consistency has already been proposed in [Wisdom, 2019]. This paper should be clearly mentioned. Also, consider renaming to multichannel mixture consistency, since you’re using P channels and RTFs. [Wisdom, 2019] Wisdom et al., Differentiable Consistency Constraints for Improved Deep Speech Enhancement, 2019. Section 4.3 Formulation is (6) is basically multichannel linear prediction with using estimated sources Z instead of the original mixtures Y. Relevant works solving a problem analogous to (6) should be at least briefly mentioned, such as earlier work in [Yoshioka, 2010] and many later works. [Yoshioka, 2010] Yoshioka et al., Blind Separation and Dereverberation of Speech Mixtures by Joint Optimization, IEEE Tr. ASLP 2011.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The main limitation is that the model does not take into account background or discrete noise, and is only demonstrated to work in presence of uncorrelated (measurement) noise. This severely limits applicability to using real-world recordings to train a separation model. It would be very useful to include results on robustness in presence of higher levels of noise. However, this is not essential and it may be out of scope of this paper. Another limitation for real-world use is the assumption of a static scenario (fixed relative IRs), which would not hold in real recordings.

Response to rebuttal

I thank the authors for clarifying some of my original points. I don't think this needs much more discussion, and I congratulate the authors on a remarkable paper!

Regarding Q2+Q3

Regarding Q2 - I think you should compare the speech separation approach rather than the speech enhancement approach. I would like to see the separation performance of classical speech separation models (e.g., Conv-TasNet or DPRNN) in the UNSSOR framework to assess the generality of UNSSOR better. Regarding Q3 - I look forward to your update on the results of MixIT's experiments to re-evaluate my recommended scores.