REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR

Summary

This paper proposes a training framework for unsupervised speech recognition models, building on top of wav2vec-u. The authors note that the performance in wav2vec-u is hindered by the quality of segment boundaries, as they show that the phoneme-error-rate can be improved significantly with ground truth segmentation boundaries. The proposed method introduces a segment boundary prediction mode and a phoneme prediction model. First, the segment boundary prediction model is trained with a policy gradient method. The reward function is based on the perplexity of the phoneme sequence. This sequence is predicted from the computed segmentation boundaries, and is compared to earlier versions of the boundary prediction model, i.e, the boundary segmentation model from a previous iteration, or wav2vec-u at iteration 0. The perplexity should be as low as possible and can be computed by using a language model from unpaired text data. Secondly, the phoneme prediction model is trained with an adversarial loss, where the phoneme prediction model acts as a generator of a phoneme sequence, and the discriminator has to distinguish between these and real phoneme sequences from the unpaired text data. These steps as then repeated until no performance improvements are observed.

Strengths

The authors clearly show that the wav2vec-u model suffers from the quality of the segmentation boundaries, and propose a well-described methodology to improve on it. The authors report results on the expected datasets, and perform a small ablation study to analyze the behavior of the proposed method.

Weaknesses

The authors propose a *very* complex methodology, with a lot of moving and finicky components, as reinforcement learning and GANs are both known to be fickle and hard to reproduce. I worry therefore that it will be hard to reproduce these results, but the authors promised in the checklist to release the code. The authors use many tricks to make the method more stable and performant: 1. Initialize the phoneme prediction model with the pre-trained wav2vec-u model 2. Introduce 2 auxiliary reward functions, which heavily punish predicted sequences which are longer or shorter than the previous iteration 3. Use behavior cloning (supervised loss on boundary pseudolabels) before the RL loss 4. boundary merging (segments with the same phoneme prediction are merged) 5. self-training of the phoneme prediction model (supervised loss on phoneme pseudolabels) These are not a weakness per se, but point towards the overall complex picture of the proposed method. I hope future work can find a simpler and more elegant formulation of the UASR problem :) The authors claim their method is iterative, but in paragraph 4.3 state they only use 1 to 2 iterations. I then wonder why the authors try to sell this as an iterative method, instead of follow-up stages to (significantly) improve on wav2vec-u. Minor comments on writing: * line 21: supervisedly trained with a ... -> trained with self-supervision using a ... * line 33: translate and -> translate, and * line 50: model: -> model. * line 74: remove "the" * line 77: GAN -> The GAN * line 95: The wav2vec-U network * line 98: to -> for * line 99: I cannot really parse this sentence, origanated to be -> originally (?) * line 102: We discuss these two related topics further in Appendix F. * line 149: The REINFORCE algorithm

Questions

Why do you state in 4.3 to only use 1 to 2 iterations, while figure 2 shows results with improvements until 5 iterations? Why do you think that your model performs better on 5 languages but worse on Portuguese compared to wav2vec-u 2.0? Why did you decide to mean-pool the segment before predicting the phoneme? I would venture that this throws away information and is not necessary?

Rating

7

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

The ultimate goal of UASR is to enable speech technology for digitally rare languages without good labeled data. Although this work studies UASR on 6 languages, they are all from rich European countries. It remains to be seen whether this method works for languages from a different continent, especially because this method relies heavily on high quality textual data, to compute the perplexity in the RL loss, and for the discriminator.

Summary

This paper addresses the challenging problem of learning speech recognition without parallel recordings and transcriptions. They build upon state-of-the-art approaches of Wav2Vec-U(2) in this area and propose to refine a pretrained model using a reinforcement learning approach. The main idea is to split the problem into two parts: segmentation and segment classification into “phonemes”. Since neither segmentation nor labels are known, they use policy gradients and leverage “phoneme” classifier rewards to learn the segmentation model. The training is stabilized by splitting it into iterations where only one of the two models is being trained while the other is frozen, and additional rewards (penalties?) that act as regularizers for the main perplexity difference reward. In essence, the model is overall encouraged to improve the perplexity vs the previous model checkpoint, but not change either the segment length or phone edit distance too much. The evaluation is performed on 100h subset of LibriSpeech and full TIMIT for English, and 100h subsets of MLS for German, Dutch, French, Spanish, Italian and Portuguese. The authors demonstrate significant improvements in WER and PER vs state-of-the-art baseline models.

Strengths

* Their main idea of framing UASR as an RL problem is original and interesting. Overall I think it is a solid contribution to UASR literature. * The proposed RL training is able to significantly improve upon Wav2Vec-U ASR results. * The evaluation setup is comprehensive enough to believe the method may be generally applicable for other languages. * The method, architecture, and experimental setup are described comprehensively. * The ablations related to reward function designs are interesting and demonstrate that the proposed edit-distance and length-difference rewards are effective regularizers for the main perplexity difference reward.

Weaknesses

* The author’s (mis)use of the word “phoneme” is problematic. E.g., p.2 l.64 mentions “segmental structures (that) are acoustic units smaller than phonemes”. Phoneme is a perceptual construct that represents all possible (sequences of) sounds (phones) for which there is no lexical distinction. Such sequence may very well be empty. Therefore it is problematic to talk about phonemic segmentation; what the authors may have had in mind is phonetic segmentation instead. This problem is discussed more thoroughly in: Moore, R. K., and L. Skidmore. "On the use/misuse of the term 'phoneme'." Proceedings of Interspeech 2019. * The authors claim that “the effect of feature segmentation on phoneme prediction is non-differentiable”, however, it can be. The authors may consult e.g. the segmental CPC approach with its differentiable boundary detection in: Bhati, S., Villalba, J., Żelasko, P., Moro-Velázquez, L., Dehak, N. (2021) Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation. Proc. Interspeech 2021, 366-370, doi: 10.21437/Interspeech.2021-1874

Questions

* Could REBORN be leveraged in a semi-supervised setup, e.g., starting from an ASR pretrained on a low amount of supervised data, to leverage a larger amount of non-parallel data for fine-tuning?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

* The proposed method depends strongly on the availability of an initial pretrained ASR model (presumably unsupervised). My understanding is that it is more of a “refinement” stage for existing models rather than stand-alone method. I think it should be presented as such - the title and the abstract don’t mention that.

Summary

The paper proposes an improvement over wav2vec-U for unsupervised speech recognition, specifically for the task of predicting phonemes. For the final WER performance, a given lexicon is used to get from the phoneme sequence to words. So the paper focuses on improving the segmentation/boundaries of the phonemes. Two model parts: The segmenter, trained via RL and various different reward functions. GAN-style training for the phoneme prediction model based on the segments from the segmenter. Stage 1 trains only the segmenter, and assumes a given frozen phoneme prediction model. In the first iteration, the phoneme prediction model is initialized from wav2vec-U. Stage 2 trains only the phoneme prediction model, taking the segments from the segmenter. In GAN-style training of the phoneme prediction model: Phoneme prediction model is the generator, and discriminator tries to distinguish a generated/predicted phoneme sequence vs a real one, using the unpaired text data.

Strengths

Good results. Code will be published.

Weaknesses

Some parts could be made more clear. (See below.) Needs Wav2vec-U for initialization.

Questions

So you train a phoneme generator model here. But it's a bit unclear, how do you get from phonemes to words? It refers to the appendix, and WFST decoding is mentioned. It mentions that it needs a lexicon. So to make this clear: the lexicon provides a given mapping from phonemes to words? If this is given, is it fair to call this unsupervised ASR then? It seems like you cheat here by giving it a crucial part for the unsupervised ASR task. While this is still an interesting problem to study then, I think this should be made much more clear. This is not really unsupervised ASR to me. Or if you really want to stick to the term, at least make it very clear (not just in the appendix, and only for readers who know what a "lexicon" is) that this is what you take as a given here. This also has further implications: The phoneme inventory is then also given. The exact definition of the phoneme predictor/generator is a bit unclear. Is this an auto-regressive model? Or non-autoregressive? How do you sample from the phoneme predictor for policy gradient? Does this involve beam search?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

-

Summary

This paper proposes a new UASR approach which explicitly learns both segmentation (phoneme boundary) prediction and phoneme class prediction. To do segmentation, the paper proposes some reinforcement-learning objectives. The phoneme class prediction follows an existing approach. It turns out the proposed approach is effective, achieving the new SOTA for UASR on public datasets.

Strengths

- The paper's presentation is clear and is easy to follow. - The proposed technique (RL for learning segmentation model) is novel and non-trivial - The proposed technique is effective. It achieves SOTA by a big margin across several datasets and languages. - There is analysis why the proposed technique works so effectively - See my concern later

Weaknesses

- In fact, ASR don't need accurate segmentation, e.g. CTC model is peaky -- the peak of a phoneme can appear at any frame for this phoneme. For this reason, it is not so clear to me why improving segmentation/phoneme granularity can help improve UASR so much. - Please justify. - For this reason, I put the rating to "border line" instead of "weak accept" - Unfortunately, the predicted segments are not more accurate than other approaches and probably are not good to be used to obtain accurate timestamps for ASR (or not? -- you may use TIMIT or Buckeye's ground-truth timestamp to check this) - L324: "some segmental structures smaller than the phonemes": does it probably mean that -- if we use sub-phoneme structures in wav2vec-U 2.0, it may be as performant as the proposed approach, as we are able to learn a smaller granularity of signals. In this case, we won't need the RL learning. - Would be better to give a real example

Questions

- Besides Table 4, it would be better to add a table for TIMIT which comes with word/phone-level timestamps. Another dataset Buckeye can be considered. - Typo or presentation suggestions: - Figure 1: probably add "1 means the start of a segment" to the caption to make it self-contained

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes.

Rebuttal and discussion needed

Hi folks, AC here. I'm trying to encourage all parties (authors and reviewers) to chip in, since there are some disagreement among all. I have read the paper and all the reviews. Here are three points I would like to hear from you. 1. It's unclear how the approach is different from Cuervo et al. (2022) (cited as [15] in the submission). Given that the authors are aware of this paper, the paper clearly fails to address the novelty (also raised by reviewer AM1A). Even the reward functions are very similar to the ones in Cuervo et al. (2022). Cuervo et al., Variable-rate hierarchical CPC leads to acoustic unit discovery in speech, 2022 2. The positive reviews are mostly due to the strong results, while being sparse on the soundness of the paper and where the improvement is coming from. Reviewer TC8W is not convinced that the improvement is coming from segmentation, while reviewer q9jx and AM1A think the improvement is from the iterative training. 3. It's a little surprising to see so few segmentation results except Table 4 (also raised by reviewer TC8W), given how much the emphasis is on segmentation. The results in Table 4 are weird. 1) Worse segmentation results actually lead to better PERs, contradicting to the claim in the paper that better segmentation leads to better PERs. 2) The proposed approach is over-segmenting a lot and does not seem to perform well compared to something as simple as k-means. 3) The paper deliberately emphasizes that the result is without boundary merging. Why not report both with and without? 4) How come using the oracle segmentation leads to a worse PER? The segmentation results on TIMIT (in the reply to reviewer TC8W) are useful, but don't seem to answer the questions around Table 4. Thanks.

Acknowledgement of rebuttal

Thank you for the explanation and details in the rebuttal. With the explanation of that segmentation is critical for UASR, I'll increase my rating.