Textually Pretrained Speech Language Models

Summary

The authors explain a method to improve the performance of a speech language model by reusing the weights of a language model trained on text.

Strengths

Using a well-trained text-based language model as an initial model seems like a good idea. If the authors' claims can be generalized, it can be used as a good starting point for speech language models, which are relatively difficult to train.

Weaknesses

The authors empirically discovered that they achieved better performance in the same training step, but there seems to be a lack of theoretical analysis. The process of analyzing, estimating, and confirming evidence for why such improvement occurs is missing. This simple empirical discovery without the analysis and validation of why it happens would have limited contribution to the conference-level community.

Questions

I would like to ask the authors for additional comments regarding the following points and the weaknesses mentioned. The audio samples provided by the authors are not sufficient. While the authors support their claims through various experiments in the manuscript, only some of the samples are included. While source code is the most clear evidence for the experiments and results, it is necessary to present other detailed experimental results when the source code is not available. Furthermore, in the field of speech synthesis, subjective evaluations hold more weight than objective evaluations, so it is essential to provide reviewers with the opportunity to listen to and evaluate the samples directly. The method proposed by the authors is a form of transfer learning. However, when observing the training phase's perplexity (PPL) curve presented by the authors, it is difficult to consider the model as having converged at the 400k step where the authors claim to have ended the training. Therefore, it is not clear whether the performance difference claimed by the authors is simply due to the difference in convergence time caused by the initialization of the initial weights or the advantages obtained from the text-based language model (such as having trained on larger data, more diverse contexts, etc.). The MMOS results are presented in a table. It is difficult to verify the confidence intervals (CI). If the CI cannot be verified, it should be indicated how many evaluators participated and with how many samples.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The limitations have been well described.

Summary

This work proposes TWIST, Textually Warm-Initialized Speech Transformer-based LMs, a technique to initialize SpeechLMs with pretrained textual LMs. Different textual LMs, tokenizers, models and dataset sizes are evaluated using TWIST. The authors find that a warm start with a textual LM helps compared to a random initialization when evaluated on a number of metrics.

Strengths

- TWIST shows how a warm start with any pretrained textual LM (OPT, Bloom, Pythia) benefits a speech-based LM. - This is one of the first works to use large-scale audio data (i.e., approximately 150K hours of speech) to train a speech-based LM. - This work presents a new evaluation benchmark for speech-based LMs based on StoryCloze.

Weaknesses

Warm start with pretrained textual LMs was found to be effective for speechLMs. A more detailed analysis of the learned representations with and without TWIST would have been useful for the reader. I elaborate on this further in my questions below.

Questions

- An analysis of the learned speech-based discrete tokens with and without the use of TWIST would be interesting to see. For example: - Visualizing the representations learned via TWIST (i.e., a warm initialization from a textual LM) and those learned via a randomly initialized model. Color-coding representations based on phoneme labels (derived from a forced alignment) might reveal that phoneme distinctions are less pronounced with using TWIST. - Training simple probing classifiers (e.g., to recognize phones or subword textual units) whose inputs are speech representations with and without TWIST and comparing their accuracies. - Due to the warm start from textual LMs that model long-term textual dependencies, one would assume that TWIST would do well on measures like sBLIMP that look at the grammaticality of entire sentences. However, from Table 3, TWIST does not fare as well on sBLIMP as prior work from Borsos et al. Could the authors comment on this? Also, it would be useful to show numbers using a randomly initialized model (in direct comparison to TWIST) in Table 3. - Are the results in Table 2 using TWIST with OPT-350M or OPT-1.3B? Please clarify. Assuming it is 1.3B, why is the sBLIMP score for TWIST-1.3B in Table 3 (60.3) different from that reported in Table 2 (59.3)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations have been addressed in the submission.

Summary

This paper proposes a simple initialization method for speech LMs named, TWIST (Textually Warm-Initialized Speech Transformer Language Models). Instead of cold-initializing a speech LM, twist initializes it with LLM weights (minus the token embeddings, which are replaced with speech vocabulary). The authors show than on OPT models of size 125M to 1.3B (+ BLOOM, Pythia of 350M size), the proposed method of initialization helps in convergence and final perplexity evaluation, as well as shows benefits over cold initialization over human evaluation (MMOS) of generated completions. The authors also create two spoken versions of the StoryCloze dataset.

Strengths

1. The proposed method is quite simple (just a straight-forward initialization trick) and the empirical results show small benefits across speech LMs of different sizes.

Weaknesses

1. The experiments could be more comprehensive, e.g., only one size of BLOOM/Pythia is considered. Similarly, the authors do not report the PPL over the transcription of the generated speech. Even if the results are high-variance, it is an important aspect of evaluation as far as the quality and diversity of speech continuations is considered. 2. The proposed method is extremely simple, it just initializes the speech LM with LLM weights. There is prior empirical justification to do this, however, the weight spaces as quite incongruent, as acknowledged by the authors, and this problem hasn't been addressed at all. As such, it is hard to consider the contribution very solid in terms of its technical merit. The authors show that text based initialization outperforms Image based initialization, but that again is not a very novel result.

Questions

1. Why not include the PPL over speech transcription results in section 3.3? If only one component is varied (cold-init model vs TWIST), then the results could be quite useful? 2. The data scaling is highlighted as a major contribution, however, except OPT other models aren't varied upto the Billion scale.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Summary

This paper studies the effect of textual LM on SpeechLMs. They propose TWIST, which initializes the SpeechLM model with a pre-trained textual LM, and then finetune with the speech datasets. This paper provides a complementary exploration of generative spoken language modeling, including front-end processing of speech tokenizer, the core component of SpeechLM, and back-end speech synthesis of Vocoder. TWIST uses a warm-start for SpeechLM and outperforms the cold-start model in all experiments. In addition, this paper also provides empirical results across various pretrained models, and presents the large-scale 7B-sized Speech language model.

Strengths

1. This paper provides insight into the design of SpeechLMs based on rapidly developing large- scale textual pretrained models. 2. This paper presents many empirical conclusions and findings, such as the HuBERT setting, results on different scales of training data. 3. This paper introduces new benchmarks for evaluating SpeechLMs.

Weaknesses

1. Some details are missing. For example, warm-start allows better performance compared to cold-start, but training costs for both methods are not reported. These results help researchers estimate the training costs of large-scale SpeechLMs. 2. Automatic evaluation results are missed. This paper reports the human evaluation of MMOS for speech generation, but no automatic evaluation results are included, like WER for ASR results. Therefore, it is different to compare results across related work. 3. Although TWIST shows powerful capability of speech understanding, SpeechLMs still lack deep semantic understanding compared to textual models, as stated in Limitation. Therefore, the proposed method is somewhat of a compromise. Despite these issues, I still believe that this work is highly valuable and can provide direction for the following researchers.

Questions

1. Does HuBERT use different vocoders to resynthesize speech for different clustering tokens and frequencies in Section A.2? 2. Can the authors provide an explanation for the poor performance of cold-start methods? Is it because there is insufficient training data to fully train the model?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Yes