DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning

Summary

The authors propose a self-supervised speech representation that combines masked language modeling, online clustering, and self-distillation. They apply these techniques using jointly-trained transformer-based teacher and student models, where the student has to guess the cluster assignment of masked input. Evaluation of the student model demonstrates a new state-of-the-art speech representation on key metrics such as phoneme error rate and word error rate. Evaluation of the learned clusters indicates some visual agreement with the phoneme set of the CMU pronunciation dictionary.

Strengths

Having not worked with online clustering or self-distillation myself, I found the explanations provided in sections 1 to 3 to be an articulate and helpful introduction to the topics—as well as how they interact. Figure 1 is a well-done visual aid for the proposed method. Thorough evaluation on multiple relevant downstream tasks demonstrates clear improvements over state-of-the-art self-supervised speech representations. Figure 2 is an especially clear demonstration of the efficacy of the proposed representation. Overall a well-organized and compelling argument for a new state-of-the-art speech representation.

Weaknesses

Section 4.6 is well done, although the “mapping phones to codewords” section could be made more compelling by utilizing the mapping to perform phoneme classification and producing a phoneme error rate. The bolding in table 5 confuses me, both because there are multiple bold items per column, and some columns have no bold items without a clear reason.

Questions

Previously, open-source releases for this research task (e.g., wav2vec 2.0) have contributed significantly to downstream research efforts. Do you intend on open-sourcing your proposed model?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Please include some discussion on negative downstream impacts of the proposed work (e.g., these types of representations are commonly used for voice cloning, so the proposed work may increase the quality of systems used for non-consensual voice cloning).

Summary

The paper proposes a self-supervised paradigm combining the methods of self-distillation and online clustering. Specifically for each frame, a teacher model's activations are clustered based on initialized codebooks. The codebooks are then updated using a momentum based method. Then a student model is trained to predict the codebook index for each of masked frames. Authors run extensive experiments on ASR, acoustic unit discovery and SUPERB tasks comparing their methods with SOTA baselines such as wave2vec2, HuBert, data2vec and WavLM. While their method is simple in the sense the codebook vectors are not updated based on gradients, they either beat or perfrom very similar to the strong baselines used.

Strengths

The key strength of the paper is simplicity with which the code-books are updated combining the which is sort of momentum based k-means clustering rather than gradient based updating such as in wave2vec2 framework. At the same time they use a different teacher network embeddings unlike HuBERT model. Combing these two ideas leads to strong state-of-art performance. The cluster analysis results such as cluster purity, codebook perplexity and number of active clusters further show the effectiveness of this method over other online and offline clustering based self-supervised methods. The plots showing P(phone|code) gives additional insight how the phones are clustered to only a few code vectors. the concentration of phones over that The paper further extends its scope to multilingual experiments. The paper is well-written and enough details are provided for reproduction of experiments.

Weaknesses

1. No explanation given for hyper-parameter tuning experiments especially why we see the huge change as the top N layers for clustering is changed. 2. Why did the authors choose to sum the loss over top N layers instead of just using one layer? Each layer has different codebook associated which leads to an increase in number of code-books to be kept track of and hence more computation.

Questions

1. Did the authors do any study to compare the effect of summing the loss over all frames vs just the masked frames? 2. How does the conditional prob P(phone|codebook) look like as one goes closer to top layers? 3. Table 7 for multilingual experiments in supplementary material is not clear to me. For layer 5, Authors show that 1.1% of code-words are shared across 9 languages but it steeply jumps to 86.5% when all 10 languages are considered. What is the implication of that? 4. Figure 3 caption seems like it should be changed as N refers to number of code-books there.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Authors have experimented mainly on English datasets and hence they acknowledge the risks associated with neglecting other languages. But they do provide some results on multilingual experiments showing acoustic units are shared across languages in an attempt to mitigate that concern to a small extent.

Summary

The paper introduces a method called DinoSR for improved speech representation learning. DinoSR combines three existing key concepts: masked language modeling, self-distillation and on-line clustering. The authors demonstrate that these components complement each other and lead to a better model for speech representation learning by evaluating the learned representations on downstream tasks. The DinoSR works as follows. First, contextualized embeddings are extracted from the input audio using a teacher network. Next, an online clustering system is applied to these embeddings, resulting in a machine-discovered phone inventory. This step helps in identifying distinct units or phonemes present in the speech data. Finally, the discretized tokens from the clustering step are used to train the student network using MLM loss. (The teacher network params are EMA of student network).

Strengths

* The paper combines different contemporary approaches for speech representation learning to achieve a better representation learning. * Removes the offline clustering constraint of HuBert like approaches.

Weaknesses

* Small dataset evaluation - The results presented in the paper use a small dataset in the context of speech representation learning. It is not clear if the benefits of the proposed approach are still relevant compared to contemporary approaches in big data regime. * Generalizability to other downstream tasks - The speech representation extracted by the proposed method seem to superior to contemporary approached in ASR/phoneme related tasks. However, it is not clear if they extend to other downstream tasks such as speaker recognition/ emotion recognition etc. (Table 3) * Generalizability to other sizes or model architectures. - The choice of target layers in teacher model is a critical hyper-parameter. It is not clear if they have to be re-tuned for every new architecture or model size for optimal downstream performance.

Questions

* How does the proposed method compare to w2v-bert approach? * Can the authors add ASR results without LM.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

None.

Summary

The paper introduces DinoSR, a method that combines masked language modeling, self-distillation, and online clustering for self-supervised speech representation learning. The authors demonstrate that these concepts complement each other and result in a strong representation learning model for speech. DinoSR extracts contextualized embeddings using a teacher network and applies an online clustering system to discover meaningful acoustic units. The discretized tokens from the clustering process guide a student network. The paper claims that DinoSR outperforms previous state-of-the-art methods in several downstream tasks and provides a detailed analysis of the model and the learned discrete units. The key innovation of DinoSR, which is the introduction of a gradient-free online clustering method that leads to meaningful acoustic units. The authors emphasize their contributions in advancing the state-of-the-art in various benchmarks through end-to-end training and providing a closer examination of embeddings from speech transformers via discrete units. They also mention future work possibilities, including structural learning with the codebook, scaling to larger models, and extending the model to different modalities.

Strengths

Novel Approach: DinoSR combines self-distillation, online clustering, and masked language modeling for self-supervised speech representation learning. Detailed Analysis: The paper provides a detailed analysis of the model and the learned discrete units. This analysis offers insights into the underlying representations and contributes to a deeper understanding of speech processing. Key Innovation: Introducing a gradient-free online clustering method, leading to meaningful acoustic units, is highlighted as a key innovation of DinoSR. This innovation adds value to the field by providing a method for discovering and utilizing important acoustic units in speech representation learning. The proposed approach does very well on the general benchmarks showing its usefulness across tasks.

Weaknesses

Limited Discussion of Limitations: The paper does not explicitly discuss the limitations of DinoSR. It is crucial to acknowledge any potential drawbacks or constraints of the proposed approach to provide a balanced view of its strengths and weaknesses. Also, no ablation based on what performance gains are achieved after each stage is not provided, which would have helped understand the different building blocks of the proposed approach.

Questions

A note on the limitations of the proposed approach would have been useful.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper does not discuss the limitations of the proposed approach.

Summary

This paper proposes DinoSR, a self-supervised training method with self-distillation and online clustering. The key contribution of this work is the online clustering which is a gradient-free method to learn acoustic unit representations. The authors demonstrate that this approach outperforms previous state-of-the-art models on resource limited automatic speech recognition and the discretizing cluster is closely aligned with the human phonetic. The paper is well-written and provides clear explanations of the proposed approach. The experimental results are extensive and demonstrate the effectiveness of proposed works.

Strengths

1) The article is well structured and details the specific methodology of DinoSR, whose combination of masked language modeling, self-distillation, and online clustering is relatively innovative and can be compared convincingly with the current better self-supervised representations. 2) The work is experimentally more complete and has a high confidence level. 3) The authors provide the code which will benefit the community.

Weaknesses

1) The contributions of this paper are somewhat limited. Self-distillation and MLM task have been used in lots of previous work such as data2vec and SPIRAL. Furthermore, the online clustering has also been used in many previous works such as wav2vec 2.0 and vq-wav2vec. 2) It would be appreciated if the authors could provide more theoretical analysis about why the proposed method works well, especially in alignment with phonetic unit and the high quality of codebook. It will make the effectiveness of the work more persuasive.

Questions

1) As the key contribution of this work is the online clustering, it is suggested to provide some theoretical analysis or experiments to clarify why online clustering can align phonetics effectively. 2) This work does not take methods to keep codebook diversity, such as diversity loss in wav2vec 2.0, but has a very good codebook active rate. It would be valuable to analyze the reason with more explanations or experiments. 3) There are numerous online clustering methods available. Is the capability of model architecture having impact on the codebook quality? For example, VQ-APC still uses RNN, but wav2vec 2.0 and DinoSR use Transformers. Is the current comparison in a fair condition? 4) The following should be further clarified: * The “contextualized” is mentioned several times. To my knowledge, wav2vec 2.0 also learns the contextualized representations. But the paper states that wav2vec 2.0 is clustering of non-contextualized representations. Why? * Line 104-105 and equation 2, I am not sure whether equation 2 is accurately defined. * The demonstration of codebook update in section 3.2 is not sufficiently clear. The initialization of s_v and n_v in equation 3 is not mentioned in either the paper or the appendix. Though I was finally able to find this information in the code, it would be beneficial to include it in the paper. * Line 238, should the "default 8" for codebook size V be some other digit, e.g. 256? * Figure 3, the legend of y axis is missing. * In section 4.6, only the 5th layer is chosen for analysis. Why is this layer selected? What about the other layers? Is the perplexity same for all the layers?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The author does not talk about the limits of his article.

Summary

This paper proposed DinoSR, a novel self-supervised learning (SSL) approach for speech representations that combines the idea of the masked language model, self-distillation, and online clustering. It improves the data2vec method by using online clustering to obtain discrete targets from the teacher model and receives better results in multiple tests.

Strengths

The paper is well-written and easy to follow. A rich set of experiments were conducted, and decent results were obtained: the proposed method sometimes achieved state-of-the-art (SOTA) results and sometimes not too far behind SOTA.

Weaknesses

1. The major weakness to me is originality when compared to data2vec. Although the main idea of the paper is presented as combining the masked language model, self-distillation, and online clustering together, the joint use of the first two was actually studied in data2vec. Thus an alternative interpretation of the novelty is perhaps an improvement of data2vec by using discrete rather than continuous targets from the teacher. 2. Some secondary weaknesses: A. In Eq. 2, which frames are chosen to generate $\tilde{z}^k_t$? B. Masked Language Modeling (MLM) -> defined twice. C. In variables like $\tilde{z}^k_t$, better to quote the superscript $k$ with a bracket (e.g. $\tilde{z}^{(k)}_t$) to avoid confusion. D. Not sure if the results of the probability-based metrics in Table 5, such as perplexity and mutual information, are comparable for the online clustering systems since different numbers of active clusters are results that lead to incomparable sharpness of distributions. E. In the reference sections: volume and pages are missing for journal references; "In" is missing in NIPS references, such as [4] and [11].

Questions

"By labelling each codeword using articulation manner classes in English, we revealed the fact that some of the acoustic attributes are embedded in the high-dimensional space. For example, vowels and silences demonstrated a high degree of concentration." Could you please explain how can this be revealed in a two-dimensional space? I personally interpret the figure from a reverse perspective: Figure 5 verified the codebooks have good correlations with articulatory features, as many belonging to the same category have a high degree of concentration.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

1. Though some initial multilingual results were presented in the supplementary materials, in the paper itself, the proposed method is only verified using English data. 2. Only clean data was used in the training and test. Perhaps worth investigating the use of the proposed method on data with general audio events and noises. 3. Since online clustering was performed on different layers, it would interesting to understand further the differences and similarities of the codebooks obtained based on different layers.

Thank the authors for the feedback

I thank the authors for the feedback. The authors responses have addressed most of my concerns raised in the previous review. I suggest the authors incorporate their responses (especially the theoretical analysis) into the new revision of the paper. I raise my score from 6 to 7.