Learning Repeatable Speech Embeddings Using An Intra-class Correlation Regularizer

Summary

Starting with the observation that supervised embeddings should vary from one class to another but not be sensitive to variations within a given class, this paper proposes to add an intra-class correlation (ICC) regularization to the contrastive loss in representation learning. This new regularizer is derived from measurement theory and encourages the embeddings to have a low intra-class variance, whereas contrastive losses pushes more the embeddings towards high inter-class variances. After a simulation illustrating the "behaviour" of these losses with respect to inter- and intra-class variance, the authors report good improvements using the proposed ICC regularizer for three different tasks.

Strengths

- The paper is overall well written, the idea well explained and easy to follow and to understand - The ICC regularizer seems to really focus on intra-class variance, i.e. make all occurences of the same class have the same embeddings, where contrastive loss does both at the same time and never really focuses on minimizing the intra-class variance. - The method is relatively simple, very well explained and easy to understand. The simulation, comparison with contrastive loss and classification error rate is quite interesting. - When the embeddings are constrained, it is indeed useful to focus on minimizing the intra-class variance to separate classes. The explanation and results are convincing.

Weaknesses

- Missing comparison with other techniques mentioned in the related work (data augmentation, etc.). Is the proposed approach competitive with these methods? Could it be complementary? How does it compare to VICReg [4] for example - In the result section, the text mainly lists the results in the tables. Maybe a reformulation would allow more space for example to study the effect of the regularization weight

Questions

Could that method apply to self-supervised approaches, where the notion of classes is less well defined? Is it competitive with other methods mentioned at the beginning of the paper? Or could it be complementary? Are the improvements with that method bigger on smaller datasets or are they independent of the dataset size? Is the method especially suited for some tasks and less for other (e.g. could that be useful for classification, or ASR?)? minor remark: - sec 3.1: "represents the i-th samples" -> sample

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

yes

Summary

The paper introduces the ICC regularizer, a novel regularization technique that complements the contrastive loss to enhance the repeatability of embeddings. The authors illustrate the reason why the ICC regularizer better minimizes the intra-class variance than the contrastive loss, offering a fresh perspective on latent representation learning. The experimental results demonstrate that the ICC regularizer enhances the repeatability of learned embeddings. Consequently, embeddings with higher repeatability exhibit superior performance in downstream tasks.

Strengths

1. This paper links the concept of repeatability, derived from measurement theory, to deep-learned embeddings, which is novel and interesting. The authors present an explanation of how the ICC regularizer effectively reduces intra-class variance compared to the contrastive loss, offering a novel perspective on latent representation learning. 2. To enhance the repeatability of the learned embeddings, this paper introduces a novel regularizer called the ICC regularizer, which is used as a regularizer for the contrastive loss. 3. The experimental results indicate that using the ICC regularizer leading to better performance in three downstream tasks .

Weaknesses

My main concerns are mainly related to the experiments. Please refer to the following Questions section for details.

Questions

1. In Section 4.2, ''GE2E Loss + ICC Regularizer'' shows superior speech naturalness on subjective evaluations. However, the objective metrics like speaker similarity scores and word error rate (WER) are missing. 2. In Section 4.2, the paper aims at zero-shot voice style conversion, which requires highly generalized speaker embedding. However, the authors train and evaluate the zero-shot voice style conversion on a small dataset called VCTK. I would like to ask if the generalization performance of the proposed method be further improved by using larger training datasets? Additional experiments on larger dataset would make the paper more convincing. 3. Although the audio samples with ICC regularizer is better, after listening to the audio samples in the supplementary materials, I think the audio quality should be further improved for practical scenarios. I would like to ask is this due to the used AutoVC model? Could the authors use the latest SOTA voice conversion model to improve the speech quality? I would like to raise my scores if the authors have fully addressed my concerns.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Some limitations of the work are discussed. The potential negative societal impact of the work is not addressed.

Summary

The paper presents a simple but seemingly effective idea to regularize contrastive embedding extractors. The regularizer aims at the intra-class correlation to ensure the repeatability of the embedding. The effectiveness is confirmed experimentally for the task of speaker verification, voice style conversion, and dysphonic voice detection.

Strengths

The presented idea is well introduced and the illustration of the ICC regularizer action compared to the GE2E loss is insightful. The evaluation is sufficient.

Weaknesses

The introduced regularizer uses 2nd order statistics only which could be a limiting factor on the embedding extractors but on the other hand this doesn't need to be required.

Questions

The contrastive loss has a direct relationship to the mutual information, one could think that ICC is automatically included in the mutual information when only 2nd order statistics are taken into account, what could be the reason that this doesn't seem to be the case.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The introduced regularizer uses 2nd order statistics only.

Summary

In this paper, authors propose to use an intra-class correlation coefficient (ICC) regularizer as a complimentary component for contrastive loss to increase the repeatability of the supervised speech embedding. Authors conduct simulation for ICC regularizer and contrastive loss (GE2E loss) to explain their similarities and differences in the optimization criteria. Based on the experimental results on the speaker verification, voice style conversion, and a clinical application for detecting dysphonic three tasks, the authors claim that adding an ICC regularizer helps increase the repeatability and further improve the performance on downstream tasks.

Strengths

1. This paper leverages the concept of repeatability and states that higher repeatability can contribute to improved performance in downstream tasks. Authors think about the importance of intra-class variance and the proposed ICC regularizer places greater emphasis on minimizing it to help increase repeatability. The simulation results in Section 3.2 explains the intuition clearly. 2. As a complementary component to contrastive loss, adding the proposed ICC regularizer successfully increase the repeatability and shows good performance in the three different tasks.

Weaknesses

1. In Task 1, the performance of SV system is not good though adding the ICC regularizer improves its performance. In this paper, VGG-M-40 is trained on VoxCeleb1 & 2. The best result (3.78% EER) reported in Table 1 is not as good as other commonly used models for SV. In Chung et al. paper, the VGG-M-40 shows much worse performance than Thin-ResNet34 and Fast ResNet34. To make the experimental solid and results and claims more convincible, authors should pick a good baseline instead of use the VGG-M-40. 2. For better replicable, more experiment details should be provided, such as the training details of Task 1. 3. To have comprehensive evaluation results for Task 1, the evaluation metric minDCF can be used as well. 4. Visualization (such as t-SNE) can be added as a comparison to see if the extracted speech or speaker embeddings have smaller intra-class variance in the space, though their ICC is decreased.

Questions

1. As this paper mentioned, the ICC regularizer requires hyperparameter fine-tuning. Are the hyperparameter tuned on a validation set or a development set? 2. As this paper mentioned in Section 3.2, though the ICC increased, do the learned embeddings are more compacted within-class and more separate between-class in the overall embedding space? Authors can use t-SNE to visualize them.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors do not analyze the potential negative societal impact.

Summary

In this paper, the concept of repeatability from measurement theory was introduced for representation learning. Intraclass Correlation (ICC) was proposed as an evaluation metric and the ICC regularizer was used as an additional element to contrastive loss for training. This aims to enhance the repeatability of deep-learned embeddings. Some synthetic examples and intuitive reasoning were used to illustrate why the ICC regularizer is superior in minimizing intra-class variance compared to contrastive loss. The ICC regularizer was assessed across three speech tasks: speaker embeddings for Text-Independent Speaker Verification (TI-SV) and zero-shot voice style conversion, along with voice feature embeddings for a clinical application. The experiments indicate that the ICC regularizer can boost the repeatability of learned embeddings and perform better in downstream tasks.

Strengths

The paper proposed a novel metric of ICC based on measurement repeatability. Although the term may be a little confusing, the motivation and intuition were well explained. The experimental results on three speaker/speech tasks are convincing.

Weaknesses

To show the repeatability of speaker embedding for TI-SV task, more controlled experiments may be necessary to prove the effectiveness of ICC loss in addition to the ICC scores: for example, adding noise to the testing sample, or switching languages of the same speaker as it was tested in the NIST Speaker Recognition Evaluation. For results on dysphonic voice detection in Table 3, other baseline models were definitely over-trained compared to the proposed model. Why do you train your own models instead of quoting the numbers from the original papers? (see line 334-335). The results in Table 3 may be unfair comparisons.

Questions

See the comments above in the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No discussion on negative societal impact was included in the paper

Summary

This paper proposes to use a traditional intra-class correlation coefficient (ICC) to assess the repeatability of speech embeddings learnt by neural network. The proposed ICC regularizer has characteristics similar to well-known contrastive loss which aims to minimize the intra-class variance and maximize the inter-class variance of training data. The authors propose to use the ICC regularizer not in isolation but in combination with the contrastive loss. This paper performs several types of experiments based on speech processing tasks such as speaker verification, voice style conversion, and dysphonic voice detection to demonstrates the effectiveness of the proposed regularizer. These tasks focus on the importance of detecting the repeatability for the target data.

Strengths

- The considered scenario is interesting and important to the community. The proposed method is simple yet effective in assessing the repeatability in the target data. - This paper assesses the effectiveness of the proposed method not only with a single speech task but also with several tasks based on voice quality evaluation. The improvements look sufficient.

Weaknesses

- ICC is very traditional metric, which is widely used for various data classification algorithms. - The proposed ICC regularizer shouldn't be used in isolation. - Lack of some experiments and details of experimental setup (see questions below).

Questions

- It looks like that the proposed ICC regularizer corresponds to the special case of contrastive loss focused only on positive sample pairs. Although there are many variants in contrastive loss, the basic concept is to increase a capability of category classification of data by leveraging both positive and negative pairs. I am wondering if the training only with conventional contrastive loss could provide some improvements similar to the proposed method, for example, by simply changing the ratio of positive sample pairs (reducing negative sample pairs) in the last half of the training stage. What are advantages that cannot be obtained by just changing the ratio of positive and negative pair data? - It is difficult to understand the effectiveness of the proposed method correctly because experimental setups including data are very briefly written. For example, in the experiment described in Section 4.1, what are the training strategy of the model such as an optimizer, a learning rate scheduling, other regularization techniques (specAugment, DropConnect, etc.), data augmentation, and so on? - In addition, a summary of data used for each experiment (a few more details) would be very much helpful.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The same comments as one of weaknesses. The proposed ICC regularizer shouldn't be used in isolation but be used in combination with a conventional contrastive loss as clearly mentioned in the paper.

Lowering the ratio of negative sample pairs

Thank you for the response and some new results. I understood that on the Monte Carlo simulation lowering the ratio of negative sample pairs (adding the negative pairs score) on GE2E loss provides undesirable impact for the problem setting in this paper. I assume that you have done the simulation with the same ratio between positive and negative sample pairs during an entire stage of training (in other words, all the epochs). However, the appropriate ratio of positive and negative sample pairs for each downstream task should be different. It can be estimated with a development set. In that sense, it is still unclear and my concern if the proposed method yields sufficient improvements over such baseline systems in actual downstream tasks.