Disentangling Voice and Content with Self-Supervision for Speaker Recognition

For speaker recognition, it is difficult to extract an accurate speaker representation from speech because of its mixture of speaker traits and content. This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is realized with the use of three Gaussian inference layers, each consisting of a learnable transition model that extracts distinct speech components. Notably, a strengthened transition model is specifically designed to model complex speech dynamics. We also propose a self-supervision method to dynamically disentangle content without the use of labels other than speaker identities. The efficacy of the proposed framework is validated via experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average reductions in EER and minDCF, respectively. Since neither additional model training nor data is specifically needed, it is easily applicable in practical use.

Paper

Similar papers

Peer review

Reviewer P6Zh7/10 · confidence 1/52023-07-03

Summary

In this paper, the authors proposed RecXi, a framework that disentangles the voice and content for speaker recognition. Also, the authors used self-supervised learning methods to improve performance. The authors focused on the fact that speaker information is static and the content information is dynamic, so the main idea is the method to decompose static information and dynamic information. To achieve their goal, they designed a three-layer Gaussian inference framework. They argued that more accurate speaker representation is decomposed from layer 1 to layer 3 because they extract the content representation in layer 2. In the experiment section, the authors compared the proposed model with previous works and show the proposed model outperforms several different datasets.

Strengths

- Disentangling different features clearly with self-supervised learning is a very challenging task in many domains. The authors sapientially solve this problem by aiming at the temporal characteristics of the speaker and content. - The proposed method outperforms the existing works.

Weaknesses

- The authors argued that more accurate speaker representation is decomposed from layer 1 to layer 3. However, it is not verified with the ablation study.

Questions

- It is a novel method but it is quite complex to understand intuitionally. Could you provide more visualized results or example audio for the supplementary material?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

- In this architecture, it is very difficult to interpret how the model decomposes different features.

Reviewer krxU6/10 · confidence 4/52023-07-06

Summary

This paper describes a novel speaker representation based on a three layer Gaussian inference network for speaker recognition. The model approach uses a static representation for layers 1 and 3, but a time-variant representation for layer 2. The authors position this network structure as disentangling speaker and content information, as speaker information is constant across a speech signal, while content information is time-varying.

Strengths

The motivation is well justified. The description of the approach is well written and relatively easy to follow. Good details are included in Table 1 about which models are reimplemented and which are reported from published results. The paper includes a nice ablation to understand the contribution of l_ssp and the availability of both \phi and \phi_t

Weaknesses

The best performing model includes \phi and \phi_t both the static and dynamic components for speaker representation. This has two limitations – first, it makes for a much larger representation vector, and the length of the vector is dependent on T (the max length) being fixed across samples, second, it somewhat undermines the central claim of the proposed technique that the static \phi should contain speaker information while the dynamic \phi_t should isolate content contributions. The introduction claims that using pretrained ASR to isolate content is suboptimal because it requires too many parameters. Therefore the aim of this work is to isolate content from the representation with fewer parameters than a full ASR model and without text labels available during training. This is a fair set of constraints, but it would be useful to have a comparison to a model that uses an explicit content-aware decomposition. That is, how well does a speaker recognition model like those enumerated in the Introduction that uses an ASR model, even at the cost of additional computation, perform at this task?

Questions

Would a 5 layer network provide improved quality by including a second layer of time-varying inference as layer 4 and a third static layer as layer 5 to further refine the static/time variant representations? (if so, what about 7? etc. Is T, the maximum length fixed a priori for all samples? If so, are RecXi representations robust to different utterance lengths with varying T? How is N tuned in the Frame-wise Content-aware Transition model G_t? Page 5 Section 3.2 re Layer 3: how expensive are the matrix multiplications such that simplification is necessary here? How much improvement to computation (FLOPS? latency?) are these optimizations providing? Table 1: Are 4 significant digits necessary to understand the results? many include only 2. It might be clearer to be consistent across all results. Any concern about other channel characteristics, noise, or reverberant environment characteristics being captured by the static representation?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are included in the last Appendix in the supplemental material. Per https://neurips.cc/public/guides/PaperChecklist "You are encouraged to create a separate "Limitations" section in your paper." I'd encourage the authors to include this section in the body of the paper, but the limitations are raised nonetheless

Reviewer 48tf6/10 · confidence 4/52023-07-07

Summary

In this paper, authors propose a disentanglement framework for speaker recognition that extract speaker representations from the mixture of speaker traits and content. The proposed framework utilizes three Gaussian inference layers to decompose the static and dynamic speech components. Additionally, a self-supervision method is also proposed to help disentangle content without relying on text labels. Based on experimental results, the authors claim that the proposed method achieves remarkable performance on the VoxCeleb and SITW datasets, which demonstrates the effectiveness of the framework.

Strengths

1. The authors conduct solid experiments and the proposed method shows good performance on VoxCeleb and SITW. 2. This paper shows detailed experimental results and a comprehensive comparison with several solid baselines. 3. The paper is well-written and easy to follow.

Weaknesses

1. The proposed RecXi model with self-supervision speaker preserving loss generative very limited improvement compared to the Xi baseline system.

Questions

I was wondering if authors evaluate the proposed method on some more challenging datasets, such as VOiCES?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors analyze the limitations in Section 6.

Reviewer 6npJ6/10 · confidence 3/52023-07-07

Summary

This paper proposes an improvement over Xi-vector speaker recognition algorithm. They propose an iterative 3 layer Gaussian inference mechanism to disentangle the content information from speaker information. The first layer gets precursor speaker representation similar to Xi-vector framework. The second layer uses the encoder embeddings and embeddings of first layer to disentangle the content information. The third layer finally gets the disentangled speaker representation by subtracting the second layer embeddings from encoder embeddings. They also take inspiration from knowledge distillation framework and use a similarity preserving loss that preserves the similarity between third layer embeddings and the difference between first and second layer embedding. This seems to improve performance in their ablation studies. They compare their model with various baselines that do not disentangle non-speaker info from speaker and that do not use the similarity preserving loss show an improvement in performance.

Strengths

A key contribution of the paper is improving Xi-vector framework to disentangle content info from speaker info. The iterative mechanism to achieve that is well motivated. The proposed similarity preserving loss designed further leads to performance improvement. Another minor contribution of the paper is changing the ResNet34 backbone to a tResNet34 backbone by changing the stride strategies based on the improvement seen in preliminary experiments. That could be a reason why the results reported here with Xi-vector with tResNet34 backbone is better than what is reported in original Xi-vector paper for SITW-eval set leading to a stronger baseline.

Weaknesses

1. The paper did not compare their disentanglement method with any other disentanglement method pre-exisitng in literature. As mentioned in their Related Work section most other disentanglement methods use ASR labels and they might perform better than this method which does not use ASR labels. But since disentanglement is a major contribution of the paper it would have been interesting to compare them in terms of accuracy, training time and model parameters used. Also an approach based on [1, 2] could also have been explored to learn nuisance free disentangled representations for speaker recognition which does not use ASR labels. 2. Another major contribution is the self-supervised similarity preserving loss. However, authors do not compare with other methods pre-existing in literature that use contrastive learning as mentioned in their related work section. 3. Utility of module "G" is not shown in the paper (as defined in "Frame-wise Content Aware Transition Model"). No ablation studies are provided as to if it is required at all. No experiments were done to show the benefit of using multiple G matrices over just one matrix. Without this module the Rec-Xi layer is same as a Xi layer and hence it is important to show its utility and what improvement we get by modifying the baseline Xi layer. [1] Liu et al. "Towards Learning Nuisance-Free Representations of Speech", ICASSP 2018 [2] Mun et al. "Disentangled Speaker Representation Learning via Mutual Information Minimization"

Questions

1. Authors do one round of content disentanglement in 2nd layer before getting the final speaker representation in 3rd layer. Theoretically, they can keep disentangling content and speaker info for more than 1 iteration. Did authors do any experiment if further rounds of content/speaker disentanglement leads to any performance improvement? 2. How much is the increase in training time when RecXi is used compared with Xi due to its iterative nature? 3. Authors try using just the "phi" and "phi" concatenated with "phi_l" embedding for speaker recognition in their ablation studies Table 3. Did they try using just the "phi_l" vector defined in eqn 24 for speaker prediction to quantify what information it carries? 4. What about adding an extra loss term that uses the precursor embedding to predict speaker info similar to Xi-vector algorithm. One can also concatenate it with the third layer's embedding to get final prediction. Did the authors try any such experiments to see if it can provide additional benefit? 5. Computation of w_t,N in equation 11 is not clear to me. Did the author convert rho_t to a N dimensional vector using the non-linear operation f and take softmax along the N dimensions of the resulting vector to get the vector w_t?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Authors discuss limitation and broader impacts of the paper.

Reviewer ffvX7/10 · confidence 3/52023-07-10

Summary

The authors propose RecXi, a Gaussian inference based disentanglement method for speech that itself is based on Xi-Vector. Overall, the methods makes sense with sufficient motivation and extensive experiments. The results are promising, and have further mileage in ASR, voice conversion and TTS. The paper's main contribution are: 1. Improve Xi-Vector to capture temporal information occurring naturally in speech. 2. A frame-wise content aware transition model to model dynamic aspects of spoken signal. 3. Gaussian inference of 3 layers to disentangle content from speaker. 4. A self-supervision trick to avoid usage of textual supervision. The authors use a number of strong baselines spanning CNN and TDNN architectures. Across 3 datasets, the RecXi shows strong performance, outperforming the baselines on EER and minDCF. The authors also do an ablation study to understand the contribution of the proposed methods. While helpful it is not clear what does the system RecXi($\tilde{\phi}$, $\tilde{\phi}_l$) entail (details below in Questions section). Rebuttal: I have read the authors' rebuttal, and they have clarified the questions I had. My question was mainly due to the presentation of the equation, and the authors have taken the feedback and agreed to update the notation they have used in equations.

Strengths

1. Well-motivated methods to the central research question of disentanglement of content and speaker. 2. Sound modeling approach with extensive experiments. 3. Good ablation study of the various components of the modeling technique. 4. Has more mileage for work in ASR, TTS and Voice conversion.

Weaknesses

1. The authors claim that disentangling speaker and content information is sufficient for getting accurate speaker representation. However, the effect of prosody is not clearly examined in this work. 2. Reproducibility since it's not clear if the code/model are shared or will be shared in future.

Questions

The concept of $\tilde{\phi}\_l$ is introduced in section 3.3; it is used for computing $l\_{ssl}$. So, what exactly does RecXi($\tilde{\phi}$, $\tilde{\phi}\_l$) entail when $l\_{ssl}$ is not used?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

As noted by the authors, the self-supervised loss makes intuitive sense but needs more investigation in terms of the loss function itself as well as some probing about why it works. Further, the number of transition models in layer 2 of Gaussian inference is a hyper-parameter whose interpretation is not clear is heavily dependent on experimentation.

Reviewer 6npJ2023-08-17

I thank the authors for providing additional experiments comparing their method with other methods for disentangling content info from speaker and self-supervised method. I appreciate the experiment showing the benefit of the proposed module G. The training time could still be a bottleneck due to iterative nature of the architecture. But I believe the efficacy shown in disentangling content information without using heavy weight ASR models outweighs that drawback. I raise my review to a weak accept.

Authorsrebuttal2023-08-17

Dear reviewer, We genuinely appreciate your positive feedback on our rebuttal! Your insights and guidance have been instrumental in shaping the revisions to our paper. We are also deeply thankful for your reconsideration and decision to raise the rating. Warm regards, Authors

Reviewer ffvX2023-08-17

Thank you for your response. It's correct that speaker embeddings obtained from static components are effective, and it's reasonable to take on investigating decomposition of prosody from dynamic components as future work. It's great that you're offering help till your internal approvals go through. However, please indicate that you intend to share code and models in near future. Re: question about loss function, I am not able to follow your explanation possibly because Latex notations aren't rendering correctly. Do you mind using plaintext as much as possible to rewrite the response to my question? Thanks!

Authorsrebuttal2023-08-18

Dear reviewer, We appreciate your thoughtful response to our rebuttal. Please allow us to address your points: 1. We are in agreement with your perspective, and we extend our sincere appreciation for this insightful suggestion. It presents a valuable avenue for future exploration. 2. Thanks for your suggestion. We will add this sentence to the paper: "The code and the pretrained models will be made available with third-party re-implementation." The internal approval may take a longer time. In any case, we will have the third-party re-implementation available. 3. We apologize that our previous response wasn't sufficiently clear in conveying our points. To ensure clarity, we provide a more comprehensive explanation in both LaTeX and plaintext formats: --- (LaTeX version) In our work, the symbol of $\ell$ denotes the loss functions, such as $\ell_{ssp}$ and $\ell_{cls}$ in Eq. 25 and 26. The usage of $\mathcal{l}$ in $\tilde{\phi_l}$ indicates that $\tilde{\phi_l}$ is obtained through a linear operation (Eq. 24). This choice of notation differentiates $\tilde{\phi_l}$ from $\tilde{\phi}$. Specifically, $\tilde{\phi_l}$ denotes the speaker representation resulting from the linear operation, while $\tilde{\phi}$ is the speaker representation disentangled by the Gaussian inference in layer 3, as discussed in Page 6, Section 3.3, line 245. **Therefore, actually, $\ell$ and $\mathcal{l}$ have different meanings in our work**. Your inquiry has brought to light the potential confusion arising from the similarity of these symbols. To avoid any potential misunderstandings, we will replace all the $\tilde{\phi_l}$ with $\tilde{\phi_{\mathrm{lin}}}$, and also change the loss function symbol from $\ell$ to $\mathcal{L}$. On the other hand, for RecXi($\tilde{\phi}$, $\tilde{\phi_l}$), when the $\ell_{ssp}$ is missing, $\tilde{\phi_l}$ is trained by the classification loss $\ell_{cls}$. It is accomplished by concatenating $\tilde{\phi_l}$ and $\tilde{\phi}$ (Line 290), and then passing this concatenation to the decoder. The results show that the $\ell_{ssp}$ loss helps in the modeling of speech dynamics. --- (Plaintext version) In our work, the symbol of \ell denotes the loss functions, such as \ell_{ssp} and \ell_{cls} in Eq. 25 and 26. The usage of \mathcal{l} in \tilde{\phi_l} indicates that \tilde{\phi_l} is obtained through a linear operation (Eq. 24). This choice of notation differentiates \tilde{\phi_l} from \tilde{\phi}. Specifically, \tilde{\phi_l} denotes the speaker representation resulting from the linear operation, while \tilde{\phi} is the speaker representation disentangled by the Gaussian inference in layer 3, as discussed in Page 6, Section 3.3, line 245. **Therefore, actually, \ell and \mathcal{l} have different meanings in our work.** Your inquiry has brought to light the potential confusion arising from the similarity of these symbols. To avoid any potential misunderstandings, we will replace all the \tilde{\phi_l} with \tilde{\phi_{\mathrm{lin}}}, and also change the loss function symbol from \ell to \mathcal{L}. On the other hand, for RecXi(\tilde{\phi}, \tilde{\phi_l}), when the \ell_{ssp} is missing, \tilde{\phi_l} is trained by the classification loss \ell_{cls}. It is accomplished by concatenating \tilde{\phi_l} and \tilde{\phi} (Line 290), and then passing this concatenation to the decoder. The results show that the \ell_{ssp} loss helps in the modeling of speech dynamics. --- Once again, we sincerely apologize for any confusion caused by the earlier explanation. We trust that this detailed clarification will be beneficial. Please do not hesitate to reach out if you have any further queries.

Reviewer krxU2023-08-19

Thank you very much for responding to all of my comments and questions. they've helped me understand the work better

Authorsrebuttal2023-08-19

Dear Reviewer, Thank you so much for your kind reply. We're truly glad that our rebuttal has been helpful. Your comments and questions are greatly appreciated and contribute significantly to the quality of our work. Your support means a lot to us. If you have more thoughts or questions, please feel free to share. Warm regards, Authors

Reviewer P6Zh2023-08-21

Thank you to the reviewers for providing proper rebuttals. If accepted, I hope the authors include the additional insights they shared through this rebuttal in the final draft.

Authorsrebuttal2023-08-21

Dear reviewer, We appreciate your response and the time you've taken to review our rebuttal. Your thoughtful suggestions on visualizations and ablation studies across the three layers have motivated us to conduct additional experiments to gather more evidence supporting our claims. Your input significantly contributes to enhancing the paper's quality. If this paper is accepted, we will include the rebuttal content in the final manuscript. Warm regards, Authors

Reviewer 48tf2023-08-21

I'd like to thank the authors for their responses and feedback. The explanation of the limited performance improvement makes sense. I will reconsider the review.

Authorsrebuttal2023-08-21

Dear reviewer, We greatly appreciate you taking the time to review our rebuttal and for reconsidering the rating. Your valuable feedback has played a crucial role in improving our work. We will include the content of the above rebuttal in our final manuscript. Warm regards, Authors

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC