Analyzing Speaker Information in Self-Supervised Models to Improve\n Zero-Resource Speech Processing

Contrastive predictive coding (CPC) aims to learn representations of speech\nby distinguishing future observations from a set of negative examples. Previous\nwork has shown that linear classifiers trained on CPC features can accurately\npredict speaker and phone labels. However, it is unclear how the features\nactually capture speaker and phonetic information, and whether it is possible\nto normalize out the irrelevant details (depending on the downstream task). In\nthis paper, we first show that the per-utterance mean of CPC features captures\nspeaker information to a large extent. Concretely, we find that comparing means\nperforms well on a speaker verification task. Next, probing experiments show\nthat standardizing the features effectively removes speaker information. Based\non this observation, we propose a speaker normalization step to improve\nacoustic unit discovery using K-means clustering of CPC features. Finally, we\nshow that a language model trained on the resulting units achieves some of the\nbest results in the ZeroSpeech2021~Challenge.\n