Summary
This study analyzes the training-inference mismatch that occurs in codec language models, a branch of personalized speech synthesis research, and mitigates it through preference optimization methods. By avoiding the labor-intensive process of collecting human preference test results, the researchers efficiently gathered data and used it to further fine-tune the model, achieving improved results in personalization.
Strengths
1. Preference optimization has not yet been sufficiently explored in personalized speech synthesis, and this study demonstrates its effectiveness for this purpose.
2. They conducted thorough observations and analyses of the problem they defined.
3. The illustrations and figures added to the paper aid in comprehension, and the writing is clear and easy to understand.
4. They evaluated various components and details they used within the paper, effectively demonstrating the impact of each component.
Weaknesses
I believe the evaluation with other baselines might be insufficient. I have included additional evaluation questions below for consideration.
Questions
1. (Questions) Distribution Gap in Section 2.3:
- You mentioned a distribution gap between golden AR tokens and synthetic AR tokens. Which attributes cause this distribution difference? Generally, discrete features are known to be more robust against error propagation compared to continuous features, and many studies have leveraged this characteristic for TTS. Does the distribution gap arise because the SpeechTokenizer fails to remove acoustic information within the AR token, leaving residual information that causes a mismatch in synthetic AR tokens? What specific information is being mismatched?
2. Scalability and Dataset Size:
- While recent zero-shot personalized speech synthesis models typically train on large-scale datasets of at least tens of thousands of hours, this study used the relatively smaller LibriSpeech dataset. This raises questions about the scalability of your methodology. Can the model achieve better performance with more data, as demonstrated by many recent models that show excellent personalized speech synthesis performance using vast datasets? Evaluating the model's potential with larger datasets is crucial for understanding its full capabilities. Although Section 5.2 provides an analysis of data size, it is limited to the amount of data from LibriSpeech and still falls short compared to recent studies in terms of data volume and data distribution (data from various source such as GigaSpeech, ...).
3. Baselines
- Given the smaller dataset size compared to recent zero-shot approaches, it would be helpful to know the relative competitiveness of your current model (trained solely on LibriSpeech) against recent personalization technologies. Even if the performance falls short compared to models trained on over 10,000 hours of data, if the gap is not significant, this could indicate an important direction of research focused on efficient personalized modeling with less data. Although recent personalized speech synthesis trends show a lack of open-source models, it might still be possible to use samples from their demo pages to compare metrics like SIM, QMOS, and SMOS.
Limitations
They addressed their limitations in the appendix.