Summary
The author proposes a zero-shot singing voice synthesis model that aims to achieve better feature space disentanglement than the current state-of-the-art. It claims to achieve this goal using the following:
1. A large-scale singing voice dataset including both human and non-human voices.
2. Use of the VITS architecture, along with multiple ASR modules for improved content feature extraction to reduce timbre leakage.
3. Inference-time speaker embedding retrieval using cosine similarity.
Weaknesses
3. The theoretical explanation is poor. In particular:
3.1 This is essentially a re-explanation of the VITS architecture, with no significant novelty found.
3.2 This section should present the main theoretical novelty of the paper; however, the algorithm, including the use of k-means and quantization, remains unclear.
Figure 3(a) suggests that the speaker encoders are directly interfacing with the spectrograms, which implies that unfreezing them during training could allow timbre leakage.
Figure 3(b) is unclear. How is the final vector produced from the clusters and codebooks? Are Q1, Q2, etc., referring to the codebooks? Do you use both methods simultaneously, or only select one?
3.3 Mapping to the closest speaker embeddings in the training set "may reduce the timbre similarity of the converted audio." However, since one of the main goals stated in the abstract is to avoid timbre leakage, what is the rationale behind using this technique?
3.4 The loss components presented are not novel, as all of them are already found in the HiFi-GAN and/or FreeVC papers.
- Regarding the dataset, while it is noted as a novel aspect, this is left unexplained in Section 3.
4. Experiments/VisualizationThe experimental details are lacking. Specifically:
4.2 Where are the training parameters? The only parameters specified are those in the loss function equation (3). Did you re-train the baselines on the proposed dataset as well? Furthermore, with such a large-scale dataset, why would you only evaluate on 11 speakers for 100 seconds each? For the non-human part, are you testing zero-shot conversion from humans to non-humans, non-humans to humans, or within each group only?
Tables 2 and 3: Although seemingly complete and acceptable metrics-wise, these contain two issues:
- Since the primary goal is reducing timbre leakage, why is there only a trivial gain on SECS in both human and non-human cases? For example, in Table 1, your gain above the original so-VITS-SVC is only 0.009 — is that really statistically significant?
- The retrieval method you proposed has a much lower SECS. How, then, does it achieve the highest MOS? Does that mean a user would rate 5/5 if I presented them with unconverted audio?
Section 4.1: The Tensor/Codes method and Table 4 do not make sense without adequate explanations in Section 3.2.
Figure 4: What is the purpose of this figure? In the text, it is mentioned that "The results show that training the speaker encoder on the SVC tasks makes a clearer distinction between the speaker embeddings from speech and singing." But isn't the primary goal to demonstrate improvements in SVC performance?