Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram\n Generation with Adversarial Style Combination for Speech Synthesis

While generative adversarial networks (GANs) based neural text-to-speech\n(TTS) systems have shown significant improvement in neural speech synthesis,\nthere is no TTS system to learn to synthesize speech from text sequences with\nonly adversarial feedback. Because adversarial feedback alone is not sufficient\nto train the generator, current models still require the reconstruction loss\ncompared with the ground-truth and the generated mel-spectrogram directly. In\nthis paper, we present Multi-SpectroGAN (MSG), which can train the\nmulti-speaker model with only the adversarial feedback by conditioning a\nself-supervised hidden representation of the generator to a conditional\ndiscriminator. This leads to better guidance for generator training. Moreover,\nwe also propose adversarial style combination (ASC) for better generalization\nin the unseen speaking style and transcript, which can learn latent\nrepresentations of the combined style embedding from multiple mel-spectrograms.\nTrained with ASC and feature matching, the MSG synthesizes a high-diversity\nmel-spectrogram by controlling and mixing the individual speaking styles (e.g.,\nduration, pitch, and energy). The result shows that the MSG synthesizes a\nhigh-fidelity mel-spectrogram, which has almost the same naturalness MOS score\nas the ground-truth mel-spectrogram.\n