This paper introduces Taco-VC, a novel architecture for voice conversion\nbased on Tacotron synthesizer, which is a sequence-to-sequence with attention\nmodel. The training of multi-speaker voice conversion systems requires a large\nnumber of resources, both in training and corpus size. Taco-VC is implemented\nusing a single speaker Tacotron synthesizer based on Phonetic PosteriorGrams\n(PPGs) and a single speaker WaveNet vocoder conditioned on mel spectrograms. To\nenhance the converted speech quality, and to overcome over-smoothing, the\noutputs of Tacotron are passed through a novel speechenhancement network, which\nis composed of a combination of the phoneme recognition and Tacotron networks.\nOur system is trained just with a single speaker corpus and adapts to new\nspeakers using only a few minutes of training data. Using mid-size public\ndatasets, our method outperforms the baseline in the VCC 2018 SPOKE\nnon-parallel voice conversion task and achieves competitive results compared to\nmulti-speaker networks trained on large private datasets.\n