Accent conversion (AC) transforms a non-native speaker's accent into a native\naccent while maintaining the speaker's voice timbre. In this paper, we propose\napproaches to improving accent conversion applicability, as well as quality.\nFirst of all, we assume no reference speech is available at the conversion\nstage, and hence we employ an end-to-end text-to-speech system that is trained\non native speech to generate native reference speech. To improve the quality\nand accent of the converted speech, we introduce reference encoders which make\nus capable of utilizing multi-source information. This is motivated by acoustic\nfeatures extracted from native reference and linguistic information, which are\ncomplementary to conventional phonetic posteriorgrams (PPGs), so they can be\nconcatenated as features to improve a baseline system based only on PPGs.\nMoreover, we optimize model architecture using GMM-based attention instead of\nwindowed attention to elevate synthesized performance. Experimental results\nindicate when the proposed techniques are applied the integrated system\nsignificantly raises the scores of acoustic quality (30$\\%$ relative increase\nin mean opinion score) and native accent (68$\\%$ relative preference) while\nretaining the voice identity of the non-native speaker.\n