The ubiquitous nature of chatbots and their interaction with users generate\nan enormous amount of data. Can we improve chatbots using this data? A\nself-feeding chatbot improves itself by asking natural language feedback when a\nuser is dissatisfied with its response and uses this feedback as an additional\ntraining sample. However, user feedback in most cases contains extraneous\nsequences hindering their usefulness as a training sample. In this work, we\npropose a generative adversarial model that converts noisy feedback into a\nplausible natural response in a conversation. The generator's goal is to\nconvert the feedback into a response that answers the user's previous utterance\nand to fool the discriminator which distinguishes feedback from natural\nresponses. We show that augmenting original training data with these modified\nfeedback responses improves the original chatbot performance from 69.94% to\n75.96% in ranking correct responses on the Personachat dataset, a large\nimprovement given that the original model is already trained on 131k samples.\n