Cross-modal Language Generation using Pivot Stabilization for Web-scale Language Coverage

Cross-modal language generation tasks such as image captioning are directly\nhurt in their ability to support non-English languages by the trend of\ndata-hungry models combined with the lack of non-English annotations. We\ninvestigate potential solutions for combining existing language-generation\nannotations in English with translation capabilities in order to create\nsolutions at web-scale in both domain and language coverage. We describe an\napproach called Pivot-Language Generation Stabilization (PLuGS), which\nleverages directly at training time both existing English annotations (gold\ndata) as well as their machine-translated versions (silver data); at run-time,\nit generates first an English caption and then a corresponding target-language\ncaption. We show that PLuGS models outperform other candidate solutions in\nevaluations performed over 5 different target languages, under a large-domain\ntestset using images from the Open Images dataset. Furthermore, we find an\ninteresting effect where the English captions generated by the PLuGS models are\nbetter than the captions generated by the original, monolingual English model.\n