An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games

Guessing games are a prototypical instance of the "learning by interacting"\nparadigm. This work investigates how well an artificial agent can benefit from\nplaying guessing games when later asked to perform on novel NLP downstream\ntasks such as Visual Question Answering (VQA). We propose two ways to exploit\nplaying guessing games: 1) a supervised learning scenario in which the agent\nlearns to mimic successful guessing games and 2) a novel way for an agent to\nplay by itself, called Self-play via Iterated Experience Learning (SPIEL).\n We evaluate the ability of both procedures to generalize: an in-domain\nevaluation shows an increased accuracy (+7.79) compared with competitors on the\nevaluation suite CompGuessWhat?!; a transfer evaluation shows improved\nperformance for VQA on the TDIUC dataset in terms of harmonic average accuracy\n(+5.31) thanks to more fine-grained object representations learned via SPIEL.\n