Compared to natural images, understanding scientific figures is particularly\nhard for machines. However, there is a valuable source of information in\nscientific literature that until now has remained untapped: the correspondence\nbetween a figure and its caption. In this paper we investigate what can be\nlearnt by looking at a large number of figures and reading their captions, and\nintroduce a figure-caption correspondence learning task that makes use of our\nobservations. Training visual and language networks without supervision other\nthan pairs of unconstrained figures and captions is shown to successfully solve\nthis task. We also show that transferring lexical and semantic knowledge from a\nknowledge graph significantly enriches the resulting features. Finally, we\ndemonstrate the positive impact of such features in other tasks involving\nscientific text and figures, like multi-modal classification and machine\ncomprehension for question answering, outperforming supervised baselines and\nad-hoc approaches.\n