Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?

We conduct large-scale studies on `human attention' in Visual Question\nAnswering (VQA) to understand where humans choose to look to answer questions\nabout images. We design and test multiple game-inspired novel\nattention-annotation interfaces that require the subject to sharpen regions of\na blurred image to answer a question. Thus, we introduce the VQA-HAT (Human\nATtention) dataset. We evaluate attention maps generated by state-of-the-art\nVQA models against human attention both qualitatively (via visualizations) and\nquantitatively (via rank-order correlation). Overall, our experiments show that\ncurrent attention models in VQA do not seem to be looking at the same regions\nas humans.\n