Seeing is Knowing! Fact-based Visual Question Answering using Knowledge Graph Embeddings

Fact-based Visual Question Answering (FVQA), a challenging variant of VQA,\nrequires a QA-system to include facts from a diverse knowledge graph (KG) in\nits reasoning process to produce an answer. Large KGs, especially common-sense\nKGs, are known to be incomplete, i.e., not all non-existent facts are always\nincorrect. Therefore, being able to reason over incomplete KGs for QA is a\ncritical requirement in real-world applications that has not been addressed\nextensively in the literature. We develop a novel QA architecture that allows\nus to reason over incomplete KGs, something current FVQA state-of-the-art\n(SOTA) approaches lack due to their critical reliance on fact retrieval. We use\nKG Embeddings, a technique widely used for KG completion, for the downstream\ntask of FVQA. We also employ a new image representation technique we call\n'Image-as-Knowledge' to enable this capability, alongside a simple one-step\nCoAttention mechanism to attend to text and image during QA. Our FVQA\narchitecture is faster during inference time, being O(m), as opposed to\nexisting FVQA SOTA methods which are O(N log N), where m = number of vertices,\nN = number of edges = O(m^2). KG embeddings are shown to hold complementary\ninformation to word embeddings: a combination of both metrics permits\nperformance comparable to SOTA methods in the standard answer retrieval task,\nand significantly better (26% absolute) in the proposed missing-edge reasoning\ntask.\n