Consensus Graph Representation Learning for Better Grounded Image Captioning

The contemporary visual captioning models frequently hallucinate objects that\nare not actually in a scene, due to the visual misclassification or\nover-reliance on priors that resulting in the semantic inconsistency between\nthe visual information and the target lexical words. The most common way is to\nencourage the captioning model to dynamically link generated object words or\nphrases to appropriate regions of the image, i.e., the grounded image\ncaptioning (GIC). However, GIC utilizes an auxiliary task (grounding objects)\nthat has not solved the key issue of object hallucination, i.e., the semantic\ninconsistency. In this paper, we take a novel perspective on the issue above -\nexploiting the semantic coherency between the visual and language modalities.\nSpecifically, we propose the Consensus Rraph Representation Learning framework\n(CGRL) for GIC that incorporates a consensus representation into the grounded\ncaptioning pipeline. The consensus is learned by aligning the visual graph\n(e.g., scene graph) to the language graph that consider both the nodes and\nedges in a graph. With the aligned consensus, the captioning model can capture\nboth the correct linguistic characteristics and visual relevance, and then\ngrounding appropriate image regions further. We validate the effectiveness of\nour model, with a significant decline in object hallucination (-9% CHAIRi) on\nthe Flickr30k Entities dataset. Besides, our CGRL also evaluated by several\nautomatic metrics and human evaluation, the results indicate that the proposed\napproach can simultaneously improve the performance of image captioning (+2.9\nCider) and grounding (+2.3 F1LOC).\n