Graph Relation Transformer: Incorporating pairwise object features into the Transformer architecture

Previous studies such as VizWiz find that Visual Question Answering (VQA)\nsystems that can read and reason about text in images are useful in application\nareas such as assisting visually-impaired people. TextVQA is a VQA dataset\ngeared towards this problem, where the questions require answering systems to\nread and reason about visual objects and text objects in images. One key\nchallenge in TextVQA is the design of a system that effectively reasons not\nonly about visual and text objects individually, but also about the spatial\nrelationships between these objects. This motivates the use of 'edge features',\nthat is, information about the relationship between each pair of objects. Some\ncurrent TextVQA models address this problem but either only use categories of\nrelations (rather than edge feature vectors) or do not use edge features within\nthe Transformer architectures. In order to overcome these shortcomings, we\npropose a Graph Relation Transformer (GRT), which uses edge information in\naddition to node information for graph attention computation in the\nTransformer. We find that, without using any other optimizations, the proposed\nGRT method outperforms the accuracy of the M4C baseline model by 0.65% on the\nval set and 0.57% on the test set. Qualitatively, we observe that the GRT has\nsuperior spatial reasoning ability to M4C.\n