Towards Efficient Cross-Modal Visual Textual Retrieval using Transformer-Encoder Deep Features

Cross-modal retrieval is an important functionality in modern search engines,\nas it increases the user experience by allowing queries and retrieved objects\nto pertain to different modalities. In this paper, we focus on the\nimage-sentence retrieval task, where the objective is to efficiently find\nrelevant images for a given sentence (image-retrieval) or the relevant\nsentences for a given image (sentence-retrieval). Computer vision literature\nreports the best results on the image-sentence matching task using deep neural\nnetworks equipped with attention and self-attention mechanisms. They evaluate\nthe matching performance on the retrieval task by performing sequential scans\nof the whole dataset. This method does not scale well with an increasing amount\nof images or captions. In this work, we explore different preprocessing\ntechniques to produce sparsified deep multi-modal features extracting them from\nstate-of-the-art deep-learning architectures for image-text matching. Our main\nobjective is to lay down the paths for efficient indexing of complex\nmulti-modal descriptions. We use the recently introduced TERN architecture as\nan image-sentence features extractor. It is designed for producing fixed-size\n1024-d vectors describing whole images and sentences, as well as\nvariable-length sets of 1024-d vectors describing the various building\ncomponents of the two modalities (image regions and sentence words\nrespectively). All these vectors are enforced by the TERN design to lie into\nthe same common space. Our experiments show interesting preliminary results on\nthe explored methods and suggest further experimentation in this important\nresearch direction.\n