Telling the What while Pointing to the Where: Multimodal Queries for Image Retrieval

Most existing image retrieval systems use text queries as a way for the user\nto express what they are looking for. However, fine-grained image retrieval\noften requires the ability to also express where in the image the content they\nare looking for is. The text modality can only cumbersomely express such\nlocalization preferences, whereas pointing is a more natural fit. In this\npaper, we propose an image retrieval setup with a new form of multimodal\nqueries, where the user simultaneously uses both spoken natural language (the\nwhat) and mouse traces over an empty canvas (the where) to express the\ncharacteristics of the desired target image. We then describe simple\nmodifications to an existing image retrieval model, enabling it to operate in\nthis setup. Qualitative and quantitative experiments show that our model\neffectively takes this spatial guidance into account, and provides\nsignificantly more accurate retrieval results compared to text-only equivalent\nsystems.\n