Deep Multimodal Image-Text Embeddings for Automatic Cross-Media Retrieval

This paper considers the task of matching images and sentences by learning a\nvisual-textual embedding space for cross-modal retrieval. Finding such a space\nis a challenging task since the features and representations of text and image\nare not comparable. In this work, we introduce an end-to-end deep multimodal\nconvolutional-recurrent network for learning both vision and language\nrepresentations simultaneously to infer image-text similarity. The model learns\nwhich pairs are a match (positive) and which ones are a mismatch (negative)\nusing a hinge-based triplet ranking. To learn about the joint representations,\nwe leverage our newly extracted collection of tweets from Twitter. The main\ncharacteristic of our dataset is that the images and tweets are not\nstandardized the same as the benchmarks. Furthermore, there can be a higher\nsemantic correlation between the pictures and tweets contrary to benchmarks in\nwhich the descriptions are well-organized. Experimental results on MS-COCO\nbenchmark dataset show that our model outperforms certain methods presented\npreviously and has competitive performance compared to the state-of-the-art.\nThe code and dataset have been made available publicly.\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC