Constructing Multi-Modal Dialogue Dataset by Replacing Text with Semantically Relevant Images

In multi-modal dialogue systems, it is important to allow the use of images\nas part of a multi-turn conversation. Training such dialogue systems generally\nrequires a large-scale dataset consisting of multi-turn dialogues that involve\nimages, but such datasets rarely exist. In response, this paper proposes a 45k\nmulti-modal dialogue dataset created with minimal human intervention. Our\nmethod to create such a dataset consists of (1) preparing and pre-processing\ntext dialogue datasets, (2) creating image-mixed dialogues by using a\ntext-to-image replacement technique, and (3) employing a\ncontextual-similarity-based filtering step to ensure the contextual coherence\nof the dataset. To evaluate the validity of our dataset, we devise a simple\nretrieval model for dialogue sentence prediction tasks. Automatic metrics and\nhuman evaluation results on such tasks show that our dataset can be effectively\nused as training data for multi-modal dialogue systems which require an\nunderstanding of images and text in a context-aware manner. Our dataset and\ngeneration code is available at\nhttps://github.com/shh1574/multi-modal-dialogue-dataset.\n