Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization

While much research has been done in text-to-image synthesis, little work has\nbeen done to explore the usage of linguistic structure of the input text. Such\ninformation is even more important for story visualization since its inputs\nhave an explicit narrative structure that needs to be translated into an image\nsequence (or visual story). Prior work in this domain has shown that there is\nample room for improvement in the generated image sequence in terms of visual\nquality, consistency and relevance. In this paper, we first explore the use of\nconstituency parse trees using a Transformer-based recurrent architecture for\nencoding structured input. Second, we augment the structured input with\ncommonsense information and study the impact of this external knowledge on the\ngeneration of visual story. Third, we also incorporate visual structure via\nbounding boxes and dense captioning to provide feedback about the\ncharacters/objects in generated images within a dual learning setup. We show\nthat off-the-shelf dense-captioning models trained on Visual Genome can improve\nthe spatial structure of images from a different target domain without needing\nfine-tuning. We train the model end-to-end using intra-story contrastive loss\n(between words and image sub-regions) and show significant improvements in\nseveral metrics (and human evaluation) for multiple datasets. Finally, we\nprovide an analysis of the linguistic and visuo-spatial information. Code and\ndata: https://github.com/adymaharana/VLCStoryGan.\n