Scene Designer: a Unified Model for Scene Search and Synthesis from Sketch

Scene Designer is a novel method for searching and generating images using\nfree-hand sketches of scene compositions; i.e. drawings that describe both the\nappearance and relative positions of objects. Our core contribution is a single\nunified model to learn both a cross-modal search embedding for matching\nsketched compositions to images, and an object embedding for layout synthesis.\nWe show that a graph neural network (GNN) followed by Transformer under our\nnovel contrastive learning setting is required to allow learning correlations\nbetween object type, appearance and arrangement, driving a mask generation\nmodule that synthesises coherent scene layouts, whilst also delivering state of\nthe art sketch based visual search of scenes.\n