Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer\n through Optimal Transport

This paper presents a light-weight, high-quality texture synthesis algorithm\nthat easily generalizes to other applications such as style transfer and\ntexture mixing. We represent texture features through the deep neural\nactivation vectors within the bottleneck layer of an auto-encoder and frame the\ntexture synthesis problem as optimal transport between the activation values of\nthe image being synthesized and those of an exemplar texture. To find this\noptimal transport mapping, we utilize an N-dimensional probability density\nfunction (PDF) transfer process that iterates over multiple random rotations of\nthe PDF basis and matches the 1D marginal distributions across each dimension.\nThis achieves quality and flexibility on par with expensive back-propagation\nbased neural texture synthesis methods, but with the potential of achieving\ninteractive rates. We demonstrate that first order statistics offer a more\nrobust representation for texture than the second order statistics that are\nused today. We propose an extension of this algorithm that reduces the\ndimensionality of the neural feature space. We utilize a multi-scale\ncoarse-to-fine synthesis pyramid to capture and preserve larger image features;\nunify color and style transfer under one framework; and further augment this\nsystem with a novel masking scheme that re-samples and re-weights the feature\ndistribution for user-guided texture painting and targeted style transfer.\n