Generating Procedural 3D materials from Images using Neural Networks

Abstract

Generating photo-realistic 3D materials from images is a challenging task in 3D modeling. In the absence of suitable 3D material assets from online material collections, texture artists have to resort to designing their own materials from reference images; a time-consuming process requiring significant expertise. Recent works on material estimation are limited by the number and resolution of 3D materials they can generate. Our work addresses these limitations through a novel framework and dataset for generating procedural materials from images. We introduce a set of 314 procedural material models each with parameters that can be varied to generate an infinite number of materials. We leverage a neural network architecture to predict materials as variations of a suitable model in our model library. Our framework is able to produce high-resolution photo-realistic 3D materials from input images. Our neural network is able to handle 314 procedural material models, a significant improvement of previous works training 17 neural networks to predict parameters of 17 procedural models. We evaluate our framework against a retrieval baseline, and show that our approach yields significant improvement on parameter estimation, and visual appearance.