Bridging the Reality Gap: Investigation of Deep Convolution Neural Networks Ability to Learn from a Combination of Real and Synthetic Data

Abstract

Recent advances in data-driven approaches especially deep learning and its application on visual imagery have drawn a lot of attention in recent years. The lack of training data, however, highly affects the model accuracy and its ability to generalize to unseen scenarios. Simulators are emerging as a promising alternative source of data, especially for vision-based applications. Nevertheless, they still lack the visual and physical properties of the real world. Recent works have shown promising approaches to close the reality gap and transfer the knowledge obtained in simulation to the real world. This paper investigates Convolution Neural Networks (CNNs) ability to generalize and learn from a mixture of real and synthetic data to overcome dataset scarcity and domain transfer problems. The evaluation results indicate that the CNN models trained with real and simulation data generalize to both simulation and real environments. However, models trained with only real or simulation data fails drastically when it is transferred to an unseen target environment. Furthermore, the utilization of simulation data has improved model accuracy significantly.