Sensor Transfer: Learning Optimal Sensor Effect Image Augmentation for Sim-to-Real Domain Adaptation

Performance on benchmark datasets has drastically improved with advances in\ndeep learning. Still, cross-dataset generalization performance remains\nrelatively low due to the domain shift that can occur between two different\ndatasets. This domain shift is especially exaggerated between synthetic and\nreal datasets. Significant research has been done to reduce this gap,\nspecifically via modeling variation in the spatial layout of a scene, such as\nocclusions, and scene environmental factors, such as time of day and weather\neffects. However, few works have addressed modeling the variation in the sensor\ndomain as a means of reducing the synthetic to real domain gap. The camera or\nsensor used to capture a dataset introduces artifacts into the image data that\nare unique to the sensor model, suggesting that sensor effects may also\ncontribute to domain shift. To address this, we propose a learned augmentation\nnetwork composed of physically-based augmentation functions. Our proposed\naugmentation pipeline transfers specific effects of the sensor model --\nchromatic aberration, blur, exposure, noise, and color temperature -- from a\nreal dataset to a synthetic dataset. We provide experiments that demonstrate\nthat augmenting synthetic training datasets with the proposed learned\naugmentation framework reduces the domain gap between synthetic and real\ndomains for object detection in urban driving scenes.\n