Semantics-aware Multi-modal Domain Translation:From LiDAR Point Clouds to Panoramic Color Images

In this work, we present a simple yet effective framework to address the\ndomain translation problem between different sensor modalities with unique data\nformats. By relying only on the semantics of the scene, our modular generative\nframework can, for the first time, synthesize a panoramic color image from a\ngiven full 3D LiDAR point cloud. The framework starts with semantic\nsegmentation of the point cloud, which is initially projected onto a spherical\nsurface. The same semantic segmentation is applied to the corresponding camera\nimage. Next, our new conditional generative model adversarially learns to\ntranslate the predicted LiDAR segment maps to the camera image counterparts.\nFinally, generated image segments are processed to render the panoramic scene\nimages. We provide a thorough quantitative evaluation on the SemanticKitti\ndataset and show that our proposed framework outperforms other strong baseline\nmodels.\n Our source code is available at\nhttps://github.com/halmstad-University/TITAN-NET\n