PANOPTIC GENERATIVE ADVERSARIAL NETWORK WITH EXPLICIT MODELING OF CATEGORY AND INSTANCE INFORMATION

Systems and methods for panoptic segmentation of an image of a scene, comprising: receiving a synthetic data set as simulation data set in a simulation domain, the simulation data set comprising a plurality of synthetic data objects; disentangling the synthetic data objects by class for a plurality of object classes; training each class of the plurality of classes separately by applying a Generative Adversarial Network (GAN) to each class from the data set in the simulation domain to create a generated instance for each class; combining the generated instances for each class with labels for the objects in each class to obtain a fake instance of an object; fusing the fake instances to create a fused image; and applying a GAN to the fused image and a corresponding real data set in a real-world domain to obtain an updated data set. The process can be repeated across multiple iterations.