Recent developments related to generative models have made it possible to\ngenerate diverse high-fidelity images. In particular, layout-to-image\ngeneration models have gained significant attention due to their capability to\ngenerate realistic complex images containing distinct objects. These models are\ngenerally conditioned on either semantic layouts or textual descriptions.\nHowever, unlike natural images, providing auxiliary information can be\nextremely hard in domains such as biomedical imaging and remote sensing. In\nthis work, we propose a multi-object generation framework that can synthesize\nimages with multiple objects without explicitly requiring their contextual\ninformation during the generation process. Based on a vector-quantized\nvariational autoencoder (VQ-VAE) backbone, our model learns to preserve spatial\ncoherency within an image as well as semantic coherency between the objects and\nthe background through two powerful autoregressive priors: PixelSNAIL and\nLayoutPixelSNAIL. While the PixelSNAIL learns the distribution of the latent\nencodings of the VQ-VAE, the LayoutPixelSNAIL is used to specifically learn the\nsemantic distribution of the objects. An implicit advantage of our approach is\nthat the generated samples are accompanied by object-level annotations. We\ndemonstrate how coherency and fidelity are preserved with our method through\nexperiments on the Multi-MNIST and CLEVR datasets; thereby outperforming\nstate-of-the-art multi-object generative methods. The efficacy of our approach\nis demonstrated through application on medical imaging datasets, where we show\nthat augmenting the training set with generated samples using our approach\nimproves the performance of existing models.\n