Semantic Image Synthesis with Unconditional Generator

Summary

This work proposes a novel method for semantic image synthesis from an unconditional generator. Specifically, a proxy mask is used to provide semantic guidance to reararange the middle layer's feature. A segmentation module is learned to map user input to the proxy mask in a few-shot manner. Experiments over different datasets show proposed method's effectiveness.

Strengths

1. Semantic image synthesis typically requires learning from a large amount of paired data. However, this work relaxes this requirement to just a single pair. This approach is innovative and has the potential to unlock new applications that enable the control of image synthesis through customized signals. 2. The authors have implemented thorough experiments over different dataset and different kinds of control signals (seg, sketch, scribble, etc.).

Weaknesses

1. The formulation is messy. For example, $m_{proxy}$ and $K$ (in Equation 2) are undefined when they are first mentioned in the paper. 2. There are many loss terms proposed. However, the authors do not provide ablation study results over them.

Questions

The image quality is not as good as previous methods for full-shot setting. Why does this happen? I hope the authors could provide some analysis and hypotheses, and it would be even better if they could conduct experiments to support their claims.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have discussed the limitations in the paper.

Summary

In this paper, authors propose to use well pre-trained StyleGANv2 for conditional image synthesis, e.g., semantic image synthesis. To achieve conditional control, a rearranging mapping network and a semantic mapper module are proposed to connect the semantic input and middle spatial feature control within the StyleGANv2 framework. Experiments show that the proposed method can generate images under various input conditions.

Strengths

1. The paper is well organized and easy to understanding. 2. The proposed method is flexible to various conditional inputs, such as semantic mask and sketch.

Weaknesses

1. As mentioned in the limitation section of the paper, the quality of generated image is not very good. For example, images (FFHQ) in Figure 5 show that the shape of hair is not well aligned with the input mask. In Figure 7, the background of cat shows artifacts and the structure of hair is not match the sketch (third column). It seems that the proposed method still can not control the spatial conditions well, although it is better than previous method, LinearGAN. 2. The ablation study can be further improved. a. Why self reconstruction loss and mask loss are used different format (i.e., MSE loss and CE loss)? b. What is the main factor for the performance of the method (self-supervised learning stage, or semantic guide stage)? If we use full-shot segmentation network, can we further benefit the proposed method? c. It seems that given a semantic mask, it is first transferred to a proxy mask, and then mapped to features ($F_{arr}$) along with the $F_{ori}$ (generated with random noise). An intuitive feeling is that when the randomly generated $F_{ori}$ is similar to the condition (proxy mask), the synthesis result will be better. So is the synthesis result sensitive to random noise? 3. Some expressions are not very clear. Authors claim that existing supervised methods require pixel-level annotations and are need to be re-trained if the number of classes changes. But the proposed method still need to re-train the semantic mapper if we want to add new semantic classes. So, I'm not sure what the author wants to emphasize. 4. With the development of large-scale vision fundament models, like SAM, we can now easily get the pixel-level annotations of images. And the results in the paper show that the performance of the proposed method is not good as full-supervised methods (OASIS). Thus, I suggest to provide more detailed analysis of the advance and potential applications of the proposed method.

Questions

My questions are listed in the weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Authors have discussed the limitations.

Summary

This paper proposed a self-supervised semantic image generation method. It adopts a pretrained generator, and rearranges the intermediate features to make the generated images align with the given input. Extensive experiments show the method achieve comparable performance with previous GAN based methods.

Strengths

The training is self-supervised and does not require paired data collection. The writing is clear. Results on diverse datasets, including faces and buildings.

Weaknesses

- The key motivation of this work is that it is difficult to collect large-scale paired data for semantic generation models. But this is not challenging for the cases shown by authors, i.e., mask-to-image, sketch-to-image. For example, we can use well-trained semantic segmentation models, e.g., SAM, to perform semantic segmentation, and use the sketch extraction method, e.g., HED, Canny, to extract sketches. In this way, we can easily obtain large-scale paired training data. Also, there are also some methods, e.g., PITI that use pretraining to alleviate the data issue. - The visual results do not show clear improvement than previous methods in Fig. 4, 5. For sketch to image generation in Fig. 7, the generated images do not align with the given inputs. - For k-means, how do you decide the cluster numbers? How would the noisy clustered mask affect the results? As the proxy mask is noisy, how would it achieve fine control of generated results?

Questions

Are G1 and G2 in Fig. 2 the same model?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

see weakness.

Summary

The paper introduces a method to adapt unconditional GAN generators to pixel-level-conditioned tasks, such as semantic image synthesis. The method is based on the observation that the intermediate generator's features contain rich knowledge about the semantics of generated objects. This knowledge is extracted to build a proxy mask via clustering, which can be manipulated by the introduced Reanranging Network. The manipulation to a user-specified conditioning mask is performed by a separate component called Semantic Mapper. Overall, the method allows to achieve high-quality, highly conrollable SIS without large paired image-mask datasets. The experiments also demonstrate a range of applications, including image manipulation, style-based SIS, and sketch-to-image synthesis.

Strengths

[Contribution] - The main strength, in my opinion, is itself the successful SIS from unconditional GANs. The literature on SIS GANs has long been somewhat detached from unconditional GANs, with its own tricks and training methods (e.g., pix2ix architectures, perceptual losses) and shortcomings (e.g., limited diversity). Apart from the data efficiency (claimed by authors), applying unconditional GANs to SIS brings the potential to improve SIS with all the recent advancements from other GAN literature. [Efficiency] - The proposed method uses a pre-trained GAN generator and one manual annotation (proxy mask -> real mask). This is a very cheap method to achieve SIS compared to conventional SIS GAN models. This can enable SIS on datasets that do not have good segmentation datasets, which can reduce costs. As the authors claimed, if the segmentation style changes, this model does not need to be re-trained, just the Semantic Mapper. [Performace] - Unconditional synthesis is naturally more difficult than SIS with paired datasets, so the quality of the model does not reach all prior SIS GANs trained on full datasets. But it is impressively not far, both in FID and mIoU. [Applications] - Some applications enabled by the model seem to extend the prior capabilities of both unconditional GANs and SIS GANs.

Weaknesses

[Limitation] - Enforcing each generated image to have K classes via K-means clustering seems rather restrictive. This works for FFHQ and Churches, but would not work for more classic SIS datasets like Ade20K, Cityscapes, or Coco-Stuff, where each image has only a subset of semantic classes. Solving this requires, probably, nontrivial extensions. [Evaluation] - One point of using unconditional GANs instead of SIS could be their better conditioning and diversity. I think it would be interesting to see more analysis on unconditional GANs vs SIS GANs, to see the trade-offs. For example, is LPIPS diversity of synthesis higher with StyleGANv2 compared to, say, OASIS? Other metrics like PPL, Precision&Recall would be also interesting. [Semantic Mapper] - It is a bit unclear how Semantic Mapper is trained. Is it correct that for a single given proxy mask, the user has to draw a real mask himself? In this case, it is also somehow a limitation, as it introduces overhead. More research is needed to find proxy-real mask alignment more automatically.

Questions

Please answer the concerns in Weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

No, more discussion is needed. I suggest adding the limitation points from Weaknesses if they remain relevant after the rebuttal. There is no discussion on societal impact at all.

Summary

This paper proposes a new method for semantic image synthesis that can generate images from given semantic segmentation maps, particularly in few-shot settings with different forms of inputs. The method trains a model to transform the input semantic segmentation maps into feature maps and cluster the output to outline the discriminativeness and boundaries in the resulting features. It then learns a conditional GAN to generate realistic images conditioned on random noise and the processed feature maps transformed from the corresponding semantic layouts. These designs enable few-shot capability and handle various forms of inputs (such as sketches and scribbles). The proposed method is extensively tested in experiments, which validate its effectiveness.

Strengths

1. The paper proposes an interesting method design that enables few-shot image synthesis by predicting a proxy mask from intermediate features predicted from the input. This approach also enables the method to handle inputs in different forms beyond just segmentation maps. 2. The experiments conducted on CelebAMask-HQ, LSUN Church, and LSUN Bedroom datasets validate the effectiveness of the proposed method. The one-shot performance on FID, mIoU, and accuracy metrics is impressive.

Weaknesses

1. The paper should discuss why certain method designs were not ablated, such as the use of k-means to generate the proxy mask from predicted features. The impact of this design choice on the effectiveness of the proposed method should also be measured and discussed. 2. The paper should include a discussion of the capacity and efficiency of the proposed model. Specifically, it should analyze how scaling the model impacts the quantitative results. 3. To provide a more comprehensive understanding of the proposed method, the paper should include an analysis of failure cases and discuss their limitations.

Questions

1. It would be interesting to investigate whether the semantic mapper could be initialized by pretrained semantic segmentation models or how its design could be improved. This could potentially improve the final performance and is worth exploring further. 2. Since diffusion-based methods can now be used for semantic segmentation, it raises the question of whether this framework could be extended to diffusion-based image synthesis. This is an intriguing possibility to explore, although there may be potential difficulties to overcome, such as the complexity of the model and the computational resources required.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

To explore the method's limitations, it would be beneficial to discuss some failure cases. This could help identify situations where the proposed method may not perform well and provide insight into potential areas for improvement.

Answer from Reviewer swNj

I thank the authors for their answers. My questions were addressed adequately, and, in addition, Appendix B.2 helped to reduce my confusion about the clustering. I remain with my positive evaluation and have no further questions for the authors at this point.