SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow

Summary

This paper presents a work for solving semantic image segmentation and image synthesis simultaneously. This work proposed to use rectified flow and adopt a VAE encoder to compress the images and pseudo masks into the latent space. By comparing the proposed method with previous semantic segmentation models and image synthesis SOTAs, the effectiveness of this method is demonstrated. The proposed method handles the randomness issue for semantic segmentation. Finally, ablation study has been provided.

Strengths

+ This proposed an interesting formulation to bridge semantic segmentation and image synthesis in the stable diffusion framework with rectified flow framework. It is an interesting work and different to previous work on unifying semantic segmentation and image synthesis, which produces segmentation masks and synthesized images at the same time. Differently, this work trains one model to convert between segmentation masks and synthesized images. + This work also shows faster generation speed for high quality images.

Weaknesses

- What is the motivation to use Euler sampler. More discussion between Euler and DDIM/DDPM will be more helpful to understand the technical details for most readers. - For image synthesis task, the proposed method had a clear gap compared with other diffusion models.

Questions

See above.

Rating

5

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

No.

Summary

The paper presents SemFlow, an approach that binds semantic segmentation and semantic image synthesis using an ordinary differential equation (ODE) model. The motivation is to use rectified flow to enable LDM as a unified framework for both tasks. The key contributions include a unified framework that jointly optimizes both tasks, the specialized designs of the frameworks, including pseudo mask modelling, bi-directional training of segmentation and generation, and a finite perturbation strategy. This approach bridges the gap between semantic segmentation and semantic image synthesis, showing good visualization results.

Strengths

1. The unified framework for joint optimization of semantic segmentation and semantic image synthesis is novel. 2. The paper is generally well-written and structured.

Weaknesses

1. Lines 133-140: Why is there a necessity to transform the semantic segmentation masks into 3-channel pseudo-masks utilizing Eq. 7? Could you provide insights on how the formulations for m1 and m2 were derived? 2. Figure 4: During the model inference process (semantic image synthesis task), will noise be added to the mask? 3. Lines 147-155: SemFlow cannot use captions to guide image synthesis, and the authors claim the usage of captions is non-casual for semantic image synthesis, however, there is still some work on it [1,2]. 4. There is still a gap in quantitative results compared to the popular model within each task. [1] Xue, Han, et al. "Freestyle layout-to-image synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. [2] Lv, Zhengyao, et al. "Place: Adaptive layout-semantic fusion for semantic image synthesis." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.

Questions

See Weaknesses

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper proposed a unified diffusion-based framework for semantic segmentation and semantic image synthesis. The proposed SemFlow applied the existing ordinary differential equation (ODE) model and modified the transport problem setting. Additionally, it incorporates techniques such as perturbation and straight trajectory to enhance model performance. The proposed method was compared with a simple diffusion-based conditional generation modeling and a transformer-based method (Mask2Former) for semantic segmentation, as well as several GAN-based and diffusion-based methods for semantic image synthesis. However, the results did not surpass existing methods, and performance in the semantic segmentation task was even 10% lower in mIoU compared to basic segmentation methods.

Strengths

1. The motivation behind this paper is well though out. It adds value by attempting to unify low-level and high-level vision tasks. 2.The writing style is clear and easy to understand.

Weaknesses

1. The literature review is insufficient and lacks citations of relevant works such as FreeMask, NeurIPS, 2023. 2. Although the motivation is thoughtful, the theoretical and practical value of the proposed method is limited. The application results are not promising, as the semantic segmentation performance is over 10% lower than Mask2Former. The method does not demonstrate advantages over existing training data synthesis methods. 3. The key components of the proposed method are commonly used in generative fields and therefore lack novelty. 4. The experiments are insufficient. It is recommended that the authors include FreeMask and other relevant methods for comparison in the semantic segmentation task, as well as ControlNet or Freestyle generation methods for semantic image synthesis. Additionally, the authors should provide complete quantitative results on all three datasets for both semantic image synthesis and semantic segmentation tasks. 5. The authors claim limited computational resources as a reason to leave training data synthesis methods for future work, yet the experiments were conducted on 8 NVIDIA A100 GPUs, which is more than the resources used in most training data synthesis studies. Also, the computational cost advantages are unclear due to the absence of computation time data.

Questions

Although I appreciate the motivation and exploration in this paper, the results do not convince me that the proposed method is promising and practical for image synthetic and semantic segmentation tasks.

Rating

4

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes