Summary
This paper proposes a novel architecture for denoising diffusion probabilistic models (DDPMs) that enables simultaneous unsupervised image generation and segmentation. The key idea is to constrain the denoising network with a structured bottleneck that factorizes the image into regions, which are then denoised in parallel. This architectural design encourages the model to learn meaningful segmentations as a byproduct of training for image generation. The authors demonstrate that their approach can generate high-quality images along with corresponding segmentations, as well as segment real images, without any supervision or additional loss terms. Experiments on multiple datasets show improvements in both image quality and segmentation accuracy compared to baselines.
Strengths
* Novel architectural design that unifies unsupervised image generation and segmentation in a principled way
* Achieves strong results on both tasks without requiring additional loss terms or supervision
* Provides insights into how the model learns to factorize images into regions
* Explores extensions to hierarchical segmentations
Weaknesses
* Limited theoretical analysis:
The paper lacks a rigorous theoretical foundation for why the proposed architecture leads to meaningful segmentations. While empirical results are strong, a deeper analysis of why factorizing the denoising process encourages semantic segmentation would strengthen the contribution. For instance, the authors could explore connections to information bottleneck principles or analyze the gradients flowing through different parts of the network.
* Scope of experiments:
The experiments primarily focus on datasets with relatively simple segmentations (2-3 regions), such as Flower, CUB, and face datasets. While the ImageNet results are promising, they are limited in scope.
The paper lacks experiments on more challenging datasets with complex, multi-object scenes (e.g., COCO, Cityscapes) that would better demonstrate the method's scalability and generalizability.
* Incomplete comparisons:
The paper misses comparisons with some recent, relevant unsupervised segmentation methods, such as PiCIE (CVPR 2021) or STEGO (ICCV 2021). Including these would provide a more comprehensive evaluation of the state-of-the-art.
For the image generation task, comparisons with other recent diffusion model variants (e.g., latent diffusion models) would be valuable to contextualize the improvements.
* Architectural choices and ablations:
The paper does not extensively explore variations in the factorized architecture. For instance, how does performance change with different numbers of parallel decoders or alternative ways of combining their outputs?
More comprehensive ablation studies would help isolate the impact of different components of the proposed architecture.
Questions
* The setting of generating both the images and the segmentation maps is interesting. However, Isn't this setup a little trival? Why we need to generate both contents at the same time? How could this feature help us?
* How sensitive is the method to the choice of number of regions K? Is there a way to automatically determine the optimal K for a given dataset?
* The paper mentions that the method can be extended to hierarchical segmentations. Could you provide more details on how this would work and what challenges might arise?
* Have you explored applying this method to more complex segmentation tasks with many object categories? What modifications might be needed?
* How computationally expensive is the proposed method compared to standard DDPMs, both for training and inference?
* The paper claims the method can be applied to segment novel images with just one forward pass. How does the runtime compare to other unsupervised segmentation methods?
* Some literature that share the similar insights of unifying generation and segmentation can be refered to:
[1] Lai Z, Duan Y, Dai J, et al. Denoising diffusion semantic segmentation with mask prior modeling[J]. arXiv preprint arXiv:2306.01721, 2023.
[2] Li C, Liu X, Li W, et al. U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation[J]. arXiv preprint arXiv:2406.02918, 2024.
Limitations
The authors have adequately addressed the limitations of their work in Section 5, acknowledging that the current results are limited to 2-3 class segmentations and discussing the need for further work on handling more complex scenes. They also mention computational costs as a potential limitation. The paper does not directly discuss potential negative societal impacts, which could be briefly addressed, though the risks seem relatively low for this type of fundamental methodological work.