SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models

Summary

This paper presents SlotDiffusion, an object-centric Latent Diffusion Model (LDM) designed for image and video data. The authors focus on improving slot-to-image decoding, a critical aspect of visual generation. SlotDiffusion outperforms previous slot models in unsupervised object segmentation and visual generation across multiple datasets. The model's learned object features also enhance video prediction quality and temporal reasoning tasks. The scalability of SlotDiffusion is demonstrated on real-world datasets like PASCAL VOC and COCO, showcasing its potential when combined with self-supervised pre-trained image encoders.

Strengths

- This direction of slot-based generation is promising. - The experimental section provides fruitful results that effectively validate the proposed method from various perspectives.

Weaknesses

- In the experimental section, the authors primarily compare their method with mixture and transformer-based baselines. As a result, the main contribution of the observed gains may primarily stem from the diffusion models themselves, potentially undermining the novelty of the proposed methods. - It would be beneficial to evaluate the proposed method on the CelebaMask dataset, as mentioned in Line 235. - In Line 96, the authors mention the differences between their work and Jiang et al. It would be useful if they also included a basic comparison in the experimental results. - Regarding Figure 11, it would be helpful to understand the performance gap between your method and the baseline at different diffusion steps. - The paper lacks additional discussion. Since slot-attention is a paradigm for identifying meaningful visual concepts, there are other self-supervised approaches, such as [1], [2], and [3], that achieve similar goals. Including a discussion relating to these approaches would be appreciated. It seems that [1] follows a similar process of obtaining semantic meaning in a post-hoc manner (e.g., sampling, observation, assigning semantic meaning). [1]. Self-Guided Diffusion Models [2].Why Are Conditional Generative Models Better Than Unconditional Ones? [3].Visual Chain-of-Thought Diffusion Models

Questions

As above. I am inclined to raise my score if my concerns are fully resolved.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

yes

Summary

The paper presents, SlotDiffusion, an object-centric model that incorporates a latent diffusion model as an image decoder. Extracted slots are used as a conditioning signal for the cross-attention layers of the diffusion UNet. Other than that, the model follows the formulation of SA/STEVE/SAVi models. SlotDiffusion is tested on segmentation, reconstruction and compositional generation tasks on various simulated and real datasets, comparing favourably against Slot Attention, SLATE, STEVE, SAVi and DINOSAUR models in their respective settings. The paper additionally considers the tasks of video prediction and VQA by augmenting the model with SlotFormer, where similarly strong results are observed.

Strengths

The proposal incorporates a latent diffusion model as a decoder, which is novel in the context of object-centric learning (up to concurrent work). The model shows strong results on a variety of benchmarks. The writing is clear and easy to follow, and the presentation is good with only a few missing details (see weaknesses/questions).

Weaknesses

While the method shows improved reconstruction quality and segmentation results over methods that do not use LDM as a decoder (and are otherwise, similar or identical), it is not clear if the LDM Unet model is comparable to a small spatial broadcast decoder (as in SA) or transformer decoder of SLATE/STEVE. Are these model components comparable in terms of e.g. the number of parameters? As stated in Appendix D.3 (L180) "We found that using a _larger decoder_ and training longer leads to better performance". Could this be the reason for the improved results? Additionally, there are some more recent works [Jie et al., Singh et al.] omitted, which might serve as better points of comparison than SA/SAVi/SLATE/STEVE. It is not entirely clear what procedure is adapted for video prediction and VQA tasks, where the SlotDiffusion is augmented with SlotFormer. Would this not remove the LDM entirely from the model? Singh et al. "Neural Systematic Binder" Jie et al. "Improving Object-centric Learning with Query Optimization"

Questions

For evaluating the segmentation, what masks are used? Is this the attention of the final slot attention iteration? It would be helpful to clarify how the masks are predicted for evaluation. How much do samples from the decoder vary? In other words, can the LDM model maintain the ability to generate diverse samples? Or does initial noise input have only limited to no effect? In the checklist, it is mentioned that results are reported as an average of 3 seeds and the std. dev. are small. Would it be possible to report them, at least for SlotDiffusion models? It is particularly interesting given the added stochasticity of the LDM. In Figure 2, are the 3 models with different decoders specific models, eg SA, SLATE and SlotDiffusion?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The limitations seem to be largely addressed. Perhaps it would be worth mentioning that it appears that the learned slots encode the position of the objects they represent. This means that moving an object in a scene would require having a slot for the same object in a new position.

Summary

A latent diffusion-based slot decoder is introduced to improve the segmentation and reconstruction quality for slot-based autoencoders. This decoder addresses issues with previous advanced Transformer-based decoders such as SAVi and STEVE which i) do not preserve the spatial dimension of images and ii) lack an iterative refinement component for capturing high-frequency image details. SlotDiffusion combines a pretrained VQ-VAE with a cross-attention denoising diffusion decoder. The key idea is to treat slots like text embeddings and therefore to use cross-attention in the decoder following text-guided latent diffusion models. Rigorous qualitative and quantitative results confirm the improvements in segmentation and reconstruction quality.

Strengths

#### Originality - This work fills a gap in the literature by exploring how to combine ideas from latent diffusion modeling with slot-based object-centric learning. - The insight that slots can be used like text-embeddings in the cross-attention module of a denoising diffusion decoder is novel and potentially quite powerful. #### Quality - The technical description of the model is sound and the experiments are rigorous (2 complex image datasets, four video datasets, 1 video prediction and reasoning benchmark, and 2 real-world image datasets). #### Clarity - The paper is well-written and easy to follow. #### Significance - I believe this work is a valuable contribution to the object-centric learning community. - In particular, the breadth of experiments in this paper help demonstrate the value of this research direction and the generality of the models being explored.

Weaknesses

I have very few concerns with this work. - While I applaud the authors for evaluating SlotDiffusion on video prediction, VQA, and real-world image datasets, the results on these benchmarks are not as impressive as on the synthetic datasets. - Moreover, scaling these models to real-world images appears to require a complex, multi-stage modeling effort with various important hyperparameters to be tuned: 1) pretraining the DINO ViT model, then 2) pretraining the VQ-VAE model, then 3) stitching all these together and training the denoising diffusion slot decoder. - These are not big issues, but it suggests that perhaps something fundamental is being missed by this approach.

Questions

- Table 2: What is the improvement with respect to? What is the Obs. % and Dyn. % for random guessing? --- Update after rebuttal: I am maintaining my initially positive outlook on this paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations and Broader Impacts are sufficiently discussed in Appendix F and G. I would encourage the authors to include this in the main text, space permitting.

Summary

The paper proposes a diffusion-based decoder for slot-centric models. Previously slot-centric models are trained with spatial broadcast or transformer autoregressive decoders, however these decoders result in a comparitively poor image generation or segmentation. The paper shows that using diffusion based decoders can result in better generation, segmentation and video prediction.

Strengths

i) The paper does a dense evaluation of it's method ii) The paper gets better peformance on almost all of the settings iii) The paper is well written and is easy to understand.

Weaknesses

I couldn't find any major weakness in the paper. Although I have a few questions that i have mentioned below.

Questions

i) In Table 1, how does the model compare against spatial broadcast decoder, instead of steve that uses transformer decoder? ii) On real world benchmarks how would this method compare against CutLER (https://arxiv.org/abs/2301.11320), a method that doesn't use any slot-based training but does normalized cuts on top of Dino features and calculates segmentation from there. iii) How does the method compare on original CLEVR, as i find all the fg-ari scores in Figure 2 to be below 0.7. iv) What about classifier-free guidance? have the authors tried that with slots? v) I would be interested in seeing (maybe in future) how does this method do when coupled with supervised labels, like evaluations shown in classical SSL methods such as MAE (https://arxiv.org/abs/2111.06377) or recent Slot-based methods such as Slot-TTA (https://arxiv.org/abs/2203.11194)

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Rebuttal Acknowledgment

I thank the Authors for their thorough and detailed response. I believe the majority of my concerns raised in the initial review have been addressed and I would encourage the authors to include the clarifications in the paper where possible. Additional results with more recent methods are encouraging as well. Finally, I would also like to suggest including the discussion above about scaling the decoders from prior work as well. Considering that incorporating LDM decoder is one of the core ideas presented in this paper, it strengthens the message to show better scaling behaviour than alternatives from prior work. I currently have no further questions but will follow any discussion that might develop and make changes to my recommendation afterwards.

Thanks for the response

The authors have provided answers to my questions and comments in a satisfactory manner. As to the question about Table 2, I do believe I was asking about the performance of random guessing! Thanks.