Summary
This work introduces a methodology for using diffusion-based models to obtain object-centric representations.
This is done using slot representations, obtained from the Slot Attention module applied to the original image, as a conditioning variable for a latent diffusion model. The diffusion model and the Slot Attention module are trained end-to-end.
The proposed model is called Latent Slot Diffusion (LSD) and is evaluated on unsupervised object segmentation, downstream property prediction, compositional image generation, and image editing. The datasets used for evaluation are ClevrTex, MOVi-E, MOVi-C and, for the first time in the field, FFHQ. The performance of LSD is evaluated quantitatively on the first 4 tasks and is compared against SLATE and SLATE+. The proposed model outperforms both baselines in the tasks according to most metrics.
Strengths
**Originality**.
The paper proposes a novel way to combine two existing models: slot attention and latent diffusion. The presentation clearly shows what are the novel elements.
**Quality**.
The method shows a successful way to leverage diffusion in object-centric learning. Especially the results on property prediction show that it has concrete benefits for the representation learning itself, which might be useful for several different tasks, not only the ones shown in the paper.
**Clarity**.
The submission neatly shows all the experiments that were carried out and the description of the underlying method is clear.
**Significance**.
The work is a necessary exploration of leveraging the generative power of the diffusion approach in the object-centric setting, highlighting the complexities related to having too-strong of a decoder and providing clear examples of its strengths, especially by using a very complex dataset (FFHQ).
Weaknesses
**Quality**.
The lack of error bars (i.e. standard deviation) in the analysis makes the quantitative analysis weaker. Additionally, it would be interesting to further explore the problems with FG-ARI, as it is currently the standard metric used in the field, and although I agree with the statements, this is only based on experience and intuition, and not a proper scientific analysis, which has not been carried out yet. Lack of comparison with traditional models such as the improved Slot Attention architecture proposed in [1] makes the performance of the model harder to evaluate. Are the good results obtained primarily due to the improved architecture (e.g. larger encoders, better decoder), or is there something fundamentally good about using diffusion (e.g. the iterative improvement) that results in better object representations?
**Clarity**
The description of the method used for unsupervised object segmentation is lacking. There is only a reference to the use of the attention masks from slot attention, it would be much better to refer directly to the nice mathematical notation used before in the text.
*References*
1) Biza, Ondrej, et al. "Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames." arXiv preprint arXiv:2302.04973 (2023).
Questions
As object segmentation is performed using the masks from the slot attention module, it is clear that it is not possible to obtain masks with the full resolution of the images in many cases. Could this be an area of improvement for future work or is it possible to already try using some unsupervised super-resolution technique to get higher-quality masks? Could be worth considering object segmentation as a visualization of what the model is doing instead of a separate task?
Do you have any insights into how the model would perform if slot attention is applied to the latent representation of the latent diffusion model instead of the original image? This would help clarify the choices made during the development of the method.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
The authors have sufficiently addressed the limitations of the proposed model, as well as the broader impact of their approach. However, this can be improved further, by being more explicit about direct effect of ethnic bias, ageism or other forms of mis-representation that the model could lead to, which is standard practice for modern diffusion model, considering the extreme high-quality of the generated images. Direct manipulation of certain characteristics enables these effects in a much easier way.