Summary
This work proposes a novel unsupervised object-centric learning method. In particular, the proposed approach enhances the slot-attention mechanism by incorporating a "top-down pathway" that highlights features relevant to objects in the image.
The method first employs a standard self-attention mechanism to extract slot vectors and the corresponding attention masks. These initial slot vectors and attention masks are then fed into another round of the slot-attention mechanism. In this round, they modulate the value maps of the cross-attention mechanism separately for each slot. This modulation is channel-wise for slot vectors and spatial-wise for attention masks, enabling the slot-attention mechanism to emphasize to features relevant to the slots discovered on the first slot-attention round.
Furthermore, the method quantizes the slot vectors from the first slot-attention round using the VQ approach before utilizing them for modulation in the second round.
To evaluate the proposed self-modulated-based slot-attention mechanism, the authors integrate it into the DINOSAUR framework and apply it to the COCO, VOC, MOVIE-E, and MOVIE-C datasets for the downstream task of object segmentation. The results demonstrate improvements over the vanilla slot-attention mechanism. Ablation studies are conducted on the COCO dataset.
Strengths
STRENGTHS:
- The idea of modulating the value maps for each slot, using slots found from a previous self-attention round, is interesting. This technique allows the subsequent self-attention round to focus more on features that are relevant to the objects discovered in the initial round. Essentially, it's an additional iterative improvement for the extracted slots, supplementing the existing iterative mechanism in the standard slot-attention.
- The proposed method has been demonstrated to improve object segmentation results on all tested datasets. Furthermore, the authors utilized challenging datasets, such as COCO, to validate the effectiveness of their method, rather than resorting to simpler one, which is often among other object-centric works.
- The paper has strong results and detailed ablation studies.
Weaknesses
- **(W1)** It's unusual that the mIoU results in the MOVIE-C and MOVIE-E datasets surpass the mBO results. The mBO is computed by assigning each ground truth mask the predicted mask with the largest overlap, and then averaging the IoUs of the assigned mask pairs. On the other hand, the mIoU metric is more strict and employs Hungarian matching (rather than greedy matching) to assign predicted masks to the ground truth masks. Consequently, mBO results should be either greater than or equal to the mIoU results. Therefore, it's highly likely that there's a bug in the computation of the mBO or the mIoU metrics on the MOVIE-C and MOVIE-E datasets. Or a typo / mistake when copying the results to the paper maybe.
- As a relevant side note for the upcoming comments, it's important to mention that the FG-ARI metric is generally viewed as unreliable for assessing unsupervised object-centric methods. This is because it only takes foreground pixels into account, which can provide a deceptive understanding of segmentation quality by disregarding the localization accuracy of predicted masks [A, B, C, 30, 42].
- **(W2)** The motivation behind the Vector Quantization (VQ) component of the method is unclear. In particular it's uncertain why quantizing the slot vectors is essential for self-modulation. In fact, Table 4 indicates that the method is sensitive to the codebook size (also acknowledged by the authors in the limitations section). The mBO performance for sub-optimal codebook sizes 128, 256, and 1024 (30.2%, 30.5%, and 29.8%, respectively) is either the same or worse than the reproduced DINOSAUR mBO performance (30.5% from Table 6). Conversely, removing VQ achieves 32.3%, which is only marginally worse than the 32.7% obtained with VQ and the optimal codebook size. Therefore, VQ unnecessarily complicates the proposed method. As mentioned earlier, FG-ARI is unreliable, and the higher differences in this metric (36.3% w/o VQ vs 37.7% w/ VQ) should be disregarded.
- It's also somewhat concerning that Table 6 shows only a minimal impact on mBO performance when using channel-wise modulation (from 32.5% to 32.7%). However, this does not really increase the complexity of the method, as VQ does.
- **(W3)** Lacks discussions on how the proposed method affects the training and test time
- **(W4)** Table 3 is missing comparisons with related works, such as Rotating Features [D] and SPOT [C], which also demonstrate strong results.
[A] Genesis: Generative scene inference and sampling with object-centric latent representations, ICLR 2020.
[B] Unsupervised Layered Image Decomposition into Object Prototypes, ICCV 2021.
[C] SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers, CVPR 2024
[D] Rotating features for object discovery. NeurIPs 2023.
[30] Bridging the gap to real-world object-centric learning, ICLR 2023.
[42] Slotdiffusion: Object-centric generative modeling with diffusion models, NeurIPs 2023.
Questions
- My primary concern regarding this work (mentioned as (W2) in the weaknesses section) is that the channel-wise and Vector Quantization (VQ) components unnecessarily complicate the method without demonstrating significant improvements in the ablation results. I would appreciate it if the authors could provide compelling arguments, such as additional experimental evidence on other datasets (e.g., VOC or MOVIE-E/C) or different settings (e.g., longer training), that show (if it is the case) the benefits of channel-wise modulation with VQ.
- Please address the concerns (W1), (W3), and (W4) mentioned in the weaknesses section.