Self-supervised Object-Centric Learning for Videos

Summary

This paper introduced a fully unsupervised method, SOLV, for segmenting multiple objects in real-world sequences. In this paper, the author employed the token drop strategy to reduce computation and enhance regularization. Additionally, Spatial-temporal Binding was proposed to aggregate the features of objects within and across frames, resulting in a remarkable improvement in mIoU. To address over-clustering, the author employed Agglomerative Clustering to merge slots. It is noteworthy that SOLV not only significantly advances the state-of-the-art on commonly used simulated data but also stands out as the first fully unsupervised method to demonstrate state-of-the-art performance on unconstrained videos from the Youtube-VIS dataset.

Strengths

1. A fully unsupervised object-centric learning method for scaling multi-object segmentation to in-the-wild videos was first proposed and achieved significant improvements on traditional synthetic datasets. Moreover, it has delivered impressive results on real-world video datasets. 2. Many details in the method, such as token drop ratio, number of slots, and variations of the visual encoder, have undergone ablation experiments, enhancing the robustness and reproducibility of the work. 3. The slot merging strategy significantly reduces the sensitivity of the model's performance to the number of slots. 4. Both temporal binding and spatial binding techniques bring significant gains in mIoU.

Weaknesses

1. A more elaborate explanation is required to clarify how the Agglomerative Clustering algorithm is employed to dynamically determine the optimal number of slots.

Questions

1. In the ablation experiments, a notable observation arises regarding the performance of Model-C and Model-D compared to their predecessor, Model-B, as they exhibit a decrease in FG-ARI. However, Model-E, which combines the spatial bind and temporal bind components from both Model-C and Model-D, ultimately achieved superior results in terms of FG-ARI. What are the possible reasons behind this phenomenon？ 2. Whether the shape of $\mathcal V_t$ at line 120 is indeed $\mathbb R^{(2n+1)\times N \times (3P^2)}$, and the dimension of $\mathcal V_t'$ in Equation (2) should be $\mathbb R^{(2n+1)\times N \times (3P^2)}$.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1. As discussed at the end of the paper, it is acknowledged that Agglomerative Clustering is non-differentiable during the training process. Therefore, it might be worthwhile to explore alternative clustering algorithms as potential alternatives. 2. Based on the visualized results, it is evident that SOLV excels in the detection and clustering of foreground objects. However, there is room for improvement in terms of the accuracy of segmentation boundaries.

Summary

The paper introduces an approach to segment multiple objects for video sequence in both real and synthetic data without utilizing any additional signals besides RGB frames. It has 3 components including a visual encoder, a spatial-temporal binding model used for grouping pixels into slots across different time, and a visual decoder to get the segmentation mask. The method also uses a slot merging strategy to address the over-segmentation issue caused by a fixed number of slots. The paper conducts experiments and it outperforms previous methods on synthetic and real data. The ablation studies are also presented to prove each component's effectiveness.

Strengths

This paper can segment complex scenes in Youtube videos and achieve satisfying qualitative results. It didn’t rely on extra signals in videos which is claimed not stable in video segmentation. The drop approach can save memory and calculation time successfully. This paper achieves great performance on MOVi-E.

Weaknesses

This paper didn't test on single object segmentation dataset like DAVIS 2017 and compare methods like CIS, Y Yang 2019.

Questions

For the video segmentation dataset like DAVIS17, how could you choose which segment mask as the object mask, not the background, used to calculate the metric? How to make sure that the merging mechanism can successfully merge the masks into a whole object? How to determine which parts should be merged into one object?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

It could be better to visualize several cases of each slot's segmentation mask results for slot=8 which is claimed to be the best in Table 4.

Summary

This paper proposes an unsupervised segmentation method in videos. The backbone network is pre-trained with the self-supervised method. Then, the model spatially binds objects to slots on each frame and then relates these slots across frames. The framework is trained to reconstruct the middle frame in a high-level semantic feature space.

Strengths

1. The proposed method is able to segment a real work video without supervision by utilizing a powerful pre-trained feartures. 2. The performance of the proposed method is better than comparison methods. 3. The method is evaluted on both sythethic and real-world dataset.

Weaknesses

1. Although the method can roughly segment the objects in videos, the method is not capable of detecting the boundary of objects. 2. The method can only be applied to videos since it relies on temporal information.

Questions

1. What's the advantage of the proposed model compared to the recent Segment Anything model? 2. In slot merging, whether the clustering method will affect the performance? 3. What if the method is applied to more complex scenes which have more than 12 objects?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

see weakness

Summary

This paper proposes a new self-supervised method for multi-object segmentation in videos called SOLV (Self-supervised Object-centric Learning for Videos). It adopts axial spatial-temporal slot attention to group pixels into slots within frames and then relates these slots across frames to track objects. It uses the masked autoencoder training objective to reconstruct visual features from latent slots. A slot merging strategy based on agglomerative clustering is proposed to address over-segmentation by dynamically merging similar slots. Experiments show state-of-the-art results on the MOVi-E synthetic dataset and Youtube-VIS 2019 real videos without requiring additional modalities like optical flow.

Strengths

The paper is well-written and easy to follow. The problem setup and overall approach are clearly explained. The performance looks great. The masked feature reconstruction seems interesting. The spatial-temporal binding and the slot merging make sense to me.

Weaknesses

A comparison with self-supervised baselines on the real data (DAVIS17 / YTVIS19) needs to be added, as there is a huge performance gap between synthetic data and real data (80 vs 30). Such a huge gap makes it questionable whether the results on synthetic data is really representative. Also, it would be great to also add recent supervised methods to help readers understand the gap between self-supervised / supervised methods. It would be great also to add an error analysis to showcase the failure mode and model behaviors. It seems that the model needs a superior pretrained backbone (DINOv2) to achieve good performance. DINOv2 is pretrained on a large, curated dataset LVD-142M, which gives the authors an advantage over the baselines. A more fair comparison should be made in Table 2 when compared with DINOSAUR, which is equipped with DINO. I encourage the authors use more text to describe the necessity of using a well-pre-trained encoder instead of simply saying using a frozen encoder in the model. Given those concerns, I feel like the current results do not show enough evidence for acceptance, I'd be happy to increase my rating if the authors could address my concerns by showing more results (especially for real data with the same visual encoder). Misc: I suggest the authors revise Fig.2 and its caption to make it more self-contained. Now the illustration is unclear and has many symbols that are not explained in the caption, making it difficult to understand without referring to the text.

Questions

Please see the weakness part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper proposes a Self-supervised Object-centric Learning framework for unsupervised multi-object video segmentation. To achieve this, this paper proposes to derive object-centric representations in a self-supervised manner to facilitate video segmentation tasks. The proposed approach adopts axial spatial-temporal slots attention to group visual regions with similarity properties, while exploiting the training techniques of masked feature reconstruction and slot merging. Experimental results on synthetic and real video datasets demonstrate superior performance against previous works.

Strengths

1. Learning object-centric representations in an unsupervised fashion is vital yet challenging for several downstream applications, such as video object segmentation or robot manipulation. The task aiming to be addressed in this paper is significant. 2. The experiment results and analysis both in the main paper and supplementary material are comprehensive. 3. The overall paper is easy to follow.

Weaknesses

My primary concern lies in the novelty and significance of the proposed framework. The novelty and contributions of the proposed approach are unclearly described in this paper. In the current paper draft, though it employs and modifies a series of existing techniques, it is not evident how the proposed approach is novel or provides significant advancements over existing methods. For example, COMUS [A] also learns self-supervised object-centric representations from DINO, aiming to achieve unsupervised object segmentation. The spatial binding is slightly modified from invariant slot attention [3], and the temporal binding is simply realized by the self-attention mechanism. In the visual decoder, the slot merging process is based on Agglomerative Clustering (AC) algorithm, while the decoder architecture follows DINOSAUR [62]. In summary, the proposed method seems to be a combination of existing approaches to perform the task of multi-object video segmentation, it is currently unclear how their combination in this framework constitutes a significant improvement or novelty over existing methods. [A] Zadaianchuk et al., Unsupervised Semantic Segmentation with Self-Supervised Object-Centric Representations. ICLR 2023

Questions

1. This proposed method utilizes a pre-defined number of slots to allow multi-object segmentation (e.g., 8 slots). Can the trained model be applicable to an image with more than 8 objects? 2. In Table 3, why did adding spatial and temporal binding (Model-C and D) deteriorate the FG-ARI score compared with Model-B? 3. Also in Table 3, without temporal binding (Model-D) just slightly drops the performance of the full version (Model-E). Does it imply that the temporal information would not be necessarily considered if the frame-by-frame segmentation is well performed?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors have listed the limitations of this proposed method and the potential future research directions to alleviate these raised issues.

Reply to Author Rebuttal

I have read other reviews and the author's rebuttal. Thanks for the response which addressed part of my concerns. I agree with the authors that SAM is a supervised model. However, it generalizes well to unseen scenes. Since we already have this kind of model, what's the point of designing from scratch instead of working based on it? I think a zero-shot segmentation method based on SAM holds greater significance than the proposed method.