Learning segmentation from point trajectories

Summary

The authors propose a loss function that seeks to group the trajectories into low-rank matrices where the motion of object points can be approximately explained as a linear combination of other point tracks. Experiments on the synthetic MOVi-F variant of the Kubric dataset and the real datasets DAVIS 2016, SegTrackv2 and FBMS show that the proposed method outperforms single-sequence methods, single-stage end-to-end methods and multi-stage methods.

Strengths

1) The authors address key issues in the field and the contribution is original even if somewhat incremental. 2) The proposed method is detailed and reproducible. 3) Experiments are relatively well conducted on synthetic and real datasets showing the superiority of the proposed method.

Weaknesses

About the presentation, please clearly state a name/acronym to the proposed method and replace "ours" by it in the comparison tables.

Questions

No questions

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes

Summary

This paper introduces a method for training a segmentation network using long-term point trajectories as a supervisory signal to enhance optical flow. It proposes a novel loss function aimed at grouping these trajectories into low-rank matrices, allowing the motion of object points to be approximately represented as a linear combination of other point tracks. The proposed approach surpasses previous methods in motion-based segmentation, demonstrating the value of long-term motion and the effectiveness of the new formulation.

Strengths

1. The introduction describes the problem in more detail when introducing the issue. 2. The structure of the article is good. 3. The experimental results of the method proposed in this paper show a significant improvement.

Weaknesses

1. The main contribution of this paper is the proposal of two losses, but the loss seems to be effective in the experiments of other segmentation methods. 2. The contribution of the paper in Subspace Clustering is not described clearly. 3. The resolution of Fig 3 is relatively low. 4. There is a lack of comparison in terms of inference speed.

Questions

1. Does the method proposed in this paper work on other segmentation networks as well, and would additional experiments on other segmentation networks help demonstrate the generality of the proposed loss function? 2. How is the training time and inference speed of the method proposed in the paper? It would be better to include some quantitative comparison experiments. 3. What is the specific contribution of this paper to Subspace Clustering? 4. The Flow Estimator and Point Tracker are both frozen during training in this work. Is it possible to also update them during training to leverage the information in the dataset?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have acknowledged some limitations of their work. I suggest the authors could further describe the limitations in the generalizability of their approach.

Summary

This paper proposes a novel loss function that allows training image object segmentation models based on object motion in videos. Motivated by recent work on self-supervised learning of segmentation using optical flow, the authors propose to use longer point trajectories as additional self-supervision signal. Related to subspace clustering, the proposed loss function encourages to predict segments whose trajectories can be well explained by few basis trajectories. The predicted segments are merged into a binary segmentation mask and evaluated on standard, real-world segmentation benchmarks. Previous methods based only on optical flow are consistently outperformed, demonstrating the effectiveness of the proposed method.

Strengths

- Unsupervised learning of segmentation is an important problem. Several recent methods approached this task using optical flow as a self-supervision signal, extending this line of research to trajectories is a well-motivated idea. - The mathematical motivation of the loss is very well explained. Without having a deep mathematical background, I could follow the derivation of the loss function without issues. - Standard benchmarks and modelling components are used for evaluation, which makes it easy to compare the proposed method to previous approaches.

Weaknesses

1. It is not described clearly enough what kind of segmentation task is targeted. From the introduction and method section it seems to me that multi-object segmentation is adressed, only at the very end of the method section it is mentioned that the predicted segments are merged into a binary segmentation in some cases. - To my understanding the task is multi-object segmentation for MOVi and binary segmentation for all other datasets. This should be clearly stated in the experiment section. - It should be stated in the introduction and method section more clearly that the main task is binary segmentation. 2. The proposed method is not compared to models that do not use optical flow for self-supervision. It would be interesting to see how the proposed method compares to other self-supervised segmentation approaches. For example - CutLER ([Wang et al. 2023](https://arxiv.org/abs/2301.11320)) and VideoCutLER ([Wang et al. 2023](https://arxiv.org/abs/2308.14710)) - DINOSAUR ([Seitzer et al. 2023](https://www.amazon.science/publications/bridging-the-gap-to-real-world-object-centric-learning)) and VideoSAUR ([Zadaianchuk et al. 2023](https://proceedings.neurips.cc/paper_files/paper/2023/hash/c1fdec0d7ea1affa15bd09dd0fd3af05-Abstract-Conference.html)) The masks predicted by these models could be merged to obtain a binary segmentation in the same way as for the proposed method.

Questions

- How do the predicted segments look like before merging? Visualization would help to better understand the capabilities and limitations of the proposed method. - The principle of common fate is not cited in the paper, a reference to the literature on Gestalt psychology would be appropriate (e.g., Wertheimer 1912). - How well does the proposed method perform on MOVi when estimating trajectories using RAFT and CoTracker? This would allow for better judging how much the proposed method could be improved in the future by using more accurate trajectory estimation methods.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors address limitations of their work in a dedicated paragraph. Their discussion is brief but adequate in my view.

Summary

This paper proposes a model to process long-term motion and short-term motion simultaneously to achieve motion-based segmentation. Specifically, motivated by subspace clustering, this work proposes a loss function that enables training a neural network to learn motion grouping from both optical flows and point trajectories. It outperforms the previous method in the unsupervised video segmentation task. The qualitative comparison also shows obvious improvement, giving a clearer boundary.

Strengths

1. The motivation and method explanation seems to be clear. The paper writing is easy to follow. 2. Using a simple sample to introduce the low-rank intuition is convincing and reasonable. Based on this core idea, other smoothing losses and regular loss from optical flow make learning more effective. 3. Experiments show the strength of the proposed strategy. A comprehensive ablation study has been performed to illustrate the impact of each factor.

Weaknesses

1. As mentioned in the limitation, the paper's principle assumes that the object is rigid. However, the task that this paper works on not only includes rigid objects -- it's a general video segmentation task. Then it seems that the low-rank theory can not extend to a general setting. And why not consider local rigid like ARAP loss? (SpatialTracker) 2. Do not give some corner cases or failure cases, especially for non-rigid objects. I hope to see some corner cases like multiple objects, where they behave similarly in the short term but different in the long term. Then it can better demonstrate the motivation of the paper.

Questions

Why does solely using long-term loss get worse performance than solely using optical flow loss (7 percent drop in Table 4)? Though the paper gives a short explanation that it is due to the sparse set of points and noise, long-term motion also has its advantage like it's more stable than short-term information.

Rating

5

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

As mentioned in the weakness, the principle the paper proposed is reasonable, but seems like it does not fully support the motivation and the ultimate goal of the task. More analysis and experiments are needed to show the right practice when applying the proposed to real-world videos.

Summary

The paper tackles video object segmentation by incorporating into the loss function not only instantaneous optical flow information but also long term pixel tracking information. Raft was used for optical flow and CoTracker was used for long term pixel tracking in the experiments. The experiments show a marginal improvement in performance when combining the two information sources in the loss function.

Strengths

The paper flows quite well, it addresses that video object segmentation is the problem space, the focus is on loss function, Figure 2, the layout appears clear as well. There are a handful of datasets and comparing methods used in the experiments.

Weaknesses

Table 2 where the experimental results are presented lists a collection of methods categorized into different groupings. Perhaps these groupings and methods could be better discussed in the lit review, it appears that the categories in the lit review do not correlate nicely and I do not know the difference of these methods unless I look at the references and read the papers myself. The improvement is incremental. IT is expected that there would be some improvement however what cases do we actually get the improvement in, a bit of more depth in the analysis would make this a better paper. I assume that the camera is static?, correct? if not, perhaps making this clearer would help. I have no idea how long the long term point trajectories were, perhaps analyzing this would help. Also depending on the trajectories, were there occlusions or other interesting factors that contribute to the loss function would be interesting.

Questions

1. I found the related works were like a laundry list. You divided the categories into unsupervised video object segmentation, motion segmentation, trajectory based motion segmentation and subspace clustering. That is find however your focus is only video object segmentation, why is that and how can you address the other problem areas? 2. I would imagine if we had 3d scene flow, by perhaps combining monocular depth and optical flow would result in good results without long term tracking? 3. why not incorporate appearance information as well? 4. Appearance information for segmentation in the examples would suffice, it would be interesting to focus on cases where appearance info is not sufficient for segmentation and we require motion information.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

I am not sure that the examples actually illustrate that motion segmentation is necessary for these cases. I would focus on cases where appearance information is not enough. Can this system deal with a moving camera or does the camera have to be static? How well does the system work under occlusion? Different motions of the object of interest will results in different performance, perhaps diving into this analysis would be informative. Both sources of info, optical flow and long term pixel tracking info are based on 2D info, the projection of 3D info. This has limitations. The paper should have explored different object movements. It does state that non rigid objects when dealing with multiple objects is an issue however an in depth exploration for a single non rigid object would be informative.