Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimuli

Summary

Authors propose a study on the generalization of motion segmentation models to random dot kinematograms, where they study various learning based (optical flow) motion estimation models vs. a classical motion energy model. Their work is the first to explore such a motion energy model for the motion segmentation task, especially with random dot scenes. They also provide human performance upper bound.

Strengths

- Good contribution and novelty to study the generalization of motion segmentation methods to random dot kinematograms. - Good experimental analysis with supportive psychophysical experiments on humans serving as an upper bound - Comprehensive analysis of 28 Optical Flow models

Weaknesses

- Missing quite relevant related work that actually utilized similar models that are not learnable but based on classical theory on these spatiotemporal oriented energy models and was evaluated on dynamic texture recognition w.r.t SOA deep learning based ones Hadji, Isma, and Richard P. Wildes. "A spatiotemporal oriented energy network for dynamic texture recognition." Proceedings of the IEEE international conference on computer vision. 2017. - Weak motion segmentation training data. I am wondering how the same study will conclude if the training data used included diversified and large-scale video object segmentation or motion segmentation datasets, e.g.: 1. DAVIS: Pont-Tuset, Jordi, et al. "The 2017 davis challenge on video object segmentation." arXiv preprint arXiv:1704.00675 (2017). 2. YouTube-VOS: Xu, Ning, et al. "Youtube-vos: A large-scale video object segmentation benchmark." arXiv preprint arXiv:1809.03327 (2018). 3. TAO-VOS: Voigtlaender, Paul, et al. "Reducing the annotation effort for video object segmentation datasets." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021. While they aren't motion segmentation datasets per se, i.e. they focus on the most appearance and motion salient objects (VOS) they still can be used to train such models as random dot kinematograms do exhibit the motion salient objects as well. Nonetheless, still quite interesting research question, contribution, novelty and study so I am still inclined towards an accept.

Questions

How is there proposed model different from SOENet cited above in the weaknesses? beyond the fact that SOENet was evaluated for dynamic texture recognition not motion segmentation.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Mainly the used training data might not be the best to conduct such study, what if their generated dataset played a role in how optical flow models generalized to such random dot scenes. A second evaluation training on large-scale VOS benchmark such as YouTube-VOS+DAVIS can help ensure the consistency of the conclusions.

Summary

The manuscript investigates whether artificial neural networks perceive moving random dots similar to humans. To this end, they propose a foreground-background segmentation framework to objectively measure several models. Two groups of models are explored: (1) driven by optical flow that finds matching points between two frames, and (2) driven by motion energy inspired by known mechanisms of the biological brain. The conclusion is that the neuroscience-inspired motion energy model matches better human perception.

Strengths

1. Evaluation of artificial neural networks with random dot stimuli offers a more controlled environment to compare networks to human psychophysics. 2. A simple energy model, inspired by visual area MT in the human cortex, can open a new line of investigation to model human motion perception.

Weaknesses

1. The comparison of accuracy between networks and humans is ambiguous. The performance difference between the human participants and (a) the best optical flow and (b) the motion energy model, is roughly the same but in different directions. This might suggest that both models that equally far away from human responses (the difference is that motion energy is better than humans while FlowNet2 is worse). Nevertheless, based on these results one cannot conclude that the motion energy model better captures human responses. 2. It is unclear why the online experiment has resulted in a "suspiciously low" quality outcome. Does this indicate large individual differences or suggest the experiment is demanding and can result in different interpretations by participants? If such concerns or similar arguments are valid, one must question their validity to test networks as well. 3. I would have liked to see more results from the Motion Energy model perhaps with a different set of conclusions that are illustrated in Figure 2. Perhaps some ablation studies to better investigate the role of area/component (i.e., V1/MT Linear/Blur).

Questions

1. Why does neither of the entries in Table 2 match the Motion Energy results in Table 1? 2. Results of Table 2 suggest that the original weights of the proposed motion energy model from Simoncelli & Heeger [35] result in the best prediction of Random Dots, and any deviation to that deteriorates the performance rather significantly. This is rather puzzling to me. It would be interesting to read the thoughts of the authors on what kind of dataset one can obtain similar weights from Simoncelli & Heeger. Essentially, how has the human brain obtained those parameters in the first place?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations of the current manuscript are well explained.

Summary

The authors attempt to use a classical evaluation of human gestalt perception, namely shape identification in moving random dot stimuli, to evaluate both a traditional model of motion perception (motion-energy model from Simoncelli and Heeger) against state-of-the-art deep learning-based optical flow methods. They find that while SoTA methods outperform the MT motion-energy model on natural video optical flow estimation, the motion-energy model in fact outerperforms these more complex methods in correctly estimating motion patterns in random dot stimuli. The poor estimation of motion in random-dot stimuli thus correlates with poor figure-ground segmentation using the SoTA deep network approaches. Due to the fact that human performance on random-dot stimuli is far greater, this suggests that curren optical flow models fall short of human performance and might be computing motion information in a way that is misaligned with human vision. On the other hand, the fact that the motion-energy model can fill this gap suggests that this more biologically-inspired model (that fits responses of MT cells in visual cortex) is sufficient to connect cortical motion processing with some aspects of human Gestalt perception.

Strengths

There are many strengths of this work and I see it as a careful and nice study of where current optical-flow methods stand relative to much simpler interpretable methods for estimating motion. The additional comparisons with human segmentation performance given random-dot stimuli are compelling and overall I think this paper provides a valuable set of experiments that can guide future work in this interdisciplinary area of human vision science and computer vision. To be more specific, I think the work is presented quite clearly and written well. I also think that the problem framing is clear and the authors successfully verify many of their hypotheses. I encourage the authors to release all of their code and data as I think a public pytorch version of the motion energy model and this human data could be valuable to the field, and could encourage the deep learning community to consider testing motion-estimation setups with the motion energy model. Finally, the idea of closing the loop between motion estimation and figure-ground segmentation provides a clean test of how well models can perform in comparison to humans on this core aspect of human perception. The experiments seem relatively complete (see below for some concerns), and I find the conclusions to be relatively well supported. Overall I see this work as quite similar to the Dapello et al. 2020 paper on simulating a V1 front-end for spatial visual models, and this work provides a complement to this in the motion processing domain.

Weaknesses

I have a few concerns regarding the experiments and some comparisons to prior work: 1) It seems that Yang et al. 2023 is a very related work that has not been discussed in the paper. Specifically Yang explore multiple models of optical flow and compare these to human performance in motion flow perception including natural scenes and random-dot kinematograms. While the conclusions and framing are quite different, I think this work deserves much more careful comparison. For example, in this work, the authors find that SoTA methods (RAFT etc) do in fact correlated better with human perception than local-pooling based methods (spatial and spatio-temporal pooling). I believe the ablation of these pooling methods would be useful in this work as it could provide additional bio-inspired baselines to compare the motion-energy model to. In this vain, I also believe more work can be done to ablate aspects of the motion-energy model (as in Section 4.2) to understand what is the key component to this improved generalization. I think isolating exactly what features of the motion-energy model lead to this generalization could enhance the work greatly. Yang, Yung-Hao, et al. "Psychophysical measurement of perceived motion flow of naturalistic scenes." Iscience 26.12 (2023). 2) While RAFT and the other methods used are strong optical flow estimators, these are still significantly older methods and perhaps comparing to a few more recent methods such as GMFlow, and FlowFormer++ (which outperform RAFT) would increase the impact of the claims Xu, Haofei, et al. "Gmflow: Learning optical flow via global matching." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. Shi, Xiaoyu, et al. "Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023. 3) I think one of the main weaknesses with the work overall is that the main results seem to suggest that the motion-energy model is much better at generalization, but significantly underperforms on the natural videos (albeit this can be improved with finetuning). I think there should be some more extended analysis of study of why this is and some more concrete proposals of how this difference can be fixed. For example, is the issue simply that the SoTA methods are very domain sensitive? For example, could you somehow finetune those methods with random-dot videos and fix the generalization gap? An analysis; however preliminary, about this would greatly increase the impact to the computer vision field in my opinion. 4) While Section 4.2 ablating finetuning aspects of the motion-energy model is quite interesting and potentially useful for the community, I believe the results are actually a bit concerning. They seem to suggest that finetuning the model does in fact overfit to the training data and then causes a large decrease in generalization to the random dot task. The authors say it is surprising that the motion-energy model (without any tuning) performs so well on the random-dot task, but I see this as perhaps a byproduct of the fact that most research on MT had been done using simple stimuli like random-dot motion and so the parameters may have initially been set to fit this data well. If this is the case, this potentially limits the scope of application of this method from the computer vision perspective as it seems like even using the motion-energy model cannot really create a "generalist model" that performs very well on multiple domains. Please correct me if I am mis-understanding, but I think the authors should comment more generally on the practical steps going forward on how this can inform creation of more human-aligned, better optical flow methods, rather than simply using this as a study to point out one flaw with current methods. At the end of the day, detecting motion in random-dot kinematograms is a great tool for understanding human and machine vision, but from a practical perspective most applications will be to natural videos or many domains. If this is the case, then there is little reason to use the motion-energy model unless you can show that there is a tangible way to use these results to improve the generality of current methods. If the above concerns are addressed I am definitely willing to increase my score.

Questions

See above weaknesses. In addition: 1) clearly based on the results in Figure 6 (appendix) random-dot shape matching is in fact significantly better for the motion-energy model than humans (while FlowNet etc. underperform). This seems to suggest that there is still a misalignment between even the motion-energy model and human perception. Do you have any thoughts on what this missing piece is and how we might be able to arrive at a better model of human perception using the motion-energy model? Perhaps because the motion-energy model doesn't really model appearance at all there is something missing? From the perspective of human vision science, I think it would be helpful to add more text discussing how this work can inform our understanding of human figure-ground segmentation and where the additional gaps may be. 2) Recent work (see below) has shown that there is a perception of opposite-direction motion in random dot kinematograms that can be verified in human vision and also seen in an application of the motion-energy model. This brings up the larger question that perhaps aligning with aspects of human vision may not be totally beneficial to computer vision systems that detect optical flow etc. Could you comment on this? Bae, Gi-Yeul, and Steven J. Luck. "Perception of opposite-direction motion in random dot kinematograms." Visual cognition 30.4 (2022): 289-303.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors do address many limitations and in the text. Other than the weakenesses/questions above, I think the authors do a reasonable job of this.

Summary

The paper measures the zero-shot generalisation of motion energy (as described and estimated by Simoncelli & Heeger (1998)) when applied to the moving object segmentation task on the random-dot moving patterns. The comparison is performed with optical flow representation. Given a synthetic dataset of single-moving object scenes, either motion energy or optical flow is estimated using the Simoncelli & Heeger method or of-the-shelf pretrained network, respectively. This is fed into the downstream segmenter, which is trained in a supervised manner. This is then applied to moving random dot images. While optical flow achieves better performance in the simulated 3D object videos, the performance degrades drastically on moving dots. Motion energy modality performs worse than many recent optical flow estimators in the original setting but retains much of the performance when transferred to moving dots.

Strengths

1) A multitude of optical flow estimators are tested, offering a wide coverage and interesting results that older, arguably, less data-driven approaches retained some performance on moving dots.

Weaknesses

#### Presentation 1) The paper lacks a description that explains precisely what motion energy is (what form it takes and what the inputs are to estimate it). Similarly, optical flow is not described either. This is problematic as a reader unfamiliar with either of these two concepts will have a hard time making sense of the study presented. #### Soundness 2) It is also not clear if the presented comparison is meaningful and sound. In the discussion (L290-292), the paper states that optical flow might not be optimal for humans, and that the truthfulness of flow is not critical for the task of segmentation. Why compare to it? 3) Similarly, as stated in the L302, motion energy is estimated from far more frames than flow. For moving dot patterns, the "true" flow is non-zero only for dots that have moved. This paper does not establish that this is enough to segment the shape. This is even before considering pretrained flow estimators operating outside the learned distribution. Similarly, the statement on L64 seems to misleading: optical flow and/or motion energy _are_ the motion information. It is just that optical flow does not work on moving dots whereas motion energy seems to still extract signal (perhaps due to many-frame observation). 4) Does running optical flow on dense moving dot patterns even make sense? It is worth considering whether obtaining "true" optical flow between two frames is ill posed in this situation. Over a larger number of frames, it perhaps is the simplest explanation of the motion consistent with the overall trend. Given just two frames with indistinguishable dots, there could be many possible "simple" motions that rearrange the dots. It is perhaps too much to expect that the appropriate optical flow could be estimated with only two frames, and that the estimate will have enough information to describe the shape (which is essentially the overall trend of motion in these simple cases). 5) There seems to be a premise in the paper that generalising to the moving dot patterns is desirable and important. This is not motivated. Is it even true? One would usually expect to apply computer vision systems more to the real-world setting than to moving dot setting. Based on the results in Table 1, recent optical flow estimators are preferable in this situation. Based on further study in Table 2, optimising the motion energy model weights to perform better in a more "real" setting reduces performance on moving dots. Why is the generalisation to the moving dot settings relevant? #### Limited Contribution 7) While the presented study is novel, its usefulness is not argued or clear. It is not obvious how the findings could be used, which essentially are the measurements performed. The motion energy model is taken from prior works, does not improve with training in moving dot patterns and under-performs in the real setting. Overall, the advancement of the knowledge presented in the paper is minimal.

Questions

The rebuttal could concentrate on answering the critical question presented above in the weaknesses section. #### Additional Remarks - The size of the datasets and evaluation protocol for "Original" and "Moving dot" settings are not described. As the datasets are new, details are required to reproduce and verify the results. - How are the inputs to the segmentation network processed? Are the motion energy/optical flow normalised and/or transformed in some way? - On L78, it is said that the study of moving object segmentation is conducted, while the focus of the study is really the moving dot segmentation setting. - The presented motion energy model outperforms all human subjects in the study. While possible, this is quite strange and could indicate that the small sample size and/or task set up are inappropriate.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

1

Limitations

The key limitations are addressed.

Response to rebuttal

I thank the authors for their very clear and careful rebuttal. I believe the authors have addressed almost all of my concerns. If the authors will follow-through on the changes they say they will make in the camera-ready version then I think this will greatly strengthen the work. My only remaining concern is that I still think there is an over emphasis on human zero-shot generalization to random dot tasks. I think it is very hard to ever claim "zero-shot" generalization for humans because human experience is so diverse. Perhaps if you had infant or children psychophysics experiments this may be closer to a "zero-shot" experiment, but even then, there is work showing that neural selectivities in early vision may be driven for example by retinal waves even before birth. Could these kinds of retinal waves simulate something that is more akin to random dots? In this vain, I just think it would be useful to show what level of finetuning it might take for a SoTA model to generalize well to the random dot task. However, I don't believe this to be a blocker for the work as it still provides a large set of good experiments and contributions that I think will be valuable to the field. I am raising my score to a 7 as a result.