Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception

Summary

This paper uses AI models to investigate how humans can accurately predict actions based solely on minimal action patterns without prior training. The authors propose a new motion recognition model called Motion Perceiver (MP). This new model first extracts patch-level semantic features from images using a DINO-pretrained ViT model, then computes patch-level optical flow from these features, and develops two pathways featuring motion-invariant representations and slot-based representations from these optical flows. They show that this new model yields more human-like generalization performance on recognizing actions from videos of joints and sequential positions than existing AI models. Their new model also shows some human-like behavior patterns when performing action recognition. They further present ablation studies showing which components in this new model are critical.

Strengths

This paper contains solid work on collecting/constructing training and testing datasets, developing a new computational model, training existing baselines on the datasets, comparing the new model to these existing baselines, and showing the influence of different architectural designs in the new model. The authors have also collected human psychophysics data through MTurk on their minimal motion stimuli, which is useful for the community. The new model proposed by the authors outperforms the existing baselines by significant and sometimes large margins. Both the new model and baselines are evaluated on various scenarios to test their generalizability performance. The paper is well written, with a lot of details in the appendix and clear descriptions in the main text.

Weaknesses

1. This paper needs stronger baselines. The authors train their baselines on the small amount of natural RGB videos collected by them. However, humans perceive a lot of videos during their development in the real world. This means a stronger and more human-like baseline is finetuning from a pretrained video model using self-supervised learning algorithms such as VideoMAE, instead of starting from random initializations. Moreover, the fact that their new model (MP) uses a DINO-pretrained visual encoder makes the existing baselines even weaker, since all the baselines are trained from scratch on the small amount of videos. Fixing this would make the comparison much more reasonable and fairer. 2. The authors also need to show how good models directly trained on these joints or sequential points videos can perform on this task. It is important to get a sense about how far away this new model is compared to models directly trained on within-domain videos. This can also help tell whether human is better than these models or not. 3. Suppose this work aims to get a more human-like model through this human-like learning curriculum compared to just training a model on the joint or sequential point videos. In that case, the authors need to show whether their new model is quantitatively more human-like compared to the other models. This can include explaining per-category action-recognition performance or the error patterns when recognizing actions. I appreciate the authors' responses to these points. The first point is well explained. Please also remember to include this training detail of the baseline models in the final version. The additional experiments on the second point are also convincing. I raise my score from 6 to 7.

Questions

See the weakness points.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discussed the limitations of the paper.

Summary

The paper introduces a novel, neuroscience-inspired approach to performing biological motion perception from videos. The videos are part of a dataset that the authors also introduce, depicting 10 different actions. The videos range from fully RGB frames to point-light displays that only cover the joints. While humans have been very successful at solving such tasks, even with limited data, AI models struggle. For this reason, the authors propose an elegant approach that involves obtaining optical flow from feature maps of an ImageNet-pretrained network and creating a system of slots to organize important features for the task. Additionally, they introduce features that account for scale and invariance. Overall, the method shows significant improvement over the tested baselines.

Strengths

The paper introduces a novel dataset designed to advance the field of action recognition in videos. This benchmark dataset challenges most of the baseline methods in the field. Additionally, the paper presents an elegant approach to combining features over time, inspired by loosely but interestingly applied neuroscience insights. This approach appears to effectively motivate a method that performs better under almost all conditions of the proposed dataset.

Weaknesses

My main question/concern is whether the baselines are fairly compared with the proposed method. It might be worth adding learning curves to indicate whether the models have effectively converged. Additionally, including a parameter count to express whether the method is more or less computationally expensive than the baselines would provide useful insight. I might have missed it but it seems that the inspiration to take ViT as a feature extraction is not well motivated. Perhaps worth mentioning if there is some rationale that harmonizes with neuroscience or if its a engineering choice. Another question/concern is that the connection back to neuroscience seems weak. However, there might be a potential opportunity arising from this approach. The ablation studies suggest a certain dependency on time resolution. It might be beneficial to show which frame in the sequence is the most important and how the start, end, or development of the action is crucial for action detection. This could tie back to neuroscience and provide testable hypotheses, thereby strengthening the paper.

Questions

I add them in the weakness section.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Summary

The paper introduces the Motion Perceiver (MP), a novel AI model designed to improve the generalization of action recognition in biological motion perception (BMP) tasks. It leverages patch-level optical flows and introduces flow snapshot neurons that learn and store prototypical motion patterns and motion-invariant neurons that maintain the consistency of motion recognition across temporal sequences. The authors create a comprehensive BMP benchmark dataset with 62,656 video stimuli across 24 BMP conditions and demonstrate that MP surpasses existing models by 29% in top-1 action recognition accuracy. The MP model’s performance closely aligns with human behavior in BMP tasks, providing significant insights into the underlying mechanisms of BMP and paving the way for more generalizable AI models in action recognition.

Strengths

- Originality: The Motion Perceiver (MP) model, which uses patch-level optical flows and novel neuron mechanisms (flow snapshot neurons and motion-invariant neurons), is a creative and original approach to improving action recognition in BMP tasks. The BMP benchmark dataset is also a significant contribution that provides a new standard for evaluating both human and AI performance in BMP tasks. - Quality: The paper demonstrates a well-designed and rigorous methodology, including detailed descriptions of the MP model architecture, the BMP dataset, and the experimental setup. The extensive evaluation of the MP model against existing AI models and its comparison with human performance provide strong evidence supporting the paper’s claims. - Clarity: The paper is well-written and clearly explains the technical details of the MP model and the experimental methodology. Figures and tables are effectively used to illustrate the key concepts and results. The paper also situates its contributions within the broader context of existing research in AI and motion perception, providing a clear understanding of the novelty and significance of the work.

Weaknesses

- Limited Contextualization: Although the paper does a good job explaining the MP model and its benefits, it could benefit from a more detailed comparison with prior work. Specifically, a deeper discussion on how the proposed method improves upon existing models in terms of architectural innovations and performance metrics would provide clearer context. Particularly, lines 56-57 state that “In contrast to many existing pixel-level optical flow models [83, 84, 79, 91, 71], ….”, but some of the references are actually representation-level flow models, e.g., [71]. More comprehensive comparisons to those prior works (theoretically and/or empirically) would make the claim more convincing. - Dataset Limitations and Biases: The paper introduces a comprehensive BMP benchmark dataset, but it would be beneficial to address potential limitations and biases within this dataset. For example, discussing the diversity of the actions, the representation of different demographics, and the potential impact of these factors on the model’s performance would provide a more balanced view of the dataset’s strengths and limitations. Most importantly, the authors designed two scenarios (joint videos and sequential position actor videos), and claim they are good benchmarks for recognizing motion patterns. This claim is not well justified. For example, why not use sketches, optical flows, or other types of motion representation? - Results Not Supportive Enough: In Fig. 4, it seems that the proposed method only outperforms on J-6P and J-5P, and is inferior on other levels. In this case, it is hard to claim that the proposed method has better generalization. While the J-6P and J-5P are most abstract and challenging, there is no evidence that those two levels are the best evaluators for generalization on action recognition. Additionally, the comparison methods are mostly dated; more recent baselines are needed. There is no mention/control on the efficiency and computation cost as well, making the comparisons less meaningful. - Contribution: The biggest weakness is the overall contribution of the paper. While the paper is strong in proposing a new neural architecture and a benchmark, the proposed task itself is not justified with theoretical or practical value. There is no valid proof that joint and SP videos are good representations of biological motion (compared to other motion representations). More specifically, the joints are from the skeleton and keypoints of a human body, which limits its generalization to other objects. Regarding human action recognition itself, it is plausible that having a keypoint modality in the training procedure could potentially solve the problem of distribution shift (RGB to abstract motion representation). While the idea is well-motivated, the approach does not strongly support the claim; the methodology, the proposed benchmark, and empirical results will all benefit a lot from further study.

Questions

Most of my questions are described in Weaknesses. Some additional or specific ones: 1. What's the runtime of the proposed method compared to baselines? 2. Have you tried using the pixel-level optical flow downscaled to patch level and compared that to the patch flow?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes.

Summary

This paper proposes a new biologically inspired architecture for action recognition. The motion perceiver computes a patch-level optical flow from DINO features which is then processed in a two-stream architecture with one pathway using slot-attention to recognize different motion patterns and the other one integration over motion to get a time-independent motion signal. The paper also contributes a new dataset which provides multipe point light versions of the videos together with human performance data. The model is trained on RGB videos and evaluated on the point light videos. The authors show that the model's performance is well aligned with human performances on the different stimulus versions and that it outperforms other ML models on the least-information stimuli.

Strengths

* very interesting study linking neuroscience, psychophysics and deep learning * Interesting modelling architecture with multiple relevant and significant contributions: * patch based optical flow on DINO features is a nice idea and seems to work very well * slot attention on the optical flow data is also a nice idea * motion invariant neurons which integrate over motion patterns * well-designed dataset * convincing and interesting results that are well presented and discussed * interesting ablation study with very good discussion that goes beyond a simple table * very well written paper with clear structure.

Weaknesses

* I think at least one relevant comparison is missing: Illic et al, "Is Appearance Free Action Recognition Possible" (ECCV 2022, https://f-ilic.github.io/AppearanceFreeActionRecognition) asks a very related question and also uses somewhat similar stimuli (white noise stimuli, but also different kinds of dot stimuli) and also introduce a dataset to that end (AFD). Illic et al claim that their twostream architecture can solve the appearance free stimuli and at least on the homepage they also mention dot stimuli close to the ones used in the present paper. However, so far I couldn't find any dot stimuli results in the actual paper. Beyond this paper, I think there are more two-stream architecture models that might be relevant to include. This is overall the main reason that kept me from increasing the rating of the paper. * the claim "Our model demonstrates human-like robustness to minimal visual information" (Figure 4) seems to strong for me. For most of the stimulus types, MP is not even the best competitor, only for J-6P and J5-P it outperforms all other models. But even then the performance drop compared to RGB seems to be 3 times more than for humans. The same holds for l282 "There is a slight decrease in accuracy when transitioning from RGB to J-6P for both humans and MP", where humans seem to be around 90% and MP around 70%. Interestingly, Figure 4 suggests that most of the performance drop comes from going from RGB to the point light display, because the drop happens already at J-26P, after which the performans of MP mostly stays constant. * I'm missing more discussion of when and where "classical AI models fail or succeed and the alignment with humans". I think there is a bit to be learned from Appendix F. Especially, I would find it interesting to compute corellations with human scores across the different stimuli. Figure 3 suggests that MP is well aligned with human BMP. But I would like to know if it is also more aligned than other models (which I think would be a very strong additional result). * notation sometimes a bit convoluted (M, \hat M, \tilde M, ...). Maybe sometimes it's worth using more descriptive names like "M_\text{invariant}" **Update**: After the rebuttal, I increased my review from 7 to 8.

Questions

* How many trainable parameters has the final model, and how are they distributed over the model parts? * given that the DINO based patch-level optical flow seems to be a major pillar of the model, I would love to get a better idea of how it performs on some example videos or images. It could be nice to have some vector field plots in the Appendix or a video in the Supplementary Material * Caption Figure 1: "AI models are trained to learn to recognize actions": Is this on purpose? I would say they are either trained to recognize actions or they learn to recognize action. Being trained to learn to recognize would imply some meta learning. * l98 "Considering Ft as a 2D grid of patches in N = H × W where H and W are the height and width of Ft " this sentence seems to be missing a subject * l127: "This dense optical flow estimation approach at the patch level captures a richer set of motion dynamics, providing a more comprehensive analysis of movements throughout the video" I find it interesting that the patch-level optical flow is defined relative to a fixed reference frame instead of, e.g. always the first frame and then actually uses all frames as reference. Did the authors check how much does this help compared to only using e.g. the first frame? Right now this sentence doesn't seem to have any supporting evidence * l158: " Every patch-level optical flow in Ô can be projected into four orthogonal motion components along +x, −x, +y and −y axes": I think I'm missing something here, but how are motions in +x and -x orthogonal? * l163 "we obtain motion invariant matrix": "we obtain THE motion invariant matrix"? * l270 " Interestingly, shuffling has a lesser impact on human performance compared to reversing RGB videos (RGB-R versus RGB-S)" technically, the two performances might be identical within the margin of error if I'm not mistaken. * Figure 4: given the size of the error bars in Figure 3, I think it would be nice to also have error bars in this figure. Since it would make the figure harder to read, I think it would be enough to have a larger version in the appendix with error bars that can be referred here. * l322: I agree that ablations A1 and A2 show that both pathways are important, but they seem to be hardly of similar importance. Removing the motion invariant pathway results in a drop of performance, while removing the flow snapshot neuron pathway essentially results in model performance collapsing. So I would say that the flow snapshot neuron pathway is the crucial part, and invariance helps further. * l329 "In A4, the model takes patch-level optical flows as inputs and directly utilizes them for feature fusion and action classification without flow snapshot neurons" Unless I'm missreading the table, A4 has flow shapshot neurons? * l349 "Psychophysics experiments on this dataset were conducted, providing human behavioral data as an upper bound": Why upper bound? I don't see why models couldn't outperform humans in principle. * references to the appendix could be a bit more clear, e.g. "see Appendix, Sec B" instead of "see Sec B"

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

Limitations are well discussed in the main paper. Societal impact is not discussed because "There are no societal impacts, to the best of our knowledge" (l980), with which I would slightly disagree. There is always potential societal impact from building more human like models (good as well as bad). But this is mostly a nitpicking point that doesn't affect my review of the paper.