Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes

Summary

It has been hypothesized that the brain builds an internal model of its environment, and uses this model to make inferences and plan actions. This works aims to understand the neural mechanisms that are the basis of such computations. To this end, the authors construct several classes of artificial neural network models, each with different inductive biases, and evaluate them on future state prediction tasks in ethologically relevant environments. They evaluate each of these models in two ways: - comparison to human behavior in an object contact prediction task - comparison to neurophysiological data from macaques playing mental-pong, a ball interception task in which the ball is partially occluded From these evaluations, they identify that dynamics modules that predict the future state in the latent space of video foundation models reasonably matches both human behavioral patterns and neural responses across diverse environments.

Strengths

Understanding the neural basis of mental simulation is an important problem both in neuroscience and AI. The approach taken here is to construct several classes of artificial networks and evaluating them on tasks that require prediction of the future state of the environment. This allows the authors to probe which inductive biases are crucial for mental simulation. The presentation is clear as well.

Weaknesses

Not really a weakness, but details about the best fitting model layers would be useful to add. It would also be interesting to see some analysis comparing the best fitting latent layers across different models. minor: - line 17, 18 in the abstract talks about the mental pong task without introducing it

Questions

- The R3M and VC-1 video foundation models (+ dynamics) are the best at neural response predictivity. However, one is a ResNet and the latter is a transformer based model. Also the training objectives are different for both models. Why is the performance of these two models similar? Is it the scale and/or type of pre-training dataset? - For the object-centric models did you try using recurrent networks for the dynamics module? - Was any analysis performed to compare the best-fitting latent representations across models?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are addressed in the appendix.

Summary

The authors compare "foundation models" of vision for mental simulation. They consider several large models including models trained on static scenes and dynamic scenes. It was found that the models optimized with self-supervision on dynamic scenes yielded the best neural predictivity.

Strengths

The work is interesting since mental simulation is relatively understudied but important for embodied models. The paper is clearly written. * The work is thorough in testing out a lot of models and model families. * Results on neural predictivity is strong for the video foundation models.

Weaknesses

* Since all the video foundation models are trained with Ego4D and the image foundation models are trained with ImageNet, it is not straightforward to disentangle the effect of dataset vs role of dynamics itself. * The models only differ in terms of neural predictivity while being similar to each other in accuracy and correlation to human responses. Previous work (Rajalingham et al. [2022b]) seem to support that model behavior should be measured in terms of correlation to human responses to disambiguate between models - with a high correlation suggesting that the model is using the same strategy as humans. In light of this, the results presented seem weaker than as claimed. * I do not see the contributions of the paper beyond "large models trained with dynamic scenes data are better at neural predictivity for a dynamic task" which is not surprising.

Questions

*Fixed encoder that is not trained during "dynamics" training seems too constraining (also unlike human brains). This might have unequal effects on the different models. For example, it might have hurt imagenet models more than others etc. Have you tried versions where you did update the encoders too? *There seem to be some models (Fig. 3 D) that are very good in accuracy but not so good in neural predictivity. Do you know why this is happening?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

NA

Summary

The paper compares a rather large variety of DL models that are able to “future predict” environmental states, including pixel-based deep networks, compositional approaches (e.g., slot-wise processing objects), as well as image and video foundation models. In the latter case, the latent space of the foundation model is used to future predict. The models are then evaluated by comparing their prediction accuracy in an object contact prediction (OCP) task with human performance and in a mental pong task with macaque neuro-biological data.

Strengths

The paper offers a huge data study that reveals model differences when predicting human behavioral performance and macaque neural dynamics data. At the moment, video-based foundation models yield best (but not very good and not much better compared to other models) performance. Generally, the generated data is interesting and should not be lost.

Weaknesses

On the negative side, the paper focuses on only two tasks, which are not very well-motivated. I have found neither a clear motivation to choose these tasks nor a clear hypothesis what the authors would expect. Further, the results that have been achieved are very far from human-to-human consistency / human accuracy and also very far from inter-animal consistency. This implies to me that the results and the performance differences between the models are not really due to actual systematic differences between the models and the model relations to how humans process the data; rather, they may be due to more fundamental design choices. Moreover, particularly in the OCP task, the model performance differences nearly seem random. Surprising to me is the fact that object-centric models yield worst performance (but all are bad), although the task itself is suitable for an object-centric setup. Maybe the authors did not tap onto the appropriate latent state representation – or it this the case because they do not allow to actually simulate dynamics with the object-centric setup? This concern goes hand-in-hand with the fact that the authors do not report an oracle-like performance in the OCP task (as they do in the mental-Pong dataset). From the CogSci work of Tenenbaum, Goodman and others it is well-known that humans do use an internal simulator to approximately solve such tasks… none of the systems the authors consider are able to actually run an internal simulation of the potentially unfolding dynamics. Rather, the authors seem to tap directly into the static latent state of the respective models. In the mental-pong task, the authors do provide an oracle comparison, which indeed yields the best neural predictivity performance, followed by VC-1 and R3M, both of which can learn to predict ball position and velocity because they learn to predict video dynamics where entities, for example, fall, fly, and roll around. Thus, I do not really see any insight created by these comparisons, except for that a model is needed that predicts ball positions and velocities… in the end, the oracle should be the baseline, which needs to be beaten by a neural network to come closer to understanding how our brain / macaque brains solve this task. Seeing this huge study, I am also quite concerned about the compute time (energy) invested here. The authors do not provide enough information about training etc. Thus, it is impossible to even estimate this investment. The final discussion stays rather superficial on the actual insights gained (because unfortunately there are no significant insights really). The mere task to match human data / macaque data even better may be interesting for some sub-community along the neuro-DL interface, but I miss both cognitive and structural insights that go beyond the fact that object dynamics (including velocities) as well as object constellations need to be encoded in the considered tasks.

Questions

A few more detailed comments: The abstract starts very superficial – it took me a while to understand what you are actually after. A full re-write seems necessary. In the introduction, the second paragraph emphasizes that “predicting the physical dynamics of environments” is critical… but isn’t this precisely what you emphasize in paragraph 1 of the introduction, that is, that mental models are important? Only mental (intuitive physics-approximating) models allow to predict physical dynamics, isn’t this per definition the case? Over-general implications are drawn in the introduction – the results are not strong enough in my opinion to come to the put-forward conclusions. Moreover, the written implications stay on a rather superficial level, although the restuls are very particular to the considered two tasks and the methods employed. Finally, on the methods side, more details would have been useful to compare sizes and understand the exact training procedure to fit the data.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

not addressed

Summary

This manuscript compared several classes of deep-learning based sensory-cognitive models in their ability to predict human behavior and monkey neural responses in tasks that require reasoning about physical relationships based on visual inputs. They find that the models that match best to **neural data** are the ones trained to predict future states of environments in the latent space of pre-trained foundation models that are themselves optimized for dynamic scenes in self-supervised way. Within these models, the ones that are trained on diverse ranges of tasks are the best. On the other hand, the models that matched **human behavior** best were the models with pixel-wise end-to-end prediction for future scenes trained on the same dataset. Overall, I think this is a timely work to help point out the gap of many self-supervised foundation models from the computation potentially being performed in the brain. It emphasizes the fact that pixel-wise prediction overemphasizes details thus losing the ability of extracting useful representation as animal brains can extract. The relative distance from human and neural data across models is useful for AI researchers to explore better models along the "gradient" in model space.

Strengths

The models being compared are comprehensive and represent the states of the art. The metrics being used are reasonable (although with limitation, see my comment for weakness). I appreciate that the authors used correlation of prediction on single stimuli (for human judgement). The environments that the models are evaluated on focus on physical understanding, which is an important gap for the current AI (e.g., chatGPT and many models lack intuitive physics) The datasets used for pretraining foundation models also come from a diverse range, from ImageNet to Ego4D and kinetics, especially Ego4D is an ecologically relevant dataset of egocentric videos.

Weaknesses

After reading, although I have obtained a good understanding of the relative performance of the models in predicting physical contact and a somewhat good picture of the relative similarity to human behavioral judgement or monkey neural data in another game (mental-pong), I still lack a good understanding of where the model perform worse and how they are different from the brain. Sure we see the accuracy is lower as in Fig 2A and they are not correlated well enough to human judgement (Fig 2B, sorry that I am describing it as a half-empty bottle), but readers like me are interested in, at least for the best model, how they are worse than humans: is there any pattern among the stimuli that the models turn to make a wrong judgment? Maybe this question is hard to answer as we may lack a good representational space to start with in which we can say the models perform poorly in certain regions, but maybe some examples to put in supplementary material, or some subjective summary from the authors' observation can be helpful, or a more fine-grained comparison between different scenarios (are some scenarios easier for networks? do networks perform more similarly to the brain in some scenarios). Fig2C may provides some hint that a model with better accuracy to start with is more correlated with human, but it still does not answer anything about the pattern of response of a particular model. The analogy of the desire for this aspect of analysis is what psychophysics has done with visual illusions: by analyzing what stimuli cause illusions (a mistake made by the brain) and what models reproduce illusions, we could come up with hypotheses of the computational principle of the brain. I think some of the metrics being used might potentially mask certain difference between model and human/neural data. For example, the fine-grain comparison of predicted hit-probability in the Physion Object Contact Prediction (OCP) task is based on Pearson correlation. But correlation may not capture the overall variance of prediction. The predicted probabilities across stimuli by a model may exhibit a much bigger or much smaller variance than those of humans while showing the same level of correlation. Models and humans may also exhibit different levels of average probabilities across stimuli which is also not captured by correlation. I think such information as mean variance of prediction may also help understand in what aspects these models are worse than or different from humans. I do have some worry that the two tasks tested on human and monkeys reflect different aspects of reasoning. The mental-pong task as displayed does not require any 3D reasoning, while the Physion task, depending on the particular stimuli, should require 3D reasoning to some degrees. I worry that the different patterns of performance between Figure 2 and 3 may partly result from this dataset difference. On the other hand, there is another confound, which is the end-to-end models are used differently in Figure 2 and 3. They were trained and tested on the same environments (but novel scenarios) for Figure 2 but tested on a different environment for Figure 3 (the authors did comment about this second confound in Lines 302-307). Given the two confounds here, I think the results in Figure 3 are more conclusive than Figure 2 since none of the models being compared were trained on the mental-pong task (the authors did emphasize the result for neural data more than that for the human data in the abstract, which I think is a fair thing to do). But if the authors want to draw some conclusion about model generality (a model trained on a diverse range performs better than those end-to-end models when tested on new tasks), then it seems that an additional comparison in which the end-to-end models are trained on a different dataset than Physion but tested on Physion would be better to be added to Figure 2.

Questions

Maybe I am missing it, but I cannot find whether the fine-grain comparison with human or neural data were performed on held-out data or training data, such as in Fig 2B and 3B,C. Related to this, I infer from Line 304 that SVF and FitVid were fitted to the same "scenario" of Physion as the scenario being tested on, but Line 301 mentioned they were tested on novel scenario. It might help clarify what exactly “scenario” means, especially given that the same word scenario is also overloaded with the meaning of the 8 different scenarios in the Physion environment. I understand that there is an urge and publication pressure to make claims such as xx model matches human/neural data. But to me, Fig 2B and 3BC all suggest a non-neglectable gap between the networks and the brain. It is this gap that is more inspiring for future research to fill. I suggest changing some subtitle such as 6. to be a relative statement (e.g., "provide a better match") instead of a binary statement of "can match".

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes. Limitations are well addressed. I want to suggest one additional limitation: a fair comparison between different model architecture/objectives would be training the end-to-end models also on the same data (Ego4D) as those foundation models before the evaluation. Perhaps one reason this cannot be done is due to the limitation of computational resource, which is not the fault of the authors and should not be considered as a negative factor for the decision of this paper, if this is indeed the reason.

still many open questions - but important things have been clarified - thank you!

From the general rebuttal reply by the authors, I am now much more convinced that this paper actually does have some merit, which I have overlooked in my review. I think what put me off was the huge analysis and the lack of a clear hypothesis. As the authors write in their general rebuttal, I highly recommend expanding on these aspects – particularly in Section 2. Additionally, the abstract and intro can be improved. I recommend getting to the point faster – specify exactly which data you are considering in your modeling work (behavioral and neural data from humans and primates). Then put forward your hypothesis and/or the main conclusions that you draw. Thereby, I am still confused why the slot-based models do not work well at all while you are suggesting that more “factorized representations of object state dynamics” will be important. Shouldn’t slot-based models produce such factorized representations? I also recommend not to emphasize the size of your data or the throughput overly much, but rather the intention you have in using it. In fact, could you explain to me (and probably many readers, I also asked some colleagues) what you actually mean by “dense” neurophysiological data? You highlight it also, for example, in the limitations and the abstract as an important factor to constrain your models. Similar, I am honestly not quite sure what you mean with “high-throughput human behavioral readouts”. Please also note that the need for out-of-distribution structural generalization is well-known in cognitive modeling (e.g., Lewandowsky, S., & Farrell, S. (2011). Computational modeling in cognition: Principles and practice. Sage Publications). I highly recommend discussing the large gap between human-to-human consistency and model-to-human consistency further --- this was one of my main concerns and still makes me hesitate to fully recommend publication. Are the models any good at all? Are model differences in terms of model-to-human correlation actually meaningful? I am still not quite sure. Please excuse that I had overlooked the limitation section in the appendix. I still find it rather short, though. With respect to the rather weak model comparison levels reaches, maybe a comparison to similar modeling approaches – say on the level of primary visual areas – would help? That is, a comparison about the levels such models are reaching and the level that you are reaching (compared to the human-to-human consistency)? The gap could tell a story about how much is missing. Thank you for clarifying the oracle value in the pong task – the ball is not visible behind the occluder – all considered models fail to track the ball, correct? Maybe a reference to the fact that babies are able to track the ball at least with 3 months of age if not earlier would be warranted. --- which again somewhat points out to me that the study is to a certain extent an overkill. Nonetheless, I fully agree to the insights and to the messages that you want to convey, which you have made now much clearer in the rebuttal. Seeing that comparisons like the one presented are en vogue, this work and the insights are timely and will gain recognition (and citations). I thus increase my score to weak accept.

response to rebuttal

Thank you to the authors for these responses. My questions have largely been addressed and I am happy to keep my rating.