A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains

Summary

This paper analyzes which ANN structures are more suitable for conversion to SNN. Based on the analysis results, the RepConv model is chosen as the baseline ANN model. The ANN-to-SNN conversion is performed using Value-range (VR) encoding and Clamp and Quantization (CQ) training, resulting in a high-performance and low-latency SNN network.

Strengths

1. The neural network model structure and training approach are novel. Compared to previous dual-stream models, this work trains the two streams separately using different tasks and combines their training by incorporating changes in attention, enabling simultaneous training. 2. The network successfully models the functional specialization of the ventral and dorsal visual pathways and demonstrates that the two streams correspond to different visual pathways. Additionally, it exhibits comparable predictive performance to other single-pathway neural networks. 3. The ablation experiments are comprehensive and provide insights into the importance of learning objectives and retinal input in the functional specialization of the human visual pathways. 4. The content of the paper is substantial, the experimental results are solid, and the writing is good.

Weaknesses

1. Some details need to be addressed. What datasets were used for training the two streams synchronously? Why is the learning rate lower when training the two streams together compared to training them separately? 2. Although the network has a recurrent structure and multiple time steps, the training dataset consists of static data. However, fMRI visual input is dynamic. Could dynamic datasets be considered for training the network? 3. Single-pathway networks with a single training objective (object recognition) such as ResNet18 and ResNet34 also exhibit good predictions of fMRI data for both visual pathways. However, intuitively, such models should not be able to predict the dorsal visual pathway well. Does this result indicate that there is no significant difference in fMRI data features between the ventral and dorsal pathways? 4. When using network features to predict fMRI data, is it possible to predict separately for each layer to examine whether each layer corresponds to different regions of the two pathways?

Questions

See the weaknesses

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Some experimental details and explanations are needed.

Summary

The main contributions is a dual stream neural network to model human visual processing. The network is composed of two CNNs (where and what) to model the dorsal and ventral pathways. The authors demonstrate that they can predict human gaze behavior and neural responses using their model, and gain new insights in to the role of the dorsal stream and how it interacts with the ventral stream.

Strengths

A dual stream CNN to model the dorsal and ventral stream is novel as far as I know and quite interesting. The authors do a good job of the motivating their work and being clear about their contributions. As they note, many approaches to comparing human and deep neural networks stop at classification CNNs, so moving beyond simple object recognition and separating out different streams of processing is an interesting contribution. In addition, the experiments appear to be well reasoned. The authors create a model motivated by properties of the human brain, and provide logical comparisons points --the authors provide both an eye-gaze and fMRI analysis to compare their model to humans and also provide a baseline single stream CNN to make it clear what benefits arise from the dual stream design. Overall, the work is very clear.

Weaknesses

The work does not have enough discussion of how their dual stream model could be useful to a more broad computer vision audience. Although the authors show their dual stream model does better than single stream at explaining brain response, the work would be strengthened by more clearly demonstrating the use for a dual stream system directly on a computer vision tasks (i.e. does object recognition or detection improve? or robustness?). The authors only consider CNNs and supervised training in their modeling. While these models are often used when comparing humans and deep neural networks, they are not top performers in the context of computer vision. Even in the context of biological modeling, more specialized models (for example, adversarially trained models) are thought to share more similarities with human visual representations than standard CNNs.

Questions

As I mentioned in weakness, could the authors touch on how their dual stream model is relevant a wider computer vision audience? What are potential performance benefits of the dual stream approach and/or some the gaze mechanisms you incorporated (better accuracy? robustness? efficiency?). Numerical results would be appreciated, but general statements/discussions are also fine. What is the size of your dual stream model? How costly is it to train? Could you point me to performance scores? I would like to know more of these details about the model. On prior work, could the authors touch on how their work relates to metrics like BrainScore (Schrimpf et al. 2018)? Also, how does your model compare to GistNet (Wu et al. 2018) which I believe takes a two pathway approach.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, the authors adequately address limitations. The authors note that their model does not address temporal sampling and does not completely capture how the dorsal and ventral streams interact with one another. I might argue that relying on small CNNs for their model and relying on tasks like recognition and detection to learn visual representation could be limiting, but I discuss these concerns in weaknesses and questions and find the work good otherwise.

Summary

This paper focuses on investigating differences between the human ventral and dorsal visual streams. The authors develop a dual-stream model that has different retinal inputs to support “what” processing (object recognition) or “where” processing (guiding the fixations). The authors then test how well the model activations are able to predict human fMRI activity and see a distinction where the “What” model better predicts ventral visual areas and the “Where” model better predicts dorsal areas.

Strengths

* Although there is a large amount of work studying the similarity of image-computable models and the ventral visual stream, there is less work done studying the dorsal stream. This paper provides an interesting analysis of the functional separation of these two pathways. * The retinal framework and its relation to biology is original and significatn, as well as the model leading to different “fixations”. * The dual-task model presented is original, and the experiments on different versions of the model provide interesting interpretation of the training components.

Weaknesses

* In the abstract and elsewhere (ie line 90 and line 248-250), the authors state that “the functional segregation of the brain’s dorsal and ventral visual pathways is driven primarily by their different learning objectives”. This is too strong of a claim and not supported by the results (a correlation $\neq$ causation fallacy), as there is no evidence presented in the paper showing that the brain itself relies on these specific learning objectives. * The “WhereCNN” is initially trained on Image Recognition and then fine-tuned on saliency maps. This makes the differences between the two models difficult to interpret, as it is *also* optimized for object recognition. * The authors detail the interesting addition of foveated vision and retinal transformations, however from the results it appears that this is less necessary for performance than the two different tasks, but the relevance of this to our understanding of biology is not discussed. It is hard to say whether this lack of a result is due to the retinal sampling being incorrect, or if it simply doesn’t matter for this level of analysis.

Questions

a) In Figure 3b, it looks like there are many ventral voxels that are predicted better with the “Where” CNN. In contrast, there are few dorsal voxels that are predicted well by the “What” CNN. Is there an interpretation for this? And are voxels from early visual areas included in this plot (if so, what color are they?) b) The colorbar on Figure 3a is misleading because there is little difference between the darkest blue and the darkest red. A diverging color map would better capture the data. c) There are control experiments presented with alternative models, but from what I can tell these alternative models always have “dual streams”. Is it necessary to have the two streams to get good predictivity for these voxels or could it be captured with a single model performing two tasks (especially given that the inputs seem to matter less than the task)? d) A fourth model with different inputs but predicting “perception” may show a different trend, and should possibly be included for completeness. Similarly a model with only wide inputs to each of the architectures. e) Additional details of training, or a direct reference to the supplement where these details are provided, would be helpful. Specifically, on line 165 it is stated that the WhereCNN “learns spatial attention from humans”. Is this separate training than the SALICON dataset? And the training dataset, updates/labels etc are not given for Stage 3 of training. f) Quantification of the similarities between human and model attention/saccadic eye movements would be nice. Also, are the images in Figure 2 from the train or test dataset? g) Are the human eye movements during the fMRI activity recorded and available? If so, how well do these match up with those predicted by the model? (Maybe beyond the scope but an interesting analysis). Minor: i) Section 4.1 seems less of a result and more like validation that the training worked, as these validations are what the networks were trained to do. ii) Lines 99-100 are technically true (in that the models do not fully replicate the way human eyes sample visual inputs) however neither does this study (it is still a model and does not capture all aspects of the eye). It would be better to explicitly state the differences between the previous studies and the model used here. iii) Clarifying Figure 4 and its caption to make it clear that the right most model is the standard model used in the rest of the paper would be helpful. iv) On line 265 there is a comment about WhatCNN and WhereCNN performing comparably to deeper models, but I don’t understand why depth matters in this context?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are addressed in the final paragraph of the discussion.

Summary

The author introduces a dual-path parallel neural network designed to simulate the hypothesized functions of the ventral and dorsal streams in the human visual system. This network consists of a WhereCNN, which has a wide view to learn spatial attention and direct gaze, and a WhatCNN, which has a local view to learn object recognition. The network is trained using two learning objectives: the WhereCNN is trained with human gaze attention data, while the WhatCNN is trained for object recognition based on local views. The study reveals that the WhereCNN demonstrates better prediction of the dorsal stream voxels, while the WhatCNN performs better in predicting the ventral stream voxels.

Strengths

The alignment of the WhereCNN and WhatCNN with the dorsal and ventral streams provides promising evidence supporting their assumed functions.

Weaknesses

Although the individual networks (WhereCNN and WhatCNN) have been previously proposed and explored for the dorsal and ventral streams, this study is the first to combine them. The paper's assumption of the dorsal stream function may be narrow, as the dorsal stream is known to process spatial information, motion, and action rather than solely focusing on spatial attention and eye fixation decisions. Given the specific training methodology used for the WhereCNN and WhatCNN, it is not particularly surprising that they show better prediction results for the dorsal and ventral streams, as suggested by earlier studies. Therefore, the results, while valuable in establishing them within a coherent and integrated system, may not be particularly surprising.

Questions

What new insights that this work provides for our understanding of the visual system? The ideas are not exactly novel, though it is nicely put together, and the alignment with fMRI data is encouraging. At the end of the day, what does this work teach us about the visual system that we don't already know -- good is supposed to illuminate and elucidate something we don't fully understand.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper discusses the limitations of the model and highlights areas for future research.

Response to author rebuttal

Thank you for addressing my concerns and clarifying various aspects of the paper. I think that the new panels of Figure 4 are a nice addition to the paper, and really seem to demonstrate that for this dataset that main thing that matters is the task and not the input representation. I've updated my score accordingly. Two minor points below (neither are influencing my score, but just wanted to respond to these for your consideration in future drafts). Re: Colormap While I appreciate the attempt to change the color map, there are issues with the non-uniformity and lack of color-blind accessibility. You might want to check out the "uniform diverging pallets" here https://seaborn.pydata.org/tutorial/color_palettes.html and overall discussion on the page about choosing color maps. See also https://colororacle.org/ for a plugin to test whether the map is color-blind friendly. Re: Depth The authors may want to look at more recent work along these lines stating that there is not a direct correlation of neural predictivity with architecture depth (for instance https://www.biorxiv.org/content/10.1101/408385v1.full.pdf). This is where my question came from, as I don't know if there is something specific for dorsal predictions where it is still believed that depth = better predictivity.

Thanks for the responses

Thank you for the articulated answers and the citations. I do recall that there is literature, for example Gustav Deco's work, arguing dorsal stream is for mediating spatial attention and ventral stream is for mediating object attention. It is still quite a surprise conjecture that dorsal stream but not ventral stream is involved in the controlled of the eye movement, as both streams feedback to V1 as well as to SC for directing eye movements.