Reproducibility of predictive networks for mouse visual cortex

Summary

This work demonstrates that overparameterized neural network models, which have many non-unique solutions, can lead to inconsistencies in representing the mouse visual cortex. It suggests a novel approach called "adaptive regularization," where the regularization parameters in the loss term are learnable rather than fixed. The study also examines other approaches, such as normal regularization and pruning, and finds that these methods are beneficial for improving representation consistency. The proposed method significantly enhances consistency. This contribution is significant to the computational neuroscience community.

Strengths

1. This work addresses a critical problem in computational neuroscience: reproducibility. It proposes solutions to mitigate this issue, which have significant impact on the computational neuroscience community. 2. The authors rigorously define the consistency of computational models and thoroughly examine how regularization strength affects neuronal properties.

Weaknesses

1. The study is limited to only one type of model, the Rotational Equivariance CNN. 2. In the Methods section, the terms "Embedding" and "Mask" are not defined. Moreover, Lp and L1 are described as functions of model parameters, but no parameters are defined in the text. This lack of clarity poses an issue for understanding. For instance, Lp could be interpreted in two different ways: as a function of the core but not the readout, or as a function of both the core and the readout. The same ambiguity applies to L1. 3. There are two types of computational neuroscience models regarding their outputs. The first type is task-optimized models, whose outputs are task-related, such as object class, object representation, or action. These models are trained to perform downstream tasks, such as supervised object classification, reconstruction, or playing games [1,2,3]. The second type is neural response fitting models, whose outputs are predicted neural responses, and they are trained to predict these responses directly. However, the reasons why the authors chose response fitting models over task-optimized models are not mentioned. Reference [1] Performance-optimized hierarchical models predict neural responses in higher visual cortex [2] Unsupervised neural network models of the ventral visual stream [3] Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments

Questions

1. Oother models that, despite lacking rotational equivariance, still demonstrate high capability in predicting neural responses measured by electrodes [1]. Why did the authors choose the Rotational Equivariance CNN? What was the motivation behind this choice? 2. Figure 2A shows the models' performances. However, what about the factorized readout with L1 regularization? Does the predictive performance decrease? 3. Why did the authors choose a log-normal prior for the learnable coefficient? 4. Why did the authors not examine adaptive regularization for factorized readout models? The implementation should be straightforward, similar to what was done for Gaussian. 5. Why does the factorized model have one γ in Table 1? Reference [1] https://www.brain-score.org/vision/

Rating

7

Confidence

3

Soundness

4

Presentation

2

Contribution

4

Limitations

This study investigates the consistency of neuronal properties and the prediction performance of regularized models. However, the models used by the authors are trained to predict neural responses. There is another type of computational model that is trained to perform downstream tasks, with neural-like representations emerging from the training. Future work could consider these latter models.

Summary

The authors present a systematic investigation of the use of deep neural network fits to biological neurons as the basis for neuron cell type classification. The authors explore how various factors such as regularization and model pruning can influence both the predictive model fit and the consistency of the neural clustering.

Strengths

This paper presents a well-motivated, calculated set of experiments for evaluating how well deep neural networks can be used as models of mouse visual cortex. With increasing interest in this area and increasingly bold claims, work like this is a breath of fresh air. The controlled experiments and attention to detail provides compelling evidence that though DNNs may be our best working computational model of visual cortex, the are likely not a global optimum.

Weaknesses

Overall, the biggest weakness I think is some lack of explanation. As someone who is very familiar with work in computational neuroscience but not a neuroscientist myself, there were a few things that felt under-explained and it wasn't clear if it was because I'm not the correct audience or because the authors did not provide enough context in the text. The authors do a good job of providing clear motivation for some of the fundamentals (L24-53) but then leave out context when things get a little more technical specialized (i.e. comparisons between readout mechanisms, significance of biological properties and tuning indices, etc) as a result it feels unclear what the reader is expected to know before reading. - The font in the figures is a little unprofessional. Comic sans? In this economy? - Figure 2G is hard to parse. I recommend splitting up the histograms - The intuitive explanation of the factored vs gaussian vs adaptive readouts is not super clear to me. It might be worthwhile to spend more time on it, especially to motivate your proposed adaptive mechanism. - minor typos ("initialiation" in L183) - L216 is nonsensical. I think some words are out of order - Confidence intervals in Table 1 would be nice to gauge significance. Also, the table format makes it a little hard to visualize. Perhaps a graphical representation would be better.

Questions

- Why is the adaptive model not pruned in the experiments shown in fig 4? I feel like it makes sense to include it in the pruning experiments if possible for consistent comparison across experiments. - I think more explanation of table 1 is warranted. First, what is the ideal case? It's not clear to me that the optimal computational model has neurons with precise tuning curves for these chosen functional biological properties. If this is obvious to the authors, then perhaps there should be explicit motivation in the text. Why these biological properties and not others? Do biological neurons in HVC demonstrate each of these properties robustly (and not others)? Is it fair to expect that neurons in silica should each individually exhibit these properties? Second, the difference in NMAE of tuning indices between the factorized and the gaussian readouts seems very large. Large enough that I would expect more discussion of it in the text. How big of a difference in score is this really, is there something that can put it in context? If the biological property fits are a big deal, then is this a nail in the coffin for gaussian readout? Is gaussian readout a hack for improving overall fit while sacrificing something more important? Or is the functional biological properties just a cool trick that would be nice to have but not essential.

Rating

7

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

Limitations are clear.

Summary

This work studies models trained to predict the responses of neurons in visual cortex. The model has a shared multilayer network core followed by a final layer which maps the core features into individual neuron responses. The key question that this paper asks is, How reproducible are the individual neuron properties and "cell-type" clusterings inferred by such techniques? The main conclusion is that sparsity-inducing regularization is important for finding consistent cell properties. Pruning certain channels in the core is also shown to improve consistency.

Strengths

Consistency and reliability of these methods for understanding visual cortical processing is important for the scientific interpretation of these models. The design of the approach seems sound. A number of metrics are used, both scores as well as response properties used in neuroscience.

Weaknesses

* I found the comic sans font and cartoon diagram style distracting in Figs 1-4. * It isn't clear that much is gained from the t-SNE plots in Fig 2. Some evidence of different clusters, yes, but it's known that these kind of plots can be deceiving. * How do you pick the optimal regularization parameters? By what metric/cross-validation procedure? (Line 242 gives $\gamma=10, \sigma=0.1$) Validation performance is reported but I'm unclear of the splitting procedure and whether there was a train/valid/test 3-way split or not. * Minor points are made in "questions" section

Questions

* Line 35 "well described" -> "well-described" typo * Line 62 "We address the identifiability problem, by proposing" typo, remove comma * Fig 1 caption "sampling receptive field positions form a Gaussian" typo * Is the clustering shown in Fig 1B truly "rotation invariant" or related to cyclic permutations of the vector indices? It is unclear from the text. The figure does not depict a rotation (i.e. action by an orthogonal matrix), since the orbit is elliptical rather than circular and it is not centered at the origin. * Fig 3A has $\sigma=01$ which should be $\sigma=0.1$ * Fig 3B axis "amount of such weights" sounds strange

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors mention that their work is limited to a particular type of core architecture. It's also limited to being applied to just one dataset, whereas open data in other animals (or collected by other research groups, e.g. the Allen Institute's work) are available. I am not suggesting the authors do this in their revision but that they mention it as a limitation.

Summary

The paper "Reproducibility of predictive networks for mouse visual cortex," explores the reproducibility of neuronal embeddings in the mouse visual cortex using deep predictive models. By introducing adaptive regularization and iterative feature pruning, the authors address key issues related to model overparameterization and provide a robust framework for achieving consistent functional representations of neurons. The work lays the groundwork for future research aimed at developing reliable and interpretable models for understanding neuronal function. The primary goal is to investigate the stability and reproducibility of neuronal function embeddings derived from deep predictive models. These models aim to predict neuronal responses to sensory inputs and have been proposed to define functional cell types via unsupervised clustering. The paper addresses the concern that deep models are often highly overparameterized, leading to multiple solutions that can represent the same neuronal function, thereby questioning the reliability of embeddings for downstream analysis. The paper demonstrates that L1 regularization, which was used in early models, is crucial for obtaining structured and consistent neuronal embeddings when newer readout mechanisms are used. A novel adaptive regularization scheme is introduced, which adjusts the strength of regularization for each neuron. This method improves the consistency of neuronal embeddings across different model fits while maintaining predictive performance. The paper proposes an iterative feature pruning strategy to reduce the dimensionality of performance-optimized models by half without losing predictive performance. This pruning improves the consistency of neuronal embeddings concerning clustering neurons. The consistency of neuronal embeddings is evaluated using the Adjusted Rand Index (ARI) of clustering partitions across models, correlations of predicted responses across models, and the consistency of tuning indexes describing known nonlinear functional properties of visual neurons.

Strengths

- Adaptive Regularization Scheme: The introduction of an adaptive regularization method that adjusts the regularization strength per neuron is a significant novelty. This approach helps achieve better consistency in clustering neuronal embeddings without compromising on predictive accuracy. - Iterative Feature Pruning: The feature pruning strategy to address overparameterization in deep models is another novel contribution. By systematically reducing the model’s dimensionality, the authors enhance the robustness and consistency of the neuronal embeddings. - Comprehensive Consistency Evaluation: The paper provides a comprehensive evaluation of model consistency across different dimensions (embedding clustering, predicted responses, and tuning curves). This thorough approach highlights the robustness and reproducibility of the proposed methods. - The paper is well-written with generally sufficient referencing to the previous methods.

Weaknesses

- Bias from Regularization and Pruning: While the adaptive regularization and pruning strategies improve consistency, they may introduce biases that affect the biological validity of the neuronal embeddings. Over-regularization, for instance, can reduce the model’s expressive power and lead to less biologically plausible representations. - The study focuses primarily on models with a rotation-equivariant convolutional neural network (CNN) core. The authors acknowledge that other core architectures, such as regular CNNs or Vision Transformers, were not evaluated. This limitation means the findings may not generalize across different model architectures. - Despite the improvements in clustering consistency, there is a trade-off between consistency and predictive performance. The pruning and regularization strategies, while improving consistency, sometimes result in a drop in predictive performance, which is not ideal. - The introduction of adaptive regularization adds another layer of hyperparameters (e.g., the log-normal hyperprior parameter) that need to be carefully tuned. This increases the complexity of the model training process and may require significant computational resources. - The focus on achieving high consistency in clustering and embeddings may overshadow other important factors, such as the interpretability and biological relevance of the model outputs. Balancing consistency with these factors is crucial for developing useful predictive models in neuroscience.

Questions

- Is ARI a metric you developed or did it exist before? If it existed before, it requires referencing. - Have you considered evaluating other core architectures, such as Vision Transformers or regular CNNs, to see if the findings generalize across different model types? What specific reasons led you to choose a rotation-equivariant CNN core for this study? - Could you elaborate on the potential biases introduced by the adaptive regularization and pruning strategies? How do you mitigate these biases? Have you explored other regularization techniques or pruning strategies that might offer a better balance between consistency and predictive performance? - Are there specific aspects of model performance that are most critical for maintaining biological plausibility? - The adaptive regularization introduces additional hyperparameters. How do you approach hyperparameter tuning to ensure optimal model performance? What are the computational costs associated with this tuning process? - Why did you choose the Adjusted Rand Index (ARI) and other metrics for evaluating consistency? Are there alternative metrics that might provide additional insights? How do you ensure that the chosen metrics accurately reflect the biological significance of the neuronal embeddings?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Response

Thank you for the authors response. I will update my review accordingly.