Trial matching: capturing variability with data-constrained spiking neural networks

Summary

This paper considers fitting high-fidelity recurrent spiking neural networks to multiple populations of neurons across sessions. The networks include LIF neurons for each fitted neuron and model within & across region interactions at fine timescales, and include a soft-thresholding mechanism for neural spiking. The model is fit to match the responses of the recorded neurons both across trials and to generate variability across trials that is consistent with the variability observed in the data. For the latter, the authors propose an optimal-transport based loss on the distributions of sampled recorded trials and trials simulated from the model. The authors validate the proposed method in simulation, showing that the trial matching term in the objective is critical for capturing variability across trial types in neural data. Then, they apply it to a large multi-population neural dataset, showing the model captures trends across trial types in this dataset as well.

Strengths

The motivation and methods are clear and novel, and the resulting model appears to be significant for many neuroscience modeling tasks.

Weaknesses

Fitting the model jointly to neurons that are not simultaneously recorded (and not necessarily from the same animal) seems to be conceptually at odds with the goal of capturing trial by trial variability. By definition, neurons from different sessions should be considered different for trial to trial variability. I would argue that a more accurate term for the loss is "condition to condition" variability. The authors show it is possible to capture responses across conditions using their objective. It would be great for the authors to expand on this point.

Questions

* Session stitching. Can the authors test their method for session stitching in a simulation, where they stitch together neurons recorded in different sessions as is done in the real data? This would help guide when to trust the results of stitching together non-simultaneously recorded neurons. * When computing the trial target $T^\prime_{trial}$ for neurons from different sessions, are the trials matched based on condition before summing neurons from different sessions? For example, `Hit` trials are combined from different sessions, but not `Hit` trials and `Miss` trials.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

The authors present a method to fit recurrent spiking neural network models to experimental data, extending prior work to multi-session recordings that involve different subpopulations in each session. To capture single trial variability, they introduce a trial matching loss function. The technique is applied to neural recordings from six cortical areas in the sensory-motor pathway during a delayed whisker detection task. The model successfully captures trial-averaged firing rates of individual neurons and population averages. In addition to the expected matching results enforced by the loss function, the model reveals an unexpected mode of variability related to movements during the delay period, resulting in two distinct clusters within hit trials.

Strengths

- Models trial variability beyond trial-averaged firing rates. - Considers a large-scale, cortex-wide multi-region dataset.

Weaknesses

- Analysis of single trial variability still relies on population averages rather than individual trial averages. - Lacks a quantitative comparison to other methods. - Represents an incremental contribution, with the closest related work being Bellec et al., NeurIPS 2021.

Questions

- Are all results based on held-out data? If so, how was the data held out? - What are the new predictions that are not expected results from the training method? Aren't the four main trial types obvious due to the optimization objective including sensory neurons that discriminate between stimulus present/absent and jaw movements that discriminate between lick/no lick, resulting in true positive, false negative, false positive, and true negative categories? - If you hold out all neurons from one or multiple entire areas, can your model accurately predict the activity in those areas? - Since you use population averages to study trial variability, wouldn't a simple population-level approach using six units, each representing a cortical area, yield the same results?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors acknowledge that this paper represents only a step towards their long-term goal of developing biophysically interpretable mechanistic models.

Summary

This paper presents a framework for training a large recurrent spiking neural network on multi-session recordings by leveraging an optimal transport-based trial matching between the real data and generated data. This model is use to model the cortical sensory-motor pathway during a tactile detection task.

Strengths

1. In neuroscience, the challenge of training a model on multiple sessions is very relevant. It is expected to record data over multiple sessions and across multiple animals, hence having tools to jointly analyze the underlying dynamics of the neural population across different regions and recordings is important and critical. The use of trial averaging, as motivated by the authors, can be limiting and fails to capture the trial variability during complex behavior. The model presented in this work enables the analysis of the neural population dynamics. The design of the model is clearly well thought-out but remains very simple and is strongly supported by 1) good arguments 2) great visualizations 3) supporting empirical results. 2. The empirical results support the effectiveness of the model at revealing the underlying modes of trial variability. The unsupervised discovery of new modes is also promising. 3. The discussion section is very thorough. In particular the comparaisons with LFADS are interesting and insightful. This work challenges the idea that low-dimensional spaces are required to be able to interpret neural dynamics.

Weaknesses

1. The main weakness of this work is that it was not directly compared to other baselines or test on other recordings during a different behavioral task. 2. The model requires the selection of multiple hyperparameters. One in particular is the number of neurons (1500). The total number of neurons across all recordings is 4415 so there is a huge decrease in the number of units that are modeled. It is unclear how this choice can be made as the size of the dataset increases or for different datasets. The current model takes 3 days of training, are there expected computational limitations for the number of neurons that can be modeled?

Questions

1. While the proposed model is promising, there are questions about how the model can be scaled to larger sets of recordings, as well as more heterogeneous recordings (example: groups of animals differentiated by age, state, disease propagation etc...) Would a single model which ignores the individual differences be adequate? 2. Most neural recordings are currently structured by trials. To truly capture the full breadth of neural code complexity, it is necessary to study these neural dynamics in more unconstrained / complex / free-behavior settings, in which case the notion of trial no longer exists. Might the authors have ideas on how their approach can be adapted to such settings?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

No limitations identified.

Summary

This paper extends previous works on generative model of neural data using a recurrent spiking neural network. The main contributions are design of new loss function based on optimal transport to match the trial-to-trial variability in the data, and the approach to extend RSNN generative model to leverage recordings from multiple sessions. The proposed method is evaluated on both synthetic and real neural data, and authors showed that the method can be used for a hypothesis-free identification of modes of trial-to-trial variability.

Strengths

- Potential usage of the method in answering neuroscience questions is appropriately illustrated from the real data results - Extension of existing RSNN to data constrained model is novel, and will be valuable step in increasing the usability of related methods in neuroscience. - Previous work is appropriately discussed and the contributions are clearly stated

Weaknesses

- Although the authors mention in the discussion section the difference in the approach between the proposed method and existing methods, the (potential) advantage of the proposed method (biophysically interpretability) is not clearly illustrated.

Questions

- Definition of $T_{trial}$ in trial matching, Equation 6, and template matching, line 256, seems to differ. Is subset of $T_{trial}$ (containing averaged response of area recorded in the particular session/trial) used to compute $L_{trial}$? - (typo) $T_{trial}$ → $\mathcal{T}_{trial}$ (line 222 and 244)

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Follow up to author response

I thank the authors for their thorough response. It appears I misunderstand how the trial matching loss is computed. As written, I thought trial statistics in eqn. 6 for summed across all neurons for a given area, whether or not these neurons were simultaneously recorded. It makes sense that the recorded statistics in eqn. 6 are computed as sums across the recorded neurons for that trial. For the generated statistics, are those computed as sums across only the neurons that are observed in a session? In your response, you describe neurons as being "hidden" during optimization. Can you elaborate on this point in the paper? I do not see the word "hidden" being used to describe the optimization in the paper (outside of architecture descriptions for the number of hidden units in an MLP).