Temporal Conditioning Spiking Latent Variable Models of the Neural Response to Natural Visual Scenes

Summary

The authors propose a new model called temporal conditioning spiking latent variable models (TeCoSLVM) that uses spiking neurons to simulate neural response to visual stimuli. They claim that this approach helps to preserve information in spike trains and adapt to temporal dependencies. In experiments with retinal data, they demonstrate that the TeCoS-LVM models produce more realistic spike activities, fit spike statistics accurately, and generalize well to longer time scales.

Strengths

Models that accurately predict spiking activity are still an open problem. The authors make an interesting contribution to this by defining a latent variable model that is trained with an information bottleneck objective. The work is, as far as I can tell, original. A few links to existing work are missing, which I mention below. The paper is mostly well and clearly written, and has an overall good quality. I have a few questions for the authors, but if they are answered satisfactorily, the paper makes a significant contribution to the development of dynamic spiking models.

Weaknesses

I have a few questions regarding the paper and a few hints for relevant literature. --- Decoder: * I couldn't find information on the actual form of the decoder in the main paper. In particular, how is $\psi_{dec}$ computed? * How are the latent states integrated into the LIF neurons? * What is also not entirely clear to me is whether the model has access to real spike trains from previous time steps or not. * From the figure it doesn't look like it, but since the decoder is not clear to me I cannot be sure. It would be nice if the authors could clarify. This is particularly relevant to make sure the comparison to the CNN is fair. Experiments: * How are the spiking rates computed? * What loss function is used to train the CNN? Poisson loss? Would it be possible to also train the spiking model with Poisson loss (i.e. do you have the rate)? The reason I am asking that is that Fig 3B shows that your TECOS models perform better in terms of spike train dissimilarity. However they were trained on that. If the CNN was trained on a different objective, the comparison is a bit unfair. Fig3: * How do you generate spikes from the CNN in Fig 3A. --- Regarding dynamic models of neural activity, these two paper might be relevant (in particular regarding the point of predicting variable length sequences): - Fabian H. Sinz, Alexander S. Ecker, Paul G. Fahey, Edgar Y. Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Xaq Pitkow, Jacob Reimer, Andreas S. Tolias Stimulus domain transfer in recurrent models for large scale cortical population prediction on video - Eric Y Wang, Paul G. Fahey, Kayla Ponder, Zhuokun Ding, Andersen Change, Taliah Muhammad, Saumil Patel, Zhiwei Ding, Dat T. Tran, Jiakun Fu, Stelios Papadopoulos, Katrin Franke, Alexander S. Ecker, Jacob Reimer, Xaq Pitkow, Fabian H. Sinz, Andreas S. Tolias Towards a Foundation Model of the Mouse Visual Cortex Regarding spiking neurons, this paper might be relevant: - Ramesh, Poornima, Atayi, Mohamad, Macke, Jakob H: Adversarial training of neural encoding models on population spike trains

Questions

I integrated the questions in the weaknesses above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are discussed in one short paragraph at the end. It could be a bit more extensive.

Summary

The authors model retinal ganglion cell responses to natural stimuli using a spiking latent variable model. They employ the (variational) Information Bottleneck (IB) method to compress the visual representation, similar to last year’s NeurIPS paper by Rahamni et al. However, this work differs in using binary responses (discretizes spike trains) instead of count responses, an input channel size of 1 instead of T, and temporal conditioning (RNN) instead of GP prior to handle temporal dependencies. They compare their results to Rahamni et al’s IB method and a CNN-based architecture on 4 salamander datasets. They show that those baselines fail to capture the spike autocorrelation, and a noisy version of their LVM model can also reproduce real firing rates more accurately.

Strengths

- good performance - scales well to long time series - modeling single trial spike trains, not just trial-averaged firing rates

Weaknesses

- Stimuli are converted to spikes, then to real valued signals (LVM), then back to spikes (ganglion cells). This does not align with visual processing in the retina: photoreceptors exhibit graded responses, as do bipolar cells (typically), whereas ganglion cells spike. - The authors stress the importance of taking temporal dependencies into account, but instead of comparison to IB-GP they compare to IB-disjoint. Whereas the former has a temporal prior over latents the latter does not. - A comparison to a method that also predicts spike trains not firing rates is lacking. Such a method could be competitive with regard to the considered spike train dissimilarity and autocorrelation measures.

Questions

- Eq (1) 3rd eq with $u_t$ on both sides, yielding $u_t=u_{reset}/o_t$, is odd. - line 103: Isn't $\tau_m$ is the membrane constant, not $\tau$? It seems $\tau=e^{-\Delta t/\tau_m}$ with bin size $\Delta t$. - Why do you compare to IB-disjoint, not IB-GP? The authors of [19] favored IB-GP (temporal prior over latents) over IB-disjoint (no temporal prior). - Does IB-GP need fewer latents that are more interpretable latents than yours? - Line 290: Where is the Fig 5 you refer to? - There are various measures of spike train dissimilarity in the literature, e.g. Victor-Purpura distance, Van Rossum distance, SPIKE-distance. You use the MMD in results and use an objective function (Eq 10) that specifically optimizes for this particular measure, thus obtaining good results. Do your conclusions hold for the other commonly used measures of spike train synchrony as well?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have discussed limitations of their work.

Summary

This paper proposes a spiking latent variable model of neural response to natural visual stimuli. The model is trained to directly predict spike trains instead of trial-averaged firing rates, and designed to work, at test time, on sequences that are longer than sequences seen during training.

Strengths

1. The authors identify two critical limitations of biophysically-realistic computational models. They propose a model that effectively address these limitations. Directly predicting spike trains using spiking neural networks, appears to be a natural solution. 2. The empirical experiments suggest that this results in a more accurate prediction of neural responses.

Weaknesses

1. A main claim of the paper is that the model generalizes to longer time scales unseen during training. It is unclear whether the model is actually leveraging long-range interaction in its prediction. The results do not demonstrate that the memory of the model is holding long-range history from the sequence. It is unclear whether a model with a sliding window (of 1s) would perform the same or worse. 2. While the authors do a great job at analyzing different aspects of the model including the impact of hyperparameters on the learned latent space, it remains unclear how this more bio-realistic model provides novel insights into sensory processing.

Questions

- The model can be applied to sensory data of other modalities, how sensitive is the model to the choice of hyperparameters? For example, the latent dimension or the hidden dimension? - Why was the warmup period similar for all four datasets (0.5s), one would expect to observe differences due to different dynamics of the stimuli.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

As stated, the proposed model is only partially realistic.

Summary

This paper proposes a model, which is composed of spiking neural networks and a conventional recurrent network, to reproduce neuronal responses, i.e., retinal ganglion cells. The author claims that the novelty of this model is incorporating the spiking network, which makes the model directly outputs spikes and is more biologically plausible. The author did simulations to show the model outperforms alternatives in reproducing retinal ganglion cells' activities.

Strengths

### Originality In my understanding, the paper has two novelties: 1) the model incorporates spiking neural networks as encoder and decoder; 2) the model uses an RNN to provide temporal conditioning which is more flexible than previous models whose kernels have fixed temporal durations. ### Quality The writing has clear explanations of the probabilistic framework, but it is quite unclear how the probabilistic framework is connected with underlying spiking networks (see my comments below).

Weaknesses

### Major I understand the variational information bottleneck framework, and the text of this part is clearly written. However, it is unclear the details of the spiking networks and how they are incorporated into the probabilistic framework. 1) Specifically, how the distributions' parameters $\phi$ and $\psi$ in Eqs. 6 and 7 are related to spiking networks? PS: I only see line 151 saying $\phi$ is a linear readout of spiking neurons' responses. 2) What are the structures of spiking networks in the encoder and decoder? Are the pure feedforward spiking network, or recurrent spiking network? 3) During the model training, did the authors only train the readout weight from the spiking network? or also train the synaptic weights inside the spiking network? Without providing these details, I am not confident in judging the novelty of this paper. I am happy to see the authors' rebuttal to explain this. ### Minor - Line 177, “Sec. C” and Line 188, “Sec. E”: I don’t know where these sections are. In the supplementary? - Eq. 3: does this equation miss a term of $\beta I_c$ in the Lagrangian multiplier?

Questions

- Conceptually, I don't understand why the model with spiking network encoder and decoder could render better results. I am really interested in some deep, theoretical discussions on this issue. I could understand if the output layer is a spiking network, it would fit the spiking neuronal responses better. Nonetheless, I don't understand why using spiking net as intermediate layers could improve the performance. Can the author provide an alternative result to replace the spiking intermediate layers by rate-based neurons and compare the performance of the two versions? - Another conceptual question. From the neural pathway point of view, the neural circuits from the retina to the visual thalamus, i.e., LGN, is equivalent to dimensionality reduction because the number of neurons decreases with the neural pathway. And then from LGN to the visual cortex the neuron number will expand a lot. In this sense, the visual pathway inside the retina, and the one from the retina to LGN looks like an __encoder__, while the pathway from LGN to the visual cortex is like a __decoder__. If my statement was true, the retina is just an encoder with less number of neurons in the output layer (retinal ganglion cells) compared with the input layer (photoreceptor). Hence, to reproduce retinal ganglion cells' responses, what is the advance of considering the proposed model architecture (Fig. 1B) with an encoder (dimensionality reduction) and a decoder (dimensionality expansion?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

See my comments in Questions.

Thanks for your clarifications

Dear authors, thanks for your clarifications. I have read the rebuttal. I will wait for the discussion with the other reviewers before adjusting (or keeping) my score.

Thanks for your clarifications

Thank you for addressing the points I raised! These clarifications and the additional work carried out in response to all reviewers further strengthens this paper. I increased my score.