Response to additional questions from Reviewer xe9o - Part 1
We thank the reviewer for getting back to us so promptly! We agree that Figure 1 should be a part of the manuscript and we will include a version of this figure in the main text of the updated manuscript. We believe there might be a misunderstanding about the goals of statistical models of neural data that we hope to clarify in the following response. Our response will be in two parts due to character constraints (we will answer Q2 and Q1 in the first comment). If we answer all questions satisfactorily, we hope the reviewer would consider raising their score.
**Q2**
We are glad the reviewer found our definition of bps helpful and we will make sure to add this in the updated manuscript. We suspect there might be a misunderstanding here about prior literature on neural population models. As described in [1], there is a rich literature of population models that utilize either Gaussian or, more recently, Poisson observation models (please see Table 2 of [1]) to predict neural spiking activity. The goal of neural population models is to predict the underlying firing rates of each neuron because the exact spike times are believed to be observations from a stochastic process. Similar to this seminal paper [2], we choose to model the observed spikes as samples from an inhomogeneous Poisson process whose rate corresponds to the inferred firing rate for the given neuron. To measure the quality of fit of our Poisson process, it is most natural to use a version of the log-likelihood such as the bps metric utilized in the Neural Latents Benchmark [3]. Using this commonly agreed upon metric makes sure that our work is consistent and comparable with existing research on neural population models.
In addition, the "bits per spike" metric is closely related to the "deviance", a standard metric generally used in Statistics, e.g., generalized linear models (GLMs) use deviance as a goodness-of-fit measure for a statistical model. The deviance compares the goodness-of-fit of the model of interest, e.g., MtM, to a baseline null model, where the goodness-of-fit is measured by the model log likelihood. The bps is simply a normalized version of the deviance metric, which compares the model predictions to the average firing rate of the neuron for the trial. The bps further normalizes the deviance by the spike count so that the metric can be comparable across neurons regardless of whether the neurons are active or inactive.
Along with the bps, we would be happy to report the R-squared in the revision to provide further intuition, but note that the R-squared depends strongly on bin size (as bin size gets small so will R-squared, since the precise timing of spikes is likely not predictable in this setting).
**Q1**
In Figure 1 of the one page PDF, the ground-truth rastermaps are the spike counts of the trial while the MtM/NDT rastermaps are the inferred firing rates. These quantities are not directly comparable because we expect the underlying firing rate of a neuron to vary smoothly while its observed spikes (which are assumed to be stochastic) may have sharp increases. A great example of this phenomena is shown in [4] where they demonstrate that when the inferred firing rates exactly fit each spike time, this is actually overfitting to the neural data. They introduce a novel dropout strategy, similar to MtM's neuron masking, that reduces this overfitting.
In fact, it has been shown by [2] (in Figure 2e) that the smoothed firing rates from a Poisson model actually allow for decoding animal behavior more accurately than the raw spike counts. Whether to smooth or not involves a bias-variance tradeoff. More smoothing means more bias and less variance. When we optimize for prediction accuracy, we are implicitly optimizing this tradeoff, and empirically we (and others) find that some smoothing (bias) leads to better predictions.
[1] Hurwitz, Cole, et al. "Building population models for large-scale neural recordings: Opportunities and pitfalls." Current opinion in neurobiology, 2021.
[2] Pandarinath, Chethan, et al. "Inferring single-trial neural population dynamics using sequential auto-encoders." Nature methods, 2018.
[3] Pei, Felix, et al. "Neural latents benchmark'21: evaluating latent variable models of neural population activity." arXiv 2021.
[4] Keshtkaran, Mohammad Reza, et al. "A large-scale neural network training framework for generalized estimation of single-trial population dynamics." Nature Methods, 2022.