Summary
In their study "Latent Diffusion for Neural Spiking Data", the authors introduce the titular LDNS model for generating realistic neural population activity, and apply it to three datasets: a synthetic dataset with true latents generated from the three-dimensional Lorenz system, and two previously published neuroscientific datasets.
The LDNS model is an instance of the latent diffusion model (LDM) introduced by Rombach et al. (2022), which is adapted here by the authors to support generation of variable-length discrete-valued time series. The autoencoder of the LDM in this case is trained with a Poisson loss to account for the discrete nature of neural spiking data. The diffusion model of the LDM operating in latent space here is constructed with layers of structured state space sequence (S4) models of Gu et al. (2021). The diffusion model can be trained to generate samples conditional on certain external covariates by conditioning the reverse mode on the covariates. The split of LDMs between the autoencoder stage and the diffusion stage is particularly convenient for the task of generating spike trains, as it avoids any adjustment to the diffusion model for generating discrete variables. A variant of the LDNS model has additional parameters trained post-hoc to incorporate the spike-time history of individual output neurons to account for refractoriness and other single-cell auto-regressive features.
According to the considered neural population statistics, the LDNS model gives an excellent fit to the synthetic data generated from the three-dimensional Lorenz system. For the two neuroscientific datasets, the LDNS performs solid in terms of matching the considered neural population statistics, outperforming the important reference model LFADS (Sussillo, 2016) in one case. On the first neuroscientific dataset comprised of human neural population data sampled during attempted speech, the LDNS is shown to handle variable sequence lengths in the training data and during sampling. The second neuroscientific dataset consists of monkey neural population data recorded during a maze reach task. Here, the LDNS variant with additional spike-history parameters fitted post-hoc is shown to also capture the temporal auto-correlation structure of neural subpopulations. The authors furthermore showcase the ability to generate samples conditionally on experimental covariates, by conditioning sampling on the initial reach angle, respectively the full velocity trajectory.
Strengths
The submitted work is original, in that latent diffusion models (LDMs) to my knowledge have not yet been used to (conditionally) generate neural population activity. I agree that LDMs are indeed a nice idea here, since the separation into autoencoder stage and diffusion model stage means we can try to use denoising diffusion models without much changes.
The work seems technically solid to me. The authors make sensible changes to the LDM for the task of generating neural population activity. The adaptations for discreteness (Poisson loss, spike-history terms) show a good understanding of best practices in neural data analysis. The inclusion of S4 layers into autoencoder and diffusion models to handle and generate variable-length sequences is interesting, and could further spread usage of these tools in the computational neuroscience community.
The manuscript is well written, explaining not only the key ideas and experiments, but also summarizing the important model parts (autoencoders, denoising diffusion models) next to the non-standard adaptations (Poisson loss, S4 layers, spike-history terms) done by the authors for this study. The figures are high quality and way above what I would expect from a conference paper.
This study follows and expands a (small) recent trend of using modern machine learning algorithms for generation of neural population activity. LDNS models / LDMs clearly have some advantages over previously suggested models, in that they combine the low-dimensional representation of autoencoders with the generative fidelity of diffusion models.
Weaknesses
The authors have adapted the latent diffusion model to the task of generating neural population activity. While I believe they did good work on that, I am less convinced of the evaluation and comparison of their model, which is important to judge its overall usefulness.
For a generative model of multivariate time-series, I find the evaluation somewhat lacking. I understand that judging spike train generation is still a much harder task than judging image generation, and that one could argue that to date we don't understand all the relevant aspects of neural population activity. But firing rates, pairwise correlations and population spike counts are features that don't take dynamics into account. Average interspike interval and standard deviations of interspike intervals are still very local temporal features. Computational neuroscience has come up with a host of methods for analysing neural population dynamics -- in particular those with low-dimensional structure as assumed by the LDNS model -- which could be used here for comparing the sampled spike trains against the data. The authors already cited both LFADS and Gaussian Process Factor Analysis, but did not compare e.g. the low-dimensional trajectories extracted from their sampled data against the low-dim. trajectories extracted from the neural recordings. If nothing else, a supplementary figure or two with a handful of additional sampled spike trains against real recordings could help readers judge if the model captures the overall neural dynamics of the data.
Another aspect that makes it difficult to evaluate the quality of their suggested model is the lack of comparison against direct competitors. The authors do an excellent job of explaining LFADS and its relevance as a comparison, but from what I understand only compare against LFADS on one of the three numerical experiments (I take that figure A4 which isn't mentioned in the text is also about the monkey data?). I understand that autoLFADS is significantly more computationally expensive to run than the LDNS model, (which I would encourage the authors to state more prominently also in the main text!) but especially for the second experiment with human data, the LDNS model seems to perform the worst of the three experiments, and I could find no alternative model for comparison.
Questions
As explained above, analysis of the full population dynamics of their sampled neural population data would be very interesting. The authors could, but do not have to follow my suggestion of analysing the sampled neural population with one of the more commonly used dynamical dimensionality reduction methods.
Can we get some comments or additional analysis of the LDNS model in cases where it doesn't work perfectly? There are several interesting supplementary figures for the Lorenz synthetic data, but for the other two experiments I am left wondering whether the non-negligible model errors arise primarily from the autoencoder or from the diffusion model. Those are the cases where I would most appreciate a comparison of the pairwise correlations from autoencoder against those of the full LDNS model, as in Fig A7 for the Lorenz data.
Limitations
I think the authors have adequately addressed limitations and possible negative societal impacts of their work.