Exploring Behavior-Relevant and Disentangled Neural Dynamics with Generative Diffusion Models

Summary

This study presents BeNeDiff that integrates behavior-informed latent variable models with state-of-the-art generative diffusion models to investigate neural dynamics during behavioral tasks. The methodology involves identifying fine-grained and disentangled neural subspaces and synthesizing behavior videos to provide interpretable quantifications of neural activity related to specific behaviors. The authors validate their approach using multi-session widefield calcium imaging datasets, demonstrating high levels of neural disentanglement and reconstruction quality.

Strengths

1. Using wide-field calcium imaging for neural-behavior analysis is both innovative and interesting. 2. The approach of employing a behavior LVM for latent factor disentanglement, followed by a video diffusion model for behavior encoding, is both clear and impressive. 3. Especially, using $Z$ produced by LVM as learning target for the classifier in video diffusion model is quiet important, avoiding the issue of "label leakage" which maybe introduced by classifier-free guidance generation. 4. The paper is well-organized, and the disentanglement results and behavior encoding results are both meaningful and useful.

Weaknesses

1. This work primarily relies on a behavior LVM for neural activity disentanglement, which can be influenced by the newly introduced total correlation (TC) penalty term with $\beta$ weighted. The impact of this penalty should be considered through ablation studies. **Not a Weakness**: There is significant research combining behavioral labels with latent variables of neural activity [1,2], considering the effects of different disentanglement algorithms on behavior encoding would further enhance this study. [1] Zhou, Ding, and Xue-Xin Wei. "Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE." Advances in Neural Information Processing Systems 33 (2020): 7234-7247. [2] Schneider, Steffen, Jin Hwa Lee, and Mackenzie Weygandt Mathis. "Learnable latent embeddings for joint behavioural and neural analysis." Nature 617.7960 (2023): 360-368.

Questions

I have no question.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

see WeaKnesses part.

Summary

The paper proposed BeNeDiff to learn disentangled neural trajectories together with a generative diffusion mode for behavior. BeNeDiff leverages beta-VAE for learning of disentangled space space Additional behavior generation module is applied for interpretation of the neural subspace. It is interesting to see that the learned latent latent trajectory is separated for different behavior types. The qualitative results and quantitative results demonstrate the superior of the proposed method.

Strengths

1.The disentangle property of neural latent space is significant compared to previous methods based on the quantitative metric. 2.The idea leveraging diffusion model to interpret learned neural subspace is interesting.

Weaknesses

1.It is unclear to me if the behavior video generation part is trained together with the disentangled subspace learning. It seems true based on the figure, but didn't see how they are optimized together from text. 2.it is interesting to learn the disentangled neural space. Some visualization of the learned latent space is beneficial. 3.Additional ablation study on different contribution of the loss function could be performed.

Questions

1.How training and testing data are split? Are they session wise or random? How the method can be generalized to different sessions or animals? 2.Does the learned latent trajectory show some clustering property, or somehow organized, could you visualize it? 3.For session 3.2, $c$ is better to be described as it is first mentioned in eq.5. Does the number of class is equals to feature dimension D and the same as the number of the behavior classes"? 4.To get figure 4, which latent variable do you use, $\hat{Z}$ or $Z$? 5.For fig.6, and fig.7, could you show the difference of the ground truth frame with the same behavior type as comparison.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

It could be better the paper include more analyze on the learned latent trajectory and see if it could provide some scientific insight.

Summary

The authors implement a VAE with behavioral labels to identify behavior-specific latent variables. They use these latents to then create a generative model for experimental video. They focus on the application of their model to a wide-field cortical recordings of a mouse during an experimental visual task. They show that their model identifies disentangled behaviorally relevant latents and can create experimental video where identifiable behavioral elements are generated from specific neural latents.

Strengths

I see two primary contributions in this paper that make it appropriate for a Neurips audience. 1) The use of Total correlation measures to disentangle latents in a VAE framework in neuroscience. 2) Using generative video diffusion models to identify what video features relate to individual neural latents. Both of these are interesting applications of existing methods but they are applied in a clever way to neural data and I think these would be of interest to a computational neuroscience audience.

Weaknesses

Each of the primary contributions to neural LVM type models (the use of total correlation and the diffusion modeling) are not sufficiently benchmarked against competing approaches. It is unclear how this model compares to the others used in neuroscience. If the authors wanted to strengthen the case that their model is an important novel contribution for behaviorally-relevant latent disentangling, they could compare how total correlation VAE compares to other methods where behavioral labels are used to help disentangle the latent states. One important comparison that I would be curious to see that isn't cited would be the pi-VAE1, but the others the authors cite could be used. Specifically, these other models could be added to figures 4 and 5. The authors could also focus on how using video diffusion is a novel application in this setting. Again, some additional benchmarking with existing deep LVMs in neuroscience and some further discussion and emphasis as to how the diffusion methodology solves an important problem in neuroscience that perhaps existing methods cannot solve would strengthen this paper. Though the authors highlight the application of the model to this specific dataset as a contribution, this alone would be more appropriate for a more experimentally-focused venue. For neurips, the focus should be on the model advancements and their applications, not the significance of the dataset. 1 - Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE - Zhou and Wei

Questions

NA

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors only minimally discussed the limits of the ability of their model.

Thank you

Dear Reviewer o3cJ, Thank you for your positive feedback and for adjusting your score. As mentioned in our rebuttal, we will incorporate the neural LVM module comparisons and the VDM module-generated video visualizations for multiple datasets into the appendix of the revised manuscript.