Neural Persistence Dynamics

Summary

This work presents a novel approach to infer the parameters of governing equations describing the collective behavior of systems like point clouds. It leverages persistent homology to capture the topological features of the system's state. These features are then modeled using a Latent ODE system, capturing the temporal evolution of the system. By analyzing the latent dynamics, the method can identify and regress the parameters of the underlying governing equations (e.g., PDE) that govern the system's behavior.

Strengths

- This work proposes a novel method based on persistent homology (PH) to infer the parameters of governing equations describing the collective behavior of systems like point clouds. - A novel application in combining persistent homology with inverse problems. - The paper discusses various aspects of combining PH with modeling temporal correlations, giving an in-depth insight into the field.

Weaknesses

- This work resonates closely with inverse problems for dynamical systems, where several works have been conducted to infer the initial parameters or parameters of interest based on observed data. Some of them are: 1. Learning to Solve PDE-constrained Inverse Problems with Graph Networks (https://arxiv.org/pdf/2206.00711) 2. Fully probabilistic deep models for forward and inverse problems in parametric PDEs 3. Invertible Fourier Neural Operators for Tackling Both Forward and Inverse Problems (https://arxiv.org/pdf/2402.11722) A comparison with these methods will strengthen the aspect of this work, as these methods do not explicitly model the topological features. - I think the paper should include some additional discussion about the computational limitations of the method. While these are briefly discussed at a high level, a more quantitative analysis (e.g., comparing the effective run times and computational complexity) seems essential to give the reader a better idea of the overhead incurred when using this method. - Despite the capability of their model to incorporate persistent homology, enabling the use of topological features, they only utilize classical persistent homology via Rips filtrations. The potential advantages of this feature are not adequately explored in the model.

Questions

- Does the model output a single point estimate for the parameters of interest or a corresponding distribution (mean and std)? - Can authors describe the regression objective w.r.t to underlying parameters of interest, as it is unclear from the paper, maybe including the correct equations they use? - Since PH does not account for the temporal changes in the point cloud, - How is the work different from this https://arxiv.org/abs/2406.03164? It seems like it can be subsumed as one can define a graph out of point clouds and operate on them. - Does the fitting of the ODE system deal with stiffness as you are fitting the topological features via an ODE, which can be non-smooth in between, leading to inefficient modeling?

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

See questions and weaknesses.

Summary

The paper addresses the challenge of predicting the specific parameters of models yielding point cloud dynamical systems known only partially from a set of observations in different time steps. This is achieved by leveraging information about the evolution of persistent homology vectorizations of the observed point clouds at different time steps. Particularly, the paper uses a specific, previously published, vectorization that is Lipschitz with respect to the standard point set Wasserstein distance. The evolution of the topological vectorization for the different point cloud dynamical systems is assumed to be governed by latent dynamic processes, which are approximated via a neural ODE and used to infer, given a set of observations of an evolving point cloud, the model parameters that produce these observations. The proposed baseline method is tested and benchmarked across three different scenarios, demonstrating the utility of its components and showing a significant superiority over the leading state-of-the-art methods.

Strengths

**Originality**: To the best of my knowledge, the idea of modeling the dynamics of vectorized persistence diagrams via a continuous latent variable model is novel and powerful. **Significance**: As stated and demonstrated in the paper, neural persistence dynamics, and the evolution of point clouds over time seen as a group rather than individual points, provide significant insights into the dynamical systems governing the behavior of the individual points. This approach has great potential in studying real-world problems, especially in "natural systems" where the interaction of individuals as a group is key to understanding their behavior (e.g., flocks of birds, insects, fish, cells in an organism). Specifically related to neural networks, the method proposed in this paper has the potential to improve the analysis of the dynamics of data going through the different layers of a neural network, as done in a more simplistic way in [1] and [2], where contradictions were found that this project might help resolve. Additionally, the method's significant superiority compared to other approaches for predicting the parameters of dynamical systems from evolving point clouds suggests that this is indeed a promising direction for further research in the study of evolving topology and in the aforementioned areas. **Clarity and Quality**: The paper is generally well-written, with a few exceptions that I will address in the weaknesses section. The comprehensive literature review significantly enhances the quality of the manuscript. Figure 1 and 2 are particularly clarifying when reading the text. [1] Naitzat, Gregory, Andrey Zhitnikov, and Lek-Heng Lim. "Topology of deep neural networks." Journal of Machine Learning Research 21.184 (2020): 1-40. [2] Wheeler, Matthew, Jose Bouza, and Peter Bubenik. "Activation landscapes as a topological summary of neural network performance." 2021 IEEE International Conference on Big Data (Big Data). IEEE, 2021.

Weaknesses

- I think that the sentence in line 32 "... due to missing correspondences between individuals..." is hard to read. Which missing correspondences? -Table 1 lacks context and is duplicated in Table 2. The experiments of this table are not explained and metrics are not justified. I think a more qualitative explanation may be better than the quantitative explanation given in the introduction. Alternatively, providing more information about the experiments would help, but it could lead to information duplicity, which might be undesirable. - The proposed approach does not track of the evolution of individual topological features of the persistence diagram. This could be important in scenarios requiring fine-grained topological information. - The authors claim that their method is more efficient than other methods, but computation times are not reported in the main text. - In the persistent homology paragraph (starting at line 154), the authors state that there is a decomposition in persistence diagrams (births and deaths) when using abelian groups in the chain complexes. As far as I know, this is not a trivial property and does not happen always [Theorem 2, reference 3] for abelian groups. The usual theorem applies to vector spaces. - Only one vectorization method is used, without significant justification. I would have appreciated a more detailed ablation on this point. - Maybe I'm wrong, but when I access the arxiv version of reference [23] in your paper, Table 4 contains execution times, and not SOTA results. [3] Jiajie Luo, , Gregory Henselman-Petrusek. "Interval Decomposition of Persistence Modules over a Principal Ideal Domain." (2023).

Questions

- Why do you use $R^2$? I thought that it was a misleading metric [4]. This is my main concern, and the core reason why the score is a borderline accept. - Can your approach be used for discrete dynamical systems (where time is discrete and not continuous?) - Since the method is very general and can work with any kind of vectorizations, not just topological ones, is there a way to regularize or add explicit bias during training to inform the training process/neural ODE that the representations are topological? [4] Li J (2017) Assessing the accuracy of predictive models for numerical data: Not r nor r2, why not? Then what?. PLOS ONE 12(8): e0183250.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

Some limitations have been addressed, although I would appreciate a better discussion, for example, adding that the evolution of individual topological features cannot be studied.

Summary

This paper considers the problem of learning some parameters $\theta$ in---roughly speaking---a dynamical system of the form $\dot X = \phi(X, \theta)$, where $X \in \mathbb{R}^{n \times d}$ (with typically $d=3$), from an observed time-discrete trajectory $X(\tau_0),\dots,X(\tau_N)$. The idea conveyed by this paper is that instead of tracking the dynamic of the whole point cloud ($n$ can be quite large) to infer the parameter $\theta$, it may be somewhat sufficient to look at a "featurization" of it---here, of a topological nature, yielding a sequence of vectors $v_{\tau_0},\dots,v_{\tau_n}$ that somewhat summarize the dynamic while being (hopefully) sufficient to infer the parameters $\theta$. The parameters of the system are then learned by existing techniques, essentially by using a low-dimensional latent dynamic $(z_t)_t$ modeled by a neural ODE and then connected through the observed $(v_t)_t$ by encoder-decoder; training being performed using ELBO maximization. The experiments are conducted on synthetic data (from rather sophisticated dynamical systems), containing an extensive ablation study, and showcasing the usefulness of the proposed method on the considered models.

Strengths

The proposed use-case of Topological Data Analysis in the context of dynamical systems feels quite convincing and refreshing. The methodology, at a global picture level, is fairly clear. The performance showcased by the experimental results, supported by an extensive ablation study, seems to robustly support the usefulness of the method.

Weaknesses

My main concern is about the clarity of the paper in the following sense: at the global picture level, the paper feels very clear and convincing. However, I think that it is not clear at the low-level scale, in that I feel completely unable to implement the method based on what is described in the (main) paper. For, the PD vectorizations are never described precisely, the final loss function (ELBO + regression) is only informally discussed in lines 226-234, I actually do not exactly understand how the parameters of the systems are encoded in the models, between which variables the $R^2$ coefficient (line 268) is computed (I infer that it is the parameters of the models wrt the one inferred at training time), etc. I understand that code and so on will be delivered and that the method will be usable by others if needed, but I believe that the impact of the paper would be strengthen if one could understand what is done _at the technical level_ (i.e., be able to roughly implement it) after reading it. Adding a minimalist proof-of-concept experiment, with a trivial dynamical system like $\dot x = \theta x$ or something like that, may possibly be helpful (or not).

Questions

1. Related to my comments above, could you make explicit the way the parameters of the systems are encoded and learn? Say that I consider the Volume exclusion model with parameters $(\alpha, R, \lambda_b, \lambda_d)$. How are these parameters actually estimated once $\texttt{Enc}_\alpha, \texttt{Enc}_\theta, \texttt{Dec}_\gamma$ are trained with the ELBO+regression rule? How are the $\hat{\beta}_t$ (showcased in Figure 3) used afterward? 2. (more genuine question out of curiosity) In some sense, the seminal point cloud $X$ contains implicitly all the possible topological information one could extract using persistent homology. It also seems that PointNet-like architectures can extract this topological information---see for instance the RipsNet model proposed in [1]. So in some sense, this suggests that turning the observed $X(\tau)$ into diagrams is "the way to go" but PointNet fails to learn that. Do you agree with this statement? If so, do you have any intuition on why? 3. Still related to the above, the paper mentions that it uses PD with homology dimension 0, 1 and 2. Did you run the experiments using $H_0$ alone? If the performance significantly decreases, that would be a nice example where non-trivial topology is useful (as frustrating as it can be, I often observed that $H_0$ alone, i.e. looking mostly at the distribution of pairwise distances, was sufficient to reach good performances). Note: please do not feel obliged to run new experiments during the rebuttal period! This is just a genuine question. 4. In the conclusion, you mention that computing $H_2$-PD with Rips filtration is quickly computationally expensive (I completely agree), and propose to use alpha-filtration instead. Is there any difficulty to do so right now? (Alpha filtrations are readily implemented in several Tda-libraries according to https://cat-list.github.io/ ). Ref : [1] RipsNet: a general architecture for fast and robust estimation of the persistent homology of point clouds, de Surrel et al.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

NA

Summary

This work considers the problem of learning the latent, continuous-time dynamics underlying time-evolving point clouds. To solve it, it leverages previous work on the persistent homology of point clouds and their vectorization, as well as the PointNet++ network, to obtain static representations of the point clouds at a set of observation times. It then makes use of the LatentODE framework of Rubanova et al. (2019) to infer a continuous-time representation encoding the observed representations. The authors employ the learned continuous-time representations to tackle the inverse problem of regressing the ground-truth parameters of a set of governing equations from which the point cloud observations were simulated. They show their model outperforms two recent baselines. They also empirically demonstrate that (i) the deep representations obtained via PointNet++ are complementary to those obtained via persistent homology; that (ii) actually modelling the continuous-time dynamics helps with the regression task; and that (iii) the regression tasks increases its complexity with that of the initial condition in the simulation. *References:* - Latent ODEs for Irregularly-Sampled Time Series. Rubanova et al. (2019)

Strengths

The paper is very well written, and provides enough information for a reader not versed in the use of persistent homologies for point cloud summarization. The authors justify their methodology, viz. the encoding of vectorized persistent diagrams into a continuous-time latent process, by using stability arguments of previous work. Their argumentation reads well and is convincing. Another strenght is that, besides empirically demonstrating that their method outperforms two recent baselines, the authors additionally perform reasonable ablation studies that further justify their methodology.

Weaknesses

The main contribution of this work could be read as a direct application of the Latent ODE framework of Rubanova et al. (2019) to the problem of time-evolving point clouds. It can be seen as incremental too, for it leverages stablished methods for point cloud representations. One could therefore argue that, despite its merits, the paper would better fit a conference or journal with a more applied character. It is also not clear how the method would perform on real-world regression tasks with empirical point cloud data. Can one, for example, use the proposed methodology to study the recorded data presented in the seminal work of Bialek et al. (2012)? It’d be nice if the authors could comment on the applicability of their method to empirical data. *References:* - Statistical mechanics for natural flocks of birds. Bialek et al. (2012)

Questions

1. Neural ODEs are well known to be very difficult to train (see e.g. Dupont et al. (2019), Finlay et al. (2020), Choromanski et al. (2020) or Pal et al. (2021), just to cite a few). Can you comment on how difficult (or easy) was to train your Latent ODE network on the synthetic data you studied? What about the training time? 2. How did you parametrize the decoder network? I don’t find it in the manuscript. 3. Did you consider adding some noise to your observations in point cloud space? As presented, your network should easily be able to handle noisy observations. Similarly, how does the persistent homology representation deal with noise? These questions are of course relevant if one wants to apply your methods to empirical data. 4. How many points from the latent path are used as input to the regressor model? Does the model work with a single point, as e.g. the last point along the latent trajectory? It’d have been nice to understand what information of the latent path is important for the regression task, specially given that completely dispensing from the latent dynamics still gives compelling results (i.e. Table 3). *References:* - Augmented neural odes. Dupont et al. (2019) - How to train your neural ode: the world of jacobian and kinetic regularization. Finlay et al. (2020) - Ode to an ode. Choromanski et al. (2020) - Opening the blackbox: Accelerating neural differential equations by regularizing internal solver heuristics. Pal et al. (2021)

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes, they did address the limitations of their method.

Thanks

Thank you for taking time answering my review, giving experimental details and clarifying the content of the paper. I like that $H_1 + H_0 > H_0$, that's somewhat of a "good new" for TDA :-) I still need some time to read other reviews / comments and discuss with other reviewers to make my mind clear but I feel more positive about this work.