We consider the problem of inferring latent stochastic differential equations (SDEs) with a time and memory cost that scales independently with the amount of data, the total length of the time series, and the stiffness of the approximate differential equations. This is in stark contrast to typical methods for inferring latent differential equations which, despite their constant memory cost, have a time complexity that is heavily dependent on the stiffness of the approximate differential equation. We achieve this computational advancement by removing the need to solve differential equations when approximating gradients using a novel amortization strategy coupled with a recently derived reparametrization of expectations under linear SDEs. We show that, in practice, this allows us to achieve similar performance to methods based on adjoint sensitivities with more than an order of magnitude fewer evaluations of the model in training.
Paper
Similar papers
Peer review
Summary
The authors develop a reparameterization scheme for affine SDEs which is advantageous in memory cost and time. Specifically, the authors consider splitting the computation into a series of different chunks such that each can be effectively parallelized and integrated individually by taking the expectation with respect to a uniform random variable over the time interval. This is possible since the SDE is linear and a mean and covariance matrix provide the full description of the data at all time marginals and individual trajectories are not needed. The points are then combined using an interpolation technique that is based upon deep kernels. The authors finally consider experiments to highlight the utility of the method. They consider synthetic and a real experiment on time series data that compares the number of function evaluations and the accuracy of the methods.
Strengths
The idea of transforming the sequential computations into parallelizable expectations is appealing and the numerical results suggest this approach provides a significant computational speed up. This is a useful extension beyond the usual sequential nature of ODE which cannot be parallelized and could be particularly useful and impactful in certain cases. Additionally, empirical results suggest that the modifications provide increased stability. The authors also show the utility of the method on a real dataset where the performance is superior compared to most of the baselines. Finally, the paper is fairly well written and the presentation is intuitive.
Weaknesses
The method only applies to SDEs where the marginal descriptions are known, as the authors make an assumption that the evolution of the latent space is given by a Markov Gaussian process. However, this is not the case for most SDEs, which makes the algorithm limited in scope. From the paper, it is not clear where such an assumption would hold that the latent distribution should be Gaussian everywhere. This question was not studied, and I think that's the major weakness of the paper. While the computations are easier to perform, it is mainly due to the simpler model. SDEs are particularly useful when sampling from a complicated distribution, but in this case the latent distribution Gaussian for all time marginals. Maybe to strengthen this, it would be helpful to show the types of stochastic processes in the observable space that can be modeled (e.g. by applying Ito's lemma to the latent process). This would the provide a more concrete statement on the expressiveness of the model. The experimental evaluation is somewhat limited since the evaluation is only on two synthetic datasets and one real dataset. I would be curious to see how the other methods compare on the pendulum dataset. Additionally, I think some more synthetic datasets would be useful to describe the utility/use of the method, particularly on non-affine SDEs and seeing the performance in approximation. It would be helpful to see a comparison to backpropagating through the solver rather than the adjoint method, which the authors mention is numerically unstable. Understanding how the behavior of backpropagation through the solver (since this is possibly the most common method when not using a fine step size or memory constraints are not a concern) compares to the proposed method. Finally, the authors mention in line 184 the linear assumption of the SDE is not a limiting factor, but the experiments do not seem to be extensive enough to validate such a claim. Other types of methods that are related and should be mentioned are [1, 2]. These methods consider similar strategies with [1] considering a gaussian process in the latent space and [2] considering a linear SDE approximating the latent evolution (using some of the same results mentioned in this paper based on casting linear SDEs in terms of their moments). [1] Fortuin et al 2020 AISTATS, GP-VAE: Deep Probabilistic Time Series Imputation [2] Duncker et al 2019 ICML, Learning Interpretable continuous-time models of latent stochastic dynamical systems
Questions
How does the interpolation between the windows affect the quality of the sample path? This point was not discussed thoroughly and it seems like it might have impact on the sample quality. Were there any comparisons to non-adjoint based back-propagation through the solver? For example, using just an Euler solver, how does the method compare with backpropagation through that? Line 147: “While the previous sections have demonstrated how to replace a differential equation with an integral …” Can the authors please expand upon this? When solving a differential equation, the integral is approximated as a sum, as the authors have done. Is this comment due to the fact that you are not using an adaptive solver but you are using an expectation to compute the integral? Or is it more that you do not require the value of $z(t)$ at some later time, only the expectation? Is there an ablation on changing the window size/number of windows? How does this parameter affect the performance? Why is the drift parameterized as a neural network of z on line 519? I thought the drift was linear in z. Is there a way to verify that the linear SDE assumption is a valid one? Are there some equivalence classes one can derive for the case studied here and a possibly more general class of SDEs?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors provide a discussion on the issues associated with the linear SDE assumption.
Summary
This paper introduces a method for identifying latent stochastic models from discrete time observations. The authors perform unbiased approximations to gradients of the evidence lower bound to identify parameters and estimate the state for latent stochastic differential equations. Thy propose to combine a recently introduced (by the same authors - provided anonymous paper) reparametrisation of the ELBO in terms of time dependent linear SDEs and amortisation to reduce the computational demands. The resulting approach does not require numerical integration of the approximate SDE or the ODEs defining the evolution of the central moments (as required by previous approaches), and seems to perform on par or better than existing frameworks. Although the method is limited by their assumptions of linear covariances and additive noise, the merging of the amortisation with the reparametrisation of the ELBO seems a rather interesting contribution. The performance of the method on the numerical experiments seem to be on par with existing approaches. However in most numerical examples the employed noise in the system was negligible when considering the dynamic range of the employed systems.
Strengths
- The authors eliminate the need for numerically integrating the approximate differential equations by replacing the initial value problem with an integral resulting in reduced computational demands. - The approach avoids numerical instabilities often encountered in adjoint methods. - the proposed solution has a time and memory cost that depends only on the number of parallel evaluations of the ELBO, i.e., scales independently with the amount of data, the length of the time series, and the stiffness of the approximation to the differential equations.
Weaknesses
- The approximate process is a linear SDE, and thus cannot probably capture effectively nonlinear dynamics that may result in multimodal transition densities. - To my understanding the proposed framework requires a diagonal covariance matrix (but I am happy to be corrected if this is not the case) for the diffusion process, as well as a diagonal time dependent covariance for the approximate process. How would you tackle problems with interactions in the diffusion component with your framework? - I consider the diffusion term for the experiment of the Lorenz system substantially small ($\sigma = 10^{-5}$) considering the state space volume the state of the system spans.
Questions
- How do you select the partitions of the temporal axis and how does this selection influence the obtained results? - In line 498 in the supplement you mention that the encoder design is interpretable. Can you explain what you mean with this? - Could you please explicitly mention in the appendix what observation model was employed in each of the numerical experiments? - The time complexity of the algorithm depends on the number of evaluations $R$. Can you provide a systematic evaluation of how the number of evaluations $R$ influences the performance of the framework? - What kind of issues do emerge in your framework if you don't split the observations into chunks? - How does the method perform for decreasing sampling frequency? - Does the method require the entire state of the system to be accessible to the observation model, or could also a lower dimensional projection of the latent state work?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
- The proposed framework is limited only to systems with diagonal covariances. - As far as I understand, they employed only a Gaussian observation model in all experiments. --- [1] Duncker, Lea, et al. "Learning interpretable continuous-time models of latent stochastic dynamical systems." International Conference on Machine Learning. PMLR, 2019.
Summary
This paper proposes a new inference method for latent SDE model. Due to the continuous nature of SDE, inference methods for latent SDE models are usually expensive and the cost would grow as the dataset goes larger or longer. Existing methods also would require differential equation solvers, which makes them unfriendly to GPU parallelization. The approach proposed in this work uses an amortized Markov Gaussian process as a variational posterior for inference over the latent SDE. The proposed variational objective allows for factorization over time steps and therefore enables alleviates the need to iterate over all time steps. Under such a framework, the cost at each iteration is only dependent on the number of Monte Carlo samples utilized for gradient estimation. The authors then empirically evaluate the efficiency of their method in a wide range of tasks, in which the proposed method shows much better performance in terms of computational cost, numerical stability, and convergence compared with existing methods.
Strengths
- The paper is well-motivated and the proposed approach, to the best of my knowledge, is very sensible and well supported by empirical evaluations. - The authors provide a thorough discussion of existing methods and include them in the experiments as baseline methods. - The paper is well written and the proposed method is presented clearly.
Weaknesses
Some technical details seem to be discussed too briefly in the main text, for example, the implementation of the encoder and the use of deep kernel for "interpolate between latent vectors". I can hardly understand the meaning without referring to the appendix. In Remark 2, the authors mention "make use of a nested Monte Carlo scheme along with stratified sampling to reduce the number of decoder evaluations and the variance of gradients." but the meaning of stratified sampling and nested Monte Carlo in this context is not discussed.
Questions
I am not familiar with the literature of latent SDE but I wonder whether the proposed variational posterior is more biased than existing approaches. Assuming computational cost is not an issue, would the baseline approaches be more favorable to the proposed method?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
Nothing I can think of other than the limitation discussed by the authors at line 182.
Summary
The present work proposes a time and memory efficient method to perform inference in a directed probabilistic model in the presence of a latent stochastic process with intractable posterior (path) distribution. Learning is performed using a variational Auto-Encoding approach, in which an approximate latent posterior process is selected from the class of Markov-Gaussian processes that are solutions to linear non-autonomous (Itô) SDEs. An evidence lower bound objective (ELBO) is derived that removes the need for numerical solvers when approximating gradients. Included experiments on toy and real world time series data approve similar performances compared to methods based on adjoint sensitivities but with two-orders of magnitude fewer evaluations and with a time and memory cost that scales independently with the amount of data, the total length of the time series and the stiffness of the approximate differential equations.
Strengths
I agree, a huge drawback of those existing models is the dependency on stochastic optimization methods that require backpropagading gradients through the numerical (stochastic) differential equation solver and hence rely on adjoint methods. It is the attempt itself tackling the road blocks by combining a variety of ideas that rely on model simplification, that I particularly appreciate. Especially under the lens of Bayesian inference it is recommended to question the need for unbounded expressivity in case of learning latent dynamics with continuous time models. This consideration is interesting on its own and adds progress to the community, e.g., regarding the applicability on low performance hardware. The authors propose improvement in time and memory cost by combining the following three major strategies: (1) simplifying the function class available for learning latent dynamics while reducing the expressiveness of the latent (SDE) models used, (2) exploiting the nature of sequential data such as time series that allow divide-and-conquer strategies such as the analysis of small sliding observation windows, and (3) introducing an optimized objective (i.e. an ELBO) by exploiting the reparameterization trick within the model used. This method builds on a novel combination of well-known techniques, and is original to the best of my knowledge. Further, the paper is well positioned to existing work, the content is well organized and, with a few exceptions, has no spelling or grammatical flaws. Empirical evidence is accessed through experiments on toy data as well as established benchmark data from real world applications; however, I would classify them small-scale.
Weaknesses
It is not the value of the approach I am concerning here, but the shortcomings related to the clarity of the submission and the technical soundness which I will address with examples in the following. 1. (2.1 Problem description) - I am missing a clear definition of the observation model, e.g., cf [Vrettas et al. 2015; Variational mean-field algorithm for efficient inference in large systems of stochastic differential equations ; Section II.A]. - Considering time $t_i \in \mathbb{R}$ instead of $\mathbb{R}_{+}$ is not wrong but unusal. - Minor ambiguities make understanding difficult in total: $dz$ vs $dz(t)$ and $\beta$ vs $\beta(t)$; the definition of $\theta$ is missing. - Is there any motivations for the stochastic variational inference approach? - I would $q_\phi(z(t)| x_i,\dots,x_{i+M}) \approx p_{\theta}(z(t)|\mathcal{D})$ as an local approximate posterior with respect to the time variable; are there necessary and/or sufficient conditions for an observed stochastic process that guarantee the validity of this approximation? 2. (2.2 Evidence lower bound) - (l. 96f) The following statement: "An added complication in our specific case comes from the fact that the latent state is defined by a stochastic process rather than a random-vector as is more typical." is poorly worded. You probably intend to say that, typically, stochastic variational inference considers vector-valued random variables to define latent states. But your approach requires function valued random variables, i.e., stochastic processes. What are those complications expected here? - It would be advantageous in combination with Eq. (1) to mention that the time dependency of the drift coefficient is necessary to account for a possible non-stationary observation process $x(t)$ that is driven by the latent stochastic dynamics of $z(t)$; cf. [32; Section 3]. - (Eq. (3)): To derive this result, the authors refer to unpublished work without stating the authors, cf. [14], but they included the reference in the supplementary material submitted. Notwithstanding the fact that I cannot call this good practice, this result is based on known facts and the derivation could have been included in the supplementary material of the present work, for example. Besides that, the derivation in [14] only concerns the case of a constant dispersion matrix $\Sigma$ with respect to time $t$ whereas $L(t)$, e.g., in Eq. (1) depends on time. - (Eq. (4)): Section A of the supplementary material indicates that this result follows by applying properties concerning a Lyapunov equation and by using the Kronecker sum. Following the arguments myself raised difficulties; in my opinion some given identities only hold in case of $A(t)$ is symmetric, i.e., $A(t) = A\top(t)$. Last but not least titling the result Theorem overstates its contribution. - (Remark 2): I would like to see the form of the prior SDE. In the setting of stochastic variational inference it is important to well design the dispersion matrix between prior and posterior SDE in order to guarantee that the KL divergence is finite in value. - Honestly I still don't get how the authors include the fact, that $m_\varphi$ and $\dot{m}_\varphi$ as well as $S_\varphi$ and $\dot S_\varphi$ are related through an ODE? Because all four are included into $r_\theta, _\varphi$ and e.g., Section G.1 only shows architecture design for $m$ and $S$. 3. (Experiments) - (Figure 2): Can you quantify "a similar validation accuracy"? Does the right plot only contain the ARCTA probabilistic prediction? Section F.1 does not contain the promised information, did you mean Section G.1? - (Section 4.2; l. 259ff.): The evaluation protocol is unclear to me; why are there multiple adjoint methods included and do you only optimize ARCTA for 2000 iterations? Can the quality be evaluated using error measures, e.g. MSE? A very similar experiment was presented in [13] proposing an adjoint method for SDEs. The results are promising. Can you please take these results into account? - (Section 4.3): From my point of view, the motion capture data set under consideration is quite delicate, since I know the shortcomings very well from my own experiments with it in the past. In my opinion it is the overall size of time series contained (16 training, 3 validation and 4 testing) that frequently causes severe overfitting. So I am still bothered with severe instabilities when reproducing the results from [30 & 13]. Therefore, I also view the results presented here with caution. There are 4 times 50 = 200 predictive posterior outputs by the model, evaluated on the test dataset; given four test time series you only provide three single dimension, that is strange. Secondly, I would like to know why the evaluation of the results is carried out in relation to the RMSE, requiring all benchmark results to be translated. Finally, I would like to reiterate my appreciation for the authors' interest in expanding the field of learning latent SDEs in the setting of stochastic variational inference. I'm sure that by working on the deficiencies mentioned, the work will gain in value and will deliver the desired contribution to the community in a new round.
Questions
- (l. 32f.) "... replacing ordinary differential equation (ODE) solvers with integration methods ..." But don't those numerical ODE solver build on integration methods? - In terms of the total number of time series samples included, the experiments dealt with are probably rather small-scale. To really highlight the added value of the proposed method, I would like to see large-scale experiments like the PhysioNet 2012 experiment in [12]. - Minor typos: - (l. 28) parareal -> parallel - (l. 54/67) time-series -> time series - (l. 66) Description -> description
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The authors address the limitations of the approach regarding the models expressivity compared to existing stochastic adjoint sensitivity methods in Chapter 3 of their work. I would appreciate some remarks on limitations concerning the recognition module, i.e., the encoder. As stated in Section D, data sparsity is only accepted to a certain extent and cf. (l 495f.) "It is worth noting that without this deep-kernel our approach struggled to achieve good validation accuracy on the datasets we considered". This suggests that the correct choice of the encoder is crucial and, on the other hand, raises doubts about the contribution of the latent stochastic dynamic. Is the proposed approach limited to interpolation and time series classification tasks or may we solve also tackle extrapolation problems? No evidence was found that the authors addressed any potential negative societal impact. Please make the effort to include this mandatory information in your work.
I thank the authors very much for their response and for the detailed discussion of my questions and concerns. (ad "The authors refer to unpublished work ...") My apologies for too strong a comment. What I addressed by "known facts" is the general form of the ELBO given in [14, Eq. (8)], which you gave references for. Indeed, new is the reparameterization for the expectations involved; presented starting at l. 468 in [14]. (ad $A$ symmetric) Given the following Lyapunov equation: $$AS + SA^\top = W.$$ Rewriting this equation and using vector representation gives $$(I \otimes A + A \otimes I) \text{vec}(S) = \text{vec}(W)$$ with $I$ denoting the identity matrix. Hence, we can obtain the following solutions $$ \text{vec}(S) = (I \otimes A + A \otimes I)^{-1}\text{vec}(W). \quad (\star) $$ Following the proof starting at l. 408, we have: $$\dot{S} = -AS - SA^\top + L\Sigma L^\top.$$ Simple algebraic transformation gives $$AS + SA^\top = L\Sigma L^\top - \dot{S} . \quad (\ast \ast)$$ Hence, $(\star)$ gives $$ S = \text{vec}^-1((I \otimes A + A \otimes I)^-1\text{vec}(L\Sigma L^\top- \dot{S})).$$ But we want to solve $(\ast \ast)$ for $A$. Therefore, transposing Eq. $(\ast \ast)$ we obtain $$S^\top A^\top + AS^\top = L\Sigma^\top L^\top - \dot{S}^\top.$$ Since $S = S^\top, \Sigma = \Sigma^\top$, we have $$S A^\top + AS^\top= L\Sigma L^\top - \dot{S}^\top.$$ Now, if $A$ is symmetric, i.e., $A= A^\top$, $(\ast)$ gives $$ A = \text{vec}^{-1}((I \otimes S + S \otimes I)^{-1}\text{vec}(L \Sigma L^ \top- \dot{S}^\top)),$$ similar to Eq. (16). This derivation was the reason for asking, if $A$ needs to be symmetric. Even I am convinced that my derivative is correct, apologies are given if you prove me wrong, here. (ad “Honestly I still don't get how the authors include the fact, that m and dm/dt ...") Sorry for my deficiency in explicitness at this point! I give it second try; in l. 41f. the authors state "... our approach removes the requirement of solving differential equations entirely." Now, following l. 102 - l. 104, necessary information for the approach, i.e., $m_\phi(t), \ S_\phi(t)$, are connected to ODEs. I am wondering, don't we still need to solve those?
Thank you for carefully reviewing the Theorem and for sharing your work. You are correct that the theorem makes the implicit assumption that the matrix $A$ is symmetric. We now state this clearly in the manuscript. This implicit assumption does not affect any results since we never parametrize $A$ directly – instead we always parameterize $m$ & $S$. Since $m$ and $S$ are parametrized as explicit functions of time we can efficiently compute $\frac{dm}{dt}$ and $\frac{dS}{dt}$ via automatic differentiation to estimate the ELBO. Note that under this parametrization the ELBO contains only a standard integral with respect to time and does not depend on differential equality constraints. This in contrast to prior approaches which implicitly parameterize $m$ & $S$ by directly parametrizing $A$ & $b$ which will require solving ODEs.
Dear authors, thank you for your continued efforts to address concerns and questions related to my review. Nonetheless, I keep my score! Overall, weaknesses in the clarity of the presentation were identified. The discussion about it has clarified this to some extent. However, I vote for further review after revision and encourage authors to seek resubmission. Sincerely 8z7R
Thank you for your response. My main concern on the linear SDE is partially addressed, and I have increased my score, and I apologize for missing that in the initial review. My other concern about the window parameter was also partially addressed, but I would recommend including additional discussion on this in the revision since this appears to be important for achieving good performance.
Decision
Accept (poster)