Modeling continuous-time dynamics on irregular time series is critical to account for data evolution and correlations that occur continuously. Traditional methods including recurrent neural networks or Transformer models leverage inductive bias via powerful neural architectures to capture complex patterns. However, due to their discrete characteristic, they have limitations in generalizing to continuous-time data paradigms. Though neural ordinary differential equations (Neural ODEs) and their variants have shown promising results in dealing with irregular time series, they often fail to capture the intricate correlations within these sequences. It is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle this problem, we propose ContiFormer that extends the relation modeling of vanilla Transformer to the continuous-time domain, which explicitly incorporates the modeling abilities of continuous dynamics of Neural ODEs with the attention mechanism of Transformers. We mathematically characterize the expressive power of ContiFormer and illustrate that, by curated designs of function hypothesis, many Transformer variants specialized in irregular time series modeling can be covered as a special case of ContiFormer. A wide range of experiments on both synthetic and real-world datasets have illustrated the superior modeling capacities and prediction performance of ContiFormer on irregular time series data. The project link is https://seqml.github.io/contiformer/.
Paper
References (64)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The authors propose a new method, a continuous-time transformer, ContiFormer, for irregular time series modelling. The proposed method extends vanilla Transformer to a continuous domain. In particular, the model incorporates the continuous dynamic modelling of Neural ODE with the attention mechanism of a transformer. Moreover, the authors show that many transformer variants designed for irregular time series modelling are a special case of the proposed ContiFormer. The authors evaluated the method on a continuous time function (spiral), irregular sampled time series for classification and irregular time series for event prediction. The results indicate that the proposed method obtains better results than the baseline models.
Strengths
Originality: The proposed work is original. The author's propose a novel continuous-time attention mechanism (CT-MHA). Quality: The authors have included also a comparison of complexity analysis of their method compared to other. The authors have tested the proposed method on multitude of datasets: irregularly-sampled time series for classification with increasing number of data dropped, as well as event prediction. Especially for the latter, the authors compare their method to existing state-of-the-art models for event forecasting (SAHP, THP) and obtain an improved performance. Clarity: Please see weaknesses. Significance: The authors provide a unique theoretical approach, and the obtained results demonstrate a benefit of the proposed method.
Weaknesses
Originality, related work: The related work is a bit convoluted. It would be better if the authors divide the related work into 3 subsections: Transformers for time-series modelling, Transformer for irregular time series, Continuous time models (NODEs). The literature of RNN is not as relevant to this work as it builds upon transformers and NODEs. With respect to NODE related work: Line 43-45, the authors say 'the recursive nature of Neural ODEs can lead to cumulative errors if the number of iterations is numerous'. This is not precise. The cumulative error is dependent of the solver used, and there are numerous works which by enforcing, for example, conservation of energy, keep the error tolerance low, for example, https://arxiv.org/abs/1909.12077. Moreover, by selecting difference step size, solver type (adaptive or fixed) one can control/adjust for the numerical error (as mentioned in https://arxiv.org/abs/1806.07366). Please rephrase. With respect to Transformer related work: In line 47-48, the authors mention that due the fixed time encoding or learning upon certain kernel functions the resulting models struggle to model complex data patterns in real-world practice. However, I do not see how this claim is supported. Even in the appendix, for given percentages of observations dropped (50%) mTAN model outperforms ContiFormer. Therefore, the claim is not supported. Please rephrase. The proposed work differs from the existing work in the field, however, some of the claims made about the previous work are not well supported in the current version of the paper. Quality: The proposed continuous time attention mechanism, eq. 3, is unclear, as the underlying functions are not clearly defined. The authors' mention that previous transformer works for irregular time series are special cases of their model, however, the proof of this only mentioned in the Appendix, I would recommend moving the core parts of the proof to the main part of the paper. Lines 281 - 283, the authors say that ContiFormer consistently outperforms all the baselines on all three settings, while in the appendix it can be seen that there are also instances were the baseline models outperform ContiFormer, therefore, I would soften the claim. For lines 303-304, I am missing a comparison to methods that are more aligned with the task at hand. The authors compare against TST and mTAN, however TST model is focused on multivariate time series (not decomposition of different dynamic trends), while mTAN although also designed for irregular time series modelling the paper's focus is on temporally distributed latent representations. For extracting temporal dynamics/trends of the data, there are other works, Autoformer (https://arxiv.org/abs/2106.13008), FEDformer (https://arxiv.org/abs/2201.12740) that would be perhaps a more fair comparison to the present method. Clarity: The submission is not clearly written, thus negatively affecting the understanding of the proposed method. Please clarify for all functions, v(), k(), q() the input, output dimensionalities, as well what these functions are. As this is unclear, it is hard to evaluate what exactly the inner product of the two function spaces is computing. Line 147. Imprecise phrasing: the authors mention that self-attention involves calculating the correlation between queries and keys, which is incorrect. The computation represents an inner dot product, which is not a correlation metric but rather a similarity measure, please rephrase. The main figure (figure 1), is also not immediately clear for the reader. Would suggest to adjust the figure. It is hard to follow all the colors and how they correspond to which input. In addition, for the NODE model compared with, as well as for the ODESolver used in author own work it is missing what kind of solver is used: adaptive or fixed step? This could greatly affect the performance, hence, it is a crucial implementation detail. In the appendix mTAN model has the wrong reference. The reference is for the Autoformer model, please correct. Lastly, for the experiments is it not clear what is the input/ouput data length for the model. Overall, the technical details of the paper are not clear. As this is the main contribution of the paper, it is essential that it is clear so that the extensive experimental results can be objectively evaluated given the method at hand. I would suggest the authors to rewrite this section.
Questions
In line 83, the authors say that previous models 'are insufficient in capturing input-dependent dynamic systems'. How is this claim supported? The previous work also is also condition on the input data. In line 99-100, the author's say 'ContiFormer, employs a non-autoregressive paradigm', however, the vanilla transformer decoder at inference time is an auto-regressive model, can you clarify this? Furthermore, to perform the ODESolver step, you perform auto-regressive P steps? Eq. (1), the formulation is unclear and incorrect: the author's define both the ODE as well as the continuous latent variable with the same parameter. Furthermore, if k_i(\tau) is the ODE, what is f()? In line 136, could you please clarify why you have modeled each observation as a separate ODE? As showed in https://arxiv.org/abs/1806.07366 NODE models can model irregular time-series with a single ODE specification, even more so, close by observations most likely follow the same ODE. In line 136, could you clarify if the ODE is a MLP or any other architecture? Line 138: would suggest the authors to adjust this assumption/claim: from a dynamical systems perspective a future event cannot affect a past event. Vast majority of works modelling dynamics are often based on Markov property. line 144 in text you mention that q(t_i) is a cubic spline approximation, however, in the appendix you clarify that linear interpolation is used due to the lower computational complexity. please clarify the approximation used in your proposed method. What is the dimensionality of Q_i? Eq. (4) Based on the introduced notation it seems like the introduced computation ends up being a matrix (Q_i), vector (k_i() \in R^d) product. Where the Q_i is an approximation of the process via cubic spline, and k_i() is the latent trajectory by an ODE, this would imply that the proposed mechanism measure the similarity of the learned approximation of the ODE to the approximation by the cubic spline function. Is this correct? Eq. (7) For multi-head attention, you also apply a linear transformation to the latent continuous state, q(),v()? How sensitive is the method of using a different approximation method instead of cubic spline? Does each latent continuous trajectory span only two adjacent time points t:[-1,1]? Meaning that depending input length L (data points), you would have L independent latent ODE trajectories? It is unclear from the text. The authors mention that their attention mechanism is measuring the correlation between different observations (line 298), therefore, I would suggest the authors to compare their work to Autoformer (that measures correlation between different time series) or Fourier-based (FEDformer) attention mechanism. Have such comparisons been made?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
1 poor
Contribution
3 good
Limitations
The authors have not explicitly addressed the limitation of the presented work. Based on the text, the proposed method models each observation with a separate ODE function. This, however, seems to be sub-optimal as data points close in time probably follow the same underlying dynamics function, furthermore, the authors have not explained why such a design choice would be necessary, as by default Neural ODEs can account for irregularly sampled time series. Moreover, it would be nice if the authors would have addressed (as an ablation) the model's performance on regular time-series data compared to existing state-of-the art methods.
Summary
This paper introduces the ContiFormer, a novel continuous-time Transformer model designed for handling irregular time series data. In this model, the keys, queries and values are vector-valued functions indexed on time. Each input observation gives rise to a key function, given by the solution of a Neural ODE started at an initial condition determined by the observation. While a similar model is adopted for the value functions, the query function is obtained using natural cubic spline interpolation of discrete query values. By replacing dot products (between discrete queries and keys) with an $L^2$ inner product, a continuous-time attention function is derived. The continuous-time output can be evaluated on a discrete time grid allowing for the stacking of such layers. To reduce the computational time, the paper leverages a reparametrization trick, which makes it possible to solve simultaneously multiple Neural ODEs on different time intervals. The ContiFormer model is evaluated on various time series tasks and datasets demonstrating its effectiveness in handling irregular time-series data.
Strengths
- Given the empirical success of transformer models on machine learning tasks involving sequential data, enhancing their capability of effectively handling irregular time series data is an important research question. - This paper proposes an elegant solution based on Neural ODEs and other ideas from the field of neural differential equations such as the use of cubic splines in Neural CDEs. - The paper is clearly written and easy to follow. - The empirical evaluation is well conducted, extensive and provides a compelling demonstration of the superior performance of the ContiFormer model.
Weaknesses
- The theoretical results are deferred to the appendix. I think the main theorem (Thm 1 in Appendix) should at least be formally stated and further commented on in the main paper. This would better motivate and support the modelling choices. - The paper is a bit repetitive at times (l100-102, l108-117, l175-178) while some components might deserve further explanation: 1. the main theoretical result; 2. although the $i^{th}$ key and value functions are well defined at $t<t_i$, it could be further commented by expanding l138, and perhaps adapting Fig. 1 with $t<t_i$; 3. the rationale for modelling the key and value functions differently from the query function. - As mentioned in the related work section, a continuous-time attention model based on Neural ODEs has previously been proposed in [10]. While several baselines are considered in the experimental section it is surprising that this model is not included without a justification. - The attention model is not fully formulated in continuous-time, in the sense that it is expressed as a sum over the number of value functions which depends on the number of observations. A natural thought of experiment for a continuous-time model is to replace the input sequences with continuous paths. Is there a way to define the ContiFormer on such paths?
Questions
- Why did you choose to model the key and value functions differently from the query function? Can the ContiFormer handle partially observed data, given how the key and value functions are defined? - Do you have an intuition/explanation for the fact that $P$ can be chosen to be very small (l 224) and the insensitivity of the ContiFormer to the tolerance error (Appendix F12, F13)? - Related to the comment above about the continuous-time formulation, does the ContiFormer offer any advantage for handling both irregular and long time series? - Although computational complexities are provided, would it be possible to report the actual time it takes to train and evaluate the model?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes, some limitations have been discussed.
Summary
The paper describes a continuous time extention of the transformer architecture using ODE blocks to propagate the effect of each observation individually through time. For computing attention values inner products between functions are used, where in the implementation the resulting integral is approximated. The paper demonstrates the method's application to irregular time-series data and in temporal point processes in different tasks including interpolation, extrapolation, classificaton.
Strengths
The paper gives a continuous-time extension of transformers capable of handling irregularly sampled continuous time series, and point processes. Computational complexity is assessed in details which is quite important for practical application, and helps the reader to better estimate the real life cost of scaling the methods to real life problems.
Weaknesses
My main problem is with the missing statistical rigor during evaluation. More specifically: In Table 3 the authors report averages over 20 datasets. I assume these datasets have different sizes, complexity, dataset imbalance etc., therefore it is hard to rigorously evaluate the average metrics.There are dataset-by-dataset level results in the appendix but there is still no standard deviation. Similarly in Table 4, no standard deviations. Probably 3 std differences are significant, but with 3 repeats I am not sure sure 2 std is very convincing. (For just a very rough estimate how big delta is needed check a 3 sample unpaired t-test table)
Questions
Please give standard deviations and mark significant differences. Also it is interesting to see that NeuralCD is outperformed by ODE-RNN here for the 30 and 50% dropped case. As there is result provided for CharacterTrajectories in the appendix (this dataset was used in the original NeuralCDE paper) I compared the results. format: present paper, vs, Kidger et al. | Method | 30% dropped | 50% dropped | 70% dropped | |-----------|-----------|-----------|-----------| GRU-D | 0.9325 vs. 0.942 | 0.8872 vs. 0.902 | **0.8433 vs. 0.919** | GRU-deltaT | **0.8558 vs. 0.936** | 0.9088 vs. 0.913 | **0.8496 vs. 0.904** | N CDE | **0.9276 vs. 0.987** | **0.9285 vs. 0.988** | **0.9241 vs. 0.986** | ODERNN | 0.9415 vs. 0.954 | 0.9506 vs. 9.960 | 0.9471 vs. 0.953 | These are serious differences on the same dataset. I can maybe explain away the case with Neural CDE as the creator of the method may be able to tune it better (this is in fact the main drawback of this entire comparison table idea we do, but this is a problem we are not going to solve here :) But results of several other baselines are quite different. This is concerning. "We generated a dataset of 300 2-dimensional spirals, sampled at 150 equally-spaced time points" <-- what is the temporal dimension? is this CD spiral a parametric curve where time is the parameter? Are you using vanilla NeuralODE in this experiment? I find it hard to understand why you get the interpolation behavior from NODEs as it is visible in Figure 2. The interpolation looks like piece-wise linear. Can you provide interpolation results with ODE-RNNs and neural CDEs? Do you see a way to avoid resampling between layers? It would be useful and quite interesting to see some discussion about the apparent contradiction in using transformers (long range interactions) and ODEs. ODEs operates with derivatives and their solutions are locally described. The assumptions of a smooth differentiable solution seems to stand in contradiction with long range interactions unmediated by values in between, or in a case of a latent-ODE decriptions by latent system states. Can you formalize the properties of systems (e.g. the mentioned stock market time series) where you expect this type of modelling give the most benefit? Minor: "To overcome the discrete nature of RNNs, a different strategy involves the exponential decay of the hidden state between observations, governed by modifiable decay parameters [5, 9, 37]." [9] is the reference for the Neural ODE paper from Chen at al., it does not apply exponential decay. typo: denoted vs donated Please stress the difference between irregularly sampled data and temporal point processes eg. in [37] more clearly (missing data vs. events). NOTE: the authors clarified some points, provided new results, and we identified a misunderstanding about the Neural ODE figure. The authors used Latent-ODE, but cited a previous paper in the table: that lead me believe that they fitted a Vanilla NODE to the trajectory. I raised my score by one.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
No systematic discussion on limitations provided.
Summary
The paper proposes a new deep learning model called ContiFormer to model continuous-time dynamics on irregular time series data. The paper argues that existing methods such as recurrent neural networks (RNNs), Neural Ordinary Differential Equations (ODEs), and Transformers have limitations in modeling continuous-time data and fail to capture intricate correlations within these sequences. The proposed ContiFormer model extends the relation modeling of the Transformer model to the continuous domain and incorporates the modeling abilities of continuous dynamics of Neural ODE with the attention mechanism of Transformers. Both numerical and theoretical justifications are provided. Different from the standard Transformer model, the proposed model is parameterized on the first-order derivative of $q$, $k$, and $v$. The real $q$, $k$ used in the attention part is approximately computed with finite sum (e.g., via Runge-Kutta 4th (RK4) order ODE solver). By restricting the function form/class of the first-order derivative on some smooth enough kernel, the proposed model could yield better interpolation/extrapolation ability. I have two concerns about the proposed model. The first one is the usage of ODE solvers may introduce computation overhead. In Table 1, the order of computation cost is already $O(N^2 \cdot P \cdot d^2)$, where $P$ is the iterative number in RK4 solver, and in my understanding, $P$ should be 4. In L580 at Appendix B. the RK4 ODE may require 80 forward passes to finish one of the RK4 round. It implies the order of computation cost for a one layer attention model should be $O(N^2\cdot 80\cdot d^2)$, which can be very computationally expensive even for models with moderate size. Moreover, if the proposed model has more than one layer (e.g., 2 layers), in order to finish the second layer's attention computation, we will need first to finish the computation of the first layer with $O(N^2\cdot 80\cdot d^2)$ cost. After that, every second layer's computation will still involve the forward pass of the first layer, which becomes $O(2\cdot N^2\cdot 80\cdot d^2)$. Therefore, for the $M$ layer attention model, the total computation cost could be $O(M^2\cdot N^2\cdot 80\cdot d^2)$. The second one the underlying construction is kind of similar to the Rotary Position Embedding in [1]. In this paper, the kernel trick is used to facilitate the computation (e.g., Eq. (27)- Eq. (32) in Appendix C). Based on the current presentation, the kernel function is only time point $t$ dependent but independent on the detailed $q,k$. Given the choice of the kernel function in Eq. (30), the effects of introducing this kernel function look like just adding a rotary positional embedding. At present, the authors have not provided sufficient evidence to demonstrate the contribution of their model, which may not be compelling enough for top machine learning conferences like NeurIPS. Despite this, the reviewer is willing to reconsider the decision after the authors' rebuttal. Overall, the paper presents an interesting approach for modeling continuous-time dynamics, but the issues raised above need to be addressed before its acceptance for a top-tier conference. Reference: [1] Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864.
Strengths
The paper proposes a new deep transformer model incoperating the continuous-time dynamics on irregular time series data. Both numerical and theoretical justifications are provided.
Weaknesses
Please see my comments in the Summary section.
Questions
Please see my comments in the Summary section.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
1 poor
Presentation
2 fair
Contribution
1 poor
Limitations
The authors include a section on the limitations of the proposed work in Appendix G at Page 30.
Summary
This paper introduces ContiFormer, a continuous time transformer-based model that leverages parallelism and can handle irregularly sampled data well, thereby removing the need to transform these datasets into discrete uniform bins. This set-up incorporates the continuous dependence on the data from differential equation based neural networks. Finally, an extensive range of experiments on irregularly sampled data are performed to show promising results for ContiFormer.
Strengths
This paper presents an original architecture class that can encompass many existing models. The discussion on attention combined with the continuous time set-up is limited and therefore this paper will be a significant contribution in this area. The paper is generally well-written and extensive numerical experiments were performed.
Weaknesses
Whilst in general a fairly well-written paper, key parts of the model can be made clearer. For example, it is not clear for line 137 how the key and values are initialised. There is also an assumption that is not explained "we assume that every observation has influence on the dynamic system even before its occurrence." Combined with the fact that the query uses natural cubic spline, does that mean the model is not causal and cannot be applied in an online fashion (see for example [Morrill et al 2022 On the Choice of Interpolation Scheme for Neural CDEs])? This paper does not appear to be comparing with the state-of-the-art models. To my knowledge, the S5 model (Simplified State Space layers for sequence modeling by Smith et al 2022), appears to already outperform quite a few of the baselines chosen in this paper, specifically, mTAN, ODE-RNN, GRU-$\Delta$t for irregularly sampled pendulum exercise. Therefore this should be added into a benchmark. The pendulum regression task used in Smith et al. seems to be used in quite a few irregular sampling task papers, and therefore it would be of interest to see how the ContiFormer performs in this case. Whilst there is an extensive range of experiments, the full results with the standard deviation across the 3 repeats for Section 4.2 and Section 4.3 seem to be missing. These seem quite key to back up some results, particularly for Table 4, where the standard deviation is referred to. Without this, it is difficult to quantify "overall statistical superiority of ContiFormer" as claimed. There does not appear to be any evaluation on regularly sampled datasets. Whilst I appreciate that the marketed strength of this method is in irregularly sampled data, it would be useful to see what the performance is on regularly sampled datasets (against benchmarks) and whether one can just choose this model to be applied on all timeseries.
Questions
Since ContiFormer requires $N^2$ ODEs to be solved, how does the $P$ relate to $L$ in Table 2? Are these on similar orders? It seems like the neural ODE performs much worse than expected even for the interpolation compared with the example we see on the diffrax website https://docs.kidger.site/diffrax/examples/neural_ode/. Is this because of the regular vs irregular sampling issue? Figure 4 in appendix does not seem to be plotting the same curve at each row? Neural ODE clearly is on a different scale or a different curve in the third row quite clearly? Minor point: Line 119: donated -> denoted
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
Limitations were not discussed in the paper. Given that the model makes use of cubic splines, the literature suggests this is will not be causal. Performance on regularly sampled dataset not clear.
Answer to Bebuttal
I would like to thank the authors for their time to answering my (and other reviewers') questions. **W1**:Being "prevalent common practice " (which is hard to assess based on 2 papers) does not automatically mean good practice. However, I appreciate the fact that if it is reported like this in previous works, you followed this way. Also you report individual results (now with error bars), which make it possible to check these. I agree with **jYof** on that, claiming "overall superiority" is a bit strong based on "average" superiority. The provided error bars and significance tests are appreciated. **Q1**: This may explain the difference. I am not sure however that rerandomizing for all test is the better approach. In a real life scenario the data is given, so there is no possibility to execute these repeats, while random restarts can be performed even in a real life scenario. So, keeping the data fixed is more lifelike. But your evaluation is still valid of course. **Q3 (General Q1)**: **[my major remaining problem]** We (you, **2E9G** and myself) would be in an easier position if you would have added code to the supplementary. You mention you use the same code as the original. Can you point to this original (git of original paper's repo, specific file describing the experiment setup most similar to your special experiment setup with the spiral - Be careful **not** to link own repo due to blind review!) My problem is, even if you use the same model code, it is still possible that the model used differently during experimental setup. Can you describe how you train and interpolate with NODE in a few points? Like: "I take the 2D time series data (x(t), y(t)), initialize a NODE with 2 hidden state, with XY architecture.., I propagate back XY error (MSE I assume)".. and so on. **Q5: [second remaining issue]** What you wrote in your reply, you already properly stated in the paper. I understand this. My question was: isn't there a contradiction here? You write for example for **jYof**'s question 1 that you assume ODE like dynamics is fundamental in your datasets, this is why transformers are not enough. Now this type of dynamics are by definition smooth and local (in time), therefore why to expect that the Markov assumption is violated in your datasets in the same time, when you assume an ODE structure is a good prior. Now I do not claim this type of data does not exist. I just asking: Can you give a convincing example?
Rely to your remaining problems
Thank you for your quick reply and explanation. **Reply to W1**: We agree that not all common practices meet standards. Error bars and significance tests will be put into our revised version. We are also committed to tempering the assertiveness of our claims. **Reply to Q1**: We understand the importance of realism in fixed data outcomes. Your input has spurred us to enhance our evaluation methodology. **Reply to Q3**: > **Point to the repo with a similar experiment setting.** We follow the code from [link](https://github.com/rtqichen/torchdiffeq/blob/master/examples/latent_ode.py) as mentioned in **General Response Q1**. For spiral data generation, following Line 31-105, where we modify Line 86-98, to generate irregularly-sample data and add noise to parameters $a$ and $b$ as claimed in the paper and Appendix E.1.2. For model training and implementation, we follow the code in Lines 181-194 to calculate the loss function and use the same code as Lines 108-159 to build the Neural ODE model. > **Describe your special experiment setup of NODE?** The training flow of Neural ODE is as follows: given a batch of 2D spirals, i.e., (200, 50, 2) for (batch size, sequence length, feature dimension). We use a LatentODEfunc with 3 hidden layers with ELU activation function and set the hidden dimension to 20. The model contains a RecognitionRNN to encode the input, followed by ODE to reconstruct the hidden state trajectory. Finally, a decoder with 2 hidden layers and ReLU activation is adopted to obtain the output. We train Neural ODE with ELBO loss as stated in Appendix E.1.3 and keep it the same as the original code. To better address your concern, we dig into the issue of why Neural ODE generates piece-wise linearity in our data setting. The main conclusions are as follows: 1. Sampling Strategy: Line 96 of the original implementation used regular sampling. Oriented toward irregular time series, we adopted irregular sampling. Unexpectedly, this may reveal linear piece-wise patterns. The impact of sampling is noted by Reviewer 2E9G (Q2). 2. Random Seed: We ran our experiments 10 times. Notably, Neural ODE may yield smooth outputs in some random tests. We suspect this might stem from training uncertainty, potentially influencing the linearly piecewise behavior. We are ready to provide the visualization by anonymous link (not recommended by NeurIPS official instruction email.) Overall, among all the random runs, we observe that the prediction result of Contiformer can outperform Neural ODE with p-value $< 10^{-6}$ in significant tests. Moreover, we alter these data settings to enhance the evaluation of irregular time series modeling and to effectively showcase method performance, as detailed in the General Response. We hope that all these explanations would help address your questions. **Reply to Q5**: > Any contradiction here? Can you give a convincing example? Thank you for your further explanation! We first express our understanding of your mentioned "contradiction". On one hand, ODE assumes data following Markov property and models the dynamical changes, which is "smooth and local" as you mentioned. Meanwhile, Transformer assumes that the data share both short-term and long-term dependencies, which seems "rough and global" and violates Markov assumptions. If it's correct, we kindly provide some examples below. 1. From a practical view, stock prices initially exhibit smooth auto-regressive patterns but are also affected by business conditions and market events. For instance, large technical companies' stock prices (e.g., MSFT and GOOG) display consistent evolving trends yet are impacted by short-term and long-term events such as the release of large language models, etc. Similarly, traffic data display both local similarities within short periods and global seasonality over longer periods (Fig. 4 in [1]). That intuitively explains why the time series will follow both the Markov property that ODE models and the global influence properties that Transformer models. 2. From the literature view, one of our baselines, the Transformer Hawkes Process [2] assumes the Hawkes process which is locally defined on the infinitesimal time interval. Besides, [2] also claims the shortcomings of Hawkes process assumption that "fails to capture complicated short-term and long-term temporal dependencies". Thus, they also incorporated other approaches leveraging long-term dependencies to close the gap. In conclusion, in our paper, we want to construct a system that is smooth in time span (like ODE-based methods) while capturing long-term dependency using powerful Transformer architecture. We hope our provided examples can address your questions. And we will also refine our description of our assumption part in our paper correspondingly. [1] HetETA: Heterogeneous information network embedding for estimating time of arrival. KDD 2020. [2] Transformer hawkes process. ICML 2020.
Will raise my score one point.
First, I would like to apologize, that I did not spot the link in your General answer when I first read it. However we had a bit of a misunderstanding, you used **Latent-ODE** from Rubanova et al, to make that spiral figure. What I mean by **Vanilla NODE**, is really that, just and end-to-end trained NODE from Chen et al. You can just fit a NODE as it would be a regular ODE just now with a black box form, to the trajectory, no EncoderRNN. In this case i would have been surprised by this behavior. In case of Latent-ODE we are on the same page, the result is of course perfectly possible. **Note that** in Table 2 in the manuscript you cite [9] Chen et al. not Rubanova et al.. Different papers, mostly same authors. Please clarify this. On the examples: Your event based disruption example on stock market is true, but it still not tell me why Transformer would solve this, it is not that this event can be predicted by attending the past, it is a disruption hard to predict. Still, we managed to clear up some of my concerns, and your method can be of interest for the community. I respect the work you spent answering my concerns. I will raise my score by one.
Reply to your remaining problems
> **Q1: You used Latent-ODE from Rubanova et al, to make that spiral figure. In Table 2 in the manuscript you cite [9] Chen et al. not Rubanova et al.. Please clarify this.** Just as you understand, we use the Latent ODE model with an RNN encoder and Neural ODE decoder to make that spiral figure. However, it is worth noticing that the Latent ODE model in our experiment is truly from Chen et al. [9]. Specifically, in Section 5 of [9], titled "A generative latent function time-series model", the author introduce the Latent ODE model. Moreover, we have conducted a comparison experiment on the Latent ODE model from Rubanova et al. [42], where it replaces the RNN encoder with an ODE encoder. This model is named ODE-RNN in our paper. The visualization result of this model from Rubanova et al. [42] has been shown in the attached PDF file (Fig. 2 ODE-RNN) in **General Response** part. We acknowledge that the use of "Neural ODE" in Table 2 is indeed confusing, we will replace the term with "Latent ODE" in the revised manuscript, but there should be no error in the citation as we understand. > **Q2: You can just fit a NODE as it would be a regular ODE just now with a black box form, to the trajectory, no EncoderRNN.** Thanks for your valuable suggestion. It is indeed possible to fit a regular ODE with an initial point (possibly the first observation) and reconstruct the latent trajectory by an ODE solver. We have also conducted extensive experiments to compare Neural ODE (w/o RNN enc), Latent ODE (w/ RNN enc), and our method Contiformer. The results are listed below with $\alpha=0.02$ and $\beta=0.1$, which follow the settings of Table 2 in our paper. From the new result table below, we can conclude that Neural ODE w/o RNN encoder yields poor prediction results, especially for *extrapolation*. Besides, from the visualization result, we can indeed observe a smooth outcome from Neural ODE. | Model | Interpolation (RMSE) | Interpolation (MAE) | Extrapolation (RMSE) | Extrapolation (MAE) | |-|-|-|-|-| | Neural ODE (w/o RNN enc) | 0.0140 ± 0.0010 | 0.0138 ± 0.0010 | 0.0259 ± 0.0021 | 0.0269 ± 0.0020 | | Latent ODE (w/ RNN enc) | 0.0209 ± 0.0022 | 0.0195 ± 0.0025 | 0.0159 ± 0.0005 | 0.0152 ± 0.0005 | | Contiformer | **0.0049 ± 0.0006** | **0.0052 ± 0.0006** | **0.0064 ± 0.0009** | **0.0065 ± 0.0008** | > **Q3: The example in the stock market still does not illustrate why Transformer would solve this, it is not that this event can be predicted by attending the past, it is a disruption hard to predict.** Thank you for your inquiry. Predicting disruptions in the stock market is undoubtedly challenging, especially considering the underlying complex market behavior. Nevertheless, we believe that the sequence patterns before the event may share some similarity of correlation to those historical sequences before those historical events, which may somehow help future prediction. Furthermore, we also provided another illustration of traffic analysis in our previous comment. We highlight the importance of local similarities within short periods and the influence of global seasonality over longer periods. In Section 4.3, we empirically showcase Contiformer's strength on the Traffic dataset, underscoring its ability to effectively capture these intricate and compound dependencies. This highlights the value of constructing a system that spans time smoothly and meanwhile still capturing long-term dependencies. This approach may hold promise for modeling irregular time series effectively in such a scenario. [9] Chen, Ricky TQ, et al. "Neural ordinary differential equations." NeurIPS 2018. [42] Rubanova, Yulia, Ricky TQ Chen, and David K. Duvenaud. "Latent ordinary differential equations for irregularly-sampled time series." NeurIPS 2019. *** In short, we sincerely thank you for raising the rating score regarding our paper. We are also open to further comments and suggestions for improving our paper. We will follow your suggestion to improve the quality of our paper.
Thank you for the detailed response to my questions voiced. I appreciate the authors effort to perform additional comparisons with Autoformer and FEDformer, and clarification on the model details, as such I will increase my score to 5.
Thank you for your reply!
We express our sincere gratitude for your thoughtful response and specific recommendations regarding our paper. Your insightful suggestions and comments have significantly contributed to enhancing the quality of our work, encompassing aspects such as related research, theorems, and technical intricacies. We sincerely thank you for raising the rating score regarding our paper. We are also open to further comments and suggestions for improving our paper.
Thank you to the authors for responding to some of the points that I raised and my questions, in particular, for doing further experiments and some comparisons with S5 and testing on the pendulum task in such a short time period. With regards to online applications, I would still like to check its practical implementation. Of course I see that the interpolation methods can be changed and you have done an ablation study responding to **jYof** to show that the results may not be so sensitive. In terms of your parallelization, it seems that each irregular time period is rescaled to $[-1,1]$ first and then functions are suitably transformed. The point of this seems to be "decoupling" the problem along the time axis, is there anything to ensure continuity in time?
Rely to your remaining problems
> **Q1: With regard to online applications, I would still like to check its practical implementation.** Thanks for your inquiry. We acknowledge that our current implementation using natural cubic spline may suffer in online applications. However, we can make the following change in our codes to support interpolation methods like Cubic Hermite splines with backward differences [1], as you suggested. For implementation, given the inputs matrix $Q$ and the corresponding time for each observation $T$, we will invoke `coeffs=torchcde.natural_cubic_coeffs(Q, t=T)` to construct the continuous path. Therefore, in the case of online applications, we can replace it with `coeffs=torchcde.hermite_cubic_coefficients_with_backward_differences(Q, t=T)`. > **Q2: It seems that each irregular time period is rescaled to $[-1, 1]$ first and then functions are suitably transformed. The point of this seems to be "decoupling" the problem along the time axis, is there anything to ensure continuity in time?** Thanks for your great point! The rescale operation used in Eq. (9) can enable the parallel computation of the calculation over time. However, this operation may induce additional numerical errors. Therefore, unfortunately, there is no guarantee to ensure continuity in the time axis. However, we would like to highlight that there are two factors that can help control the error, and therefore relieve influence of the continuity issue of the latent trajectories in time. 1. Error in ODE Solver: When utilizing a numerical ODE solver, inherent errors may arise during the solution process. Nevertheless, the mitigation of such errors is attainable by adopting a smaller step size or error tolerance, the errors can be effectively controlled [2]. 2. Error in numerical approximation. As shown in Eq. (9) of the paper, we use a numerical approximation method (e.g., Gauss-Legendre Quadrature approximation) to approximate an integral from $[-1, 1]$. However, the approximation error can be bounded. Besides, we can increase $P$, which is the number of intermediate steps for integral approximation, to achieve a lower approximation error. Overall, as we discussed in Appendix D Line 658-660, the output of ContiFormer can be considered “continuous” only if we overlook the approximation and numerical errors in the ODESolver. Also, our framework allows the user to trade off speed for precision. Furthermore, the empirical findings presented in Section 4 illustrate that Contiformer consistently delivers commendable performance across diverse tasks with numerical error. In the revised manuscript, we will provide a more detailed discussion about the continuity of the Contiformer. We hope these explanations have solved your concern. [1] On the choice of interpolation scheme for neural CDEs. TMLR 2022. [2] Chen, Ricky TQ, et al. "Neural ordinary differential equations." NeurIPS 2018.
Thank you to the authors for addressing my remaining questions. I maintain that this is a paper with a novel method which will have moderate-to-high impact and as such should be accepted to neurips. My score remains at 6.
Thank you for your reply!
We sincerely thank you for your positive comment regarding our paper. In our final version, we are committed to integrating supplementary experiments to more comprehensively showcase our model's capabilities. Moreover, we are dedicated to augmenting the lucidity of our explanations to ensure a clearer presentation.
Thanks for your comments.
Thanks for the authors' comments. After reading the authors‘ feedback and other reviewers' comments. Most of my concerns are addressed and I'll increase my rate accordingly. My remaining concern is mainly about the computation cost, the usage of RK4 solver can be viewed as an implicit layer that is typically very time composing for large-scale problems or large-scale model configurations. However, given the capacity of modern deep-learning hardware and the scale of real-world time series problems, I believe the computation cost of the proposed method won't be a major issue. It would be great if the authors could include a stress test in the final version on the computation cost v.s. problem scale/model size and discuss the *sweet zone* of the proposed method.
Rely to your remaining problems
> **Q1: The usage of RK4 solver can be very time consuming.** Thank you for your insightful feedback. Utilizing an ODE solver, such as the RK4 solver, requires performing multiple forward passes, which can lead to computationally demanding operations. In our reply to reviewer GCXG, we can see that the time for training and validation is roughly $6$ times as vanilla Transformer model, which is somehow acceptable, with RK4 step size equal to $0.1$ and hidden dimension as $8$ in our experiment. We can also leverage some other techniques and packages to accelerate the ODE solver procedure, like diffrax as mentioned by reviewer 2E9G. These efforts are orthogonal to our work and we leave that as one important future work. Thank you for your suggestion. > **Q2: It would be great if the authors could include a stress test in the final version on the computation cost v.s. problem scale/model size and discuss the sweet zone of the proposed method.** Thanks for your valuable suggestion. Although we have measured the time cost on a single dataset, a stress test is necessary for a complete evaluation of the time cost and scalability of our model. In our revised manuscript, we are committed to addressing this aspect by incorporating a stress test that assesses how our proposed method performs across various input lengths and hidden sizes. Also, we intend to report the precise training/inference time for a wide range of datasets. *** In short, we sincerely thank you for raising the rating score regarding our paper. We are also open to further comments and suggestions for improving our paper. We will follow your suggestion to improve the quality of our paper.
Reply to authors rebuttal
Thank you for thoroughly addressing my questions and for your comprehensive rebuttals that are substantiated with further experimental results. Your rebuttal work will certainly enhance the quality of your paper and increases my confidence in my evaluation.
Decision
Accept (poster)