Non-adversarial training of Neural SDEs with signature kernel scores

Summary

This paper proposes a signature kernel-based Neural SDE by using signature kernel scores. The proposed method eliminates existing state-of-the-art methods' mode collapse and instability using adversarial training.

Strengths

1. In all experiments, the proposed method outperforms existing methods. 2. The explanation in chapter three is clear and reproducible because the code is also provided.

Weaknesses

1. This paper's main contribution includes using signature kernel scores for the discriminator, but there needs to be more motivated to use signature kernels. 2. There is no sub-caption for each of Figure 1, so it is unclear which generator it is.

Questions

1. Is that why you chose the signature kernel among the various kernels? Also, if different kernels are adaptable, I would like to know the results of your experiments with other kernels. 2. Which generator does the sub Figure in Figure 1 correspond to? Minor corrections 3. The unconditional_nsde.ipynb included in the code provided need to be corrected? ``` #cond_ = data_type == "rBergomi" cond_ = data_type == "gbm" ```

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Summary

In this work, the authors proposed to generate time-series using neural stochastic differential equations using the scoring rules-based training objective which is on signature paths computed from signature kernels. Combining the idea of generative adversarial network, this work also employ the generator-discriminator pair, but solve them as a system of linear PDEs where the adjoint method is applied to reduce the memory cost. Experiments are conducted accordingly on sequential data, including financial datasets.

Strengths

1. This paper is written in high clarity. All formulations and derivations are clearly elaborated. 2. The idea is promising and novel to combine generative adversarial networks and neural stochastic differential equations, avoiding adversarial training which can cause instability in training. 3. Rough path theory is a relatively new topic in mathematics. It has penetrated in the machine learning community since only 2019. It is interesting to utilize it in time series generation in machine learning.

Weaknesses

1. In experimental design, the baseline models are not sufficiently explored. The experiments should include diffusion models for time series generation, as they are commonly used generative models nowadays as a very special variant of SDE and would provide a useful comparison to the proposed approach. 2. The space complexity and time complexity seem to raise the concerns, as signature transform is applied where the dimensionality is largely increased. 3. The scale of the datasets in the experiments are of relatively low dimensionality in time series dat. It would be hard to prove the efficiency of the adjoint method applied to reduce the memory cost.

Questions

1. How should the "order" of the signature, k, be selected? Is there any specially designed optimization method to help with it? 2. Why are the neural networks in neural SDE the regular neural networks? I am very curious if more sophisticated deep learning models, such as u-nets, are examined and what their impacts to the performance are.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations of this work is not adequately addressed. It can be possible to consider the application scope of this work, where it is only for time series generation, especially for the cases where the dimensionality is not high.

Summary

This paper introduces a novel approach for training Neural SDEs (Stochastic Differential Equations) as generative models for sequential data without using adversarial techniques. The authors propose a novel class of scoring rules based on signature kernels, which offer stability and avoid issues like mode collapse that are common in GAN-based training. The new formulation allows for memory-efficient adjoint-based back-propagation and enables the generation of spatiotemporal data, demonstrating superior performance over alternative methods in various tasks, including simulation of rough volatility models, conditional probabilistic forecasts of forex pairs, and mesh-free generation of limit order book dynamics.

Strengths

- The authors propose a new method with rigorous theoretical justification and verify it with numerical experiments. - The proposed method does not require any adversarial training, so presumably training the generative model with the method is more stable than GANs. - The proposed method achieves significantly better performance than SDE-GAN without adversarial training.

Weaknesses

I am not expert of this field, stochastic differential equations (SDE) for continuous-time generative models. So it is difficult to point out some critical weakness of the proposed method. - I highly recommend that authors elaborate the evaluation metrics (e.g. KS-score) in detail for those who are not familiar with this field. `

Questions

- Is the proposed method applicable to generating images? - How is the proposed model sensitive to hyper-parameters and what would be rule of thumbs to tune those hyperparameters?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

Since my expertise is not stochastic differential equation, it is hard for me to point out critical limitations.

Summary

This paper considered the training of neural SDEs, where are continuous-time generative models for sequential data. State-of-the-art approaches train neural SDEs in an adversarial manner and suffer from instability. In this work, a non-adversarial training approach is proposed for stable and effective training of neural SDEs. Particularly, a new class of scoring rules based on signature kernels is defined and used as objective for training. Extensive evaluations are performed to demonstrate the effectiveness of the proposed training objective.

Strengths

The differences between the related works and the proposed method are clearly discussed. The proposed method seems solid in theory. The possible extensions of the proposed method are extensively discussed.

Weaknesses

The significance of the work is somehow unclear to me. In the related work section, two works [PADD21] and [BO21] are discussed that are closely related to the proposed method. They are different from the proposed method in terms of the data type (discrete/continuous) and the setting of continuous-time processes (general/neural SDE specific). However, it is not clear how significant the proposed method is from the above two aspects. The related work section is hard to follow. Lots of works are discussed from different perspectives: Neural SDE; score-based generative models; and scoring rules for generative networks. For me, these discussions are a bit disjointed. They are mainly discussed to show the differences between them and the proposed method. I think it is necessary to clearly locate the proposed method in the related literatures and then discuss. If there is a table or figure to visualize the overlaps between the proposed method and the other related methods, it will be clearer. The proposed method is heavy in math. The notations are not clearly explained. For example, in line 95, what is Omega, F, and P? What is the law representing in line 97? More intuitive descriptions and clear definitions will greatly help the reading. Though the mathematical foundation is solid, the empirical evaluation of the proposed method compared to SOTA methods is weak. Only SDE-GAN is considered for comparison. Score-based generative models (SGMs) are also leveraging scoring rules. Also, [BO21] introduced scoring rules for continuous-time processes. Since these works are closely related to the proposed method, it is necessary to compare with them. If can’t, it is necessary to explain the reasons, which I think also help clarify the significance of the proposed method. There is a lack of experimental analysis. In section 4.1., there is no explanation or analysis for the values shown in Table 1. Why SDE-GAN is better at t=19? Similarly, in section 4.4, the conclusions of the evaluation discussed in section 4.4 are unclear to me. The authors are claiming that the proposed method requires less memory compared with SOTAs. There is a lack of evaluation in terms of computational cost or the memory cost to support this claim.

Questions

Please see the details questions in the weakness section. I would love to change my rating if the authors can address my concerns about the significance and the evaluation.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The limitations of the proposed method are well discussed.

Summary

This paper introduces a novel approach to training Neural SDEs using non-adversarial methods based on signature kernel scores. The authors demonstrate that the signature kernel score is strictly proper and provide consistent estimators for such scores. The effectiveness of their approach is demonstrated in various tasks.

Strengths

1. The method is novel as it offers a non-adversarial alternative to traditional GAN-based training methods for Neural SDEs. This work is likely the first to introduce the signature kernel method to the Neural-SDE model. It achieves superior performance compared to the GAN method while being easier to train. 2. The theoretical part of the paper is well-written, with clear mathematical formulations and adequate definition of symbols. The proof of strict properness and consistency, although direct consequences of prior works, offers robust guarantees.

Weaknesses

1. Limited comparisons: The authors claim that their procedure outperforms alternative ways of training Neural SDEs, but they only compare it with the SDE-GAN. In the related works, the authors mention the latent SDE, which is easier to train than SDE-GAN, and claim that it yields worse performance than the SDE-GAN. While I acknowledge that GAN generally has better model capacity than VAE, can you provide quantitative results to substantiate this claim? The original paper only compares with the latent ODE from 2019. 2. Lack of visualizations: Although the paper visualizes the generated distribution of paths conditioned on a previous path using their model, it would be beneficial to include a comparison with the paths generated by the SDE-GAN. This would provide a clearer understanding of how your outputs capture more characteristics of financial markets. Furthermore, the generated paths exhibit a large variance, which could be addressed or explained. 3. Explanation of the LOB experiments: The simulation of LOB dynamics and the decision not to use autoregressive prediction are interesting, but how is the proposed design better than the five autoregressive generators that are listed? The paragraph discussing this topic appears to be hastily written. The reported average scores alone do not provide sufficient information.

Questions

1. Can you provide quantitative results to show latent-SDE performs worse than SDE-GAN and your model? 2. Can you visualize conditional path distributions generated by SDE-GAN and explain why your generated paths are better? 3. Could you provide more details on how your model is useful in limit order book (LOB) prediction and elaborate on the design choices you made?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have well discussed the limitations. The work assumes the noise is continuous without jump on event arrival, therefore is not applicable to noise governed by point processes. The work pose certain regularization conditions on the sampled paths. The work could benefit from including more varied evaluation metrics to assess the performance of the generated data.

On the diffusion models as baselines

Thank you for your response. It generally answers all my questions, except one point as follows. In terms of the baselines, your rebuttal to Reviewer xQxq mentioned that diffusion models suffer from the limitations on path generation with invariance to resolutions. The reason lies in the score-based nature of diffusion models. In specific, these models do not possess the canonical Lebesgue measure in infinite dimensions and hence do not have a coherent notion of density. Would you mind further explaining this reason with more details and less assumption of the audience's mathematical background knowledge? Thanks a lot!

Response to the rebuttal

Thank you for your answers. I keep my current score but with low confidence score as before.