On the Constrained Time-Series Generation Problem

Summary

The authors propose a set of methods for generating synthetic time series, with or without constraints. They defined the different types of constraints possible, including soft and hard constraints. They developed a COP approach for generating time series from a seed, then presented ways of adding constraints, and delivered an all-inclusive method based on diffusion models: GuidedDiffTime.

Strengths

The paper is very clear and pleasant to read. Everything is very well explained, including the context, the issues and the solutions implemented. The content is substantial and shows a lot of hard work on the part of the authors. The results have been tested on several databases and show the advantages and disadvantages of their solutions.

Weaknesses

Nothing in particular.

Questions

1) What kind of constraints are not included in those you describe? How would you go about adding them and taking them into account? 2) Have you thought about less "traditional" time series that look more like "functions of time"? More precisely, how would your method adapt to very different curve shapes?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

See questions.

Summary

The paper proposes a model for generating synthetic time series data given user-specified constraints. The constraints can be either soft (e.g. following trend) or hard ((in)equalility). The first proposed method is framed as a Constrained Optimization Problem (COP), it inputs a seed time series which is either real or sythetic and maximizes or minimizes the L2 distance, respectively, while satisfying the constraints. The second method, called DiffTime, is a conditional diffusion model -- where the soft constraint (trend) is conditioning the diffusion process. The hard constraint (fixed point) is incorporated in the reverse process. Thirdly, general constraints are included in the diffusion loss s.t. the model reproduces the sample x_0 with all the constraints satisfied, at every diffusion step. Finally, sampling from a diffusion-based model can be guided with gradients that come from the constraints directly.

Strengths

The paper is well written. Approaches are straightforward and seem valid. Each extension in Section 3 comes naturally as an answer to a weakness of the previous approach. There is some novelty in the presented methods.

Weaknesses

The originality of most of the approaches is somewhat limited. The first three approaches seem like simple baselines that are applying existing methods to constrained generation. Only the last approach with guidance cannot be directly applied to a GAN or some other generative model. Motivation to solve this problem. Although the paper mentions reasons to perform constrained generation, the experiments do not support this. A case study or showing benefits on a downstream task would significantly improve the argument. Conserning the results, in Figure 2 and 3 t-SNE is not very convinving, and in general using t-SNE is not very rigorous. GT-GAN seems to be outperforming DiffTime in Table 1 (upper part). An interesting baseline would be a naive approach that adds noise to data samples or mixes time series or does some other simple transformation. This would show how much is relative difference between methods significant. The experiments would benefit from having more datasets and baselines.

Questions

- In COP-method, is L2 objective the best proxy for obtaining diversity/similarity? Could simple pointwise difference guarantee these properties? - Section 4.1, how is COP exactly "trained"? As pointed in this section, since it's using real data seed, the comparison is not really fair. - Can you guarantee that the final samples follow true (conditional) data distribution? I imagine some of the proposed methods can create samples that satisfy constraints and are unrelated to data.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

Performance limitations are discussed.

Summary

This paper proposes multiple approaches, including a Constrained Optimization Problem (COP) method and a conditional diffusion model called DiffTime, to cope with the problem of generating constrained time series data that adheres to statistical properties and specific user-defined constraints. It also provides a comparative evaluation against existing time series generative models using real-world and synthetic datasets.

Strengths

This paper provides a strong motivation for studying the constrained time series generation problem by highlighting its practical applications and the need for constraint-compliant synthetic data. Additionally, it offers comprehensive experiments against existing models using real-world and synthetic datasets.

Weaknesses

1. The clarity and definition of certain terms in the paper require improvement. For example, terms like "seed" in line 133 and "desired property" in line 145 need clear definitions to enhance understanding. Additionally, in line 154, what does “TS property is a vector mean?” I assume it refers to the autocorrelation of a time series with various lags. It is recommended that the authors focus on the presentation and ensure better clarity in their writing. 2. I find the experiments regarding COP in unconstrained time series generation unclear. I am unsure how COP is applied in such settings and what the specific objective and generative model used are. 3. An ablation study is lacking in the paper, particularly in relation to DiffTime. Although DiffTime is designed based on diffusion models, no diffusion models are included as baselines for comparison. Furthermore, would the performance of GAN baselines be improved with the soft and hard constraints?

Questions

I have a concern regarding the potential optimization issues that may arise when incorporating soft and hard constraints during the training of generative models. I wonder if additional constraints prolong the tuning process or affect the convergence of the models, requiring additional time to train effectively.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The presentation and writing require significant improvement.

Summary

The authors propose multiple methods to solve the constrained time-series generation problem: Given a time-series and a set of (hard or soft, local or global) constraints, the goal is to generate a time-series which is realistic, adheres to the input constraints, yet sufficiently different from the original inputs. The authors propose multiple methods starting from simpler constrained optimization to more complicated diffusion model based solutions relying on recent advances in that space. The authors then compare their models to some state-of-the-art methods and demonstrate that their method's efficacy.

Strengths

I really like this work. It is well-written, motivated and clear. I am not an expert in diffusion models, but the ideas, the problem formulation and the solution are interesting and make sense to me. The authors do a good job at evaluating their methods and demonstrate superior performance in comparison to state-of-the-art methods. The authors use standard datasets to evaluate their methods.

Weaknesses

I do not see any weaknesses in the proposed method.

Questions

N/A

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have not mentioned any limitations of their proposed work. I would encourage them to add a discussion about the same.

Summary

In this paper, the authors investigate the problem of time-series generation under different types of conditions. Since the existing methods usually require re-training when the conditions should be changed and the rejection sampling might require heavy computation, the authors use the conditional diffusion model to generate the time-series data with hard and soft constraints. The authors evaluate the proposed method and several datasets and obtain good performance.

Strengths

The problem of time-series generation that the authors investigate is interesting and important. And the authors devise a flexible framework for this problem. Moreover, the proposed method achieves ideal performance and is is environmentally friendly.

Weaknesses

1. According to equations on Pages 4 and 5, the authors assume that the data points in each timestamp follow the Gaussian distribution and that the time-series data might be stationary. However, some time-series data, e.g., financial data, are influenced by unexpected policies, so the noise or the generation process of the time-series data might be changed, and the data become nonstationary. How can the proposed method address these challenges? 2. Some implementation details of the proposed method are not clear. For example, what is the architecture of $\epsilon_\theta(x_t, t)$ in equation (4)? Can the authors provide some figures for the proposed method? 3. According to Algorithm 1, the data are generated via a linear generation process, which might be a strong assumption. Can the proposed method be extended to other more challenging cases? 4. The compared methods might be too weak and the authors only consider one latest methods. It is suggested that the authors should consider more latest compared methods like [1][2][3]. 5. I think the authors should consider more metrics for the proposed method. For the trend constraint as shown in Figure 1(b), it is suggested that the authors should employ DTW to measure the distance between the original trend and the generated data. Moreover, since the authors consider the sinusoidal data, which is seasonal, it is suggested that the authors should employ spectral transformation methods like Fourier transform and compare the basis. [1] Deep Latent State Space Models for Time-Series Generation [2] Generating multivariate time series with COmmon Source CoordInated GAN (COSCI-GAN) [3] Towards Generating Real-World Time Series Data

Questions

Please refer to the weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to the weaknesses.

Thanks for the response!

I would like to thank the authors for their work. I stand by my score.

"Deep Latent State Space Models for Time-Series Generation" - Comparison

Following the reviewer's suggestion, we have successfully obtained the source code for the paper 'Deep Latent State Space Models for Time-Series Generation.' We're now testing it on our datasets and preparing to adapt it for constrained time-series generation. To the best of our knowledge, the suggested work proposes a novel state-of-art method for continuous-time and long-sequences generation. However, it does not address constrained time-series generation. To study its performance in our setting, we plan the following straightforward changes: - we introduce conditional variables (for trend constraint); - we employ a penalty term to regularize the decoder output and improve the constraint satisfaction (of hard constraints). However, we note that adding such penalty terms may complicate the model training, and, most important, re-training is required when the constraints change. Conversely, our Guided-DiffTime incorporates constraints at inference without requiring specific (re)training steps. We thank the reviewer again for the suggestion. We will provide an extended related work in the supplementary material, mentioning the main differences between our work and suggested methods.

Official Comment by Reviewer DGpF

I appreciate the authors took the time to address my concerns. I now tend to accept the paper.