CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting

Summary

In this paper, the authors propose a dataset distillation approach by decomposing the distillation loss into two terms, the value term and the gradient term. Then, they derive bounds for these two terms that enable more effective dataset distillation for time-series forecasting. Strong performance is demonstrated in the conducted experimental evaluation

Strengths

+ The paper is well-written and easy to follow. + Authors attempt to support their methodology through theoretical explanations + Strong performance is obtained in the experimental evaluation.

Weaknesses

- p4. l121. The authors do not explain why this non-optimizable $\epsilon$ error occurs. - l. 137 in principle we could directly optimize this gradient term without deriving any bound. Unless I am missing something, this is supported by DL frameworks (e.g., PyTorch). On the other hand, I understand that there might be computational benefits from deriving a bound that does not involve the gradients. However, authors should justify their choice (why we need to derive this bound) and if this is related to computational efficiency. Ideally, experiments should be provided to support their choice. - At various points there are implicit assumptions the effect of which is not appropriately discussed, e.g., Eq. (9) is essentially supported by an experimental result in a Figure and a verbal description. I do not (empirically) disagree with the conclusion, but I can see cases where this would not hold (e.g., in financial time-series forecasting, where using the same data we can have vastly different (non-linear) hypotheses). Whether this alters the conclusions/and or the method, is not appropriately discussed. - The updating process in (12) essentially moves the targets towards the predictions of the model. This is a reasonable result that is well-known to work. However, the way this is applied is now very close to target smoothing/model distillation approaches that essentially do the same thing (also applied in the context of time series forecasting as far as I know). There is some literature on model distillation approaches (and/or self-distillation) in the context of time-series forecasting and/or regression, which essentially demonstrate that this kind of "smoothing" can indeed increase performance. Therefore, this raises some questions on whether the method should be compared experimentally (or at least theoretically) with such approaches. - Even though some ablation studies were provided in the Appendix, I didn't manage to find perhaps the most important result: What is the impact of the gradient term and what is the impact of the value term? What would happen if we remove the L_param loss in l. 9 of Alg 1? What would happen if we removed the value term? - Isn't the method also applicable in the case of regression tasks? If I am not missing anything, I think the same methodology without any change (apart from the input to the models) can be applied also to handle generic regression tasks that also expand over time-series forecasting. Minor: l.147-152: I think $k$ is used before being introduced.

Questions

Please respond to the comments provided in the drawbacks. Overall, strong performance is demonstrated, but there are several aspects that should be improved.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes.

Summary

This paper explores dataset condensation, which generates a small dataset for training of deep neural networks, for time series forecasting. Specifically, it proposes a one-line dataset condensation plugin designed specifically for time series forecasting. It first formulates the optimization objective of dataset condensation for time series forecasting, and based on which a reformulation is proposed and further theoretical analysis is performed. Based on the proposed theoretical analysis, this paper proposes a simple data condensation plugin CondTSF. Experiments on 8 time series datasets are performed to demonstrate the effectiveness of the proposed CondTSF.

Strengths

1. It proposes a simple dataset condensation plugin. 2. A theoretical analysis is provided.

Weaknesses

1. Dataset condensation is a strategy specifically designed for training large dataset. However, the data size in time series is usually significantly smaller than that in NLP and CV. In my experience, the computational demand to deal with time series data may not be an urgent request in real-world time series applications. The topic discussed in this paper seems interesting given the current program of different foundation models, but I highly doubt it applicability in the field of time series. 2. A related problem is that how the time series foundation model performs remain unclear. Exploring the foundation model for time series may be a promising direction, but it is not widely accepted and deployed in real-world applications. In fact, considering the semantics of time series is highly dependent on the underlying applications, so far I have not observed any foundation model can provide good zero-shot learning capability.

Questions

Please refer to the weakness part.

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Please refer to the weakness part.

Summary

In this paper the authors study the question of dataset condensation / distillation in the context of time series forecasting. They propose a decomposition that could be used to empirically estimate bound the point-wise forecast MSE loss. Based on this decomposition the authors proposed a condensation plug in on top of existing data condensation method that updates the distilled dataset with the estimated full model forecast. Through empirical study they show that with this plug in the resulting condensed dataset leads to better models across different model classes and benchmark forecasting tasks.

Strengths

The paper attempts a new and simple solution to an important training data efficiency problem. It also attempts a novel prospect of analyzing the dataset distillation through the loss decomposition.

Weaknesses

The major weakness is the writing. It is hard to follow in its current shape, in particular the math description of the research question should be more rigorous. To name a few: - What are the sample space of x and s? They now look like a single univariate time series. It is the case? - What is t' in Eqn (5)? Is it summed over? If so is it double sum within sum_t? - How is Eqn (7) and (8) optimized (also in Algo 1). Similarly regarding writing, the paper does not motivate the research question well, and has not provided enough background details on its base distillation methods (Line 200 Model Settings), especially MTT(3). As a result it is hard to evaluate the novelty of the paper when the main idea is not clearly delivered. Though mentioned as a limitation, the assumption of linear model trivializes the theories, which ties the proposed method to DLinear expert, and makes it hard to understand how the learning from the synthetic data generalizes beyond the expert's model class. There are similar simplifications, e.g. the empirical approximation as in Eqn (9) that's observational.

Questions

Besides the questions in the weakness: - While the distill ratio in Table 2, e.g. condensing the datasets into one training example, can be used to show the extremity of the distillation process, it is questionable what this one-shot learning means for the MLP, LSTM, CNN involved. Any insights, especially why to choose them over the few-shot study in, e.g. Appendix F (this one has visualization as well) to be in the main paper? - As now the expert model is more actively involved through the two decomposition steps to touch the synthetic data, how does the capacity of the expert impact the fidelity of the synthetic dataset versus the full one? What about at least one non linear expert just to get some insights? - The method proposed here does not utilize any time series concepts. If so this is an application on multivariate regression with linear expert. What's the novelty beyond that for time series exclusively?

Rating

4

Confidence

3

Soundness

3

Presentation

1

Contribution

3

Limitations

The authors addressed the limitations.

Summary

This paper provides a simple fix on the parameter-matching based dataset condensation methods for time series forecasting tasks. Besides matching the parameters of the model $M_f$ trained on the full training set and $M_s$ trained on the smaller synthetic dataset, it additionally induces the $M_f$ to perform well in the synthetic data too. It theoretically shows that the additional regularization contributes to an surrogate upper bound of the original objective, and empirically proves that the fix help improve various dataset condensation methods.

Strengths

1. The paper is well-written with clear structure and proper organization, and the motivation of the proposed condTSF plugin is solidly grounded by theoretical analyses 2. The proposed method is simple and easy to implement. 3. Impressive improvement are observed in the experiments on various baselines and benchmarks.

Weaknesses

A few details are not clear. See questions.

Questions

1. In line 162, the author claims that it is "observed" that models initialized with different samples of parameter values predict similarly after trained on the full dataset given arbitrary input. Is it supported by any evidences, either theoretically or empirically? 2. Some clarity on optimizing gradient term can be provided. For example, when the parameters are matched and $M_s$ are well fitted, why the label error of $M_f$ on $s$ will increase?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation that the analysis only applies to linear models is discussed in the paper.