Scaling Law for Time Series Forecasting

Summary

The authors propose a set of scaling laws on time series based relating model size, data size, and the "history dependence" of the time series. They end with a power law which they then try on different datasets.

Strengths

The topic is interesting and it would be a good thing to study.

Weaknesses

The presentation is really bad and it makes the understanding difficult. The theoretical results have assumptions that are sometimes not properly stated as assumptions, and sometimes hard to justify. Some of the theoretical preliminaries are either questionable, or badly explained. If this is solved I might look into the actual derivations more in detail: - Quasi-isometric assumption: it seems to imply that the mapping is either the identity or close to it. Imagine you have a language where there are two synonyms; then the only way in which the equation holds is when alpha is very small. Which is fine, but then the value of alpha can be arbitrarly small! They should justify when this is valid. - A similar limitation appears in assumptions 4 and 5, which seem to indicate that the data must follow some markovian structure. And I can definitely find cases where they would not apply: imagine a language with X letters forming Y words of length Z. If Y<<X^Z (which almost always happens) when L<Z you could easily end up with the rank of the covariance matrix for sequences (of length less than Z) being higher than the rank of words. In more practical settings, there are languages (ex: german) where the sense of a word is tied to a word that happens much later in the sentence (in german, verbs with preposition when there is a nebensatz), which would relate to the toy example I mention before, because the rank of the covariance decreases as the length increases. - In line 132 they assume that the data follows Zipf's law. This is a big assumption, and while it could apply to languages, it is hard to argue that it applies to every time series (and see Clauset et al. "Power-law distributions in empirical data"). They should at least mention why they think it applies to the datasets they are using. - In sec 3.2.4 they assume that the model partitions the space uniformly, and claim that this is the worst case. It is not clear why would that be the case, and don't seem to actually analyze other scenarios later on (if they do, refer to it on the same sentence). I haven't checked this in detail, but it seems to me that with assumptions 1 to 3 the model is either linear or very similar to it, and by 4 and 5 the data might need to be implicitly autoregressive. Taking those two together and assuming that the data has a power law distribution, it seems almost necessary that the loss or model size would follow the laws that they propose. For the experiments: - From looking at Fig 1, what I could conclude is that different datasets/models follow different distributions. - I think that from the fit is very hard to make any proper justification. MOst of the trends could be fit by some exponential or a simple polynomial, so they should compare to some sensible alternatives, and use some valid statistic criterion (akaike, for example) to compare. - In the same line of thought, fitting on a logscale is often hard, and I don't see how they did the fit. The fit should be more properly explained (and likely done, check http://bactra.org/weblog/491.html for example) - Fig. 2, from looking at it I would expect the double-descent to relate to those results. - I did not see in the theory any indication that the losses could scale negatively (as in some cases of Fig. 2). THis seems to be important, as some data has such trend. If their model is not suposed to overfit, this is a crucial assumption? - In line 85, a less relevant point: "[33] shows that the average intrinsic dimension for a time series sequence should converge to a finite value, which indicates that the intrinsic dimension of a sequence should be proportional to its length." This sentence is wrong as far as I can tell: if the average intrinsic dimension converges to a finite value, it is not proportional to its length. For example, imagine the sequence 01010101010.... irrespective of the length, the intrinsic dimension is fixed. The notation is also problematic. They use L for both length (line 100) and loss (Eq. in line 144), and sometimes they just write "loss" instead of L. I pointed out a few problems with the writing. There are more, but at this point it's on the authors to fix them: - Lines 21-22 Neural Nets utilize different model architectures, including FFN-based, Transformer-based and Convolution-based neural nets have been proposed. - Sometimes they refer to "scaling laws" in plural, and sometimes as "the Scaling Law" in singular as if there was a single one. - Line 67: "There have been many works analyzing on different mathematical properties " - Line 1667: if the data samples are "sparse". I assume they mean that there are few data points.

Questions

Check weaknesses. Also, how would the authors' results relate to the double-descent curve? This would put the two cases in context. Also, their scaling law seems to be tied to ever-decreasing loses

Rating

7

Confidence

3

Soundness

3

Presentation

1

Contribution

3

Limitations

They should really address the limitations of their assumptions and discuss when are they valid.

Summary

This paper introduces scaling laws for DNN analysing MTS data, involving dataset sample size, model size, look-back horizon, dataset covariate size (dimension), and noise/uncertainty in the dataset. It relies on an axiomatization of the intrinsic information space (linearity, isomorphism) and bayesian optimal model (lipschitz-continuity) to derives an approximation of the Loss in different scenarios, and tests aspects of this laws in experiments.

Strengths

The paper enters the difficult discussion about power-laws in DNN to the time series forecasting field, with a clear approach motivated by mathematical assumptions on what time series intrinsic information should respect. Linearity with respect to horizon especially. The identification of the influence of model overparametrization is explicited. While I am not entirely clear on which approaches are completely new and which other are inspired by scaling law studies of other data types, I believe that the resulting theoretical laws are novel and take into account the particular of TS data. Experiments cover several important parameters in the laws, and show matching results with the derived scaling laws.

Weaknesses

It is hard to understand where the assumptions on the intrinsic space (3.2.1) come from. Especially lipschitz-continuity and space quasi-isomorphism. Are those properties 1) required for results to hold? 2) observed in practice in meaningful representations of (multivariate) time series data? 3) Are all the described properties completely new, or inspired from other works on Deep Neural Networks?

Questions

Since the paper relies on an intrinsic space formulation, did authors study the representation learned by the neural networks? For instance, using PCA and/or tSNE on intermediary vectors. Could we verify that the assumptions made in 3.2.1 are plausible in such a way?

Rating

7

Confidence

2

Soundness

3

Presentation

4

Contribution

4

Limitations

Several limitations are identified in the paper: need for larger datasets, impact of prediction horizon (fixed in this work), supervised forecasting task specific analysis. I would also add limitation/concern of mine: the evaluation datasets come from a same MTS benchmarking effort, and current DNN structures where built to optimize their performance on these datasets (among others). It is unknow if this induces bias compared to datasets generally not tested during standard benchmarking. It could happen that our current TS DNN models are overoptimized for the common benchmarking datasets. Optimally, the power scaling law should involve a few datasets not typically evaluated in benchmarking models.

Summary

This paper proposes a theory for scaling law in time series forecasting that accounts for dataset size, model complexity, and data granularity, with a particular focus on the look-back horizon. This paper empirically evaluates various models across diverse time series forecasting datasets to verify the validity of the scaling law and validate the theoretical framework, especially regarding the influence of the look-back horizon.

Strengths

1. this paper is complete with good organization. 2. several theatrical analyses are provided for the justification.

Weaknesses

1. the main argument is not novel. I think it is well-known that a larger horizon may not bring up a better performance in time series forecasting. In some cases, it can but the larger horizon would also bring up more noise that can hinder the forecasting, 2. the contribution is kind of weak. I don't find some quite useful conclusions or contributions for time series forecasting. With the scaling laws the author verified, there is no further improvement of new models on state-of-the-art benchmarks.

Questions

See weakness

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

1

Limitations

See weakness

Summary

This paper proposes a theory that explains why complex models do not necessarily outperform simpler models even under the presence of larger amount of data, and why longer inputs hurt performance of some models. The authors consider the data size, model complexity, data granularity and the look-back horizon. Together with the theory here proposed the authors present empirical evidence verifying the theoretical insights.

Strengths

The authors present an analysis of scaling laws for time series forecasting by adding components that are characteristic of time series forecasting, i.e. the fact that a limited amount of information of the past is provided to do inference/forecasts. Whereas in other domains it is usually assumed that the larger the look-back window then the better the estimations, in time series forecasting this is not always the case. Based on this the authors proceed to present a framework based on intrinsic spaces where slides of time series can be represented as vectors - they consider a suitable space for the look-back window and the forecasted values. Perhaps the most relevant contribution of the authors is the theoretical framework considered, together with numerical verifications.

Weaknesses

- The authors present a theoretical framework that is interesting but lacks of precision. The authors constantly derive equations but at the end it is unclear what are the hypothesis required to be sure that the corresponding results hold. - The authors present results in different models, but these results seem to be done on models trained on individual datasets, and hence it is unclear if the results here presented hold for the case of pretrained models, which are pretrained in multiple time series datasets. - In line 193 the authors claim that their results are well approximated by the laws here derived. But this is completely unclear a the derived laws are not plotted together with the empirical ones per model. This leaves the verification task completely to the readers. This further makes it more challenging to check the correctness of the presented results. - Appendix A.1: The proof presented is not a formal proof but a sketch of a proof. - Typos: - - line 113: “ and thus We can scale” [there is a wrong capital letter] - - line 122 “s[h_1] to s_[h_2-1]” there is a wrong positioning of a bracket

Questions

How do the presented result hold for the case of pretrained models, or any other setting where models are trained on multiple datasets? Since the recent developments of large models for time series this kind of results would be of high value. It would be nice if the authors could provide more details on the derivations presented in 3.2.2. Although the main sketch is there is rather unclear how this is done.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors clearly state that the analysis on pretrained models remains for future work. This is a valid statement from the authors as one can assume that novel challenges arrive for this setting, and arguably the notion of scaling law for single-dataset models is not fully answered yet, being this work an interest step towards this topic.

Summary

The paper introduces a theoretical framework for scaling laws in time series forecasting, focusing on the impact of dataset size, model complexity, and look-back horizon on model performance. The authors make two main contributions: They propose a novel theory that explains scaling behaviors in time series forecasting from an intrinsic space perspective, emphasizing the critical role of the look-back horizon. The theory identifies an optimal horizon and demonstrates that beyond this point, performance degrades due to inherent limitations of dataset size. Conducting comprehensive empirical investigations to verify scaling behaviors related to dataset size, model size, and look-back horizon across various models and datasets. These experiments establish a robust scaling law for time series forecasting and validate the proposed theoretical framework. The paper examines how the optimal look-back horizon changes with different amounts of training data and model sizes. It also explores the relationship between channel-dependent and channel-independent models in the context of the proposed theory. The authors test their theoretical predictions using a range of popular time series forecasting models (including linear models, MLPs, iTransformer, and ModernTCN) on multiple datasets (such as ETTh1/h2, ETTm1/m2, Traffic, Weather, and Exchange). Their experiments demonstrate the existence of scaling laws in time series forecasting and provide evidence supporting their theoretical framework, particularly regarding the influence of the look-back horizon.

Strengths

The paper demonstrates several notable strengths. Its originality lies in introducing a novel theoretical framework for scaling laws in time series forecasting, particularly addressing the impact of look-back horizon, an aspect previously unexplored in scaling law theories. The quality of the work is evident in its combination of thorough theoretical analysis with extensive empirical validation across multiple models and datasets, including Linear, MLP, iTransformer, and ModernTCN models tested on various datasets such as ETTh1/h2, ETTm1/m2, Traffic, Weather, and Exchange. The paper maintains clarity through clear mathematical derivations, well-defined assumptions for the theoretical framework, and presentation of experimental results with error bars and statistical analyses. The significance of this work is apparent in its potential to inform future model design and hyperparameter selection in time series forecasting, which could lead to performance improvements across various applications.

Weaknesses

Despite its strengths, the paper has a few areas that could be improved. The authors acknowledge that their experiments are conducted on datasets smaller than some recently proposed large datasets, which may limit the generalizability of their findings. The theoretical framework relies on several simplifying assumptions about the intrinsic space properties, which may not always hold in practice. A more in-depth discussion on the implications of these assumptions could enhance the paper's robustness. Additionally, while the theory provides valuable insights, the paper could benefit from more concrete guidelines on how practitioners can apply these findings to improve their models, which would increase the practical impact of the work.

Questions

It would be interesting to understand how the proposed theory extends to multi-variate time series forecasting and if there are any specific considerations for such cases. The paper mentions that downsampling can sometimes improve performance, and it would be valuable for the authors to elaborate on how this relates to their theory of optimal horizon. Lastly, given the changing nature of many real-world time series, it would be beneficial to know how sensitive the optimal horizon is to changes in the data distribution over time (concept drift) and whether the theory accounts for non-stationary time series.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors demonstrate transparency by addressing limitations. They acknowledge that their theory primarily covers time series forecasting and may not generalize to other time series tasks. They also note that their experiments are conducted on datasets smaller than some recently proposed large datasets, and that the theory doesn't consider self-supervised or pretrained models. Additionally, they mention that the impact of prediction length on optimal horizon is not thoroughly explored. While the authors are commendably upfront about these limitations, they could potentially enhance this section by discussing implications for computational resources and energy consumption related to finding optimal horizons in practice. Given the theoretical nature of the work, the absence of a discussion on potential negative societal impacts may be appropriate, but a brief mention of this could further strengthen the paper's consideration of broader impacts.

Preliminary assesment

I will check the responses more carefully in the next days and decide if I change my grading, but for now I mention a couple of points. - 1.1 Fair, I will check it later. - 1.2: It's unclear to me if you are saying that it's fine to use the autorregressive assumption (1.2.2) or that you are not using it (1.2.1). For the record, I am fine with making this assumption, but you should explain in plain words that you're using it. - 1.3: This is more of a clarification of the statements. If you are assuming Zipf's law on the data, you should state it clearly as one of your key assumptions. I agree that is important to show that it applies to the data you are using, but that does not mean that it applies to every dataset. So besides verifying it for your data, you should add it to your assumptions in the bullet points. - 1.4. Line 85. The problem is not whether the claim is empirical or relates to previous works, but rather that it is not stated properly . If D(L) is the average intrinsic dimension of a sequence of length L and converges to a finite value, this does not imply or suggest that the intrinsic dimension grows with the length.This is only the case if you are refer to the average as lim_{L->inf} D(L)/L, but usually when you say the average of a quantity you would refer over many samples (so intrinsic dimension for many samples of a sequence of a fixed length L). I still think that the empirical validation is not really supporting the theory, in the sense that it's few points per line. THe main problem is on Fig. 1, where fitting a 3-parameter function to something with 8 points is ok if that's the data you have, but there are very few points for the number of parameters that you have and therefore you really should check that the fit is significant, or if other simple models (with the same number of parameters, not driven by the theory) give a similar fit.

Papers cited (continue)

These papers are cited in our dicussions and could possibly act as reference or supplement: 10. U. Sharma and J. Kaplan, “A neural scaling law from the dimension of the data manifold,”, 2020, published on JMLR, 2022. 11.Eric J. Michaud and Ziming Liu and Uzay Girit and Max Tegmark, "The Quantization Model of Neural Scaling", in NeurIPS, 2023. 12.Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma, “Explaining neural scaling laws,” 2021. 13.Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown and Others, "Scaling laws for neural language models", 2020. 14.Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya and Others, "Training Compute-Optimal Large Language Models", 2022.

Thank you, Authors!

Dear Authors, Thank you for your responses to Rviewers and your categorized overall response. On our end, we will collectively continue to review these. I hope the Reviewers' reviewers and discussions have helped strengthen your work and highlighted its significance and areas for improvement. Thank you for your submission and continued effort in the review process! Best, Area Chair Fwdj

Summary of Revisions

We would like to express our sincere gratitude to all reviewers and ACs for their patience and detailed feedback. Their comments are very valuable and constructive and have enhanced our work. As summarized in our global rebuttal, in our work, we introduce a theoretical framework of scaling laws in time series forecasting, focusing on the impact of dataset size, model complexity, and look-back horizon; experimentally, we validate the existence of Scaling Law for Time Series Forecasting and further validate our theory framework. As stated both in our submission and during the global rebuttal session, our theory and experiment results could have positive impacts on the community, two of which are listed as follows: (1) We provide a more comprehensive understanding of the impact of the look-back horizon that has not been completely understood yet. (2) We provide novel perspective toward commonly-used components like down-sampling. We hope our work may inspire not only large foundational datasets and models, but also new models targeting datasets of limited size in the field of TSF. We truly appreciate constructive concerns and suggestions raised by all reviewers, according to which we have made supplements and enhancements to our new version of the paper. Our new version of paper features enhancements in the following aspects compared to our original submission: **Theory: clearer presentation and more supplement for Theory** T1.**We state our assumptions more clearly**. Especially, we would add the Zip-f assumption into our bullet points, and make more precise discussions about intrinsic dimension. Meanwhile, we present theory deduction in a clearer way, both in main paper and in appendix. (Reviewer 7rY6, Reviewer Vwbb) T2.**We further propose weaker assumptions that could lead to similar results**, the process of which is added to appendix. (reviewer 7rY6) T3.**We explain more in the theory deduction part in main paper about how our theory adapts to multivariable cases**. (T3 and E1 would supplement each other, providing a more comprehensive view.) (Reviewer urDc) T4.**We extend more discussions about Mixed datasets**, and the conditions required to apply our theory in these cases in appendix, hoping to provide insights about understanding works on these datasets (T4 and E1 would also supplement each other, providing a more comprehensive view). (Reviewer Vwbb) T5.**We give a more detailed explanation about the down-sampling case** in appendix. (Reviewer urDc) **Experiment: more precise and comprehensive Experiment** E1.**We include additional PCA results** for intermediate vectors for iTransformer on Multi-variable Non-linear Channel-Dependent cases, and for MLP on a mixed dataset case, further validating the Zip-f distribution assumption for multivariable cases and encouraging future research on large mixed dataset. (Reviewer oD7J, Reviewer urDc, Reviewer 7rY6, Reviewer Vwbb) E2.**We would use more comprehensive metrics** and **compare against valid functions**, further **justifying the fit of our theory model** on the experiment results. (Reviewer 7rY6) **Supplement: amendment for miscues presents, emphasize for main argument and contribution** S1.**we have checked the typos carefully and further polish our paper.** (Reviewer Vwbb, Reviewer 7rY6) S2.**We add discussions about possible limitations related to over-optimized Time Series DNN models** in the limitation part discussed in the conclusion section. (Reviewer oD7J) S3.**We discuss more the computational resources** related to optimal horizon searching in appendix. (Reviewer urDc) S4.**We emphasize more about our main argument, as well as contributions and potential novel impacts** in introduction and conclusion sections. (Reviewer SJaN) Overall, the valuable suggestions from reviewers are very helpful for us in revising our paper to a better shape. We would be delighted to have further discussions if possible.