The vast majority of time-series forecasting approaches require a substantial training dataset. However, many real-life forecasting applications have very little initial observations, sometimes just 40 or fewer. Thus, the applicability of most forecasting methods is restricted in data-sparse commercial applications. While there is recent work in the setting of very limited initial data (so-called `zero-shot' forecasting), its performance is inconsistent depending on the data used for pretraining. In this work, we take a different approach and devise ForecastPFN, the first zero-shot forecasting model trained purely on a novel synthetic data distribution. ForecastPFN is a prior-data fitted network, trained to approximate Bayesian inference, which can make predictions on a new time series dataset in a single forward pass. Through extensive experiments, we show that zero-shot predictions made by ForecastPFN are more accurate and faster compared to state-of-the-art forecasting methods, even when the other methods are allowed to train on hundreds of additional in-distribution data points.
Paper
References (63)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The work present the ForecastPFG a zero-shot forecasting method trained using only synthetic dataset and it is evaluated on several real world dataset.
Strengths
The paper is well written and the steps well described. Moreover, I think that it could be an interesting approach when you have very few data.
Weaknesses
Weakness are inserted in the Question section
Questions
Rows 71-74: This a repetition of what authors wrote in rows 16-20. This part should be improved with some details. The same in rows 83-85 and rows 23-24. The relation between Zero shot forecasting, Transfer Learning in the forecasting context and the concept of Global Models is not discussed properly. I suggest authors to discuss more about this relation. Regarding global models authors can refer to recent works as: Januschowski et al., "Criteria for classifying forecasting methods", 2020; Buonanno et al., "Global vs. Local Models for Short-Term Electricity Demand Prediction in a Residential/Lodging Scenario", 2022; Montero-Manso, P.; Hyndman, R.J. "Principles and algorithms for forecasting groups of time series: Locality and globality", 2021, etc. Rows 102-104 Authors use a multiplicative decomposition ($y_t = \gamma(t) \cdot z_t$). Why this choice and not the additive decomposition ($y_t = \gamma(t) + z_t$)? How this choice impact on the results of the work? Row 109-110 Could you clarify this tense? The $y_t = \gamma(t) \cdot z_t$ is always valid not only when $\gamma(t)$ is deterministic. Moreover, if it is present $z_t$ sampled by a Gaussian, there are different possible output values and not only $y_T$. Do I miss something? Rows 113-114 The athors wrote: "We optimize the PFN’s parameters $\theta$ by making predictions on a held-out test set from D"--> Usually with test set is named the set used exclusively for the evaluation and that is not involved in learning of the model. In this case authors define a test set used to evaluate the loss used for the parameter tuning. I think that the name is misleading. Authors can evaluate to use different names (e.g. Training Input and Training Output) to identify the two parts of the timeseries used in the training phase. Rows 129. Is not crucial to know the prior distribution of timeseries in order to have a good result on real dataset? In this work authors use a large dataset of timeseries that try to cover different situations but how the choice of the prior distribution impact on the results? Maybe this can be a idea for future works. Rows 144 Is there a motivation on the usage of Weibull distribution for the noise? How a choice of a different noise distribution impacts on the results? Rows 146-147 Now $\gamma(t)$ contains two contributions: trend and seasonal. In the row 102, instead, is written only trend. I got that in 102 authors want to introduce the general frame of the problem but I would not use the name trend in row 102. Rows 169 "Predict the MSE of this query" --> Does not the architecture predict the output? The next rows in fact the authors wrote "the output is a single prediction at a future time". Also at row 178 it is written "the loss" and not the output. Rows 180-183 It is not clear to me why the classic scaling techniques are not appliable in this case. As discussed after, the problem is the outlier that has to be excluded since they can be very different across several timeseries. Please express better this issue. Rows 188-191 Is MSPE the mean squared prediction error? Could you describe mathematically the issue raised in this section because I didn't get it. Row 196 How authors evaluate the convergence? Row 235-236 Not always the forecasting models are evaluated in retraining mode. For having a fairer comparison authors could force the forecasting model to not retrain during the evaluation test. This condition is, in fact, more similar to what happen in the real context where you don't retrain continuously your model once it is deployed. Moreover, this modality (no retrain) is followed in several works about forecasting. Using this modality (no retrain) we should have a fairer comparison between ForecastPFN and other models. Row 255 Cross product --> do you mean cartesian product? Figure 4 Is the Time Budget the same of wall clock budgets defined in row 243 Row 301 Is not more important to compare classic standardization (without outlier removal) with robust scaling? The authors, instead, compared robust scaling with min-max scaling.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
A limitation I see is the maximum number of training data used for the other models, limited to 500. It looks clear that with more other models outperform the ForecastPFN (Fig.5) while with few data ForecastPFN looks a good solution. Moreover, the training of ForecastPFN requires 30hr so it has a cost that should be taken in account for having a fair comparison with other approaches.
Summary
The paper proposes to pre-train a deep learning model on synthetic data which follows characteristics of real world time series data. This model can then be used for any downstream forecasting dataset. The authors performed experiments to show that the proposed method performs well compared to existing classical and deep forecasting models in constrained scenarios (restricted number of training data, restricted amount of time for training).
Strengths
The paper proposes an exciting direction of pre-training deep forecasting models with synthetic data, which can be quickly adapted in a Bayesian manner on a new unseen real time series, and be able to perform accurate forecasts. The synthetic data is proposed using a prior distribution which has properties similar to real world data. Such a direction is exciting and of huge significance, unlocking the power of deep learning models to be pre-trained on large datasets. The paper is well written, neatly organised in a logical flow, and contains comprehensive experiments.
Weaknesses
* While the premise of the paper is indeed exciting, I believe the claims made in the abstract and introduction are greatly exaggerated. Claims are made that the proposed method, ForecastPFN is able to beat SOTA forecasting methods. However, the experiments are only performed on severely handicapped scenarios (SOTA models are only given access to 100s of data points or maximum of 2 minutes of training). More effort should be made to highlight this handicapped scenario. * Claims that "the zero shot methods are at a disadvantage because the other six methods were allowed to train on data from the evaluation time series" (lines 280 - 282) seem a little disingenuous since the zero shot methods were pre-trained extensively (30 hours of pre-training for ForecastPFN), whereas in certain cases the six methods were only allowed 1 second of training? * The claim in line 173 - 175, that existing transformer models have prediction length to be fixed is inaccurate. This is a feature of Transformers in the long sequence time series forecasting setting, but not true of time series Transformers in general. See [1] for one such example. * Only number of wins/ranking is given for the results, but not the actual MSE scores. The actual MSE values should be given for readers to understand whether the comparisons are meaningful at all. * Following up on the previous point, 2 more very simple baselines should be added - the naive forecast (current value is taken as the forecast) and the historical mean. The number of wins/rankings doesn't really matter if any of these models can't beat such naive baselines (which is a fair comparison regardless of the train/time budget). [1] Rasul, Kashif, et al. "VQ-TR: Vector Quantized Attention for Time Series Forecasting." (2022).
Questions
1. Are the existing methods (FEDformer, etc.) also pre-trained on the synthetic data? If not, it seems quite artificial to claim wins when these methods are only given 50 datapoints (1 batch?) for training, or 1 second to train. In fact, it seems unfair to compare to ForecastPFN which has 30 hours of pre-training on a huge synthetic dataset. 2. How was Meta-N-BEATS trained on the M4 dataset? In the paper, 1 model was trained for each frequency. Which subset of M4 was the baseline trained on? 3. What does number of MSE wins mean? 4. What does training on 1 second mean? How is this implemented? Was a single gradient step even performed in this setting? 5. What are the experiment details of the baseline models? What machine was used to run these baseline experiments? It seems a little iffy to compare the time budget in terms of seconds. How to compare 120 seconds on a H100 GPU vs a K80 GPU or even a CPU? 6. How was the time restricted training for ARIMA/Prophet implemented? I understand for deep learning models you can easily stop the training, but what about ARIMA/Prophet if the model has not been fitted? Please consider using number of batches/epochs trained instead of wall clock seconds.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
1 poor
Presentation
3 good
Contribution
3 good
Limitations
The paper doesn't adequately address limitations. The limitations given in the limitations section aren't really true limitations, but rather things that are out of scope of this paper. Some suggestions of limitations to consider: 1. What if the proposed prior for synthetic data is completely different from the downstream real world data? Perhaps some sort of adversarial forecasting task? 2. ForecastPFN doesn't compute the posterior predictive distribution, it is just an approximation. 3. Does not consider probabilistic forecasting, which has more implications on the synthetic prior (not just the shape of time series, but also the distribution of data, need to consider some autocorrelation in the errors)
Summary
This paper proposes a zero-shot prior-data fitted network (PFN) for time-series forecasting. Existing works have challenges in designing a general and flexible PFN for general time-series distributions, and tuning an architecture and training scheme. This work overcomes these by designing a novel synthetic time-series distribution in training and using a transformer architecture for queries to arbitrary future timestep. Experiments have been presented to demonstrate the improved performance of the proposed model over comparison models.
Strengths
1. The proposed work tackles an interesting setting where few initial observations are available in forecasting. 2. The transformer-based network aims to predict arbitrary future time steps and the robust scaling deals with the extreme scale of time series, which increase the novelty of the methodology. 3. The results seem compelling as well, with a clear outperformance of the proposed model over comparison baselines.
Weaknesses
1. In Section 3.2, it is not clear what motivates the author to choose a multiplicative data generation model with seasonal and trend components as the synthetic prior. It would be appreciated if the author could elaborate more on that. Also, the ablation study could be more sufficient: the diversity of the synthetic data generated for training should be explored since the model relies on a general synthetic data distribution. 2. The author could provide more details about the architecture in Section 3.3: 1) How does the network learn diverse temporal features of the input data? 2) How does the network deal with different input sizes? 3) What is the benefit of using transformer-based layers vs residual blocks or recurrent neural networks? 3. In Section 4, the author should explain the definition of train budget and time budget. Are highlighted texts “training time” and “training data” related to them? It would be better to make them consistent.
Questions
Please see the weaknesses mentioned above.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Yes.
Quick reminder and new ablation
Dear reviewer 5AR9, thank you once again for your insightful review. We appreciate that you find our work is novel, that it tackles an interesting setting, and that we achieve compelling results. We would like to check in to see if you have any follow-up comments on our rebuttal. We replied to your three weaknesses on the motivation for a multiplicative data model, on the details of the architecture, and on the definitions of train budget and time budget. We would also like to mention that we now have preliminary results for a synthetic data ablation study (which you mentioned in part 1 of your weaknesses). We trained ForecastPFN with three scales of our noise parameter, $m_{\text{noise}}$: 1 (original ForecastPFN), $2/3$ (low noise), and $1/6$ (lowest noise). We do not change the scale of the noise in the validation data. Below, we report the train and validation MSE on synthetic data for each model. Training MSE Loss: | Epoch | 0 | 1 | 25 | 50 | 100 | 200 | 300 | |---| ---| ---| ---| ---| ---| ---| ---| | ForecastPFN with lowest noise | 23.71 | 5.009 | 2.326 |1.991 | 1.484 | 0.8781 | 0.5912 | | ForcastPFN with low noise| 0.3593 | 0.294 | 0.08648 | 0.08536 | 0.04981 | 0.03857 | 0.3835 | | ForecastPFN| **0.2074** | **0.1061**| **0.06425** | **0.06642** | **0.04882** | **0.03696** | **0.03196** | Validation MSE Loss: | Epoch | 0 | 1 | 25 | 50 | 100 | 200 | 300 | |---| ---| ---| ---| ---| ---| ---| ---| | ForecastPFN with lowest noise | 1.13E6 | 1.58E5 | 2.78E4 | 1.31E5 | 7.12E4 | 4760 | 1.35E4 | | ForcastPFN with low noise | 5.06E4 | 2.65E4 | **3.09E4** | **1.09E6** | 3.52E4 | 4487 | 1.57E4 | | ForecastPFN | **4.94E4** | **2.28E4** | 4.2E4 | 1.88E5 | **1.66E4** | **4267** | **1.22E4** | Recall that we remove noise in the ground-truth *predictions* of the training data as a design decision that improves performance (lines 200-203 and Appendix E.6), which is why the scale looks different for the train and validation losses. This new ablation study complements our ablation study in Appendix E.6 which considers noise in the predictions of the training data. We will continue updating this ablation study by adding more values of noise and more parameters. Please let us know if you have any additional questions or comments. Thank you!
Summary
ForecastPFN is a zero-shot forecasting model designed to overcome the limitations of traditional forecasting methods when dealing with data-sparse applications. Unlike most approaches, ForecastPFN is trained solely on synthetic data, which captures diverse time series patterns and incorporates multi-scale seasonal trends, global trends, and noise. This prior-data fitted network approximates Bayesian inference and enables accurate and fast predictions in a single pass. Extensive experiments presented in the paper demonstrate that ForecastPFN outperforms state-of-the-art methods across multiple datasets and forecast horizons, even when the latter are trained with significantly more data points.
Strengths
1. The manuscript is well-written, and the presented ideas are easy to follow. 2. The concept of training a foundational time series model solely on synthetic data is innovative and exciting. This approach not only addresses the issue of lower forecasting accuracy in data-sparse scenarios but also has positive environmental implications. 3. The execution of this work is commendable. The paper provides detailed information on the generation of the synthetic dataset, the model architecture, and the training process. Additionally, the authors conducted an extensive set of experiments, exploring various model configurations, prediction horizons, and datasets from multiple domains
Weaknesses
The paper predominantly relies on plots to present the results, without including tables. While it is understandable that summarizing all the experiments in tabular form would be challenging due to the extensive nature of the research, it can be difficult for readers to extract meaningful information from a large number of plots. Additionally, certain plots, such as Fig 5, are scaled down to a point where they become challenging to interpret. The same issue extends to the plots presented in the appendix. To enhance clarity, it is recommended that the authors include some key results in tabular form and adjust the scale of the plots to improve readability for readers.
Questions
1. Considering the focus on commercial forecasting applications in this work, it would be valuable to understand the reasons behind modeling ForecastPFN as a point forecaster instead of a probabilistic forecaster. 2. In real-world forecasting applications, the presence of exogenous features can provide additional information that directly affects the accuracy of forecasts. How does ForecastPFN handle the incorporation of exogenous features? 3. It would be interesting to know if the authors explored the option of fine-tuning ForecastPFN on the target data. This analysis could provide insights into the impact of fine-tuning on the performance of ForecastPFN in downstream forecasting tasks.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
See above
Thank you authors for the time and effort in crafting an extensive rebuttal. Overall I still have major concerns regarding the fairness of the comparisons. 1. I am still confused what is the point of comparing the settings for 1 - 10 seconds of training. Loading the data probably takes more than 1 second? 2. Why not let the fallback forecast when the model fails to output any predictions be the naive forecast? 3. There is also another simple baseline called the seasonal forecast. https://otexts.com/fpp2/simple-methods.html 4. Can we get some indication on how often in your plots, does ARIMA and Prophet fail to output a prediction? 5. I am wondering how realistic / in what real world settings is the case of having a limited time budget. It may be the case for ARIMA, whereby you usually would want to retrain the model each time new observations come in. But for deep learning models, usually we train the model once, and can use it for a long time, i.e. no need to train the model again when new observations come in. 6. I agree that number of steps/epochs is not the right measure of compute resources in this case, since the setting being explored is when time is a constrain. However, wall clock is still not a reliable measure. There are so many unknowns as a reader - what was the batch size picked? Was the most efficient batch size picked for all methods? Was the most efficient implementation of ARIMA used? Does Prophet have a GPU implementation? I find it hard to draw any conclusions from the results without knowing any of these details. FLOPs seems to be a much more suitable measure of compute resource here, as it allows us to ignore all these questions about efficiency of implementation. 7. You mentioned that the ForecastPFN architecture is non-trivial. I am a little confused regarding this - based on my understanding, the architecture is a standard Transformer. Apart from the synthetic data generation and some scaling pre-processing, what is new in the architecture?
Second reply to 3SYZ, part 1
Thank you very much for replying and giving us a chance to further clarify and improve our submission. Overall, we are happy to see that your new questions are relatively minor, such as correctly handling low-runtime baseline predictions, clarifying the metrics, and discussing real-world applications for the low-resource setting, and we especially appreciate your pointer to the new baseline, which we have now added. **1. Data loading time.** Loading training data can take longer than 1 second. However, our training time, as in the TabPFN paper, only starts the runtime after the data and model are loaded and gradient computations start. See #5 for motivation for why it is important to evaluate 1-10 second of training. **2. Fallback forecasts when the model fails.** After further thought, we believe that the best way to handle this issue is to do the same thing as the original [TabPFN paper](https://arxiv.org/abs/2207.01848) (Fig. 5): for each baseline, start showing results when the baseline is able to make non-trivial predictions on **all** datasets/settings. We will also include raw results for each dataset for completeness. This is the best way to update our paper so as not to cause confusion about the low-runtime setting, and it is backed up by existing work. (For the specific runtimes of baselines, see answer #4.) You bring up a great point that in real life, if using a model such as FEDFormer, then the practitioner would use FEDFormer for high budgets and the seasonal baseline for low budgets. The performance of this combination can easily be inferred from our plots by looking at the two corresponding methods, and we will make a point of this as a comparison to ForecastPFN in our updated manuscript. **3. Seasonal baseline.** Thank you, we are happy to add more baselines. Many of our low-resource experiments involve series that are fewer than 365 days long, so instead of monthly we use weekly, which uses the historical mean from the same day of the week as the current prediction. This baseline now becomes the 2nd best method in low train size and low runtime settings, 2nd to ForecastPFN, across all axes (MSE wins across train and time budgets, mean MSE rank across train and time budgets). See part 3 of our reply for these results. We have updated all plots in our paper to include this baseline. **4. ARIMA and Prophet minimum times.** We have calculated the percentage of configurations (dataset, prediction lengths, seeds) that failed, for each time budget and each model which requires training or being fit. We see that Arima is the model which struggles the most at low time budgets. | Model/Time Budget | 1.0 | 5.0 | 10.0 | 15.0 | 30.0 | 45.0 | 60.0 | 120.0 | |:------------|------:|-------:|-------:|-------:|-------:|-------:|-------:|--------:| | Arima | 9.68 | 77.42 | 88.71 | 92.9 | 100 | 100 | 100 | 100 | | Autoformer | 70.97 | 97.74 | 100 | 100 | 100 | 100 | 100 | 100 | | FEDformer | 71.94 | 99.68 | 100 | 100 | 100 | 99.35 | 100 | 100 | | Informer | 72.58 | 100 | 100 | 100 | 100 | 100 | 97.42 | 100 | | Prophet | 88.71 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | | Transformer | 66.77 | 100 | 100 | 100 | 98.39 | 100 | 100 | 100 |
Second reply to 3SYZ, part 2
**5. Motivation for the low runtime setting.** First, we emphasize that there is a wide range of applications for a limited training set size, as cited in our paper (lines 23-24, 83-88). Next, there is also a wide range of applications which require a low time (or computational) budget. As you say, deep learning models can train once and be used for a long time for in-distribution data. But, they will need to be retrained for new applications (out-of-distribution), unlike ForecastPFN. As an example of a nice property, ForecastPFN can be loaded onto a CPU (a standard function of tensorflow) and used on edge devices to make predictions. As an example of a specific application, ForecastPFN can then be used to forecast personalized utilities usage across households and across towns in developing countries. Other deep learning models would need to retrain when the data becomes out of distribution. Another application is as a forecasting data-exploration tool, allowing users to see *instant* forecasting visualizations as they navigate their dataset. **6. Runtime.** We respectfully disagree with your statement that “wall clock is still not a reliable measure”. - First, recall that on lines 218-220, we mentioned that all of the baselines use the hyperparameters from their official release, and we specified the ARIMA implementation. We also open-sourced our entire codebase (line 61), so that readers can see the exact code and reproduce experiments if they want. - Second, there is an enormous precedent throughout the ML literature to use wallclock time as a metric. Wallclock time is used in many prior works and is mentioned in best practices guides. We do agree that we will specify the fact that ARIMA and Prophet do not use GPUs, and the CPU version (it's an N1 series from GCP). We are happy to include additional plots that use FLOPs instead of wallclock time, but we will need to add caveats that FLOPs can sometimes be misleading. As a quick example, a vanilla transformer is highly GPU-memory bottlenecked, and would appear to be slower in FLOPs than an LSTM, even though LSTMs are substantially slower in wall clock time. (This issue is solved by using Flash Attention for transformers). **7. ForecastPFN architecture.** The transformer is more flexible than existing transformers for forecasting, due to the input that can take in any timestep and value (not just a sequential input), and robust scaling. Our main point is that *training* a universal forecasting model is highly nontrivial. Notably, the form and the hyperparameters of the synthetic data distribution, especially the noise parameters, must be set precisely in order for the model to achieve non-degenerate performance. Thank you very much once again, for your time in reviewing our paper.
Second reply to 3SYZ, part 3
Here, we include results with the new "Weekly" baseline (from our answer to question 3). It often achieves 2nd place behind ForecastPFN in the low train budget and low runtime settings. Train Budget vs. MSE Wins: | Model | 50.0 | 100.0 | 150.0 | 200.0 | 250.0 | 300.0 | 500.0 | |:--------------|-------:|--------:|--------:|--------:|--------:|--------:|--------:| | Arima | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | Autoformer | 1 | 3 | 8 | 9 | 8 | 10 | 6 | | FEDformer | 2 | 4 | 5 | 7 | 13 | **16** | **16** | | ForecastPFN | **32** | **30** | **26** | **23** | **20** | 11 | 11 | | Informer | 0 | 0 | 1 | 0 | 1 | 8 | 12 | | Last | 9 | 9 | 9 | 9 | 9 | 9 | 9 | | Mean | 5 | 4 | 3 | 1 | 2 | 0 | 0 | | Meta-N-BEATS | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | Prophet | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | Weekly | 13 | 12 | 8 | 8 | 6 | 3 | 2 | | Transformer | 0 | 0 | 2 | 5 | 5 | 4 | 5 | Time Budget vs. MSE Wins: | Model | 1.0 | 5.0 | 10.0 | 15.0 | 30.0 | 45.0 | 60.0 | 120.0 | |:--------------|------:|------:|-------:|-------:|-------:|-------:|-------:|--------:| | Arima | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | Autoformer | 0 | 3 | 4 | 5 | 9 | 6 | 6 | 6 | | FEDformer | 2 | 5 | 10 | 11 | 10 | **18** | **18** | **18** | | ForecastPFN | **30** | **23** | **21** | **19** | **14** | 12 | 11 | 11 | | Informer | 1 | 1 | 3 | 4 | 9 | 10 | 10 | 10 | | Last | 10 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | | Mean | 5 | 4 | 3 | 2 | 1 | 0 | 0 | 0 | | Meta-N-BEATS | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | Prophet | 5 | 5 | 5 | 5 | 5 | 4 | 4 | 4 | | Weekly | 9 | 7 | 3 | 2 | 1 | 0 | 0 | 0 | | Transformer | 1 | 6 | 5 | 6 | 5 | 4 | 4 | 4 | Train Budget vs. MSE Rank: | Model | 50.0 | 100.0 | 150.0 | 200.0 | 250.0 | 300.0 | 500.0 | |:--------------|-------:|--------:|--------:|--------:|--------:|--------:|--------:| | Arima | 7.44 | 8.29 | 8.59 | 7.96 | 7.97 | 8.26 | 8.26 | | Autoformer | 6.33 | 5.52 | 3.65 | 3.33 | 3.69 | 2.78 | 3.26 | | FEDformer | 4.89 | 4.78 | 4.43 | 3.3 | **2.65** | **2.54** | **2.83** | | ForecastPFN | **2.03** | **2.42** | **2.85** | **2.98** | 3.13 | 3.47 | 3.56 | | Informer | 7.71 | 6.21 | 6.15 | 6.26 | 6.04 | 4.9 | 4.27 | | Last | 2.78 | 3.04 | 3.71 | 4.3 | 4.35 | 4.94 | 5.01 | | Mean | 2.36 | 2.68 | 3.15 | 3.72 | 3.98 | 4.56 | 4.64 | | Meta-N-BEATS | 3.2 | 3.41 | 4.14 | 4.7 | 4.72 | 5.29 | 5.41 | | Prophet | 9.11 | 9.13 | 8.9 | 8.88 | 8.97 | 8.83 | 8.96 | | Weekly | 2.95 | 3.07 | 3.95 | 4.51 | 4.61 | 5.25 | 5.15 | | Transformer | 6.12 | 6.33 | 5.47 | 5.05 | 4.9 | 4.19 | 3.64 | Time Budget vs. MSE Rank: | Model | 1.0 | 5.0 | 10.0 | 15.0 | 30.0 | 45.0 | 60.0 | 120.0 | |:--------------|------:|------:|-------:|-------:|-------:|-------:|-------:|--------:| | Arima | 5.03 | 6.78 | 7.23 | 7.53 | 8.19 | 8.34 | 8.28 | 8.38 | | Autoformer | 6.83 | 5.72 | 5.31 | 5.22 | 3.83 | 3.29 | 3.26 | 3.32 | | FEDformer | 4 | 4.28 | 3.6 | 3.34 | **3.45** | **2.85** | **2.72** | **2.77** | | ForecastPFN | **2.03** | **2.75** | **2.9** | **3.04** | **3.45** | 3.6 | 3.59 | 3.66 | | Informer | 5.86 | 5.99 | 5.63 | 5.72 | 4.74 | 4.48 | 4.38 | 4.33 | | Last | 2.78 | 3.66 | 4.06 | 4.18 | 4.85 | 5.06 | 5.07 | 5.13 | | Mean | 2.42 | 3.36 | 3.79 | 3.84 | 4.44 | 4.63 | 4.58 | 4.76 | | Meta-N-BEATS | 3.21 | 4.13 | 4.52 | 4.6 | 5.24 | 5.46 | 5.47 | 5.53 | | Prophet | 6.09 | 7.28 | 7.62 | 7.66 | 7.9 | 7.95 | 7.89 | 8.01 | | Weekly | 3.05 | 3.87 | 4.27 | 4.36 | 5.04 | 5.31 | 5.3 | 5.4 | | Transformer | 6.12 | 6.3 | 5.77 | 5.4 | 3.77 | 3.98 | 4.32 | 3.71 |
Thanks authors for the extensive reply, I have raised my score to borderline accept since my concerns about fairness of comparison in the setting of low train time has been addressed. My reason for not giving a higher score is that I am still doubtful about the low train time setting -- the out of distribution case is different from low train time setting. If OOD is our concern, then we should consider OOD methods... such as AdaRNN and many more. I think there is a huge onus on the authors' end to update the papers with much more extensive empirical results. I acknowledge that FLOPs is also not sufficient alone, as such reporting results on both wall clock and FLOPs would give readers a much better understanding of the comparisons. I would also like to point out that existing Transformer architectures for forecasting can already take in any timestep and value, based on the existing formulations.
Thank you for your reply. We are glad that we addressed your points about the comparisons in the low train time setting, and we are happy to see your more favorable view of our work. We would like to very briefly mention a couple more points regarding the low train time setting. We agree that the more compelling setting, which has more applications in the real world, is the low train size setting. We believe that ForecastPFN’s success in the low train size setting is a notable contribution by itself, and we would be happy to make the necessary edits to emphasize this setting slightly more than the low runtime setting (although we think the low runtime setting is still important to include in the paper). We agree with you that FEDFormer and the other models can be reused many times without retraining, and we also agree that models such as AdaRNN can handle the out of distribution case. Instead of “out of distribution”, a better phrase for us to use would have been “new dataset” or “different distribution.” We were thinking of applications that require many low runtime / low computation forecasts which are all very different from one another. We are currently still working on using FLOPs as a metric our experiments. For now, we can report the parameter count of the deep learning models, which is very roughly correlated with FLOPs. | Model | #Params | |---| ---| | Autoformer | 1.05E7 | | FEDFormer | 1.44E7 | | Informer | 1.13E7 | | Transformer | 1.05E7 | |ForecastPFN | 1.50E6 | Finally, we also included a preliminary [ablation study](https://openreview.net/forum?id=tScBQRNgjk¬eId=IkeN4k7sXF) on the scale of the noise in the synthetic data. Thank you once again for your time!
Thanks for the rebuttal and doing the additional experiments. I would suggest moving these tables into the main paper as well. After reading the other reviews and the corresponding rebuttal discussions, I would like to maintain my original score.
Second reply to MwwS
Thank you very much once again for your review. We appreciate that you found our work innovative and exciting, with positive environmental implications, and also that our work is well-written and easy to follow with extensive experiments. Thank you for your update. Yes, since you listed the lack of tables as a primary weakness, we created tables with [raw MSE values](https://openreview.net/forum?id=tScBQRNgjk¬eId=Z6mb79DbuS), [MSE wins and MSE rank](https://openreview.net/forum?id=tScBQRNgjk¬eId=WZZvHCWdd0), and [percentage completion](https://openreview.net/forum?id=tScBQRNgjk¬eId=F2RWvstySO), and we added all of these into our paper. We would also like to mention that we just added a preliminary [ablation study](https://openreview.net/forum?id=tScBQRNgjk¬eId=IkeN4k7sXF) on the noise of our synthetic data, and we included [parameter counts](https://openreview.net/forum?id=tScBQRNgjk¬eId=OexP4WpgRv) for the deep learning models. Please let us know if you have any additional follow-up questions or reservations about our work, and we will be happy to reply. Thank you for your time!
Thanks for the answers and clarifications. I decided to increase the my evaluation.
Author update
We thank all four reviewers for replying during the discussion period. We were happy to complete all suggestions, such as adding 3 new baselines to our experiments, adding new ablation studies, adding tables of the raw MSE values of our experiments, and answering clarification questions about our experimental setup and synthetic data. We once again appreciate that all reviewers found our work novel and exciting: the first forecasting model to be trained purely on synthetic data. Thank you once again, for all of your comments and suggestions!
Thank you for making an effort to address the reviewers' concerns which includes conducting more experiments. AC
Decision
Accept (poster)