Multivariate Probabilistic Time Series Forecasting with Correlated Errors

Summary

Briefly summarise the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary. The paper proposes an extension to multivariate forecasting model that takes into account the error auto-correlation. The issue of remaining auto-correlation residual is first illustrated clearly in an example with a multivariate forecasting method and the author then introduces their method. The method consists in modelling the cross-covariance through a learned weighted sum of kernels matrices that are projected to obtain the final estimated correlation. This approach leverages previous work on low-rank parametrisation of multivariate forecasting and the authors manages to find an approach that remains tractable despite the complexity of the task. Experiments are conducted on various real-world datasets where their method is shown to improve several forecasting metrics such as CRPS-SUM, RMSE or energy scores.

Strengths

* the paper is well written, even though the approach is very technically involved, the authors did a great job at explaining it well * the paper is well motivated: insighful data-analysis are made to illustrate the presence of autocorrelated residual and motivate their approach * large coverage of experiments and empirical evaluations: the experiments are done on a large set of datasets covering many metrics and are also well assessed qualitively

Weaknesses

* potential gaps in experiments: * matching budget: the method is currently much more expensive that the baseline considered * some details of Hyperparameter Optimization are currently missing (but will be added at the camera ready according to the rebuttal) * the paper lacks clarity on whether the method will be competitive with better likelihoods. Currently the method may work well only because of non-competitive likelihood is used (Gaussian) as opposed to Student-t or Non-paranormal which would get better residuals

Questions

I am separating my questions between the most important ones that would help to raise my score and minor comments. ## Most important **Lack of clarity whether the method will be competitive with better likelihoods.** Currently, the method only studies a Gaussian noise. However, this noise is a poor fit for the dataset studied and as such is inflating the residuals. This leaves open the question of the method works just because the initial residuals are very poor (and adds a smoothing effect which helps their estimation) and not so much because of the reason given by the paper (that the approach models the autocorrelation aspect). I can see two ways to remove this potential cofounding factor, one is to include experiments with paranormal distribution which would be more robust from [2], the other is to consider student-t distribution which should also be tractable. **Details of Hyperparameter Optimization.** The authors mentions: ``` The best hyperparameters for a base model—dataset combination are selected based on the best validation loss. Once the optimal learning rate and hidden size are determined, we apply the same hyperparameters for models both with and without applying our approach ``` How does it work exactly? Which model are you using to determine and fix the hyperparameters? (Selecting the best hyperparameters for your method may favour their transformation) **Matching budget.** The method proposed by the author is roughly 10x slower on average (and up to 30x slower for exchange-rate from table 10). This point is important and currently a bit buried down, for instance the authors mentions: “* However, there is no evidence that our method slows down convergence. On the contrary, it accelerates convergence for 5 out of 9 datasets for GPVar.” However, the wallclock time are clearly much slower I think this is fine for the paper overall but it should be stated clearly as a limitation (the method proposed by the authors would likely be worse than 10 ensemble of the baseline for instance, I don’t expect this particular point to appear in text or experiment but I want to explain why this is important). **Citation of multivariate evaluation metrics.** The authors uses CRPS-sum as their main error metrics (in addition to many other metrics including energy scores in their appendix), I would highly recommend citing this paper which discuss the limitation for this metric when assessing multivariate methods (not mine): *Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts* Étienne Marcotte, Valentina Zantedeschi, Alexandre Drouin, Nicolas Chapados *Proceedings of the 40th International Conference on Machine Learning*, PMLR 202:23958-24004, 2023. To be clear, I do not think this is a large problem given that the authors also conducted many high quality qualitative analysis of their results. ## Details - It would be very useful to clearly spell out the dimension/numbers of projection parameters in a table, possibly in the appendix - I highly encourage the authors to release their code (the intention is mentioned in the supp, the work will be hard to reproduce w/o given that it is quite technical, and the work will be more impactful as other will be able to directly compare to their setup) - a small point regarding "real-world data frequently exhibit significant error autocorrelation and cross-lag correlation due to factors such as missing covariates", I agree that missing covariates is indeed a potential use-case but it feels very niche to me (none of the dataset have covariates for instance) compared to model mispecification

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes

Summary

The paper proposes an approach for multivariate time series forecasting, modelling the correlation between the multivariate errors in close time steps. The authors show that indeed multivariate errors in close time instants are correlated. They generalise the work of Zheng et al (Aistats 2024) where the idea of modelling the correlation between errors in different time step has been used only for probabilistic univariate models. Experimentally, they consider two different deep architectures suitable for multivariate time series forecasting. In both cases they report better performance compared to the traditional approach which does not model the correlation between the multivariate errors in different time steps.

Strengths

Generalizing the model of the temporal correlation of error from the univariate to the multivariate case is a challenging problem. The proposed approach, based on modelling the correlation of the error as the sum of kernels with different length scales, can be applied to any deep network model suitable for multivariate forecasting. The approach is tested with two different neural architectures and positive results are reported in both cases. The paper is well written.

Weaknesses

None in particular.

Questions

* You show a better CRPS for the models trained with correlation compared to those without correlation. Could you elaborate if the improvement of CRPS is due to better point forecasts, better variance of the predictive distribution, or both? * line 169-171: could you clarify (also in the final paper) the sentence: Since the parameters of mapping functions are shared for all time series, we can view z_t as a Gaussian process assessed at points hi_t?

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

None in particular.

Summary

The paper proposes a method to improve multivariate probability forecasting by accounting for potential temporal dependencies of the residuals. The paper introduces a dynamic covariance using a small number of latent temporal processes. The method is evaluated on standard dataset of multivariate time-series on two forecasting models, with the CRPS as a metric.

Strengths

The method proposed in the paper is original and clearly motivated. The paper shows improvements of the CRPS accuracy over a number of time-series for two models.

Weaknesses

The paper partly relies on the premise that incorporating time-dependency in the residuals is crucial for providing better uncertainty estimates. Demonstrating that the new residuals generated by the paper's method are less auto-correlated than those without the method is therefore crucial. However, the paper lacks a comprehensive metric for this evaluation. The theoretical introduction could be enhanced for mathematical clarity and efficiency.

Questions

How does the number of time-series influence your method? In particular, it could be interesting to measure the decorrelation of the residuals with your method (with an appropriate metric that averages over all the time-series), with methods without accounting for it. Minor comments. - l18 "vital", this is a strong word - equation (1) is not accurate. Indeed, to simplify, let's take $P=1, Q=2$, and $x = 0$ (in order to remove it). To simplify further, let's assume $z$ takes discrete values. Writing $A=\{z_T=z^0_T\}, B=\{z_{T+1}=z^0_{T+1}\}, C=\{z_{T+2}=z^0_{T+2}\}$, equation (1) implies the equation $\mathbb{P}(B\cap C | A) = \mathbb{P}(B|A)\mathbb{P}(C|B)$ where $\mathbb{P}$ is the probability measure. This identity is wrong. Indeed, one just need to take $A\cap B\cap C = \emptyset$ but $A\cap B \ne \emptyset$ and $B\cap C \ne \emptyset$.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

4

Limitations

The limitations of the method should be explored.

Summary

This paper introduces a method for multi-variate time series forecasting where the residual errors are not assumed to be independent and modeled to be correlated using a Gaussian process model. Standard time-series approaches assume that the errors are temporally independent, however, this assumption does not hold in many datasets due to model misspecification. It is shown experimentally that the residuals have a non-zero correlation between different time steps. The proposed methods accounts for the correlation between the time steps using a Gaussian process approach. The correlation matrix of the GP is designed to be both temporal and spatial. Spatial correlations occur due to a low rank assumption of the correlation matrix in the spatial dimension for a particular time step. Temporal correlations occur due to a non-identical correlation matrix for the latent variables across different time steps. Experimental results on benchmark datasets show improved performance with temporal correlations than without. An extensive analysis and interpretation of the results are provided in the appendix.

Strengths

- The presented approach is well motivated as experimentally shown in the extensive analysis. Residual errors are indeed correlated and hence this is an interesting and significant problem in the time series community. - The proposed method is technically sound. Gaussian processes are an ideal approach to learn correlations between dimensions and hence a natural choice to account for the temporal correlations. - Experimental results show improved performance on standard benchmark datasets when modeling the correlations. A full set of experiments and extensive analysis shows potential of impact.

Weaknesses

- The proposed method may not be a completely novel approach to time series forecasting. Gaussian processes have been used for time series predictions extensively and a simple way to account for temporal correlations is the following: - Learn a deterministic base model on the data, compute the residual and fit a Gaussian process on the residuals. - The Gaussian process kernel is a product of two kernels, one along the time dimension and the other across time series. - It is unclear how this simple baseline will perform compared to the proposed method which is complex. - Some strong baselines are missing from the experiments. Just to give some examples: - Rangapuram, Syama Sundar, et al. "Deep state space models for time series forecasting." Advances in neural information processing systems 31 (2018). - Salinas, David, et al. "DeepAR: Probabilistic forecasting with autoregressive recurrent networks." International journal of forecasting 36.3 (2020): 1181-1191

Questions

- How is the model actually learned? A full algorithm listing all the steps would be very useful. - Are the base models and residual GP model trained independently? Or are they trained end-to-end? - How is inference actually done? Let's say we have historical data up to step t, and would like to predict step t+1. The correlation between the residual errors and information about the residual errors till step t helps predict a more accurate residual for step t. Decreasing correlation over longer time ranges means that the advantage of modeling correlated errors will be small for long horizon predictions. Why don't we observe this phenomena in Figures 19 and 20?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Please comment on the usefulness of the method for various time ranges. It seems that the method may not be an advantage when predicting over longer time ranges.