TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables

Summary

The paper introduces `TimeXer`, a novel approach for time series forecasting with exogenous variables. Its main contributions include a new Transformer-based model architecture designed to effectively integrate and utilize exogenous information alongside endogenous time series data. The paper demonstrates the effectiveness of `TimeXer` through extensive experiments on various real-world datasets, showcasing its superior performance compared to **state-of-the-art** methods.

Strengths

- The paper introduces a novel approach to **integrating exogenous variables** into **Transformer-based** models for time series forecasting. - The model design is well-thought-out, and the experiments are carefully executed. - The paper is well-written and clearly articulates the problem, solution, and experimental results. - The work has potential implications for improving forecasting accuracy in various real-world applications.

Weaknesses

- There is no comparison between native models that can be used for forecasting with Exogenous Variables.

Questions

- How is the memory consumption of the model measured? It seems that there is no implementation in the provided code. - The prediction task of TimeXer and previous prediction task such as `iTransformer` and `DLinear` are essentially different types of prediction targets. More details need to be disclosed on how to transfer previous prediction tasks in the text.

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

- The paper adequately discusses the limitations, such as the need for more extensive experimentation and the potential challenges with a large number of exogenous variables.

Summary

The article designs TimeXer, Empowering Transformers for Time Series Forecasting with Exogenous Variables. TimeXer effectively utilizes exogenous information to enhance the accuracy of time series forecasting. Extensive experimental results validate the effectiveness of the proposed method.

Strengths

S1. Unlike general multivariate time series forecasting, this paper introduces a new task for time series forecasting by integrating exogenous variables to predict endogenous sequences, which is a meaningful research direction. S2. The experiments are comprehensive, with detailed validations on datasets from different domains and under various prediction horizons. S3. The method is simple and intuitively effective. S4. The scalability validation in Figure 4 is interesting and convincing.

Weaknesses

W1. There is a potentially slight inconsistency between the description of the method and the scale of the data used in the experiments. The method section suggests that the length of the exogenous variables differs from that of the endogenous variables. However, in the datasets used for both long-term and short-term forecasting, the lengths of the exogenous and endogenous variables are the same. W2. Although the authors extensively discuss related work on considering relationships between variables in time series forecasting, some highly relevant literature is missing [1,2]. It is recommended to include these in the camera-ready version. [1] CrossGNN: Confronting Noisy Multivariate Time Series Via Cross Interaction Refinement. NeurIPS 2023. [2] FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective. NeurIPS 2023. W3. Figure 5 presents the memory usage of TimeXer and some baselines. Essentially, TimeXer and iTransformer share a similar model structure, yet TimeXer consumes less memory. What do you believe accounts for TimeXer being more memory-efficient than iTransformer? W4. Table 4 presents a comprehensive ablation study. However, I would like to see the complete removal of the exogenous variable enhancement module, for instance, by setting the exogenous variables to zero or random numbers, to explore TimeXer's adaptability to exogenous variables. If the performance deteriorates, it may indicate that TimeXer's results depend on the quality of the exogenous variables. If the performance remains stable, it suggests that TimeXer has good adaptability.

Questions

Q1. Have you considered exploiting large language models (LLM) and leveraging rich textual knowledge, to improve the model's ability to understand exogenous variables at multiple granularities? Q2. In most forecasting scenarios, do you think that the historical information of the target sequence itself is more useful, or is the information from exogenous variables more valuable? You can discuss on it freely.

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have thoroughly discussed the limitations of this work in the appendix.

Summary

This article proposes a method based on Transformer modeling to enhance the prediction accuracy of endogenous variables by incorporating exogenous variables.

Strengths

1. The research topic is interesting and has strong practical value. 2. The paper is well-structured, making it easy for readers to understand. 3. The experiments are extensive, demonstrating the effectiveness of the method across various datasets and experimental settings.

Weaknesses

1. The implementation details on how to extend TimeXer to multivariate time series prediction are not sufficiently detailed (lines 190 - 196). 2. The article does not provide a specific explanation for why the non-overlapping patch method is used instead of the overlapping method. 3. Figure 3 shows the prediction performance corresponding to different step lengths of exogenous/endogenous variables, but it lacks experimental results for some extreme cases, such as reducing the step length of exogenous/endogenous variables to 1. 4. It is recommended to explain more clearly in the methods and introduction sections why the global token is applied to the time series prediction task.

Questions

The influence of exogenous variables on endogenous variables is multi-grained, including aspects like periodicity, trend, noise, and stability. Does the Transformer have the capability to capture such multi-grained influences, or does the author think that more promising methods (such as LLMs) could be exploited to capture these associations?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation is discussed sufficiently in appendix.

Summary

The paper presents TimeXer, a Transformer-based model for time series forecasting that integrates exogenous variables using innovative attention mechanisms. Unlike traditional models that either focus solely on endogenous variables or treat all variables equally, TimeXer integrates exogenous information using patch-wise self-attention and variate-wise cross-attention mechanisms.

Strengths

1. Given the growing emphasis on multivariate time series forecasting in recent research, this paper distinguishes itself by focusing on exogenous variables. By integrating the strengths of recent studies and introducing novel ideas, it advances the field in a meaningful way. 2. Unlike other models that specialize in specific types of forecasting, TimeXer presents a versatile framework applicable to univariate forecasting, multivariate forecasting, and forecasting with exogenous variables. This adaptability makes it a comprehensive tool for a wide range of forecasting scenarios.

Weaknesses

1. Methodological Gaps in Handling Practical Situations While the paper presents various practical situations involving exogenous variables, the methodology, specifically the variate embedding section, only proposes embedding through learning parameters. It lacks a detailed methodological approach to address these practical issues effectively. 2. Lack of Intuitive Examples for Exogenous Variable Utilization The paper does not provide intuitive examples of how well the model learns and uses exogenous variables. Including examples where TimeXer can effectively learn from other variables in scenarios where information is only available from external variables would significantly emphasize the model's necessity and practical relevance. 3. Lack of Causality Between Variable Similarity and Prediction Accuracy There is an inherent lack of causality between the similarity of shapes among variables and the accuracy of predictions. In Figure 5 (left), the paper demonstrates score differences when two series look similar versus when they do not. However, this does not convincingly show that the presence of similarly patterned time series aids in prediction accuracy.

Questions

1. The paper would benefit from more intuitive examples demonstrating how TimeXer effectively learns and utilizes exogenous variables. Can you provide practical examples or case studies where TimeXer successfully learns from exogenous variables in scenarios where only external information is available? 2. In Figure 5 (left), the paper shows score differences when two series look similar versus when they do not, but it does not convincingly establish that the presence of similarly patterned time series aids in prediction accuracy. Can you provide more evidence or analysis to support the claim that similarity in variable patterns enhances prediction accuracy? 3. In the experiments, univariate forecasting with exogenous variables and multivariate forecasting were not distinctly separated. Why did you choose not to clearly differentiate between these two types of forecasting scenarios in your descriptions?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are discussed briefly.