BasisFormer: Attention-based Time Series Forecasting with Learnable and Interpretable Basis

Summary

To be effective, a basis must be tailored to the specific set of time series data and exhibit distinct correlation with each time series within the set. As far as we know, the state-of-the-art methods are limited in their ability to satisfy both of these requirements simultaneously. To address this issue, the authors propose BasisFormer, an end-to-end time series forecasting architecture that leverages learnable and interpretable bases. Firstly, the authors acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series as two distinct views and employs contrastive learning. Secondly, the authors design a Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional cross-attention. Finally, the authors present a Forecast module that selects and consolidates the bases in the future view based on the similarity coefficients, resulting in accurate future predictions.

Strengths

1. The authors acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series as two distinct views and employs contrastive learning. 2. The authors design a Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional cross-attention. 3. The authors present a Forecast module that selects and consolidates the bases in the future view based on the similarity coefficients, resulting in accurate future predictions.

Weaknesses

1. The authors claim that this is a self-supervised learning model, and according to the reports of existing studies, the performance should be weaker than that of supervised models. I hope the authors can compare some time series prediction models based on self-supervised learning. Forgive me for being skeptical about the effect at the moment. minor comment: 2. I would like the authors to be more clear about Eq. 7, why explainability is related to the smooth term, and to use the common l2-norm.

Questions

The authors addressed all my concerns. So, I have no further comments.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

This paper addresses the problem of finding effective bases for time series forecasting models. Current methods are limited in their ability to satisfy the requirements of being tailored to specific time series data and exhibiting distinct correlation with each time series. To tackle this challenge, the authors propose BasisFormer, an end-to-end architecture that leverages learnable and interpretable bases. Bases are obtained through adaptive self-supervised learning, where historical and future time series sections are treated as distinct views and contrastive learning is used. The proposed architecture includes a Coef module that calculates similarity coefficients between time series and bases using bidirectional cross attention, and a Forecast module that selects and consolidates bases for accurate future predictions. Extensive experiments on six datasets demonstrate that BasisFormer outperforms previous methods for both univariate and multivariate forecasting tasks, achieving considerable improvements in performance.

Strengths

This paper has studied a time series forecasting problem. The paper is well written. The significance of this work lies in addressing the limitations of existing methods in time series forecasting by proposing BasisFormer, an architecture that leverages learnable and interpretable bases. This approach allows for tailored modeling of specific time series data and distinct correlations with each time series, leading to improved forecasting accuracy.

Weaknesses

However, I still have several concerns towards this paper. First, the illustration and motivation for the basis/bases is not very clear. What are the basis in essence? How can you define basis in time series? Are basis really needed for time series forecasting? What good effects can basis learning bring up? How to verify it? Second, in the main contribution, though the authors combine several techniques together, including self-supervised learning, contrastive learning, cross attention, basis selection. However, I don't find the necessity for each component. Third, the experimental results are less convincing. The proposed method adopts contrastive learning, but it does not compare the performance with other self-supervised learning methods, such as [1] [2]. Other works adopt periodic basis are also not discussed in related work [3]. [1] Cost: contrastive learning of disentangled seasonal-trend representations for time series forecasting. ICLR 2022. [2] Time-Series Representation Learning via Temporal and Contextual Contrasting. IJCAI 2021. [3] DEPTS: Deep Expansion Learning for Periodic Time Series Forecasting. ICLR 2022.

Questions

see weakness

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

n/a

Summary

This paper studies basis learning for time series forecasting for which the past and future basis representations are aligned. Contrastive learning is used to build the time series basis and similarity between the past values and basis is used for time series prediction. The experiments on several time series forecasting datasets show improvement, especially for multivariate forecasting, when the input length (history) is short.

Strengths

- The paper is overall well written - The idea of using contrastive learning to learn basis is sound - The proposed approach improves over baselines especially for multivariate case

Weaknesses

- There is a strong emphasis on the consistency of the representations for past and future but this is directly experimented/evaluated - Reproducibility is questionable without source code and scripts to run as the approach composed of several components and steps of training - Proposed approach is particularly improves over baselines when the history to predict is shorter, however the motivation/need for a shorter history is not supported

Questions

- Basisformer seems to have more advantages over baselines for the multivariate time series forecasting, how do you explain this? - How models are trained for the longer input sequence? Is it only inference time change? - Increasing the input length gives more advantage to the baselines. Do you conduct further analysis of the inference time versus performance for different length of input? - Do you plan to share the implementation?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

This paper proposed BasisFormer which is an end-to-end time series forecasting model that leverages learnable and interpretable bases. BasisFormer treats the historical and future sections of the time series as two distinct views and using contrastive learning. By making use of Coef module and Forecast module, the BasisFormer outperforms previous state-of-the-art methods for univariate and multivariate forecasting tasks.

Strengths

1. Contrastive learning objective is applied for basis learning which guarantees the consistency between the historical and future sections of the time series. And when applying the SSL module to other frameworks, there is a performance improvement of approximately 5%, which suggests the general application of the learnable basis. 2. Based on the experiment results on six datasets, the proposed BasisFormer model outperforms previous SOTA methods on univariate and multivariate forecasting tasks. 3. The network architecture of BasisFormer is carefully designed and well analyzed through ablation studies and model comparison. The paper is well written and easy to understand.

Weaknesses

In 4.3, the author analyzed the interpretability of the learned bases by visualizing the time series and the corresponding learned basis. An additional visualization of the attention distribution in BCAB module can be helpful for checking different weights assigned to each basis in different attention heads, and thus understanding the network behavior.

Questions

At this point, I don't have specific questions to ask. The paper is clearly written.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The authors of the paper propose a time series forecasting architecture with a self-supervised method for basis learning, called BasisFormer. Their assumption is that the selection of basis for a time series is consistent across both historical and future sections of the time series. They introduce a Coef module that measures the similarity between the time series and the basis in the historical view via bidirectional cross-attention, and a Forecast module that consolidates vectors from the basis in the future view according to the similarity yielded by the Coef module. In their evaluation they demonstrate improvements in forecasting tasks.

Strengths

1. The paper is well-written with clear explanations of the proposed architecture. 2. The empirical results, including the ones that were presented in the supplementary material, support the original claims across the manuscript.

Weaknesses

1. In order to further validate the claims presented in this work, I would expect seeing another comparison to methods involve discretization of time-series, such as [Moskovitch, R. and Shahar, Y., 2015. Classification of multivariate time series via temporal abstraction and time intervals mining. Knowledge and Information Systems].

Questions

1. What happens if bases are not learnable directly from the time-series, but require a domain expert's KB?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The novelty of this work is limited to cases where bases do exist and are detectable automatically in the input time-series, where in many cases the provided input is shorter than the length required for identifying cases of seasonality, for example.

Thanks to the author reply, which addressed all my concerns.

I have no further questions and I have improved my confidence score.