Peri-midFormer: Periodic Pyramid Transformer for Time Series Analysis

Summary

This paper introduces Peri-midFormer to capture multi-periodicity in time series data. Specifically, it designs a pyramid structure and attention mechanisms to effectively model complex temporal variations. The proposed methods demonstrates great performance in several time series analysis tasks in the author's experiments.

Strengths

1. This paper compares a variety of cutting-edge methods. 2. Overall, this paper is solid and the authors conducted thorough experiments.

Weaknesses

None.

Questions

1. Were the results of DLinear and PatchTST obtained with a lookback of 96? From the results in Table 12, it seems they were achieved with the default input of 336. Correspondingly, your assertion on line 236 is incorrect. As far as I know, DLinear, PatchTST, and FITS do not use a default input length of 96. FITS has a default lookback of 720. 2. Regarding the Anomaly Detection experiment, did you use point adjustment techniques and manual thresholds? Combining point adjustment techniques with very low manual thresholds (e.g., 0.5%) can easily achieve an F1 score greater than 90 on these datasets [1], even with a randomly generated anomaly score list. Please do not simply state that you followed previous work, as this only exacerbates the legacy issue. Clearly state your stance and clarify this issue appropriately in the paper. 3. Was the ablation study in Table 5 conducted only on the ETTh2 dataset? As far as I know, the ETTh2 dataset is small in scale, has relatively weak periodicity (compared to ETTh1, or multi-period datasets like Electricity and Traffic), and suffers from significant distribution drift. Under such circumstances, any hyperparameter adjustment or other random factors can greatly affect model performance. 4. The Exchange dataset belongs to the financial domain and typically exhibits weak periodicity (due to the unpredictability of financial data). This is evidenced by findings in the DLinear paper, where simply copying the last point can achieve SOTA performance. This contradicts two claims in your paper: (1) Peri-midFormer improves predictive accuracy by extracting periodicity, so how did it achieve SOTA performance on this weak-period dataset? (2) Your Limitation section mentions that Peri-midFormer is not good in scenarios with weak periodicity, yet your experimental results show that your method performs well in this scenario. [1] Wagner, Dennis, et al. "Timesead: Benchmarking deep multivariate time-series anomaly detection." Transactions on Machine Learning Research (2023).

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

As mentioned above, the authors have already discussed the limitations, though I have some concerns for it.

Summary

In this paper, the authors proposed a new method Peri-midFormer, which uses the multi-periodicity of time series and modeling the periodic part of a time series in a pyramid way. They further proposed an attention mechanism to use the neighborhood relation in the pyramid. Extensive experiments on different tasks show the effectiveness of the proposed method. After reading the rebuttal, I raise my rating from 5 to 6.

Strengths

- The idea of using attention in the pyramid structure seems to be novel. - Extensive experiments are conducted for different tasks on benchmark datasets. - The proposed model is light-weight.

Weaknesses

- Some important related works are missing. - There are other related works utilizing the idea of modeling time series in multi-scale, e.g. [1] [2] [3]. - Experiments could be improved. - The authors should clearly state whether they reproduce the results for comparison methods or they copy the numbers from the paper. - In the original paper of PatchTST, the authors use a longer context window than 96. The authors are suggested to tune this parameter for all comparison methods (as many papers did) and provide the best results, rather than fix the context window. - The authors should provide the script for hyper-parameter tuning to improve the reproducibility of the paper. - The ablation study (Table 5) is only shown on one dataset (ETTh2). The authors are suggested to include results in more datasets. - The training and inference complexity and actual time could be analyzed, as well as memory efficiency. [1] TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. ICLR 2024. [2] PERIODICITY DECOUPLING FRAMEWORK FOR LONGTERM SERIES FORECASTING. ICLR 2024. [3] Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting. TKDE 2024.

Questions

- Apparently, the proposed method focused on periodic part forecasting. However, it works pretty well on the non-periodic forecasting task (short-term forecasting). For instance, there is no periodic in the 'Yearly' dataset, the proposed method still performs quite well. Can authors explain the possible reasons? - It is not clear why the proposed methods have a smaller number of parameters than PatchTST. Can the authors explain more? - It seems that the trend part is not used for classification tasks (from Figure 7). Can the author explain more?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Focused only on periodic signal

Summary

The paper introduces Peri-midFormer, a novel transformer-based architecture designed for time series analysis. By leveraging the multi-periodicity inherent in time series data, the model constructs a Periodic Pyramid structure that decouples complex periodic variations into inclusion and overlap relationships among different periodic components. The proposed method incorporates self-attention mechanisms to capture dependencies between these periodic components, achieving state-of-the-art performance across five mainstream time series tasks: short- and long-term forecasting, imputation, classification, and anomaly detection.

Strengths

S1. The concept of decoupling time series data into a Periodic Pyramid is a valid point. This new representation seems to capture the multi-periodicity of time series effectively. S2. The effectiveness of the proposed method is extensively verified on five tasks.

Weaknesses

W1. While the Periodic Pyramid and self-attention mechanisms are well-explained, the model's complexity might pose challenges for practical implementation and scalability, especially for users with limited computational resources. It would be useful if the authors could report in the main body of the paper, at least the main conclusion of the training and inference time for the proposed method, compared with existing baselines, across the five tasks. W2. The improvements provided by Peri-midFormer seem to be insignificant when compared with Time-LLM and GPT4TS on forecasting, imputation, and anomaly detection tasks. It would be better if the author could further discuss this in the main body of the paper (e.g., discuss the time-accuracy tradeoff as shown in Appendix E.4 Complexity Analysis).

Questions

Q1 (cr. W1). Add in the main body of the paper a discussion about the training and inference time, as well as some critical results and main conclusions. Q2 (cr. W2). Add in the main body of the paper a discussion about the significance of the improvements or about the time-accuracy tradeoff for Peri-midFormer against Time-LLM and GPT4TS on forecasting, imputation, and anomaly detection tasks.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

L1. It is not clear how the proposed periodic pyramid can be effectively and efficiently integrated into multi-dimensional time series.

Summary

The abstract succinctly introduces the Peri-midFormer, a novel approach designed for time series analysis, acknowledging the challenges posed by the discrete nature of time series data and the complexity of capturing periodic variations directly. It proposes a method to address these challenges by decomposing complex periodic variations into hierarchical periodic components, termed the periodic pyramid. This approach leverages inclusion and overlap relationships among these components, mimicking the natural pyramid structure observed in time series data.

Strengths

Innovative Approach: The concept of a periodic pyramid to model time series data is innovative and promises to address the limitations of traditional methods that struggle with capturing complex periodic patterns. Hierarchical Representation: By representing time series as a pyramid with progressively shorter periodic components, the model potentially enhances the understanding of multi-scale temporal relationships. Self-Attention Mechanism: Incorporating self-attention into the periodic pyramid allows capturing intricate relationships among periodic components, which is crucial for tasks like anomaly detection and forecasting.

Weaknesses

Complexity and Scalability: The introduction of a hierarchical pyramid structure combined with self-attention could potentially introduce computational complexities and scalability issues, especially with larger datasets or real-time applications. Addressing these concerns in the paper would strengthen its practical utility. State-of-the-Art Comparison: It is not explicitly stated whether Peri-midFormer achieves state-of-the-art (SOTA) performance when compared to strong baseline models. Without clear comparative results, it is difficult to ascertain if the proposed method truly represents an advancement over current leading approaches in time series analysis. Interpretability: The abstract lacks interpretability regarding why the periodic pyramid structure exists in the applications and why it plays a critical role in improving forecasting. Providing a rationale or theoretical justification for the efficacy of the pyramid structure in capturing temporal patterns would enhance the understanding and acceptance of the proposed method.

Questions

see weakness

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

see weakness