PGN: The RNN's New Successor is Effective for Long-Range Time Series Forecasting

Summary

This paper proposed a Parallel Gated Network (PGN) as a successor to RNN, featuring a Historical Information Extraction (HIE) layer to directly capture information from previous time steps. Additionally, it introduces a Temporal PGN (TPGN) framework with two branches to capture both long-term periodic and short-term semantic patterns, demonstrating state-of-the-art performance in long-range time series forecasting.

Strengths

1. This paper compares a variety of cutting-edge methods. 2. The experiments are generally thorough.

Weaknesses

1. The major issue with this paper is the lack of analysis and comparison with significant literature. The entire paper's premise is the traditional RNN's failure in long-term sequence problems due to the long information propagation paths of its recurrent structure. However, as far as I know, SegRNN[1] has already addressed these shortcomings of traditional RNN in long-term forecasting through segmented iteration and parallel prediction. Yet, there is no discussion on this in the paper. Please compare your method with it and clarify your differences and advantages. 2. In Section 2, you should distinguish between Linear-based and MLP-based methods. The former has only single-layer parameter connections, while the latter has multiple layers and can learn non-linear features due to the presence of activation functions. Methods like DLinear and FITS should be classified as Linear-based methods. 3. The description of HIE is unclear: (i) The process shown in Figure 2(a) suggests first performing linear mapping and then zero-padding, which conflicts with Equation 1 in the paper, where H = HIE(Padding(X)), and the actual code. It is recommended to modify Figure 2 to make this clearer. (ii) Line 169 describes that “HIE(·) is a linear layer,” but in practice, the behavior of HIE is more like a sliding aggregation operation of CNN (or TCN) rather than a sorely linear mapping. Given **(ii)**, calling the proposed method RNN-based is debatable since it is more likely TCN-based. 4. You should include an ablation analysis of the normalization layer, explaining its impact on TPGN achieving state-of-the-art results. 5. Although the authors provide source code, it does not include the hyperparameter settings required to reproduce the key results in the paper, meaning there is no directly runnable script. Are the hyperparameters in the main results all defaults? For instance, is TPGN_period=24? If not, providing a complete script file that can be run directly is necessary. [1] Lin, S., Lin, W., Wu, W., Zhao, F., Mo, R., & Zhang, H. (2023). Segrnn: Segment recurrent neural network for long-term time series forecasting. arXiv preprint arXiv:2308.11200

Questions

See Weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have already described the limitation of this work in the paper, namely the lack of modeling multivariate relationships. To some extent, this is not a significant issue because many current cutting-edge studies have demonstrated that focusing solely on univariate temporal relationships can also be effective in multivariate tasks.

Summary

This paper focuses on long-range time series forecasting problems. To address the limitations of RNNs, a novel paradigm called PGN is introduced as an alternative, providing shorter information propagation paths. Building upon PGN, the paper further presents a generic temporal modeling framework named as TPGN, which effectively captures both long-term and short-term periodic patterns, as well as local and global information, through a dual-branch design. The experimental results in this paper demonstrate that TPGN exhibits excellent performance in time series forecasting.

Strengths

S1: This paper proposed a novel paradigm called PGN, which effectively tackles the inherent issues of RNNs through a simple yet powerful design. PGN exhibits a high level of innovation and holds the potential to replace traditional RNNs. S2: TPGN primarily focuses on modeling the temporal dimension. Its dual-branch design makes sense as it captures both long-term and short-term periodicity, as well as the local and global characteristics of time series. Additionally, it is reasonable to set up different univariate forecasting tasks to evaluate TPGN's performance. S3: This paper is well-written, and the presentation of the figures and tables is clear, making it easy to understand and follow. The experimental comparisons are comprehensive, including numerous advanced baseline models such as iTransformer, ModernTCN, FITS, TimeMixer, PDF, WITRAN, and Basisformer.

Weaknesses

W1. For tables with a large amount of content, such as Table 1, it may be beneficial to consider using different colors for highlighting, as it could enhance clarity. Additionally, another option to consider is moving some of the experimental results to an appendix. W2. While TPGN exhibits some advantages in terms of efficiency, I have noticed that it still appears to be challenging to reach the optimal level. Specifically, I have noticed that as the input sequence size increases, the efficiency of TPGN may gradually become inferior to that of iTransformer.

Questions

Q1. Why was the Gated Mechanism designed in PGN this way in Figure 2 (a)? Can this part be replaced with GRU or other RNN variants? Q2. Is it necessary to have two Linear layers in the design of the short-term branch in TPGN?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Summary

The paper introduces a new model paradigm which aims to solve the traditional bottlenecks of RNN models, such as non-parallel computation, gradient explosion/vanishing issues, etc.

Strengths

1. An important problem is studied in this paper. 2. The overall representation is clear and easy to follow. 3. A comprehensive summary of the related work is provided.

Weaknesses

1. The overall contribution is not very significant. 2. Some questions regarding the time complexity and experiments need to be clarified.

Questions

The model proposed in this paper is pretty neat and easy to follow. My questions mainly focus on the time complexity and experiments: 1. I think the total amount of computation done in the PGN should be O(L^2) because it is 1 + 2 + … + L, which is O(L^2). Although the PGN removes half of the computations of the self-attention, the self-attention and PGN still share the same asymptotic complexity. Thus, technically, replacing the PGN with self-attention won’t change the time complexity asymptotically. But I agree that it should be faster than RNN since it enables parallel computation as self-attention does. Also, this is the reason why I think the overall contribution is less exciting than the paper title claims. Can the authors kindly address this issue? 2. The above discussion can also be tested in the Efficiency of Execution experiment. 3. Regarding the experiments, only an input length of 168 is tested. Why did the authors choose to fixate on this input length instead of testing some other options? 4. In the ablation test, it seems that on some datasets (e.g., ETTh1), TPGN’s improvement is very slight compared to its LSTM/GRU/MLP ablations. Can the authors provide some analysis on these cases? 5. I am also interested to see some analysis on what would happen if the PGN is replaced by self-attention.

Rating

5

Confidence

4

Soundness

2

Presentation

4

Contribution

2

Limitations

N/A

Summary

This paper proposes a new network called PGN to capture the long-term dependencies of time series. Based on PGN, this paper further design TPGN for long-range time series forecasting. TPGN consists of two branches to respectively capture the long-term periodic patterns and short-term information of time series. Extensive experiments are conducted to show the effectiveness and efficiency of the TPGN.

Strengths

S1. This paper is easy to follow. The motivations are clearly described by figures. The authors thoroughly analyze the information propagation modes of different time series forecasting models and explore new information propagation path to improve TS forecasting effectiveness. S2. The design of the PGN is novel, which is a completely new network architecture and can effectively solve the inherent problems of classic RNN models. Both experimental results and theoretical analysis show the effectiveness and efficiency of PGN. S3. This paper proposes TPGN upon PGN, which capture both the long-term and short-term characteristics of the time series with low computational complexity. S4. Experiments are sufficient. Five benchmark datasets are evaluated and the most representative models proposed recently are included in the experiments.

Weaknesses

W1. The computational complexity of TPGN is not well discussed in this paper, and it would be better if the inference efficiency was adequately discussed as the time series size increases. W2. Some presentation needs to be improved. For example, it is difficult for readers to quickly get important conclusions on Table 1 and Table 4.

Questions

In table of experiment comparison, could you explain why TPGN-long outperform TPGN in some cases.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors have fully discussed the limitations of their work.

Thanks for rebuttal

I have carefully read the rebuttal and all my concerns have been well addressed. I will accordingly raise my rate. Thanks.