CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patterns

Summary

This paper proposes a novel and effective technique to enhance long-term time series forecasting. The proposed technique, Residual Cycle Forecasting (RCF), directly models periodic cycles with learnable parameters, decomposing the learning of time series into periodic cycles and residual components. The residual can be learned with a simple architecture like Linear or MLP, and this technique can be integrated into existing forecasting models. Results show significant improvement achieved by using this technique.

Strengths

* The proposed method is simple and effective. It not only improves accuracy but is also more parameter efficient and can be easily integrated into different backbones. * The paper is well-written with good clarity. Both the problem and analysis are presented clearly. * The experiments are conducted with high quality, providing extensive and comprehensive analysis of the proposed technique. The results are consistent with prior related work. Limitation of the proposed method is also clearly stated.

Weaknesses

The notations are not strictly consistent. The notations for instance normalization (in Section 3.2 and Algorithm 2) are independent from the whole framework and not consistent with the previous problem definition.

Questions

1. Since Figure 2 depicts the entire CycleNet architecture, instance normalization should also be included. 2. The description of alignment and repetition of $Q$ (Lines 152-157) can be further improved. The link to Algorithm 1 should be added, and a schematic figure (even in Appendix) would make it clearer to understand. 3. How extreme points affect RCF in the Traffic dataset should be further explained (Line 255). Extreme points can also affect other methods. It is unclear why this would specifically deteriorate RCF.

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

See Weaknesses and Questions

Summary

The paper introduces CycleNet, a novel time series forecasting method that enhances long-term prediction accuracy by explicitly modeling the inherent periodic patterns present in time series data. The core contribution of paper is introducing the Residual Cycle Forecasting (RCF) technique, which leverages learnable recurrent cycles to represent these periodic patterns and predicts the residuals, significantly improving upon the performance of existing models with reduced computational complexity. CycleNet demonstrates state-of-the-art results across various domains, such as electricity, weather, and energy forecasting, while offering over 90% reduction in parameter quantity, highlighting its efficiency and effectiveness in capturing long-term dependencies for accurate forecasting.

Strengths

1. The author tries to model the period information explicitly in the time series prediction task, and the motivation is intuitive and reasonable. 2. The method designed by the author is reasonable and closely related to motivation. Combined with the experimental results, the author gives a simple but effective method. 3. The author exposes the code and gives a detailed description, which increases the reproducibility of the model. 4. The limitations of this method are clearly discussed and the possible problems are pointed out.

Weaknesses

1. In the introduction, the author's statement establishes a close relationship between long-term prediction and periodic information. In the absence of some experimental support, this is not rigorous. Periodic information may be useful for long-term forecasting in certain situations, but it is not appropriate for all tasks, nor is it the only important information that these tasks require. The author slightly obfuscates these to highlight the motivation of this article. 2. The authors select data sets with different periodicity to illustrate the validity of the model. However, the authors only demonstrate the validity, and experiments can be added to show how the proposed method performs differently on periodically different data sets and discuss the underlying rules. At the same time, it is also worth showing how CycleNet performs on data sets where there is no obvious periodicity. 3. Combined with the results in Table 2 and Table 4, the results predicted with Linear alone are even better than some well-designed methods, whether this is due to the particularity of the data set or some other reason.

Questions

Please refer to the weakness

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to the weakness

Summary

This paper presents a novel technique for improving the accuracy of multivariate long-term time series forecasting. The technique, called Residual Cycle Forecasting (RCF), involves learning the cyclical patterns of time series through recurrent cycles, which can be used as a pre-processing step for any forecasting model. The authors also propose CycleNet, a linear-based model that uses RCF to enhance its predictions. CycleNet first uses RevIn to account for distribution shift, then subtracts the learned RCF from the input data. The backbone of the model predicts the future residual, adds the learned RCF, and reverses RevIN from the outputs to obtain the final prediction. The proposed method is evaluated on eight multivariate time series datasets and compared against several baselines. The results show competitive performance and resource consumption.

Strengths

* Clear and concise writing style * Novel approach to time series decomposition * Comparison of performance and resource consumption with baseline methods * Ablation study and parameter impact analysis (e.g., $W$) * Easy-to-understand model design and components * Informative figures to illustrate model and results * Thorough discussion of results, including strengths and limitations * Code and data provided for reproducibility and transparency.

Weaknesses

* Lack of clarity to specify which results are from the authors (reproduced or produced) and which ones are collected from previous papers (if so, which ones) * Incomplete comparison with existing time series decomposition baselines, such as LD, TDFNet, or SparseTSF (even though this one was in the related works) * Model was not compared to RLinear (also relying on RevIN)

Questions

## Baselines The paper fails to properly compare the proposed CycleNet model with appropriate baselines. Since CycleNet is based on RevIN, it would be more complete to include other RevIN-based baselines such as RLinear [1] for linear-based models and for instance RevInformer (from the RevIN paper) for transformer-based solutions. Although PatchTST uses RevIN, including these additional baselines would avoid any doubt and provide a more comprehensive comparison. Additionally, the paper misses important baselines such as LD [2], which also uses RevIn and also argues for using learnable decomposition rather than moving average. According to the published results, CycleNet appears to be behind LD. But it would be interesting to compare LD against backbone + RCF. Potential references to [3] is also missing. [1] https://arxiv.org/pdf/2305.10721 [2] https://arxiv.org/abs/2402.12694 [3] https://arxiv.org/pdf/2308.13386 Finally, despite being describes as main baselines, why SparseTSF was not included in this paper? Especially, as SparseTSF seems to produce better results than proposal especially for ETT and not for Electricity and Traffic. What would be the reason for such differences? Including such comparisons is required to correctly position CycleNet with all the other baselines and further discuss why CycleNet offers better performance with some datasets and not others. Such discussion is important to further investigate the benefit of time series decomposition for forecasting tasks. ## Visualization The visualizations of the learned cycles are informative, but it is unclear whether these cycles change depending on the prediction horizon or lookback. If they do, it would be interesting to see the differences and whether they can explain the variation in performance for different prediction horizons. Since the cycles are learned along with the backbone model, there could be differences in the learned cycles for different experiments. It would be helpful to plot the cycles for different prediction lengths, such as 96 and 720, and for different lookback lengths, such as 48, 96, and 336. In addition, the cycle may change depending on the backbone. It would be interesting to show the difference if any when RCF is used with iTransformer, DLinear, etc. This would allow readers to better understand whether there are any differences in the learned cycles and whether they are due to the parameters, backbone model or other factors. Figure 3(f) is surprising, and it is unclear whether it represents a household or an industrial building with specific working hours. In my opinion, it looks like there are only 3 main weekly patterns (the large pattern on the left seems to have ~50 time steps = 2 days) and weekends (one pattern seems to have ~24 time steps = 1 day). Therefore, it could be a factory or industrial building with specific working hours (6 days a week). It would be interesting to discuss how RCF impacts previous backbones, such as DLinear. Specifically, it would be helpful to know how much DLinear's performance changes with the addition of RCF and plot it similarly to Figure 4. Finally, it is important to provide Figure 4 and Table 4 for other datasets to give readers a global view of the impact of RCF/CycleNet depending on the dataset and its nature. This would help avoid generalizations that could be incorrect based only on results depicted in the current version, especially for datasets such as Traffic and Solar Energy where the impact might be drastically different. ## Discussion “Traffic dataset exhibits spatiotemporal characteristics and temporal lag characteristics, where the traffic flow at a certain detection point significantly affects the future values of neighboring detection points.” However, the authors do not explain why this is not an issue for the Solar-Energy dataset, where weather conditions at one location may slowly impact neighboring locations in the future. Can authors comment on that? Regarding the electricity consumption dataset, the authors state that "a user’s electricity consumption thirty days ahead does not directly correlate with their consumption patterns in the past few days." However, it is possible that long-term habits may still influence consumption patterns, for instance with a monthly routine. The authors suggest that "the cycle length W depends on the a priori characteristics of the dataset and should be set to the maximum stable cycle within the dataset." However, they do not address how to account for yearly cycles, such as those found in weather datasets that repeat each year with the seasons or datasets influenced by human behavior, such as electricity consumption, which may increase in winter and summer due to heating and cooling needs. It would be helpful to clarify whether RCF is only suitable for mid-range stable cycles or if there is a way to account for longer cycles. This might need to be included in the limitations section. ## Additional points The paper lacks a computation cost study to determine the overhead imposed by RCF on the existing backbone in terms of memory and consumption. It is important to evaluate whether the gain in accuracy justifies the increase in complexity and associated cost. ## Proof-read * “a single-layer Linear or a dual-layer MLP” maybe find another way to express it to avoid repeating too much (cf., abstract, introduction, etc.) * “surpassing complexly designed deep models”-> redundant, ”complex design” or “deep models” * “leveraging this discovery to enhance” should change this as the fact that time series have cycle is not new leveraging this knowledge is the innovation

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Authors have discussed some limitations of their proposal and especially when each channel have different period cycle.

Summary

This paper proposes a learnable Seasonal-Trend Decomposition method (CycleNet) to improve the prediction performance of current long-term multivariate time series forecasting models. Specifically, it firstly model the periodic patterns of sequences through globally shared recurrent cycles and then predicts the residual components of the modeled cycles. Extensive experiments are conducted to evaluate the proposed method.

Strengths

1. The proposed method is a model-agnostic solution applicable to different kinds of models. 2. Although it is simple, it is able to achieve good performance improvement in many cases. 3. Extensive experiments are conduct to evaluate the proposal.

Weaknesses

1. It seems that the proposal (CycleNet) does not work well for complex datasets, e.g., Traffic. It is better to show more results on the same kind of datasets like the PEMS datasets used in the iTransformer paper. 2. The Section 3 is not well written and lacks a lot of details, especially about the Learnable Recurrent Cycles. The authors may consider to reorganize Section 3 and Appendix A.1 to make Section 3 more clear.

Questions

Please refer to the weaknesses.

Rating

7

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Additional Rebuttal (Optional Reading) # Part 1

**Dear Reviewer 24Zo,** Thanks again for you careful review. **It’s the discussion phase now, so we would like to continue to address the uncovered points from the rebuttal phase** (*due to the text length limits in the rebuttal*). *If you find these additional comments any inappropriate (e.g., out of rebuttal limits), please ignore these comments.* > Finally, despite being describes as main baselines, why SparseTSF was not included in this paper? Especially, as SparseTSF seems to produce better results than proposal especially for ETT and not for Electricity and Traffic. What would be the reason for such differences? As mentioned in the rebuttal, we noticed SparseTSF, which was recently accepted at ICML 2024, because it emphasizes the importance of periodicity in long-term forecasting tasks, similar to our work. Thus, we included it in the related work but did not perform a complete comparison due to the submission deadline constraints. We will supplement a complete comparison in the revised paper. Additionally, SparseTSF performs better on the ETT dataset rather than the Electricity and Traffic datasets for two reasons. First, **the ETT dataset is smaller and noisier**. In such cases, the sparse techniques proposed in SparseTSF help the model focus more directly on corresponding elements in historical cycles, reducing interference from less relevant elements and improving overall performance. Second, the ETT dataset's dimensionality (number of channels) is significantly lower than that of the Electricity and Traffic datasets, with the former having only 7 channels compared to the latter's 321 and 862 channels. Note that SparseTSF fundamentally employs a channel-independent modeling approach (parameter-shared) and linear-based methods, which limit its performance on high-dimensional datasets. This is because such scenarios require non-linear capabilities to remember different patterns of multiple channels, or a separate linear layer for each channel (non-parameter-shared) to model each channel's patterns separately [1]. *In contrast, CycleNet’s RCF technique is a fully channel-independent modeling scheme (non-parameter-shared)*, meaning it models each channel's periodic pattern separately. In this case, even with Linear as the backbone, it performs better on high-dimensional datasets because it enhances the model's ability to capture different patterns of multiple channels. Therefore, as shown in our comparison results in *Table 1 of the attached PDF*, **our RCF technique overall still outperforms the Sparse technique** (even on the ETT dataset). *We will include full experiments on more datasets in the revised paper.* [1] Li, Zhe, et al. "Revisiting long-term time series forecasting: An investigation on linear mapping." *arXiv preprint arXiv:2305.10721* (2023). > Figure 3(f) is surprising, and it is unclear whether it represents a household or an industrial building with specific working hours. In my opinion, it looks like there are only 3 main weekly patterns (the large pattern on the left seems to have ~50 time steps = 2 days) and weekends (one pattern seems to have ~24 time steps = 1 day). Therefore, it could be a factory or industrial building with specific working hours (6 days a week). Yes, you are correct. This might represent the typical working hours of a user, for example, working on Monday, Wednesday, and Friday, with the other days off. Indeed, this interesting phenomenon further reveals *the potential value of our proposed RCF technique*, as it may be **a superior way to help data engineers analyze patterns in time series data**. > “Traffic dataset exhibits spatiotemporal characteristics and temporal lag characteristics, where the traffic flow at a certain detection point significantly affects the future values of neighboring detection points.” However, the authors do not explain why this is not an issue for the Solar-Energy dataset, where weather conditions at one location may slowly impact neighboring locations in the future. Can authors comment on that? The Traffic and Solar-Energy datasets are indeed quite different.