Frequency-domain MLPs are More Effective Learners in Time Series Forecasting

Summary

This paper presents FreTS, a framework that addresses both channel-wise and time-wise dependency learning in the frequency domain for time series prediction. FreTS introduces a specially designed frequency-domain MLP structure that processes the real and imaginary parts of the frequency components interactively. The experimental results on 13 benchmarks demonstrate the superior performance of FreTS compared to other state-of-the-art methods.

Strengths

- The two-stage framework and the frequency-domain MLP module introduced in FreTS exhibit some novelty. - The experimental results across 13 benchmarks show the potential of the proposed method.

Weaknesses

- The authors failed to motivate their method both theoretically and in relation to prior work. 1) The authors did not clearly define and explain the terms "point-wise mappings" and "information bottleneck", not elaborate on how existing methods suffer from these issues, and not clarify how the proposed method addresses and alleviates these problems. 2) While the authors claim that their method has a "global view", it did not highlight how it differs from previous MLP-based and Transformer-based methods in capturing long-range dependencies. 3) The claim of the advantage of “energy compaction” is not supported by experimental evidence.   - The related work section lacks logical organization and fails to provide a comprehensive summary of previous methods.   - The method is not well presented. 1) For the Frequency Temporal Learner, the validity behind performing DFT along the channel dimension needs further discussion. It is more likely to be based on engineering considerations than a theoretical grounding. 2) The rationale behind the design of the proposed frequency-domain MLP is unclear regarding why it calculates the new real and imaginary components as Eq. (7).   - Additional experiments are needed to highlight the benefits of the proposed approach, e.g. the learned global periodic patterns, and the robustness towards noise. Including these experiments will strengthen the empirical evidence for the effectiveness of the proposed method.   - Certain phrases or sentences lacks clarity, and need improvements. E.g., “stacked MLP layers”, “learning in the frequency spectrum helps capture a greater number of periodic patterns”, .etc.  

Questions

- Are the “Domain Conversion” and “Domain Inversion” in Fig. 2 differentiable? How is the network trained end-to-end? - What are the meanings of the axes in Fig. 1? Whether the results are from a time series or a dataset?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors are encouraged to discuss the computational complexity after introducing frequency transform and inverse transform into the framework, and its performance on irregular time series.

Summary

This paper investigates time series forecasting in the frequency domain. By utilizing MLPs in the frequency domain, the proposed FreTS effectively captures the patterns of time series with a global view and energy compaction. Frequency learning is applied to both inter-series and intra-series scales, allowing FreTS to capture channel-wise and time-wise dependencies. Extensive empirical experiments demonstrate the effectiveness of FreTS in both short-term and long-term forecasting tasks.

Strengths

1) This paper redesigns MLPs in the frequency domain to effectively capture both time-wise and channel wise correlations. The use of simple MLPs ensures high efficiency and helps mitigate overfiting issues compared to existing sophisticated deep models. 2) This paper is well-organized, with comprehensive discussion and experiments.

Weaknesses

1) The authors mention that learning time series in the frequency domain has the nature of energy compaction, as the energy concentrates on the smaller part of frequency components. However, the proposed FreTS still retains all the frequency components when performing MLP in the frequency domain, thereby not fully leveraging this advantageous characteristic. 2) Frequency domain modeling is advantageous for capturing temporal dependencies due to the inherent periodic patterns in time series. However, the suitability of frequency domain modeling for exploring channel-wise correlations requires further discussion and verification. Moreover, conducting channel-wise modeling for each time point is inefficient, and should be performed at a coarser granularity, such as segment-wise or series-wise. 3) The dimension extension block takes an input $X_t \in \mathbb{R}^{N \times L}$ and produces an output $H_t \in \mathbb{R}^{N \times L \times d}$. However, if the number of variables $N$ or the input length $L$ is large, the memory usage of $H_t$ can become very high.

Questions

1) Short-term forecasting comparison shown in Table 1 is suggested to include N-BEATS[1] and N-HITS[2] as baselines as well. 2) Figure 4(b) demonstrates that the parameter number of FreTS remains constant as the prediction length increases. However, the projection block described in subsection 3.1 includes learnable parameters $\phi_2 \in \mathbb{R}^{d_h \times \tau}$ and $b_2 \in \mathbb{R}^\tau$, and the number of these parameters will increase with the prediction length $\tau$. 3) From which learner (frequency channel or frequency temporal), and at which layer are the weights used for obtaining the visualizations? Reference [1] Oreshkin B N, Carpov D, Chapados N, et al. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting[C]//International Conference on Learning Representations. 2019. [2] Challu C, Olivares K G, Oreshkin B N, et al. NHITS: Neural Hierarchical Interpolation for Time Series Forecasting[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(6): 6989-6997.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

In this paper, the authors investigate the problem of time-series forecasting. Since the frequency domain can preserve the information from a global view and enjoy the advantage of energy compaction, the authors propose the FreTS model, which is composed of the Frequency Channel Learner and the Frequency Temporal Learner. The authors further prove the reasonableness of the frequency-domain MLPs and evaluate the proposed idea on several datasets.

Strengths

1. The authors investigate the time-series forecasting problem from the frequency domain and provide convincing reasons for the advantage of the frequency domain. 2. The authors devise the frequency channel learner and frequency temporal learner, which address different challenges in a unified framework. 3. The authors investigate the proposed method on several datasets and achieve the ideal performance.

Weaknesses

1. To capture the dependencies among channels, the authors propose the frequency channel learner, which applies the FreMLP on the variables from each timestamp, e.g. $H_t^:$. However, Since it is strange to consider $H_t^:$ as a time series, I think it might be not a good idea to employ FreMLP to capture these dependencies. In my opinion, I think the CNNs, the attention mechanism, and GNN might be good choices. It is suggested that the author should try different methods to better capture these dependencies. 2. The authors address the frequency-domain MLP from time-series forecasting, but some time-series data might be not seasonal and contain some monotonic tendency, can the proposed method address this problem?

Questions

Please refer weaknesses

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Please refer weaknesses

Summary

The authors argue that MLP-based forecasting methods suffer from point-wise mappings and information bottlenecks and explore an interesting direction of applying MLPs in the frequency domain for time series forecasting. They further analyze the inherent characteristics of frequency-domain MLPs and propose the FreTS model via stacking frequency-domain MLPs for time series forecasting. The paper provides theoretical guarantees and extensive empirical evaluation to analyze the advantages of frequency-domain MLPs and show the superiority of FreTS, verifying the authors’ arguments and the effectiveness of frequency-domain MLPs.

Strengths

1. The authors study an interesting neural network of frequency-domain MLPs and analyze the advantaged effectiveness of frequency-domain MLPs compared MLPs in the time domain. This potentially inspires the MTS forecasting community to pay more attention to frequency analysis. 2. Frequency-domain MLPs are theoretically proved to be energy compacting and equivalent to global convolutions. FreTS based on frequency-domain MLPs is straightforward in architecture and theoretically sound. 3. The experimental results are comprehensive and impressive. Extensive results on 13 datasets validate the superiority of FreTS and the effectiveness of frequency-domain MLPs for both short-term forecasting and long-term forecasting. Model analysis and efficiency analysis are also provided to investigate the advantages of FreMLP over MLP and the higher efficiency of FreTS over SOTA baselines. 4. The visualization analysis provides quite interesting patterns and clearly shows the distinct characteristics between the time domain (MLPs) and the frequency domain (frequency-domain MLPs). Especially, the visualizations of the learned weights in the frequency domain show highly concentrated values in the diagonal, verifying the energy compacting and learning efficiency of frequency-domain MLPs. 5. The paper is well-organized and easy to follow, and the main contributions are quite clear and solid. In my opinion, it studies an interesting topic and provides a new perspective on incorporating frequency analysis into time series analysis.

Weaknesses

1. The experimental results are convincing, but only one MLP-based baseline is compared in the experiments. Additional MLP-based baselines help to verify the advantages of FreMLPs over MLPs. 2. The superiority performance of FreTS verifies the effectiveness of (one layer) frequency-domain MLPs, but the authors did not investigate how well multi-layers of frequency-domain MLPs perform in MTS forecasting.

Questions

1. The learned patterns on the frequency spectrum show that FreTS can capture more obvious periodic patterns. But when there are no periodic patterns in MTS data, can FreTS still perform well? That is, whether FreTS is suitable for non-periodic MTS data. 2. Why FreTS contains only one layer of FreMLP in either frequency channel learner and frequency temporal learner? How well does FreTS work when stacking with multiple FreMLPs? === I have read the rebuttal and would like to keep my rating.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper studies time series forecasting problem under the deep learning paradigm. Authors propose a new network architecture with MLPs in the frequency domain to capture both inter-series and intra-series correlations.

Strengths

1. The paper is well-written and easy to follow. 2. Learning the spatio-temporal correlations in the frequency with MLPs seems reasonable and shows lightweight in the experiment part. 3. Experiments with diverse datasets and solid comparison methods demonstrate the effectiveness of the proposed method.

Weaknesses

1. As mentioned in the related work part, there are already several works in the frequency domain for time series forecasting. It should be clearly discussed what’s the difference of this work. 2. Since the efficiency is a core merit of the proposed method, it would be better to analyze the theoretical computation complexity compared with other method. 3. While the frequency method may improve the efficiency, it is not clear whether these kind of methods would mainly focus on the low-frequency part but ignoring the high-frequency part, which is more important for time series forecasting.

Questions

Please check the weakness

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Reply to rebuttal

Thanks for the authors' effort in providing the rebuttal. This helped clarify my concerns. I am happy to keep my current rating.