From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection

Summary

This paper introduces a time series forecasting framework, where LLM-based agents are employed to sift out relevant news to time series of interests and the news are utilized to enhance the accuracy of time series forecasting models.

Strengths

1. The idea of filtering and utilizing news to enhance time series forecasting is innovative and interesting. 2. The whole framework is well designed. Each component is with reasonable motivation. 3. The experimental result is impressive, showing great superiority of the proposed framework on time series forecasting.

Weaknesses

1. Time consumption of the framework should be discussed. 2. The components in the framework are inherited from existing works, making the model itself not as innovative as the idea of the paper. 3. Collecting-then-filtering mechanism of the proposed framework might have negative effects on the real-time performance of the model.

Questions

For real-world application, we may have to collect up-to-date news related to time series of our interests, do the authors have any suggestion of the amount of news?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are well discussed in the manuscript.

Summary

This paper proposes a new framework for time series forecasting. This framework fine-tunes a generative large language model (LLM) to improve forecasting accuracy by integrating news and supplementary information with numerical data and introducing iterative self-evaluation through LLM-based agents.

Strengths

Strength 1: This paper has good originality to identify a unique challenge in time series forecasting task, which is the lack of effective modeling to address the distortions induced by additional random events with time going by. Strength 2: Through Figures 1, 2, 3, and 4, this paper has good clarity to describe the proposed framework and the detailed procedures to complete the time series forecasting task. Strength 3: This paper conducts comprehensive experiments, using time series datasets across multiple domains, to demonstrate the effectiveness of the proposed framework.

Weaknesses

Weakness 1: There could be some statistical analysis on random events compared with normal events which represent a universal knowledge distribution with time going by. Weakness 2: It seems a bit redundant and contradictory for methods 1) and 2). The description of the three-phase prompting could be better organized. Weakness 3: Ablation studies and sensitivity analyses are encouraged. Based on the description of the four scenarios from line 290 to line 297, the news and the supplementary information are always integrated throughout the experiments. Weakness 4: There could be more detailed analysis for Table 2, especially regarding the roles of the evaluation agent.

Questions

Question 1: Apart from prediction accuracy, how to demonstrate the improved reliability? From line 144 to line 145, do sudden shifts embedded in random events from news context help the framework improve prediction reliability? Question 2: From line 216 to line 217, is the understanding of time series influencers, or the sorting based on impact and duration, developed manually by people or automatedly by the LLM agent? What is the difference between such an understanding and a given reasoning logic mentioned from line 218 to line 219? Question 3: From line 277 to line 278, apart from news articles, are there any ablation studies studying the accuracy differences between keeping and removing partial components of or the entire supplementary information? Question 4: According to Table 2, why would introducing more rounds of reasoning selection decrease the forecasting accuracy?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have adequately addressed the limitations and the potential negative societal impact of their work.

Summary

This paper introduces a novel approach to enhance time series forecasting using Large Language Models (LLMs) and Generative Agents. By integrating news content with time series data, the method aims to align social events with fluctuations in time series to provide enriched insights. The approach involves filtering irrelevant news and employing human-like reasoning to evaluate predictions, continuously refining the logic of news selection and the robustness of the model's output. The results show significant improvements in forecasting accuracy by effectively harnessing unstructured news data.

Strengths

- The proposed framework integrates unstructured news data with numerical time series inputs, enhancing the contextual understanding and responsiveness to real-world events. - The use of LLM-based agents for dynamic news selection and analysis is interesting. The agents effectively filter and analyze news content, continuously improving their logic based on forecasting results. - The incorporation of news data significantly improves prediction accuracy across various domains such as finance, energy, traffic, and bitcoin, demonstrating the model's ability to navigate complex real-world dynamics.

Weaknesses

- The performance heavily relies on the relevance of the news data selected. I'm worried that Inaccurate or irrelevant news can degrade the forecasting accuracy. - The method may not perform as well in domains requiring highly localized or specific news data that is not available in general news sources.

Questions

- How does the model ensure the relevance of the selected news items? Could the authors provide more details on the filtering criteria and logic used by the LLM agents? - How does the model mitigate the impact of irrelevant or misleading news on the forecasting results? Are there any mechanisms in place to detect and exclude such news?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed some limitations, such as the dependency on the relevance and quality of news data and the complexity of integrating textual and numerical data.

Summary

The paper proposes a novel method to integrate event news as external information into the time series forecasting system.

Strengths

1. An important problem is studied in this paper. 2. An innovative idea of an automatic relevant news extraction mechanism is proposed. 3. Overall, the presentation is clear and good.

Weaknesses

1. Some questions regarding the experiments need clarification.

Questions

1. Though the idea that relevant event news may positively benefit time series forecasting is intuitively correct, this is not well demonstrated in the paper regarding the datasets used. Specifically, for many datasets tested, it’s hard to imagine what kind of news could dramatically affect them. Of course, it’s impractical to manually evaluate the news considered relevant. Besides the examples already provided, the authors could also test the average number of relevant news items per time window. This could give a rough idea of the distribution of relevant news, which can be used to approximate if the LLM works as expected. 2. Conflicting news could exist within a time window. For example, Elon Musk may praise or criticize a cryptocurrency within a short time frame. It seems that the model doesn’t consider this situation. Can the authors elaborate more on this issue? 3. The iterative analysis results show that the performance after each iteration is somewhat random. Though, in general, the final results are better than the initial iteration, it’s actually hard to predict if one iteration will be better or worse between two adjacent iterations. This raises a concern since the huge computational cost may seem unnecessary. 4. Another concern is that only LLaMA 2’s behavior is tested. Therefore, the sample set of LLMs tested is quite small, making it difficult to predict the behavior of other LLMs using the proposed method.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Thanks to the ACs and the Reviewers.

We sincerely thank the Area Chairs and the Reviewers for their time and effort during the discussion and review process. We are pleased to see that this paper has received consistently positive feedback from the reviewers. In this work, we integrate news events into time series forecasting by fine-tuning a large language model. Additionally, we leverage LLM-based agents to iteratively filter out irrelevant news and apply human-like reasoning and reflection to evaluate predictions. We hope that this paper offers valuable insights into time series forecasting tasks and aids in the application of language models to real-world scenarios. Bests, The Authors