Approaching Human-Level Forecasting with Language Models

Summary

This paper introduces a forecasting system based on Language Models that aims to achieve human-level forecasting capabilities. It presents a system that autonomously searches for relevant information, generates forecasts, and aggregates predictions. Through collecting a large dataset of questions from competitive forecasting platforms, the authors test the system's end-to-end performance. The results indicate that the system performs nearly on par with the crowd aggregate of competitive forecasters and surpasses it in certain scenarios.

Strengths

Strengths： 1. The paper introduces a novel and well-conceived retrieval-augmented LM system, effectively combining information retrieval, reasoning, and prediction aggregation to enhance forecasting accuracy. 2. The paper proposes a self-supervised fine-tuning method that leverages the model's own forecasts to generate training data, thereby improving the accuracy of predictions. 3. The result is based on a comprehensive dataset of questions from multiple forecasting platforms, enhancing its breadth and reliability.

Weaknesses

Weaknesses: 1. Compare to baselines, the system requires significant computational resources due to its summary and multi-sampling operations. Although the authors use some methods to save the cost, report token statistics and cost used by the system and baseline may be necessary. 2. The system prompt base model 3 times and fine-tuned models 3 times, however the baseline is 1 time. Whether this creates an unfair comparison, an obvious baseline might be to sample baseline 6 times and then vote or average. 3. Some description is confused, like in section 6.1, “our averaged Brier score is .179, while the crowd achieves .149”, I don't see the paper say which table the results come from.

Questions

N/A

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Summary

The authors contribute a novel system that approaches human-level forecasting performance. The authors also contribute a dataset of forecasting questions submitted to various human forecasting websites. The authors show that their system generally approaches human crowds. In some settings, where the LLM can selectively submit forecasts, they find that their system even outperforms humans. The authors conduct a series of ablations across their system, and highlight how each component contributes to the overarching forecasting ability.

Strengths

This is a well written paper that tackles an interesting problem for LLM based systems. The dataset it contributes is also quite useful. The ablations are careful, and the Appendix does a good job detailing aspects of the dataset, ablations, prompts, and other design decisions. Providing a system that can forecast events at a level that can rival humans and serve as a complement across decision making also has important practical applications (that the authors also carefully discuss).

Weaknesses

I would cite some crowdworking papers from other fields (e.g. HCI) just to highlight the effectiveness of crowd work in the related work section. Minor: I would’ve liked more qualitative examples at each step of the system in the text: e.g. what the retrieved articles looked like, other questions, etc. instead of hunting through the appendix. Closed models: I wonder how far Llama 2 / 3 could go, given the same finetuning setup. Given that the OpenAI models are closed, it would’ve been nice to see if we could push open source finetuning to achiever similar deltas. How good are the humans on these markets, exactly? They seem public in nature. Is the average forecaster on Metacalculus an uninformed person? I think that is worth emphasizing in the paper—or at least qualifying some of the findings with a crowd of human amateurs. Many of the weaknesses are formulated as questions in the question section.

Questions

- Will the authors release the dataset if/when the paper is published? I think the collected dataset, even if it becomes stale soon, is very useful! - A benefit of this system is that rationale---even if unfaithful---can be extracted from the LLM. Is there any kind of analysis on the underlying reasoning that might shed more light on the types of evidence an LLM finds "important" across forecasting tasks? In general, a more thorough error analysis of the final system would have been nice! - In the selected prediction setting, are there certain subareas or domains where the LLM is more likely to submit a forecast? - Also, is there an ablation with just finetuning and no IR? I’m curious to see how much the LLM system actually uses the retrieved docs. - Purely for curiosity reasons: would the authors expect this to work on stock market predictions? Why or why not? - I’m happy to raise my score a bit if some of these points are addressed!

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes

Summary

The authors benchmark LLMs ability to perform on the task of forecasting, or predicting the outcome of future events. They test several methods and find that ensembling pretrained and fine-tuned LLMs which have access to news sources produces predictions similar to the accuracy of humans.

Strengths

Authors collect dataset and human baseline to benchmark task Authors investigate many model designs and abrasions to understand which factors lead to high accuracy. Authors present a model which is comparable to human accuracy on a challenging new task Authors provide analysis of the difference between model and human predictions distributions. Authors benchmark a number of different models.

Weaknesses

Dataset is relatively small, as there is limited data in existence for which is there a human baseline.

Questions

none

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

yes

Summary

The authors develop a forecasting system that uses news article retrieval and reasoning to predict future events. The system performs with near-human capability and is also complementary to humans. Thorough ablations and evaluations are done to identify that each component of the paper's method provides meaningful improvements to the prediction accuracies of the system.

Strengths

Originality: Tackles a highly impactful and important field of predictions. To my knowledge not many efforts as well-organized as this work have been made towards this. Quality: Methods, evaluations, etc. are done extremely carefully including hyperparameter search, testing multiple dates of prediction, and fine-tuning multiple models. Clarity: Writing is clear and easy to understand. No issues. Significance: Accurate predictions are highly applicable to almost any macro-level problem. This paper and its results are very significant, and the improvement (though small) is better than humans which is already a big deal.

Weaknesses

Overall it is already very great. Kudos to you. A few suggestions to improve the paper: 1. More motivation in the introduction/general paper on why prediction is important: Other reviewers/readers may not understand the degree of importance prediction tasks have in fields such as social science. 2. More generally, I feel that the paper could benefit from having more focus on a story of why this matters, such as a short discussion section. In particular, discussion on scalable oversight/broader impact and how to manage models that have these capabilities would be appreciated. (not necessary): Would be interested in seeing comparisons against human experts if there are any such datasets out there.

Questions

See weaknesses.

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

No Limitations.