Sample Complexity of Forecast Aggregation

Summary

The authors study the problem of forecast aggregation in a Bayesian setting. Here n experts observe an individual signal that is correlated with the truth, and each expert reports their posterior belief to the principal. The principal aggregates these reports and outputs a prediction, whose quality is assessed using square loss. The object of interest is the minimax sample complexity necessary to obtain an additive excess error of $\epsilon$ compared to the best aggregator that knows the conditional distribution of truth given reports. The authors derive results for arbitrary distributions $P(\omega, s)$ on truth and signals, as well as distributions that factorize as $P(\omega,s) = P(\omega) \prod_i P(s_i|\omega)$. They show that there is an exponential gap in sample complexity between these two cases, and in the latter case the complexity does not depend on the number of experts $n$.

Strengths

- Well written and appears technically sound - Introduces the study of sample complexity to the forecast aggregation literature - Interesting and general results that cover multiple natural settings - A novel lower bound construction for distribution estimation that allows reduction from estimation to aggregation

Weaknesses

- Non-matching upper and lower bounds in most cases - Main technical difficulty lies in construction of lower bound, the upper bounds don't require significant new ideas - Techniques seem limited to the specific loss function + discrete truth/signal setting. Commenting on possible extensions in these directions would be interesting

Questions

Have the authors thought about results for alternative loss functions (e.g. different weight for type2 vs type1 errors) or continuous truth/signal distributions where for example the experts report the condition mean of the truth given their signal?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors address limitations.

Summary

The paper initiates the study of sample complexity of forecast aggregation under a Bayesian forecasting model. In this problem, n experts, each observe a private signal about an unknown binary event, and then report their posterior beliefs about the event to a principal, who then aggregates the reports into a single prediction. The underlying joint distribution is unknown to the principal, but he has access to i.i.d. samples from the distribution. Using these samples, the principal aims to find an epsilon-approximate optimal aggregator, where optimality is measured in terms of the mean squared error between the aggregated prediction and the real event. The authors show that the sample complexity grows exponentially in the number of experts n, but that if the experts’ signals are conditionally independent, then the sample complexity does not depend on the number of experts at all. They further consider the case of non-binary events and weakly/strongly informative experts.

Strengths

The paper is very elegantly written. It presents the setup of the problem clearly, motivates it thoroughly, and initiates an interesting discussion on the fundamental limits of the problem. The proof sketches are quite intuitive and convincing and their implications are reflected and discussed.

Weaknesses

One weakness for me is in the particular choice of mean squared error as an optimality measure. It seems counter-intuitive, given that the experts report their posterior beliefs, which is in essence a minimum error probability optimality measure. It would seem more natural for the principal to look for an aggregation that minimizes the probability of error given the exerts reports. Another weakness is the gap between the upper and lower bounds with respect to epsilon, which might follow from the relatively simple upper bound proposed in the paper. This also bleeds over to the conditionally independent variant. Finally, the choice of averaged squared error as an optimality measure in the multi-outcome events section brings forth quite bizarre looking results. It follows, under this measure, that is the number of events is omega(1/epsilon^2) our task succeeds with zero samples! The authors do clarify this in the appendix, but I would remove it altogether from the paper (or change the measure to additive MSE).

Questions

Did you consider using the maximum a posterior optimality measure for the principal? An added explanation on why MSE is a good choice can benefit your work.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately addressed the limitations of their work

Summary

The paper studies the aggregation of expert opinions in a Bayesian setting for discrete (binary) distributions. The setting further limits each expert to have at most m different opinions, making the problem fully discrete. This reduction enables the analysis of the sampling complexity of the problem, i.e., the minimum number of observed aggregation events (expert opinions and true outcome) required to get an epsilon tight estimate on the true distriution in the total variation norm.

Strengths

The paper approaches the more and more important problem of how to combine different "expert" opinions in a thorough and interesting way. It is well written, clear and, as far as I managed to go into detail, correct. Their approach sparks many new ideas on how to approach aggregation problems and opens up interesting questions on the relevance of (historical) data in aggregation.

Weaknesses

The proof of the lower bound in Section 4.2 doesn't seem intuitive. The mere construction of an example is of course sufficient for a formal proof, but an explanation of why the authors had the idea for this specific construction might be more illuminating. The authors sometimes speak of the "number of signals an expert can possibly observe." Maybe this is better explained as the "cardinality of the signal the experts observe"

Questions

Is it essential in the proof that the experts decide on Bayesian arguments? Or does it suffice to assume that each expert always just makes one of m possible reports, i.e., can we focus only on the joint distribution of r^t and omega and ignore s^t? In Section 6, does bigger Omega really reduce the complexity? This is counterintuitive. What is the source of Theorem 7.1?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors clearly state the limitations of their work, are honest and clear about the current mismatch in their bounds in epsilon, and don't oversell their results. I do not see potential societal impact.

Summary

This paper studies the problem of forecast aggregation: There are n experts who are each given signals about some unknown binary event, and each output a posterior probability based on these signals. We have access to k such reports (corresponding to k events). The task is to aggregate these reports to produce a prediction that is close to the unknown event in expected squared distance. The paper shows that there is an exponential gap in sample complexity (ie k) between the cases when the experts' signals are independent conditioned on the event, versus the general case. In addition, a (slightly loose) upper bound is provided for the arbitrary case, and a lower bound is provided for the conditionally independent case.

Strengths

This is a very clearly written paper, and it studies an interesting problem. The story of the exponential sample complexity difference in the arbitrary signals vs conditionally independent signal setting is a compelling one. Overall, I think this is an interesting paper that deserves to be accepted to NeurIPS. More comments: - It studies a natural extension where the aggregator sees multiple samples from each expert. This is a more realistic setting than the one-shot problem. - There are natural connections to distribution estimation here that are interesting. For this reason, I think this paper will appeal to the CS theory community. - The future directions mentioned are compelling, and will likely lead to several interesting results.

Weaknesses

I don't see any real weaknesses.

Questions

- Is it possible to say something about when you expect the signals to be conditionally independent in the real world? For instance, your paper points to a Kaggle dataset - is there any evidence that there is conditional independence there? Do you expect your algorithms to work well on that dataset? - Can you draw any connections between (variants of) this problem and distribution testing?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Response to rebuttal

I have read the rebuttal, the authors addressed the comments and questions raised thoroughly.