No-Regret Learning with Unbounded Losses: The Case of Logarithmic Pooling

For each of $T$ time steps, $m$ experts report probability distributions over $n$ outcomes; we wish to learn to aggregate these forecasts in a way that attains a no-regret guarantee. We focus on the fundamental and practical aggregation method known as logarithmic pooling -- a weighted average of log odds -- which is in a certain sense the optimal choice of pooling method if one is interested in minimizing log loss (as we take to be our loss function). We consider the problem of learning the best set of parameters (i.e. expert weights) in an online adversarial setting. We assume (by necessity) that the adversarial choices of outcomes and forecasts are consistent, in the sense that experts report calibrated forecasts. Imposing this constraint creates a (to our knowledge) novel semi-adversarial setting in which the adversary retains a large amount of flexibility. In this setting, we present an algorithm based on online mirror descent that learns expert weights in a way that attains $O(\sqrt{T} \log T)$ expected regret as compared with the best weights in hindsight.

Paper

References (36)

Scroll for more · 24 remaining

Similar papers

Peer review

Reviewer XNSv6/10 · confidence 3/52023-07-03

Summary

This paper studies the logarithmic pooling method for prediction using expert advice. At each step, $m$ experts report distributions $p^1, p^2, \ldots, p^m$ over a size-$n$ domain. The goal is to make predictions with a vanishing regret in terms of the log loss. The usual logarithmic pooling returns an aggregated distribution $p^*$ such that each $p^*(j)$ is proportional to $\prod_{i=1}^m[p^i(j)]^{1/m}$, the geometric mean of the predicted probabilities. This paper considers a generalization in which the exponent for each expert is replaced by $w_1, w_2, \ldots, w_m$ that sum up to 1 (instead of $1/m$ identically), and the goal is to learn the right set of weights to achieve a low regret. Formally, each round of the prediction proceeds as follows: First, algorithm selects weights. Then, adversary chooses the experts' forecasts and the outcome subject to a *calibration* condition. The two steps are in sequential order, so that the forecaster is prevented from outputting an arbitrary distribution. The main result of the paper (Theorem 3.2) states that the regret can be upper bounded by roughly $m^{3/2}n\sqrt{T}\log T$ in the $T \gg m, n$ regime. The algorithm uses online mirror descent with the Tsallis entropy regularizer.

Strengths

The problem setting is quite natural and well-motivated. The authors did a good job in explaining why certain assumptions are necessary by providing illustrative examples. While the solution is based on online mirror descent, the analysis of the algorithm requires several new ideas that seem to be non-trivial and of independent interest.

Weaknesses

- The scope of the setting is limited to logarithmic pooling and the log loss. - The tightness of the bound is unclear; an $\Omega(\sqrt{T})$ lower bound is proved for a restricted class of algorithms, while the polynomial dependence on $m$ and $n$ might be sub-optimal. - Several aspects of the setting don't seem sufficiently convincing; see questions below.

Questions

- The setting assumes that the adversary chooses the forecasts and outcome after seeing the weights selected by the algorithm. If both parties act simultaneously, would the setting become significantly easier? (In particular, the obstacle of Example 1.1 still seems to be there?) - What if the weights don't need to sum up to 1? Would that change the expressivity of the aggregation? (Multiplying $w_1, \ldots, w_m$ by the same factor seems to give a different distribution.) - Lines 63--70 argue that we need to keep the algorithm from seeing the experts' advice, so that the algorithm cannot "output an essentially arbitrary function of the forecasts". What would be the practical motivation/justification of this choice?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The main limitations are the assumptions made regarding the setting (e.g., the adversarial choices of outcomes and forecasts must satisfy the calibration condition). These have been clearly pointed out in the abstract and introduction and, the assumptions are formally stated in the paper.

Reviewer n3Hc6/10 · confidence 3/52023-07-05

Summary

Summary of the paper ==================== * The prediction setting explored in this work is harder than the usual "prediction with experts advice" (henceforth abbreviated PwE) setting (e.g. as in the Cesa-Bianchi and Lugosi book), since the learner is required to reveal the expert weights (w_t) *before* observing the expert advice (adversarially chosen by the environment) and the outcome (in the standard PwE model, the mixture prediction is to be done before observing the outcome but after observing the advice). This is crucial for the lower-bound in Example 1.1. * The form of weighted logarithmic pooling considered in this paper to aggregate the advice of the experts (Lines 47-48) is also natural and considered in the previous literature (as minimizing weighted average KL divergence to the expert advice, having external Bayesianality, maintaining log-concavity of densities etc). * The algorithm proposed is again efficient and well-studied in the literature; online mirror descent with the Tsallis entropy regularizer (as in Zimmert and Seldin, 2018). * To beat the lower-bound (Example 1.1) and achieve sublinear regret, the condition imposed on the adversarial joint-sampling of expert advices and outcome is that the all the experts must be calibrated (for each expert, the conditional probability of the outcome conditioned on that expert's advice must match the advice distribution). * With the calibration assumption and an appropriate step-size schedule, OMD+Tsallis is shown to achieve $\tilde{O}(\sqrt{T} \log T)$ regret (for $T \gg m$).

Strengths

* The adversarial prediction setting explored in this work is harder than the usual PwE setting. The form of weighted logarithmic pooling considered is well-studied in the literature (especially in the context of combining Bayesian priors) and this work seems to give the first non-trivial online adversarial prediction regret bound for the logarithmic pooling with respect to the the log loss (to the best of my knowledge). * The algorithm proposed (OMD with Tsallis regularizer) is efficient and well-studied in the literature. * The calibration condition proposed for the experts is extensively studied in the Bayesian inference literature (in the sense that Bayesian posteriors are naturally calibrated). * Quite a bit of detail has been provided for the proofs in the main paper, and these are reasonably sound and well-written.

Weaknesses

* Not much motivation is given for the harder adversarial setting (requiring the aggregator to predict the weights before seeing the expert forecasts). * While the calibration condition is well-motivated in the existing literature w.r.t Bayesian posteriors, the authors have not provided sufficient motivation for why it is practical when treating experts as learners (in a prediction-with-expert-advice setting). It may be particularly problematic when the experts are learners/prediction models which are decoupled from the generation of the adversarial outcomes. With the log loss, no-regret online predictors are of course possible even with adversarial outcomes (using the Laplace or Krichevsky-Trofimov estimators for instance, as in the Cesa-Bianchi and Lugosi book Chapter 9), but it may not be very useful to aggregate such no-regret experts with logarithmic pooling. * There are no experimental results in the paper, nor are any practical applications discussed.

Questions

* Lines 66-70 show how forcing the aggregator to predict before seeing the expert forecasts disallows some form of manipulation by the aggregator (mimicking a linear mixture). Could you please elaborate on why such a situation should be disallowed (when using the aggregator for either theoretical or practical purposes)? * Could you please provide some discussion on why calibration is still practical in the PwE setting when the experts are learners which are decoupled from the adversarial generation of outcomes (i.e. the expert $i$ has fix the forecast distribution $p^i_t$ at time $t$ before seeing the adversarial outcome $j_t$)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Not applicable.

Reviewer RmFD5/10 · confidence 3/52023-07-07

Summary

This paper investigates the logarithmic pooling method for minimizing log loss and introduces the OMD algorithm utilizing Tsallis entropy as a regularizer to update weights for the logarithmic pooling method. By assuming calibrated forecasts, the paper demonstrates that the proposed algorithm ensures a sub-linear regret. Additionally, this paper establishes a corresponding lower bound, indicating that their regret bound is optimal in terms of its dependence on the number of time steps $T$.

Strengths

1. Learning weights for logarithmic pooling studied in this paper is novel as far as I know, which appears to be a crucial direction, particularly due to its natural alignment with log loss. 2. The results and methodologies presented in this paper seem to be novel and the techniques used in this paper may be of independent interest to the community. 3. This paper is well-written. The algorithm is well-motivated, and the theorems presented are rigorous and sound.

Weaknesses

1. It is important to include a discussion of the relevant literature on online portfolio selection in the related work, as it addresses a similar problem to this paper, namely learning parameters using log loss. 2. It is worth noting that the lower bound only applies to OMD with a constant step size, which may limit its effectiveness. Additionally, the optimality regarding the number of experts (m) and outcomes (n) should be discussed in this paper. 3. Providing a more intuitive explanation for the choice of Tsallis entropy as the regularizer would enhance the paper's clarity and value. 4. This paper can be improved by presenting a detailed example to illustrate the significance of the logarithmic pooling method with log loss.

Questions

1. How about the efficiency of the algorithm? Can the algorithm be executed efficiently? 2. Why there is no need to project our decision, as stated in footnote 5?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Reviewer 2zyV7/10 · confidence 4/52023-07-09

Summary

The paper studies no-regret learning in the setting of logarithmic pooling of experts with the logarithmic loss. In this setting, there is a set of experts each outputting a distribution $p_i^{t}$ over outcomes in some finite set $Y$. The task of the learner is to output a vector $w^{t}$. An outcome $y$ in $Y$ is then observed and the learner suffers loss $\sum_i w^{t}_i \log p^{t}_i (y)$ where the sum is over the experts. The loss is then compared to the best choice of weights $w$ in hindsight. Under an additional assumption that the expert probabilities are "calibrated" (which the paper argues is necessary), the paper presents $\sqrt{T}$ regret algorithm. Further, the paper is the first to consider regret in the logarithmic pooling setting and to connect calibration of experts to regret in this setting.

Strengths

The main contributions of the paper in my opinion is the initiation of study of logarithmic pooling + logarithmic loss to the study of online learning. The paper justifies this setting by noting various nice properties this method of aggregation is known to satisfy. Further the relationship of calibration to the regret in this setting is also interesting. It is the key condition under which their algorithm has non-trivial regret.

Weaknesses

The main issue I have with the setting is that beyond the presentation in terms of pooling, this setting seems like an instance of standard online linear optimization. As the paper notes the main difference here is the apriori unbounded sizes of the loss vectors. Viewed in this light calibration is the same as assuming that the losses are bounded ( ). The issue really is whether this is a priori clear or not. The point that makes me lean towards the fact that the calibration is "just assuming" boundedness is that no average notion of calibration (for example calibration on average across time steps) seems to be sufficient (it seems that the counterexample can be modified to be calibrated on average). Thus, the interpretation just as having "good experts" is less clear and seems more like assuming bounded losses.

Questions

- It would be very helpful to elaborate the usual setting of learning with the log loss for comparison since that is well established area of online learning and information theory. The reason this would be helpful is that in the usual setting the actions are unrestricted but it turns out to be (essentially) optimal to consider "linear pooling". The fact that linear pooling comes up naturally in that setting would be a good constrast and good place to compare linear and logarithmic pooling. - The comment about linear pooling and logarithmic regret needs to be considered a bit more carefully. In the usual notion, the learner does not compete against all convex combinations of experts. In fact, there can be polynomial difference between the two cases.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Reviewer 2zyV2023-08-18

We thank the author for the detailed response and apologize for the delay in the response. 1. Regarding the linear pooling, I do not know of an explicit reference but people do study complete against "general" experts and the comment I made was about even simple cases where "taking the convex hull" increases the regret. see https://arxiv.org/pdf/2303.07279 for an example. 2. I agree with the point about approximate calibration. 3. The reason I was saying that this was similar to linear optimization is the following: the actions are the weights $w$ and the loss suffered is $ \sum_i w_i log(p^i_j)$ right? I understand that the loss has some semantics related to predictions but in the end is the game not similar to predicting $w$ and suffering against a loss vector $ (.. log(p^i_j) ... )$ ? This is what I was saying about in this notation calibration seems like a bound on the "size" of the loss. Am I misunderstanding something?

Authorsrebuttal2023-08-18

Response to 1: Thanks for the reference! We will take a look. Response to 3: Ah yes, I believe there's a misunderstanding, and I have a guess about where the misunderstanding is coming from. The issue is that the normalizing constant $c$ in the formula for the logarithmic pool depends on the weight vector. Recall (line 30) that $c$ is the normalizing constant that makes the probabilities add to $1$. Since the logarithmic pool $p_j^*(\mathbf{w})$ is defined as $c \prod_{i = 1}^m (p_j^i)^{w_i}$, the normalizing constant $c$ can be written as $$c = \frac{1}{\sum_{j = 1}^n \prod_{i = 1}^m (p_j^i)^{w_i}},$$ which depends on $\mathbf{w}$. The loss suffered by the aggregator is $$\log p_j^*(\mathbf{w}) = \log \left( c \prod_i (p_j^i)^{w_i} \right) = \log c + \sum_i w_i \log(p_j^i).$$ If $c$ were independent of $\mathbf{w}$, then it would be correct to say that our problem is an instance of linear optimization. However, since $c$ depends on $\mathbf{w}$ (according to the formula above), this is not the case. I think the paper would be made clearer from us clarifying this point, and we will do so for the final version, should the paper be accepted. Thanks!

Reviewer 2zyV2023-08-19

Thanks much for the clarification. This helps a lot. I am convinced that the paper is interesting. But the sake of the reader I would recommend the following changes: - Add to the paper justification for logarithmic pooling - Similarly for calibration. The present presentation makes it appear like logarithmic pooling is a bit ad hoc and calibration is a technical fix that get around an issue. My current understanding is that this is not the case - A more thorough review and exposition of the connection and motivation to learning with the log loss (linear pooling as you call it). The reason this would be helpful is that the linear pooling notion is very well motivated in terms of compression and density estimation. Comparisons would hopefully explain settings where one would be better than the other and maybe also hint at whether there is an interesting deeper theory for the case of logarithmic pooling as well. - Explicitly mention the loss and the challenges that arise due to the normalization term. For example, if possible explicitly relating the challenges to the corresponding linear optimization problem. I trust that the authors will make the appropriate versions of the above changes and I am increasing my score.

Authorsrebuttal2023-08-19

Thanks for the back-and-forth and the suggestions. We will indeed make these changes.

Reviewer XNSv2023-08-18

Thank you for your reply!

I would like to thank the authors for the clarification. I don't have further questions, and my overall evaluation remains positive.

Reviewer RmFD2023-08-19

Reply to authors

I would like to thank the authors for the detailed response. I would keep my score as a (weak) accept.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC