Smoothed Analysis of Sequential Probability Assignment

We initiate the study of smoothed analysis for the sequential probability assignment problem with contexts. We study information-theoretically optimal minmax rates as well as a framework for algorithmic reduction involving the maximum likelihood estimator oracle. Our approach establishes a general-purpose reduction from minimax rates for sequential probability assignment for smoothed adversaries to minimax rates for transductive learning. This leads to optimal (logarithmic) fast rates for parametric classes and classes with finite VC dimension. On the algorithmic front, we develop an algorithm that efficiently taps into the MLE oracle, for general classes of functions. We show that under general conditions this algorithmic approach yields sublinear regret.

Paper

References (65)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer 4UMy6/10 · confidence 4/52023-06-07

Summary

This paper focuses on the contextual sequential probability assignments, and specifically examines cases where the contexts $x_{1:T}$ are generated by $\sigma$-smooth adversaries as introduced in [Haghtalab et al. 2021], and where the labels $y_{1:T}$ are adversarially generated. The primary findings pertain to the minimax regrets under the logarithmic loss, as compared to a generic set of experts $\mathcal{F}$. More specifically, the paper demonstrates the following: 1. If the class $\mathcal{F}$ possesses a scale-sensitive VC-dimension of order $\epsilon^{-p}$, then the information-theoretical minimax regret grows as $T^{p/(p+1)}\cdot \text{poly}\log(T/\sigma)$, which is tight up to polylogarithmic factors. 2. An efficient algorithm exists, given an MLE oracle, that attains regret of order $T^{2/(2+w)}\sqrt{1/\sigma}$, provided the Rademacher complexity of $\mathcal{F}$ is of order $T^{-w}$. For the VC-class, specifically where $w=1/2$, this algorithm yields a regret bound of order $T^{4/5}$. 3. For the VC-class, it demonstrates that the information-theoretical regret is of the order $d\log(T/\sigma)$ up to a constant factor (the same result was also found in concurrent work [Wu et al. 2023]). The principal contribution of this paper that distinguishes it from previous work on contextual sequential probability assignments is the introduction of an oracle-efficient algorithm that achieves sublinear regret. Although the primary techniques are rooted in the FTPL (Follow The Perturbed Leader) framework as outlined in [Haghtalab et al. 2022], this unique contribution sets a new precedent in this line of research.

Strengths

The primary strength of this work lies in the oracle-efficient algorithm, which presents the first known instance of computationally efficient sublinear regrets under log-loss for general non-parametric classes, albeit using an MLE oracle. This is likely to inspire further research in this field.

Weaknesses

The main weakness is the novelty of the techniques used in this paper. From a purely technical standpoint, Algorithm 1 and its analysis seem to merely mimic those used in [Haghtalab et al. 2022]. In fact, there are many places in the proof that appear to be verbatim copied! There are also some presentation and technical issues, which I outline below: - There are many places where the index of summation is $i$ but the summand is written as $s_t$, for instance, in Lemma 4.2. I suggest the authors conduct a thorough proofreading, including the appendix, to correct issues like this; - I recommend that the authors include a proof for Corollary 4.1.1. It's unclear to me how Theorem 4.1 implies this bound for certain values of $n$, $\alpha$, and $m$; - The authors claim that a $T^{3/4}$ bound could be achieved for an alternative oracle. This is not clear to me at all. Are you refering to the Theorem 7 of [Block et al., 2022]? If so, it should be properly cited and clearly explained by what you mean "mixed objective function"; - In line 636 (appendix), it appears that Lemma 3.3 from [Haghtalab et al., 2021] requires $\epsilon> \frac{\sigma}{T\log T}$? The choice of $\epsilon=\frac{\sigma}{T^2}$ does not align with the condition stated in that Lemma. I believe a stronger concentration result, such as Lemma 21 from [Wu et al., 2023], would be required in this context; - In the proof of Lemma 4.4, presented in Appendix G, you assert that the distributions $P, Q$ are solely dependent on the ensembles $\\{n_0(x),n_1(x)\\}$. However, I contend that this is not a self-evident fact. It is essential, at the very least, to posit that your OPT oracle is permutation invariant with respect to the input sample sequence. This assumption, it should be noted, does not necessarily hold for all optimization methods; stochastic gradient descent, for example, may not exhibit this property. These are the primary reasons I give a "weak accept".

Questions

I have the following questions for the authors: 1. It seems that your FTPL algorithm relies on an exact ERM oracle. I'm curious, how would the performance or operation of the algorithm change if the oracle was only approximately optimal? 2. Is there any computational lower bounds as in [Haghtalab et al. 2022] that can be proved for the log-loss? 3. Your bound in Theorem 4.1 seems to rely solely on the Rademacher complexity. Does this suggest that your analysis isn't capable of achieving a $\log(T)$ dependency for the VC-class? For instance, consider the linear function $|\langle w,x\rangle|$ with $w,x$ in a unit ball (of a Hilbert space). This class admits a $1/\sqrt{T}$ Rademacher complexity, but it's known that the information theoretical lower bound is $T^{2/3}$. 4. Is it possible to achieve better dependency on $T$ with your FTPL algorithm if the contexts are generated from an i.i.d. process (i.e., with $\sigma=1$)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

No issue with negative societal impact.

Reviewer 9j8s6/10 · confidence 2/52023-06-19

Summary

The paper studies the sequential probability assignment problem with a smoothed adversary. The learner sequentially assigns a probability given contexts, which the adversary generates from a distribution that can be far from a known base distribution by a factor of $1/\sigma$. For the problem with the i.i.d. setting, there is an $O(\log T)$ regret method, while we must incur $\Omega(T)$ regret in the pure adversarial setting. Thus, it is well motivated to study what regret is possible in the smoothed setting with respect to $T$ and $1/\sigma$. The authors provide a general regret bound via reduction to transductive learning. E.g., if the class of functions that map contexts to a probability has the VC dimension of $d$, the regret is $O(d \log (T/\sigma))$, achieving the logarithmic dependence on $T$ and $1/\sigma$. The authors also present an FTPL-style algorithm that is efficient in terms of calls to MLE oracles. It is shown that the algorithm attains $T^{4/5}\sqrt{d/\sigma}$ regret for the VC class.

Strengths

1. The paper studies an interesting online learning problem under the smoothed setting, which is well motivated by the known results in stochastic and adversarial settings. 2. The results imply some interesting consequences. Although I'm not an expert, I found Section 3.2.2 to be interesting, where the authors discuss the benefit of their covering-argument-free idea.

Weaknesses

1. Theorem 3.1, the reduction to transductive learning, seems to be relying heavily on the coupling lemma by Haghtalab et al. [2021]. 2. The analysis of the oracle-efficient algorithm in Section 4.1 also appears to be relying on Haghtalab et al. [2022].

Questions

1. I would appreciate it if the authors could describe intuitive technical differences from Haghtalab et al. [2021, 2022]. 2. The scale-sensitive VC dimension appears to be related to the fat-shattering dimension. Is there some connection between them? #### Minor comments - In Algorithm 1's input, capitalization is inconsistent. - In Algorithm 1, line 2, a period or a line break is missing before "Call the oracle..." - I prefer references arranged in alphabetical order. - Hyperlinks do not work.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Although limitations are not explicitly discussed, the range to which their results apply seems clear from the description.

Reviewer X6XT6/10 · confidence 4/52023-07-05

Summary

The paper considers the sequential probability assignment problem, which is the following: The algorithm (forecaster), based on past context and outcomes, must assign probabilities to 0-1 values for the next outcome given the latest context. The forecaster competes against a reference class of predictor functions (experts) and aims to suffer low regret compared to the best of these functions. For a logarithmic loss incurred for incorrect predictions, regret is defined as the difference in the total log-loss of the forecaster and the best of the experts. When the contexts are drawn independently from an unknown distribution, and the class of predictor functions is special (say, has a bounded VC dimension), the sequential probability assignment has low (i.e. sublinear) regret. However, if the contexts are adversarial, there are impossibility results even for simple classes of predictors. Therefore, following Haghtalab et al. (2021), the authors study a smoothed version of the problem where contexts can only be chosen adaptively from the set of sigma-smoothed distributions. They study how the properties of the class of the predictor functions like bounded “scale-sensitive” VC dimensions affect the regret. The main tool used is a powerful coupling lemma from Haghtalab et al. (2021) that converts the problem with an adaptive sequence of t contexts to a problem with t*K uniformly and independently distributed contexts (the length of the sequence increases but they are no more adaptive). Thus, results for i.i.d. samples may be applied where the regret is essentially logarithmic in T (number of time steps), which is the first result of the paper. The second question studied is an algorithmic approach to solving the problem using calls to MLE. A follow the perturbed leader type algorithm using ideas generalized from Haghtalab et al. (2022) is used. Here, however, the regret is not logarithmic but T^0.8. This is the second main result in this paper.

Strengths

The paper is well-written and the considered problem is natural. The results are interesting.

Weaknesses

It seems the first main result of this paper is already presented in a paper that has been accepted at COLT 2023 (Wu et al.) Even though this seems to be independent work of a different group of authors, I would still find it strange if the same result was now also accepted at NeurIPS, in particular because the proofs are quite similar. Both rely essentially on the coupling arguments from the work of Haghtalab et al. I also have to say that my impression is that the known coupling argument is really the key ingredient and the proof in this submission is "only" an adaption of the arguments by Haghtalab to the log-loss setting. Maybe the authors could point out more clearly if there were significant challenges in this adaption. The second main result is not contained in the COLT paper. However, it unfortunately only shows a regret of T^0.8 and no logarithmic one and it also follows known algorithmic approaches from the literature like follow the perturbed leader.

Questions

--

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

n.a.

Reviewer oQa87/10 · confidence 3/52023-07-08

Summary

The paper discusses smoothed analysis of probability assignments in an online setting. The paper shows how the problem can be reduced to a transductive setting and obtains an upper bound on the regret using covering numbers which is further bounded by the scale sensitive VC dimension. The results are instantiated for various classes of functions, including VC function class and the non-parametric class where the scale sensitive VC dimension grows as \epsilon^{-p} at scale \epsilon. On the algorithmic side, the authors propose an oracle efficient FTPL style algorithm to predict the probability assignments and show weaker regret bounds for the same (by bounding the Rademacher complexity)

Strengths

- The paper is very well written and is very clear and easy to follow. - The paper is the first to consider Oracle efficient algorithms for the sequential prob. assignment problem - The paper creatively combines earlier ideas (Coupling Lemma) to reduce the smoothed adversary setting to a transductive setting.

Weaknesses

Please see questions below

Questions

- The authors can elaborate the results of Wu.et al [2023] and highlight the differences and similarities (both in terms of results and techniques). - Is Algorithm 1 actually implementable in practice - especially without the knowledge of \sigma, the smoothness parameter? - How do the results change in case the adversary is realizable w.r.t to the class F that the regret is measured against? - How critical is the assumption on uniform distribution over X. For instance if X does not support a uniform distribution, how does Algorithm 1 and it's guarantees change?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Area Chair KTBK2023-08-18

Dear Reviewer oQa8, Could you please acknowledge the author's rebuttal? Thank you, Your AC

Reviewer w3DK7/10 · confidence 3/52023-07-25

Summary

This paper studies the problem of sequential probability assignment in the smoothed setting. In particular, a learner receives labelled examples sequentially, where the contexts are drawn from some smooth distribution which can otherwise be chosen adversarially (in each step) and the labels can be chosen adversarially. The goal of the learner is to minimize the regret (minimax rate of excess error with respect to the optimum hypothesis on the subsequence seen so far). The first main result provided is that the optimum regret for the problem is characterized by the optimum regret of a different problem where the adversary chooses a priori the set of contexts that they will have to choose from throughout the interaction with the learner. Using the demonstrated relationship between the two problems, the authors provide an upper bound on the regret in the smoothed setting, which involves the scale sensitive VC-dimension of the hypothesis class and a number of free parameters that can be chosen accordingly to adapt the bound to different regimes for the VC-dimension. For VC-classes the provided adapted bound is essentially tight. The second main result provided is an oracle-efficient algorithm achieving sublinear regret on VC-classes (and more generally on classes with polynomially decaying Rademacher complexity), provided access to an ERM oracle for the hypothesis class considered. The algorithm follows the approach of Follow-the-Perturbed-Leader, which takes advantage of the trade-off between the stability of the algorithm's intermediate states and the excess error due to perturbing the current set of samples (by adding a number of hallucinated uniformly random data points).

Strengths

The paper demonstrates that the smoothed analysis framework is relevant to providing provable guarantees for the problem of sequential probability assignment, hence initiating (and motivating) the (further) study of the problem through the lens of smoothed analysis. The results provided include essentially tight bounds on the (smoothed) regret for VC-classes and two main conceptual contributions: First, characterizing the minimax regret for the problem considered in terms of the minimax regret of another relevant problem (rather than some combinatorial notion of dimension, which might even be hopeless in general). Second, repurposing ERM as an oracle to be exploited by algorithms with sublinear regret. Overall, the results are presented with clarity and sufficient detail, and provide bounds for the considered problem that are fairly general and adaptable to hypothesis classes with different properties.

Weaknesses

The first weakness of the paper is that the oracle-efficient algorithm proposed is not necessarily efficient, due to the potentially high complexity of the ERM oracle, especially since the labels are chosen adversarially. In other words, it is not clear whether assuming oracle access to an ERM is reasonable. A discussion on existing or simple positive results regarding the oracle's implementation or even a pointer to empirical results that demonstrate its success on a relevant setting would be appreciated. Furthermore, many of the technical contributions of the paper are not discussed in the main text. For example, the only hint provided for the proof of Theorem 3.1 in the main text is that it uses Theorem 2.1. In particular, it is not clear what (if any) technical obstacles arise when one tries to apply Thm 2.1 to this setting, or if adapting some approach from prior work is sufficient. Similarly, a proof sketch for Theorem 3.2 would be helpful for the reader. Overall, while the problem considered is well-motivated and the results are concrete, it is not clear whether there are strong conceptual and technical contributions relatively to prior work (smoothed analysis has already been applied to online learning and most of the tools used to demonstrate the results existed in the literature, at least in some less general form -- e.g., Theorem 2.1 and the idea of decomposing the stability term in Haghtalab et al. [2022]). -- Most of my main concerns where addressed in the rebuttal and for this reason I increased my score from 6 to 7.

Questions

My main questions are related to the two main points raised in the weaknesses section. Given the concreteness of the provided results, I would be happy to increase my score if the issues I pointed to are sufficiently addressed by the authors' answer. 1. Are there some simple function classes for which the ERM oracle can be efficiently implemented under the marginal assumptions that correspond to the considered setting? Alternatively, is there a(n empirical, conceptual or theoretical) reason for which one might expect to heuristically obtain an approximate ERM oracle? 2. What are the main technical hurdles arising when one tries to instantiate prior techniques in the considered setting? In which ways does the technical work differ from prior work? I have also found a small number of typos: - line 294: likliehood $\to$ likelihood - lines 306-307: i=1 $\to$ t=1 (3 times, one for each summation) - line 318: at in $\to$ in - line 321: refers $\to$ refer - line 344: achieve $\to$ achieved

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Yes

Reviewer 4UMy2023-08-11

I thank the authors for addressing my questions, I have no additional questions.

Reviewer X6XT2023-08-16

Thank you for the clarification. I understand now that the overlap with the paper by Wu et al. is not as large as I had originally thought. Hence, I have raised my score.

Reviewer 9j8s2023-08-16

I appreciate the authors' thorough response. My concerns have been adequately addressed. While my expertise in this field is limited, I believe that this paper is technically solid and deserves acceptance.

Reviewer oQa82023-08-20

Thanks for the author's response. I have read the rebuttal. The clarifications on certain points in the rebuttal such as the practical implementation, arbitrary domain, etc can be added to the paper/supplementary as appropriate.

Program Chairsdecision2023-09-21

Decision

Accept (spotlight)

© 2026 NYSGPT2525 LLC