Likelihood Ratio Confidence Sets for Sequential Decision Making

Certifiable, adaptive uncertainty estimates for unknown quantities are an essential ingredient of sequential decision-making algorithms. Standard approaches rely on problem-dependent concentration results and are limited to a specific combination of parameterization, noise family, and estimator. In this paper, we revisit the likelihood-based inference principle and propose to use likelihood ratios to construct any-time valid confidence sequences without requiring specialized treatment in each application scenario. Our method is especially suitable for problems with well-specified likelihoods, and the resulting sets always maintain the prescribed coverage in a model-agnostic manner. The size of the sets depends on a choice of estimator sequence in the likelihood ratio. We discuss how to provably choose the best sequence of estimators and shed light on connections to online convex optimization with algorithms such as Follow-the-Regularized-Leader. To counteract the initially large bias of the estimators, we propose a reweighting scheme that also opens up deployment in non-parametric settings such as RKHS function classes. We provide a non-asymptotic analysis of the likelihood ratio confidence sets size for generalized linear models, using insights from convex duality and online learning. We showcase the practical strength of our method on generalized linear bandit problems, survival analysis, and bandits with various additive noise distributions.

Paper

References (41)

Scroll for more · 29 remaining

Similar papers

Peer review

Reviewer XFVf6/10 · confidence 3/52023-07-06

Summary

This paper proposed to use the likelihood ratio approach to provide an any-time valid confidence sequence, which is suitable for problems with well-specified likelihood. It discusses how to provably choose the best sequence of estimators and sheds light on connections to online convex optimization. To counteract the initially large bias of estimators, they propose a reweighing scheme. They also provide a non-asymptotic analysis of the likelihood ratio confidence set size for GLM, and performed numerical simulations.

Strengths

The idea of using the universal inference method in bandit problems is interesting.

Weaknesses

In bandit problems, the target of interest is regret, and the validity/length of the confidence set is a tool of analysis. One shortcoming of this paper is that there is no regret analysis (of the bandit problem) of the proposed method. One important question is, what is the regret of the proposed method on simple bandit problems such as the K-armed bandit (UCB method gives \sqrt{KT} regret) and the linear bandit (LinUCB gives \sqrt{d^2 T} regret). Since bandit is the focus of this paper, without a concrete regret bound (showing that it matches the best existing method), the benefit of the proposed method is not convincing. The method proposed in this paper is similar to the OMLE method [1], which has a provable regret guarantee in many RL problems. One interesting question is to study whether the proposed algorithm has benefit over the OMLE algorithm. [1] Optimistic MLE—A Generic Model-based Algorithm for Partially Observable Sequential Decision Making. After reading the rebuttal: I am convinced that a regret bound can be obtained for this method. I trust the authors will add this in the revised version.

Questions

NA

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

NA

Reviewer 4g2j6/10 · confidence 4/52023-07-06

Summary

This paper examines the confidence set of an estimator, defined by a likelihood ratio. The contributions stated within the work, alongside my corresponding queries, are outlined as follows: * For generalized linear models, we theoretically analyze the geometry of the LR confidence sets under mild assumptions. We show their geometry is dictated by Bregman divergences of exponential families (Chowdhury et al., 2022). *We show that the size of the confidence set is dictated by an online prediction game. The size of these sets depends on a sequence of estimators \{\theta_s\}_{s=1}^t that one uses to estimate the unknown parameter \theta_*. We discuss how to pick the estimator sequence in order to yield a provably small radius of the sets, by using the Follow-the-Regularized-Leader algorithm, which implements a regularized maximum-likelihood estimator. We prove that the radius of the confidence sets is nearly-worst-case optimal, and accordingly they yield nearly-worst-case regret bounds when used in generalized linear bandit applications. However, due to their data-dependent nature, they can be much tighter than this theory suggests. *We analyze limitations of classical (un-weighted) LR sets when the underlying conditional observation model is not identifiable. In this case, the resulting (inevitable) estimation bias unnecessarily increases the size of the confidence sets. To mitigate this, we propose an adaptive reweighting scheme that decreases the effect of uninformed early bias of the estimator sequence on the size of the sets downstream. The reweighting does not affect the coverage guarantees of our sets, and utilizes an elegant connection to (robust) powered likelihoods (Wasserman et al., 2020). * Thanks to the adaptive reweighting scheme, our sets are very practical as we showcase experimentally. We demonstrate that our method works well with exponential and non-exponential family likelihoods, and in parametric as well as in kernelized settings. We attribute their practical benefits to the fact that they do not depend on (possibly loose) worst-case parameters.

Strengths

The research question posed in this paper is notably intriguing and holds significant potential for various applications. Furthermore, the innovative approach adopted by the authors represents a substantial contribution to the field.

Weaknesses

While the paper offers several valuable insights, I noticed that the overall integration of its contributions could be improved. For instance, claimed contributions 1 and 3, which could potentially have been effectively merged, were instead treated separately. This compromise the contribution significantly from my perspective. Furthermore, I have concerns about the clarity and accuracy of some claims. For more specific observations, please refer to the "Questions" section. I believe addressing these issues would greatly enhance the coherence and validity of the work.

Questions

* The relationship between LR confidence set geometry and Bregman divergences is briefly discussed on page 6. It would be beneficial to have this connection elaborated upon more thoroughly in the main body of the text. * Section 2 provides a clear explanation on how to select a sequence of $\{ \theta_s \}_{s=1}^t$ to minimize the radius of the confidence set. However, this connection becomes less clear in Section 3. Moreover, the paper claims that the radius of the confidence set is nearly-worst-case optimal. Could you please provide further explanation as to why this is the case? While it's clear that the FTRL can lead to nearly-worst-case optimal regret for the online optimization problem, $\mathcal{R}_t$ defined in $(3)$ appears to be not identical, but smaller than the definition of regret in the online optimization problem. If $\mathcal{R}_t$ is significantly smaller than the regret of the FTRL algorithm, might our confidence set not be optimal? * The introduction of a re-weighting scheme is intriguing. However, given that the re-weight $w_t$ is difficult to compute, could you provide any theoretical performance guarantees? Additionally, in Theorem 2, the bias estimate depends on \norm{\theta_*}_2^2. Could this potentially conflict with our objective to estimate the confidence set of \theta_*? Furthermore, in Section 3, we ascertain the geometry of the linear models assuming $w_t = 1$. Is it possible to perform a similar analysis incorporating the re-weighting scheme? Otherwise, the value of the re-weighting scheme might be discounted if we cannot incorporate it into the analysis of simple model * A query arises from Theorem 3, where \theta \in \mathcal{C}_t satisfies a certain inequality. I am curious about the application of this inequality in estimating the confidence set of \theta_*, given that the Bregman distance involved depends on $\theta_*$, which is unknown. Is this result primarily of theoretical interest, or is it of practical use in this context?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

NA

Reviewer DGax6/10 · confidence 3/52023-07-07

Summary

The paper proposes to use likelihood ratios to construct confidence sequences that facilitate the downstream online decision making under uncertainty. The weighting and corresponding bias estimation are proposed to avoid regret blow-up in low-noise setting. The paper offers theoretical insights for the bias estimation, the geometry of the confidence set for generalized linear model (GLM), and regret in the online optimization in GLM. The paper offers the corresponding empirical results on bandits problems.

Strengths

1. The paper incorporates the likelihood ratios into online decision-making. The LR confidence set reduces the requirement of conventional concentration results and allows existing algorithms to solve more generalized problems as long as the likelihood is well specified. There is room for studying its application in various sequential decision-making tasks. 2. The adaptive weighting scheme is well justified with practical bias estimation. 3. The geometry analysis could be helpful to justify the usage of the LR confidence set rather than alternatives.

Weaknesses

1. In general the paper is clearly organized. However, the scattered yet closely related notations in sections 2 and 3 make it challenging to follow. I would recommend further highlighting important definitions and formulas or trying to concentrate on them to allow easier revisits. 2. The experiment results shown in Figure 2 lack statistical significance. The fact that the figure shows median values rather than mean values indicates the stability problem of the experiment results. 3. The mismatch between the theoretical results in section 3 with regards to the online optimization and the empirical study mostly focused on bandit problems in section 4. As only the downstream cumulative regrets are shown.

Questions

1. Minor issue: the product on the denominator of eq (1) seems to be falsely indexed, could the author double-check? 2. In the bandit problems, how are the confidence levels specified? 3. Could the author briefly comment on the potential impact of misspecified likelihood (robustness to misspecification when there is a uniform error bound for the misspecification)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Discussed in the comments above.

Reviewer CfDQ8/10 · confidence 3/52023-07-07

Summary

This paper proposes a new construction of confidence sets for a parametric setting, where the likelihood of the noise process is explicitly given. The proposed method is based on a weighted variant of the sequential likelihood ratio (LR) statistics, which was proposed in universal inference (Wasserman et al., 2020). The confidence set is a function of the choice of estimators that construct the sequential LR statistics, and the paper proposes a specific method to construct such estimators by the means of Follow-the-Regularized-Leader (FTRL) algorithm in online learning, by viewing the confidence parameter minimization as a regret minimization problem. Further, the paper proposes a new adaptive reweighting scheme, which uses a powered LR statistics to make the resulting confidence set more “robust”. The proposed reweighting scheme is based on the bias of the used estimators. While the bias may not be directly estimated, they provide a computable upper bound on the bias for a certain form of distributions. For generalized linear models, the size of the confidence set is analyzed under regularity assumptions. To make the bound fully concrete, they also provide a high-probability regret guarantee of the used FTRL estimators (for GLMs). The effectiveness of the proposed method is demonstrated with linear and kernelized bandit experiments.

Strengths

The paper is very well written. The proposed framework and analysis are a nice and careful combination of several different techniques (universal inference, FTRL, Bregman information gain, …) in statistics and online learning. I think this could be indeed useful in bandit problems but it can be also applicable whenever parametric models are available.

Weaknesses

I do not find a crucial weakness of this paper, except a few questions detailed below.

Questions

- I find the sentence in line 107 a bit confusing. It says “If the confidence parameter goes to zero, only $\theta_\star$ is in the set.” This is not the case, as the LHS (of the inequality inside the set notation) in the equation after line 103 is 1 if $\theta=\theta_\star$, and so $\theta_\star$ will be excluded if the RHS goes to zero. What should be a proper explanation? - I cannot make much sense of the specific reweighting scheme in line 162 right away, though I can understand the intuition thanks to the Gaussian example. Can you elaborate this choice? A possible typo: - Line 153: depending only $\theta_\star$

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Limitations are not discussed.

Reviewer DGax2023-08-15

I appreciate the authors' detailed response. I believe my questions have been well answered, and incorporating the corresponding discussion into the paper would enhance the overall presentation. I have carefully considered the responses to my concerns about the mismatch between the theoretical results and the later empirical study on regrets of bandit problems, as similar questions have also been raised by other reviewers. While I believe the answers provided could alleviate the issue, I still hold the belief that establishing a more direct connection to justify the benefits of the proposed method in the bandit problem is crucial for the paper's coherence. In general, I value the theoretical contribution of the paper, but I do have concerns about its completeness and presentation, which could potentially be better addressed in the revised version. Therefore, I will maintain my current score.

Authorsrebuttal2023-08-16

Thank you for engaging in the discussion. We would like to respectfully disagree with your conclusions: - We provided the arguably most useful demonstration of improvement in bandit problems by showing that empirically the regret decreases when using our method. This is almost always the relevant and targeted metric in the field. - As we discussed, the mismatch the reviewer is pointing out is inherent for adaptive sequences. One first needs an algorithm to generate the adaptive sequence which makes the generation of calibration plots application dependent. We believe they are not the optimal way to showcase our method accordingly, which is why we refrained from doing so in the first place. - We provided calibration plots in response to the review nonetheless. One last possible addition would be to directly plot the radius of the confidence sets and compare them to a baseline ellipsoidal set. However, this is only satisfactory in the Gaussian case as it‘s the only case where our sets are actually ellipses -- essentially our banner plot. We are interested to know if the reviewer would be satisfied with this. - We provided an explanation that a regret bound follows automatically from our confidence sets. However, the main contribution is not improving bandit rates. We do not improve any known rates. We match the existing rates via our universal setup. We invite the reviewer to look at this paper more holistically instead of looking purely for theoretical regret bounds. This paper provides a novel and unique way of looking at the problem of adaptive confidence sets in machine learning applications. We hope that matching regret rates in known settings will encourage practitioners to use our methods in different and novel settings and enjoy the empirical benefit of this provable method.

Reviewer XFVf2023-08-15

Thanks for the clarification of the authors. I am satisfied that the regret bound can be obtained for the linear bandit, and I trust that the authors will add it to the revised version. I will raise my score.

Reviewer CfDQ2023-08-16

I appreciate the authors' response and I read other reviews as well. I think that the authors can incorporate the changes in the revision to make the contributions clearer. Especially the regret bound for linear bandits (as pointed out by Reviewer XFVf) and a better intuitive explanation on the reweighting scheme will strengthen the paper. Hence I will keep the score.

Authorsrebuttal2023-08-17

Discussion period ends soon

Dear reviewer, the discussion window is closing soon. We believe we have addressed your questions and concerns and are happy to engage in further discussion. Please reconsider your score.

Reviewer 4g2j2023-08-20

I concur with the authors regarding their well-articulated responses, which addressed my queries. As a result, I am inclined to raise my review score. Nonetheless, I share the sentiment expressed by some of my fellow reviewers that the manuscript would benefit greatly from a thorough revision for readability. Enhanced clarity and structure could not only facilitate better comprehension but also underscore the significance of the paper's contributions.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC