Fast and Regret Optimal Best Arm Identification: Fundamental Limits and Low-Complexity Algorithms

This paper considers a stochastic Multi-Armed Bandit (MAB) problem with dual objectives: (i) quick identification and commitment to the optimal arm, and (ii) reward maximization throughout a sequence of $T$ consecutive rounds. Though each objective has been individually well-studied, i.e., best arm identification for (i) and regret minimization for (ii), the simultaneous realization of both objectives remains an open problem, despite its practical importance. This paper introduces \emph{Regret Optimal Best Arm Identification} (ROBAI) which aims to achieve these dual objectives. To solve ROBAI with both pre-determined stopping time and adaptive stopping time requirements, we present an algorithm called EOCP and its variants respectively, which not only achieve asymptotic optimal regret in both Gaussian and general bandits, but also commit to the optimal arm in $\mathcal{O}(\log T)$ rounds with pre-determined stopping time and $\mathcal{O}(\log^2 T)$ rounds with adaptive stopping time. We further characterize lower bounds on the commitment time (equivalent to the sample complexity) of ROBAI, showing that EOCP and its variants are sample optimal with pre-determined stopping time, and almost sample optimal with adaptive stopping time. Numerical results confirm our theoretical analysis and reveal an interesting"over-exploration"phenomenon carried by classic UCB algorithms, such that EOCP has smaller regret even though it stops exploration much earlier than UCB, i.e., $\mathcal{O}(\log T)$ versus $\mathcal{O}(T)$, which suggests over-exploration is unnecessary and potentially harmful to system performance.

Paper

References (45)

Scroll for more · 33 remaining

Similar papers

Peer review

Reviewer SqBM7/10 · confidence 3/52023-06-21

Summary

This work focuses on simultaneously achieving regret minimization and best arm identification in multi-armed bandits, which is called Regret-Optimal Best Arm Identification (ROBAI). That is, the goal is to identify with high probability and as fast as possible / within a certain time frame the optimal arm, and play until round T. In practice, authors propose three algorithmic contributions, which have guarantees on the cumulative regret, the time before commitment to exploiting a single arm (called stopping time in the paper) and the error on the identification of the optimal arm. The first two algorithms are asymptotically optimal regretwise for Gaussian bandits, and respectively tackle the pre-determined stopping time case and the adaptive stopping time case. They consist in exploring arms using the UCB principle, and committing to the arm which maximizes the LCB. The last one is a variant of the previous algorithms which achieves asymptotical regret optimality for subgaussian distributions, by leveraging KL confidence intervals.

Strengths

- Originality: This paper tackles an interesting problem which has been seldom considered. Although the algorithms reuse well-known principles (UCB / LCB), their analysis allow to prove very good results (regret optimality, interesting bound for adaptive “stopping” time). Table 1 is clear and shows that the work is well-grounded in prior literature. - Quality: The results seem technically sound, although I did not check the appendix in detail. - Clarity: The submission and the proof sketches are clearly written. - Significance: This paper provides and substantiates an interesting insight on the behavior of UCB algorithms with respect to over-exploration

Weaknesses

- Significance: I am not 100% convinced about the real-life applications of this framework. If the “stopping” time is fixed because of budget limits (predetermined setting), I don’t see what ROBAI brings more than classical regret minimization (i.e., why playing pessimistically at the end of the sampling phase is better), especially since it is not feasible in practice, as the related algorithm requires the knowledge of Delta_min to be optimal.

Questions

- What does ROBAI bring more than classical regret minimization (i.e., why playing pessimistically at the end of the sampling phase is better) in the predetermined stopping time setting?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

This paper deals with theoretical work, and does not raise significant concerns about negative impacts.

Reviewer T4df6/10 · confidence 3/52023-07-06

Summary

This paper studies the classical multi-armed bandits problem with the goal to design algorithms achieving tight regret bound and tight sample complexity to identify the best arm simultaneously. To this end, three algorithms are proposed based on upper confidence exploration and lower confidence commitment for three settings: Gaussian bandits with known and unknown suboptimality gap $\Delta_{\min}$, general bandits when suboptimality gap (in terms of KL divergence) is known. It is proved that all the three algorithms enjoy tight regret bound up to the constant in front of the dominating term ($\log T$), and they also have polylog sample complexity scaled linearly with $1 / \Delta_{\min}^2$. There is also a lower bound on the sample complexity given the regret to have a tight constant in the $\log T$. The lower bound states that the rate of sample complexity is tight when $\Delta_{\min}$ is given, and can also be tight when $\Delta_{\min}$ is unknown as long as the order of remaining terms in the regret is smaller than $\log T$.

Strengths

1.The paper addresses the classical settings of regret minimization and best arm identification problems in bandit theory. The investigation of their combination opens up new avenues for fundamental methods in this field. The paper stands out for its clean and clear presentation, making it accessible to readers and facilitating understanding of the proposed techniques. 2.The paper introduces several novel techniques that enable the simultaneous bounding of regret and sample complexity. The incorporation of lower confidence bound (LCB) commitment following upper confidence bound (UCB) exploration is a particularly interesting departure from the traditional approach of UCB commitment. This novel technique offers fresh insights and potential improvements in bandit algorithms. Additionally, the stopping criterion employed in the unknown gap setting is highly appreciated in theory by leveraging the deep connections between the number of pulls across different arms and the total pulls of suboptimal arms. 3.The analysis of lower bounds in the paper sheds light on the fundamental limits and trade-offs involved in identifying the best arm. These findings provide valuable insights into the design of algorithms, emphasizing the need to balance regret and sample complexity. The paper's techniques offer the potential for achieving a near-perfect balance between these two factors in practical applications of multi-armed bandits.

Weaknesses

1. The unified achievement of both optimal regret and optimal sample complexity is an important problem in theory and application. However, I am not fully convinced of the motivation of this paper to study the constant-level tight regret bound but rate-level tight sample complexity. It somehow looks like a beam search of different settings in multi-armed bandits that are not fully addressed, since algorithms with both rate-optimal regret and rate-optimal sample complexity have been extensively studied. The vanilla multi-armed bandits is a fundamental model in theory, and the algorithms of multi-armed bandits provide prominent insights in many other problems. However, the UCB and LCB algorithms, which are conceptual in nature, require significant domain-dependent modifications to be applicable in real-world scenarios. This lack of naturalness in the setting detracts from the paper, despite the appreciation for the introduction of new techniques. 2. Two of the three proposed algorithms require the minimal gap to be known, which I believe is not very possible in application (except for the synthetic task). There are two reasons for it. Consider some tasks that aim to identify the user's favorite items: the expected reward is 1 if the item is liked by the user, and 0 otherwise, with some noise to model random attributes of the user. In this case, the gap is known and large, which greatly simplifies the problem. In general, it takes a very short period of time to find the favorite items even in the presence of noises, so it is totally acceptable that the regret is rate-optimal instead of constant-optimal. When the gap is not determined manually and small, it is not likely one can obtain the exact value of the gap. People often guess this gap by some binary search routines. However, the Algorithm 2 suffers a $\log^2 T$ sample complexity, where $\log T$ is the dominating term. This means the overall sample complexity of this algorithm is even not rate-optimal, though the regret is constant-optimal. It would be better in this case to use other methods such as elimination to keep both regret and sample complexity rate-optimal. 3. The experimental results presented in the paper have limitations in their scope. Firstly, comparing and plotting cumulative rewards instead of regret would provide a more convincing demonstration, as the sum of rewards and regret is a linear function rather than a constant. Additionally, the experiments should be conducted in more diverse environments, including scenarios with a higher number of arms. The current results do not suggest a significant improvement over the vanilla UCB algorithm, raising questions about the practical advantage of the proposed EOCP and its variants (see the questions below).

Questions

1. Why you only consider the setting where the regret should be constant-optimal while the sample complexity can be rate-optimal. From the perspective of theory, is it possible to obtain both constant-optimal algorithm or constant-optimal sample complexity and rate-optimal regret? 2. How is the identification accuracy of the vanilla UCB algorithm at the step it gets the same regret of EOCP? Is this accuracy comparable to the commitment accuracy of EOCP?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See above.

Authorsrebuttal2023-08-11

Dear Reviewer T4df: We want to follow up to see whether our response addresses your concerns. Please don't hesitate to let us know if you have any other questions/comments. Thanks! We also want to comment on weakness #2 raised by the Reviewer. ### Require Known Minimum Gap ### For the weakness #2 raised by the Reviewer, we would like to comment that our algorithm only needs a value that is no larger than the minimum gap $\Delta_{\min}$ instead of the exact value. While it is not completely "model-free", it is much easier and practical than knowing the exact value of $\Delta_{\min}$. For example, we don't need to use binary search routines to estimate the exact value of $\Delta_{\min}$ as suggested by the Reviewer. In practice, the agent can choose a small number $\epsilon$ to replace $\Delta_{\min}$ in the algorithm. For any problem where $\Delta_{\min}$ is larger than $\epsilon$, the algorithm will be regret optimal and commits in $\mathcal{O} (\log T)$ rounds. In order to have an algorithm that works for all $\Delta_{\min},$ we proposed the adaptive stopping time setting and Algorithm 2, which however has a higher sample complexity. But on the other hand, according to our Theorem 3 in adaptive stopping time setting, it is not possible to maintain constant optimal regret and commit at $\mathcal{O} (\log T)$ rounds at the same time.

Reviewer T4df2023-08-16

I thank the authors for your efforts to address my concerns. I really appreciate the additional experimental results that illustrates the effectiveness of LCB commitment strategy in a UCB algorithm. I do agree this is a technically solid theory paper studying an important fundamental problem in bandits, so I decided to slightly raise my score. However, I am still concerned with the motivation and setting of the paper. The UCB (or LCB) is a conceputal algorithm useful in theory, but the industry uses a much more complicated algorithm pipeline for their purpose based on the principle of UCB (or LCB). Given the motivating example of the paper, I am not sure about why requiring the regret to be constant-level optimal is so important **in a theory paper**. Moreover, the reply to the Weakness #2 seems to contradict with the purpose of the paper. Say, you have an $\epsilon$ in your algorithm that equals $\Delta_{min}/2$, then the regret would not be constant-level optimal anymore, since now your have a dependency like $2\log(T)/\epsilon = 4\log(T)/\Delta_{min}$. I hope the reviewer to give some intuitions on how to decide the $\epsilon$ to keep the regret to be constant-level optimal.

Authorsrebuttal2023-08-16

We are glad that our additional experimental results address the concerns. We thank the reviewer for the additional comments and for raising the score. We would like to clarify that substituting $\Delta_{\min}$ with $\epsilon$ will not affect the regret performance in Theorem 1. Namely, with $\epsilon$ smaller than $\Delta_{\min}$ in the algorithm, the regret performance remains to be $2\log (T) / \Delta_{\min}$. However, the sample complexity result (Corollary 1) will be affected. Instead of having $\mathcal{O} ( \log(T)/\Delta_{\min}^2)$ commitment time, it becomes $\mathcal{O}(\log(T)/\epsilon^2)$. The regret is still constant-level optimal.

Reviewer WGiE6/10 · confidence 3/52023-07-07

Summary

The work studies how to design an algorithm with an asymptotically optimal regret rate such that it will also commit to the best arm with high probability after a stopping time (e.g., O(logT)). The paper proposes two algorithms in the Gaussian bandits setting: one with pre-determined stopping time (EOCP), which requires the knowledge of the minimum reward gap, and another with adaptive stopping time (EOCP-UG). The authors prove both algorithms are asymptotically optimal and will commit to the best arm with confidence O(1/T) after O(log(T)) and O(log^2(T)) respectively. The paper further shows the corresponding lower bounds for commitment times (expected stopping time) and finds EOCP is sample optimal and EOCP-UG is nearly sample optimal. In addition, the authors extend the EOCP to KL-EOCP for general bandits, whose commitment time also matches the lower bound.

Strengths

1) In general, the studied question is interesting and the paper is well-written and easy to follow. 2) The proposed algorithms are novel which are based on well-designed stopping rules. 3) The theoretical results and the proofs are clear. The numerical results also show the advantages of the proposed algorithms compared to the benchmark algorithms, including UCB and BAI algorithms in the literature.

Weaknesses

1) The problem setting is the fixed-confidence BAI setting (note for the setting of EOCP, the stopping time is pre-determined by the algorithm instead of by the environment). It would be great if the authors can give some discussions on the fixed budget setting (the stopping time is given), like the challenges of extending the proposed algorithms. 2) In the general bandits setting, the authors only extend EOCP to KL-EOCP, which needs the KL_min of arms. The authors demonstrate if KL_min is not known, an extension of EOCP-UG can deal with this setting. But there is no pseudo-code or provable results on it. More details or discussions would be helpful. Minor comments: In Table 1, it might be helpful to add another column ‘Setting’, since ‘Optimality’ is a bit confusing for explaining ‘Gaussian’ or ’General’.

Questions

See above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Na.

Reviewer tQKP3/10 · confidence 4/52023-07-08

Summary

The paper delves into the study of the 'Explore Then Commit' (ETC) policy, where the algorithm is divided into two stages: exploration and commitment. During exploration, the algorithm is permitted to switch actions, while the commitment phase restricts the algorithm to pulling only the commit arm. The primary objective of the algorithm is to reduce the expected commit time, denoted as $T_c$, and ensure that at time $T_c$, $\hat a \neq a^*$ holds true with a probability of $O(1/T)$. Three variations of the ETC policy are introduced in this paper: EOCP, EOCP-UG, and KL-EOCP. For EOCP, it's structured for a known gap setting, achieving regret in the order of $2\log T/\Delta_i$. The authors assert that this outcome is asymptotically optimal. However, this claim appears to be incorrect. As per reference [On Explore-then-Commit Strategy], the asymptotic optimal regret for a known gap should be $\log T/(2\Delta_i)$. Furthermore, the authors have not discussed this setting's reference in a comprehensible manner. For instance, DETC with a known gap achieves exact asymptotic optimality. For EOCP-UG, the authors show that it achieves asymptotic optimality with a commit time of $\log^2 T$. As for KL-EOCP, it necessitates knowledge of certain parameters for the design of the pre-determined stopping time. In summary, the claim that the EOCP and KL-EOCP algorithms are asymptotically optimal appears to be overstated. The upper bound for EOCP-UG and its associated lower bound is the same as DETC. As for the lower bound, the assumption seems excessively strong. The authors did not clearly indicate whether algorithms must adhere to the equation given in Line 241. Primarily, this paper targets two-armed bandit problems, focusing on asymptotic regret. However, I am intrigued by the finite-time bound. I would appreciate an algorithm that is not only asymptotically optimal but also demonstrates a robust finite-time bound. In addition, the paper only considers a fixed horizon T setting. What about the case with an unknown $T$? From my understanding, DECT and its variant [Almost Optimal Anytime Algorithm for Batched Multi-Armed Bandits], are also suitable for an unknown T setting. Furthermore, the necessity of the assumption in Line 141 is not clearly articulated and requires clarification. Regarding experimental results, the performance of ETC and DECT presented in this paper is inconsistent with previous papers. This discrepancy might be due to parameter adjustments in these algorithms. I would appreciate seeing empirical results with parameter tuning for both ETC and DETC. ---- # At the end of discussion phase The authors fail to address my primary concerns. Below, I elaborate on these issues: 1. Line 174 of the manuscript asserts that in the pre-determined setting (where the gap $\Delta$ is known), $\frac{2\log T}{\Delta}$ is asymptotically optimal. This claim is misleading. Algorithm 4 in reference [21] shows that in the pre-determined setting with $T_c = \log^2 T$, there exists an algorithm with a regret of $\frac{\log T}{2\Delta}$. Moreover, the known lower bound (Theorem 6 in [20]) for known gap setting is $\frac{\log T}{2\Delta}$. Therefore, stating $ \frac{2\log T}{\Delta}$ as asymptotically optimal is an overstatement. In their response, the authors attempt to equate the lower bound of the pre-determined setting with the lower bound of the known gap setting in the ETC strategy [20]. This comparison is problematic and has the potential to mislead other readers. Specifically, Table 1 in the authors' response erroneously claims that $4\log T/\Delta$ is the lower bound for the known gap setting, whereas this is actually the lower bound proven for the **unknown gap** setting in Theorem 4 of [20]. Additionally, they assert that $2\log T/\Delta$ is the lower bound for the known gap and pre-determined settings, which contradicts Algorithm 4 in [21] that shows regret $\frac{\log T}{2\Delta}$. 2. As I pointed out in my initial review, DECT [21] (Algorithm 5 and 2) can be directly applied to the unknown gap setting. The results for DECT in an unknown gap setting are identical to those presented in this paper. Given the paper's significant overstatements and the paper's limited contributions, I am revising my score from 4 to 3.

Strengths

In this paper, the authors propose an algorithm called Explore Optimistically then Commit Pessimistically (EOCP) to solve the Regret Optimal Best Arm Identification (ROBAI) problem. It first uses an optimistic modified UCB algorithm to explore actions with a slightly larger exploration function , and then commits to actions according to a pessimistic LCB algorithm when the exploration ends. The main contributions include designing new stopping rules with both pre-determined stopping time (vanilla EOCP) and adaptive stopping time (the EOCP-UG variant), which provably balance the trade-off between regret minimization and optimal action identification

Weaknesses

Please refer to the Summary.

Questions

Please refer to the Summary.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The main consideration of mine is that the proposed algorithm considers asymptotic bounds other than finite $T$ (probabily unknown). This limits the contribution of the paper heavily.

Reviewer SqBM2023-08-14

Response to the rebuttal

I thank the authors for their detailed answers. My apologies, I had misunderstood the pre-determined stopping time setting and the differences with regret minimization. However, the answer to the former question raises a concern (already mentioned by Reviewer T4df in Weakness #2 and left unaddressed in the rebuttal) of practicality due to the necessary knowledge of the minimal gap for that specific setting. As such, I will keep for now the score as it is.

Authorsrebuttal2023-08-15

We are glad that our response clarifies the difference between the two settings. For the concern the reviewer raised (and the weakness #2 by reviewer T4df), we would like to comment that our algorithm only needs a value that is no larger than the minimum gap $\Delta_{\min}$ instead of the exact value. While it is not completely "model-free", it is much easier and practical than knowing the exact value of $\Delta_{\min}$. For example, we don't need to use binary search routines to estimate the exact value of $\Delta_{\min}$ as suggested by Reviewer T4df. In practice, the agent can choose a small number $\epsilon$ to replace $\Delta_{\min}$ in the algorithm. For any problem where $\Delta_{\min}$ is larger than $\epsilon$, the algorithm will be regret optimal and commits in $\mathcal{O} (\log T)$ rounds. In order to have an algorithm that works for all $\Delta_{\min},$ we proposed the adaptive stopping time setting and Algorithm 2, which however has a higher sample complexity. We thank the reviewer again for the positive review and quick response!

Reviewer tQKP2023-08-21

I'm skeptical about the claim of asymptotic optimality in the pre-determined setting as presented in Line 174. In Algorithm 1, the gap, $\Delta$, is utilized to define the parameter $T_c$. This leads me to question the assertion that EOCP is asymptotically optimal. A more fitting comparison would be with the known gap lower bound rather than the unknown gap lower bound. In their response, the authors draw parallels between this setting and the fixed-design setting referenced in [20], where the regret lower bound is given as $\(4\log T/\Delta\)$. I find this comparison misleading for several reasons: 1. The ETC strategy in [20] ensures that both arms are pulled an equal number of times up until the point of commitment. Contrarily, the algorithms examined in this paper allow for varying numbers of pulls for each arm. As a result, the lower bound outlined in [20] for ETC isn't applicable to the present study. Invoking the result $\(4\log T/\Delta\)$ in this context is not justifiable. 2. Algorithm 1 inherently caters to the known gap setting since it mandates the specification of $T_{c}$ based on the known gap, $\Delta$. In contrast, $\(4\log T/\Delta\)$ is recognized as the lower bound for the ETC strategy with an unknown gap.

Authorsrebuttal2023-08-21

We thank the reviewer for the response, and we want to point out the following misunderstanding the reviewer had regarding our rebuttal and our paper. ### Utilizing $\Delta$ in Designing Parameter $T_c$ ### We want to point out that our algorithm **only requires a lower bound of $\Delta$** to define $T_c$ instead of the exact value. While it is not completely "model-free", it is much easier and practical than knowing the exact value of $\Delta$. This lower bound does not provide all the information that $\Delta$ has. In practice, the agent can choose a small number to replace $\Delta$ in the algorithm. For any problem where $\epsilon$ is larger than $\Delta$, the algorithm will have exactly the same performance as in Theorem 1 and Corollary 1. On the other hand, ETC strategy in [20] in the known gap setting requires **the exact value of $\Delta$**, and replacing it with any value other than $\Delta$ would either harm the best arm identification accuracy or harm the sample complexity and regret. Based on this essential difference and the fact that our algorithm does not require the exact value of $\Delta$. We consider it not fair to compare our algorithm 1 to the lower bound $\log T/ 2\Delta$. ### Comparison to the Fixed-Design Setting ### It is true that in the fixed-design setting, all arms need to be pulled uniformly, and the commitment time is pre-determined, which results in the $4\log T/ \Delta$ regret. The reason why our algorithm 1 can achieve lower regret than $4\log T/ \Delta$ is due to the adaptive arm sampling rule. However, our design of commitment time is still pre-determined. We said that our pre-determined setting is more similar to the fixed-design setting, instead of the known gap setting. Based on the reviewer's argument, we found the known gap setting lower bound $\log T/2\Delta$ is as not applicable to the present study as the lower bound $4\log T/\Delta$ in fixed-design setting. The full comparsion of all settings are presented in the original rebuttal. Given we indeed provide general results for bandits with more than two arms and also finite-time bounds for all algorithms in the original paper, which are the main limitations the reviewer is concerned about, we would appreciate it if the reviewer could re-evaluate the rating and review based on our response.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC