High-dimensional Contextual Bandit Problem without Sparsity

In this research, we investigate the high-dimensional linear contextual bandit problem where the number of features $p$ is greater than the budget $T$, or it may even be infinite. Differing from the majority of previous works in this field, we do not impose sparsity on the regression coefficients. Instead, we rely on recent findings on overparameterized models, which enables us to analyze the performance of the minimum-norm interpolating estimator when data distributions have small effective ranks. We propose an explore-then-commit (EtC) algorithm to address this problem and examine its performance. Through our analysis, we derive the optimal rate of the ETC algorithm in terms of $T$ and show that this rate can be achieved by balancing exploration and exploitation. Moreover, we introduce an adaptive explore-then-commit (AEtC) algorithm that adaptively finds the optimal balance. We assess the performance of the proposed algorithms through a series of simulations.

Paper

Similar papers

Peer review

Reviewer 57ot5/10 · confidence 3/52023-06-10

Summary

The paper addresses the problem of high-dimensional linear contextual bandits where the number of features is greater than the budget or even infinite. Unlike previous works, the paper does not impose sparsity on the regression coefficients but instead leverages over parameterized models. The authors propose an explore-then-commit (EtC) algorithm and an adaptive explore-then-commit (AEtC) algorithm to tackle this problem. The performance of the proposed algorithms is evaluated through simulations, and the optimal rate of the ETC algorithm is derived, showing that it can be achieved by balancing exploration and exploitation.

Strengths

The paper studies a new problem in high-dimensional bandits when there is no sparsity in parameter vector. EtC is first proposed based on the minimum-norm interpolating estimator with the optimal exploration rate, and then AEtC is introduced to deal with the model-dependent parameters on the covariance.

Weaknesses

1. Lower bound of the problem is not discussed in the paper. Since this is the first work on the high-dimensional bandit without sparsity, a lower bound can help people understand how good your algorithm is in theory. I acknowledge that authors mention this point as limitation in the appendix, but I believe this is an important issue. 2. Assumptions are strong and more explanations will be better. Some assumptions are given in the paper, e.g. Assumption 1, which may not be realistic in practice. For example, I can hardly understand the first part of Assumption 1. What is the intuition behind it? Why you used spectral decomposition to construct $X^{(i)}(t)$? Some examples on which these assumptions hold should be given. 3. More explanation on definitions should be presented. Authors use some definitions in this work that are not commonly used in the bandit community, e.g. effective bias, effective variance, coherent rank, effective rank, benign covariance. Some more explanations on these definitions and maybe correlation between them should be given in the paper. 4. The regret bound in Theorem 4 depends on some $\alpha > 0$, but this value is not clearly explained in Theorem 4. I think when authors write a formal Theorem in the paper, it is better to explain every term in detail (e.g. how to formulate $\alpha$ in general). 5. (minor): in the first page, the footnote 1 is under the sentence "submitted to 37th Conference on .......", which seems unusual. Maybe the authors change the margin of the template a little bit?

Questions

Besides the questions in the above Weakness section, I have some questions here: 1. For your experiments, in several cases (DGP 1, DGP 2) it seems that the cumulative regret of your algorithm and other existing algorithms is linearly increasing according to the plots. Could authors explain why sublinear pattern is not observed? 2. Can your algorithm still work well when $p$ is greater than $T$ in practice? It might be more comprehensive if such case could be considered in your simulation studies.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

No negative societal impact.

Reviewer C3HD6/10 · confidence 4/52023-07-05

Summary

This paper considers a high-dimensional contextual linear bandit setting and uses recent results on the risk of minumum norm linear interpolators to evaluate the regret of two variants of an explore-then-commit algorithm. The first variant makes use of the risk results to balance regret in the "explore" vs "commit" phases, assuming oracle knowledge of the population covariance of the contexts (which determine the risk, hence regret, of the linear interpolator and derived arms). The second variant uses simple sample-covariance-based estimators that do not require such oracle knowledge to inform the stopping of the explore phase, making the algorithm adaptive to the instance.

Strengths

The paper's ideas are straightforward, and relatively easy to understand once one is familiar with the work analyzing the risk of interpolating linear estimators. The paper also lays out the results and proof ideas well. The algorithms are also easy to motivate from that context, and amenable to quick implementation. The setting of high-dimensional linear contextual bandits without sparsity is also interesting, pointing towards potentially infinite-dimensional contexts.

Weaknesses

In my view, the papers main limitations are related to its reliance on the interpolating estimator work: 1. It is unclear to me why the paper is focused on covariance structures of the type in Proposition 1. Presumably this is because these are the covariances for which the minimum norm interpolators have non-trivial risk guarantees, but that seems like a weak reason to focus solely on them; e.g. one could consider ridge estimators, lasso estimators (notwithstanding the lack of sparsity, given it can still have good prediction error), etc. 2. The authors provide a lower bound for explore then commit algorithms showing that their upper bound is tight, restricted to this class of algorithms. However, it is not obvious that ucb or thompson sampling algorithms would face the same obstacle in performance. An information-theoretic lower bound would be convincing on this score.

Questions

1. Can you relate/situate this work with e.g. the paper "Model selection for Contextual Bandits" by Foster et al. and the references therein? The settings are similar, but different in crucial ways (e.g. in their work context covariances are assumed bounded below). 2. Can you elaborate on the regret dependence in terms of $\theta_\max$? Presumably knowledge of this affects the optimal choice of $T_0$? 3. Can the class of covariance structures considered "benign" be expanded if one does not use interpolating estimators? 4. Can the example of Theorem 5 provide a "hard" example for more than explore-then-commit algorithms?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See above

Reviewer dtiS5/10 · confidence 2/52023-07-07

Summary

The authors studied the high-dimensional linear contextual bandit problem where the number of features 𝑝 is greater than the budget 𝑇, or it may even be infinite. Problem setting: They assumed each arm has an individual hidden parameter, and the contexts are drawn from nature for each round. They argue that only by using the notion of effective dimension, in some cases when data distributions have a small effective rank, explore-then-commit algorithm can address this problem.

Strengths

They adopted the new techniques from [Bartlett et al., 2020] to design an explore-then-commit algorithm with this 'tough' environment - where the dimension $p$ is large compared to the number of samples. Seems like this observation is original. Plus, they proved that their regret bound is tight by proving several lower bounds. If their arguments are all correct, these results can be significant.

Weaknesses

I am unsure whether this paper is clear enough. 1) It's not straightforward whether they assumed 'benign covariance matrix' in proposition 1 or their assumption 1 implies benign covariance matrix. 2) I cannot understand how one can take an inverse matrix in line 117 when $p>n$. 3) Eventually, seems like they made only two major assumptions - Assumption 1 with sub-gaussianity, and Assumption 2 which is about the error rate of the OLS. Especially, for AEtC, they only used Assumption 1. I am strongly doubtful that they achieve a meaningful regret bound without any additional assumption on high-dimensional space - As in the first paragraph of Chapter 7, 'High-dimensional statistics' by Martin Wainwright, if the model lacks any additional structure, then there is no hope of getting a consistent estimator in the high-dimensional environment.

Questions

0) In line 117, you mentioned that $\hat{\theta}$ can be achieved by simple OLS (ordinary least square) operation. However, you mentioned that $p>n$. How can you define $(XX^\top)^{-1}$ in this case? The authors also mentioned that their analysis can be executable even when $p=\infty$ (the first sentence of the abstract.) Please help me to understand this. 1) The authors use the word 'optimal balance' when they describe AEtC. Here's the sentence: > Moreover, we introduce an adaptive explore-then-commit (AEtC) algorithm that adaptively finds the optimal balance. I want to ask what the word 'optimal' means. I cannot see any lower-bound proof of the exploration in this paper. Did you prove that none of the EtC algorithms can beat the AEtC algorithm? If not, I believe that one should be careful of using the word 'optimal.' 2) According to my understanding, AEtC is an algorithm that relieves the assumption on the knowledge of $\Sigma^{(i)}$ (and naturally Assumption 2) from EtC, not the algorithm that improves the order of regret, am I right? 3) I want to know whether you 'assumed' the covariance matrix to be benign, or Assumption 1 (or 2) implies the covariance matrix to be benign. 4) In your proposition 1, I want to check that your $\lambda_k^{(i)}$ is exactly $k^{-1 + 1/T^a}$ or $k^b$, not the orderwise notation, am I right? 5) Maybe a silly question, but suppose that we regard your problem as a K-armed bandit problem, without considering the context. Then, isn't this mean that we already have a regret bound of $O(\sqrt{KT})$? Even for EtC, we have $O(T^{2/3})$ regret bound for K-armed bandit with EtC. What is the main factor that separates your problem from the naive K-armed bandit problem?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

There's no negative societal impact of their work. I also think they didn't state their limitations or future works properly.

Reviewer pLAf6/10 · confidence 3/52023-07-07

Summary

This paper addresses the linear contextual bandit problem in high dimensions where the number of features is greater than the budget. While the recent works impose an assumption about sparsity on the regression coefficients, this paper uses the minimum-norm interpolating estimator which is inspired by recent results on overparameterized models when data distributions have small effective ranks. The authors propose an explore-then-commit algorithm and derive an optimal regret for the proposed algorithm under their setting. This algorithm requires model-dependent parameters on the covariance, which can limit the applications in practice. Therefore, they further propose an adaptive explore-then-commit algorithm by estimating these parameters and show that their adaptive version also achieves the optimal rate. Finally, they provide some simulations to verify the efficiency of the proposed method.

Strengths

- Unlike previous works, the paper solves the high-dimensional linear contextual bandit problem without sparsity. This seems to be the first work in the field of high-dimensional contextual bandits. This paper also proposes new algorithms in the form of the Explore-then-Commit strategy in their setting and derives the optimal regrets for proposed algorithms. - The organization and the presentation are almost clear to follow.

Weaknesses

- Although the paper introduces a new direction for the high-dimensional linear contextual bandits, the authors seem not to analyze deeply the difference between their work and the recent works using sparsity (e.g., Kim and Paik, 2019, Bastani and Bayati, 2020, Hao et al., 2022). Without sparsity, their work needs to impose a condition to be benign on the covariance matrix as well as Assumption 2 in the paper. I am wondering if this assumption is weaker than the assumption on the sparsity of the recent works. Thus, I think that a section of the related works and a comparison of assumptions with recent works are needed. - The regret bounds in Theorem 4 and Theorem 9 are in terms of $\alpha$. However, the definition of $\alpha$ is not clear at all. For examples 1 and 2 in the paper, $\alpha$ can be understood but in general, what is $\alpha$? - Although the regret bounds are provided for the proposed algorithms, I think that a comparison of regret bounds with the recent works (e.g., Hao et al., 2022) is also needed. - The used techniques are adapted from the recent work of Bartlett et al., 2020.

Questions

- In Theorem 4, they impose Assumptions 1 and 2. Thus, where is the assumption about the benign covariance? - Also see my questions in the Weaknesses section

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes

Reviewer h4Ff6/10 · confidence 3/52023-07-12

Summary

This paper analyzes the Explore-then-commit algorithm for the linear contextual bandit problem in high dimensions, when the number of features is higher than the number of rounds T. The authors use recent studies on overparametrized models for linear regression to analyze the algorithm. Since this algorithm requires the knowledge of the underlying covariance matrices of the contexts, the authors provide an alternate algorithm where they adaptively determine the extent of exploration required.

Strengths

The paper presents a thorough analysis of the high-dimensional linear contextual bandit problem by providing upper and matching lower bounds on the estimation error. The Adaptive algorithm presented in the paper further adds to the paper's strengths and provides a practical way to solve the problem. The simulations provided demonstrate the the adaptive algorithm works practically in both the benign and non-benign cases.

Weaknesses

The authors could improve the writing of the paper in the following ways: 1. The introduction of the term benign covariance is a bit abrupt and very technical very quickly. The authors could provide more intuition about what it means before defining the bias and variance terms. 2. The effective rank definitions used in this paper vary slightly from the standard definitions in literature. The authors could maybe visualize the spectra of a matrix and illustrate what these quantities correspond to.

Questions

1. Could the authors define the conditional number of a matrix? If they are referring to the standard condition number, how can this number be negative?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Authorsrebuttal2023-08-11

Thank you for the update

We sincerely value the time you've dedicated to reviewing our work. We acknowledge that some parts of our paper might require more explanation, particularly due to our use of tools that may not be commonly known within the bandit community. We are more than happy to provide further details or clarification to address your concerns.

Authorsrebuttal2023-08-14

Thank you for your time & our update plan

Dear reviewer, We will make significant changes to the introduction of the setting as described in the global reply. If you still have concerns about the setting, we are happy to address them. Thank you for your time and efforts.

Reviewer h4Ff2023-08-18

Thank you for the clarifications! I do not have any further comments.

Authorsrebuttal2023-08-21

Thank you for all questions

Thank you for taking the time to review this paper! If your assessment aligns with the paper being above the acceptance threshold, would you consider increasing the score to weak accept that reflects the actual decision threshold? I understand that this comes toward the end of the process, and making changes may not be comfortable. If you prefer to keep the current score, we completely understand. Your inquiries, particularly those related to the clarity of the assumptions and the robustness of our findings, are greatly valued. They are helpful in our future revisions.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC