Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification

In linear bandits, how can a learner effectively learn when facing corrupted rewards? While significant work has explored this question, a holistic understanding across different adversarial models and corruption measures is lacking, as is a full characterization of the minimax regret bounds. In this work, we compare two types of corruptions commonly considered: strong corruption, where the corruption level depends on the action chosen by the learner, and weak corruption, where the corruption level does not depend on the action chosen by the learner. We provide a unified framework to analyze these corruptions. For stochastic linear bandits, we fully characterize the gap between the minimax regret under strong and weak corruptions. We also initiate the study of corrupted adversarial linear bandits, obtaining upper and lower bounds with matching dependencies on the corruption level. Next, we reveal a connection between corruption-robust learning and learning with gap-dependent mis-specification, a setting first studied by Liu et al. (2023a), where the misspecification level of an action or policy is proportional to its suboptimality. We present a general reduction that enables any corruption-robust algorithm to handle gap-dependent misspecification. This allows us to recover the results of Liu et al. (2023a) in a black-box manner and significantly generalize them to settings like linear MDPs, yielding the first results for gap-dependent misspecification in reinforcement learning. However, this general reduction does not attain the optimal rate for gap-dependent misspecification. Motivated by this, we develop a specialized algorithm that achieves optimal bounds for gap-dependent misspecification in linear bandits, thus answering an open question posed by Liu et al. (2023a).

Paper

Similar papers

Peer review

Reviewer DxTV5/10 · confidence 3/52024-07-09

Summary

This paper studies corruption-robust linear bandit optimization and characterizes the regret bound in terms of both weak and strong corruption measures. Under the stochastic setting, this paper proposes a phased elimination algorithm, and the regret bounds match the lower bound. Under the adversarial setting, the paper proposes two individual algorithms for the two corruption measures respectively. In addition, this paper studies gap-dependent misspecification setting through reduction, and discusses a use case for linear MDPs.

Strengths

- The regret bounds in terms of both corruption measures are provided, where the regret bound depending on $C_\infty$ is first introduced in this paper. - The theoretical results are supported with detailed proof. - This paper is generally well-written.

Weaknesses

- The algorithms are efficient regarding regret bound, but the computational complexity is not discussed. - A conclusion section could be added.

Questions

What is the computational cost of the proposed algorithms?

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Some limitations are discussed.

Reviewer TGCM5/10 · confidence 3/52024-07-13

Summary

In this work, the authors characterize the problem of learning the presence of reward corruption in the linear bandit setting. They provide matching upper and lower bounds in the corrupted stochastic setting, and initiate the study on the corrupted adversarial setting, for which they obtain optimal scaling in the corruption level. Not only that, the authors prove a general reduction that efficiently handles gap-dependent misspecification with corruption-robust algorithms. They were able to show that linear MDPs with gap-dependent misspecification are efficiently learnable. While this reduction is general, interestingly they denied the possibility to obtain the tightest rate for gap-dependent misspecification. This observation leads them to develop a specialized algorithm which, in the linear bandit setting, obtains the optimal rate. According to their argument, this resolves the open problem of Liu et al. (2023a).

Strengths

- Interesting results - Deterministic algorithm cannot avoid suboptimal regret (Proposition 1) - Matching upper and lower bound on the stochastic setting, by just changing deterministic sampling to stochastic. - Solving an open problem of instance-dependent misspecified setting. - Clearly state the limitations of previous works and their technical novelties. - Easy to understand their contributions.

Weaknesses

- (Minor) The algorithms are not seriously different from the previous works as they mentioned, but this is just a minor point - every theoretical improvement is important. - Not clear what they tried to say on page 9 - Why Theorem 6.2 shows that $\rho \leq \frac{1}{d}$ is not optimal? - Impossibility result (from line 304): so basically what authors are trying to say is, that 'their' reduction is not applicable for a tighter result, right? It is not about any reduction from corruption to misspecification. - No future works.

Questions

- Unclear points in the weakness section. - It would be great if authors could explain why the gap-dependent misspecification assumption (Assumption 1) is necessary. ### Minor - Theorem G.1 in line 296 - is it correct? - Corollary 6.2.1 in line 209 - it seems like it is the result for the MDP...

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately addressed the limitations. It would be great if authors provide possible future works. There is no potential negative societal impact of their work.

Reviewer QWPG6/10 · confidence 3/52024-07-14

Summary

This paper studied the corrupted linear bandits. The authors propose four different metrics to evaluate the total corruption in Eq. (1). Many settings are considered in this paper. For stochastic LB, the proposed algorithm achieves a regret bound of $d\sqrt{T}+\sqrt{d} C_{\infty}$. For adversarial LB, the proposed algorithm achieves a regret bound in the order of $d\sqrt{T}+\sqrt{d} C_{\infty}$ or $d^{3}\sqrt{T}+d^{5/2} C$. The authors also consider the gap-dependent misspecification, where the misspecification level of an arm $a$ can be evaluated by $\rho$ times the gap of arm $a$.

Strengths

See summary.

Weaknesses

**Weaknesses and Questions:** 1. At lines 107-109, the authors claim that the strong adversary is equivalent to the CM viewpoint. This doesn't seem right. For regret, the strong adversary is harder than the CM viewpoint. Thus, it is unfair and wrong to compare He et al. (2022) in the same way. 2. At line 131, adversarial linear bandits are discussed. However, no problem definition of this problem is introduced before line 131. 3. This paper studies the fixed action set, while the previous works He et al. (2022) and Foster et al. (2020) allow the action set to be chosen by an adaptive adversary, which is much harder than this paper. Table 1 is not fair. He et al. (2022) is for the adaptive adversarial viewpoint, which is totally different from the stochastic LB. For a fixed action set, the optimal regret without $C$ should be $\sqrt{d T \log k}$, where $k$ is the number of arms. 4. Assumption 1 is not very reasonable.

Questions

See weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

N.A.

Authorsrebuttal2024-08-09

Thank you for the comments! We would like to elaborate more as follows. **Regarding the first concern:** Our AA and CM viewpoints are exactly what you describe. However, we would like to draw the reviewer's attention to the definition of the corruption measure in the CM viewpoint: $C=\sum_{t=1}^T |\epsilon_t(a_t)|$. This definition only accounts for the corruption for the "chosen action" $a_t$, but not that for other actions $a\neq a_t$. How does this property allow us to simulate a strong adversary in the AA viewpoint? Below, we illustrate it through a concrete example as in your response (below, we use "he" for the adversary and "she" for the learner). Consider a $d$-dimensional linear bandit with the action set being the canonical basis $e_{1:d}$. From the AA viewpoint, suppose the adversary wants to corrupt the $i$-th dimension of $\theta^*$, and his strategy can be described as the following: *If $a_t=e_i$ is observed, then set $\epsilon_t=c$*; *If $a_t\neq e_i$ is observed, then set $\epsilon_t=0$*. Clearly, this is a strong adversary in the AA viewpoint since the adversary decides whether he wants to corrupt $a_t$, and to what extent he wants to corrupt it, after observing the chosen action $a_t$. From the CM viewpoint, the equivalent adversary would specify the following "corruption function" $\epsilon_t(\cdot)$ *before* seeing $a_t$: *$\epsilon_t(e_i)=c$, and $\epsilon_t(e_j)=0$ for $j\neq i$*. After the learner chooses $a_t$, the learner experiences a corruption of $\epsilon_t(a_t)$. We clarify two points: 1. The learner has exactly the same observations in the two viewpoints: if she chooses $a_t=e_i$, then she suffers a corruption of $c$; if she chooses $a_t\neq e_i$, then she suffers no corruption. 2. The total corruption $C$ is the same in the two viewpoints: In the AA viewpoint, $C=\sum_{t=1}^T |\epsilon_t|=\sum_{t=1}^T |c|\mathbf{1}\\{a_t=e_i\\}$. In the CM viewpoint, $C=\sum_{t=1}^T |\epsilon_t(a_t)|=\sum_{t=1}^T |c|\mathbf{1}\\{a_t=e_i\\}$. One key observation is that: in the CM viewpoint, even though the corruption function might be a non-zero function (i.e., $\epsilon_t(a)\neq 0$ for some $a$), the amount of corruption counted towards $C=\sum_{t=1}^T |\epsilon_t(a_t)|$ is zero as long as $\epsilon_t(a_t)=0$. In other words, it is possible that $\epsilon_t(a)=0$ and $\epsilon_t(b)\neq 0$ for some arms $a,b$, which could simulate the AA strong adversary who decides NOT TO corrupt after observing $a_t=a$, but decides TO corrupt after observing $a_t=b$. This might address the reviewer's concern, "*If $a_t$ is not known, the only option for the adversary is to add corruption to each arm and hope the algorithm picks the arm that has a high influence on the dimension it wants to corrupt.*" From the discussion above, we know that a CM adversary can simply do the following: set high corruption for the actions with high influence on the dimension, and set low/zero corruption for the actions with low influence on the dimension. Then, after $a_t$ is decided by the learner, the corruption level $|\epsilon_t(a_t)|$ will naturally adapt to the choice of $a_t$, i.e., the corruption will be high if $a_t$ has high influence on the dimension, and will be low/zero if $a_t$ has low influence on the dimension. A mathematical conversion between the two viewpoints is as follows. Assume that the strategy of the AA strong adversary can be decribed as a function $\epsilon_t = f(h_{t-1}, a_t)$, where $h_{t-1}$ is the history up to time $t-1$ and $a_t$ is the chosen action at time $t$. Then in the CM viewpoint, the "corruption function" would be defined as $\epsilon_t(a) = f(h_{t-1}, a)$ for every $a$. Notice that the function $\epsilon_t(\cdot)$ depends only on the history up to time $t-1$. **Regarding the second concern:** If the corruption is always zero, our formulation (Line 74 - 79) is the same as the standard adversarial linear bandit setting formulated in Section 27 of Lattimore and Szepesvari (2020), where the linear reward vector $\theta_t$ changes over time. The only extension we make is that we consider additional reward corruption. We can refer to it as "adversarial linear bandits with corruption" to make things even clearer. (Lattimore and Szepesvari, 2020) Bandit Algorithms. **Regarding the third concern:** Thanks for your suggestions. We will add the bound $\sqrt{dT \log k}$ to our Theorem 4.1. We will also discuss more about our bounds compared with previous works.

Authorsrebuttal2024-08-11

We thank the reviewer for the time and effort spent reviewing our paper. As the discussion phase is about to end, we would like to make sure our responses have sufficiently addressed your concerns. We look forward to your feedback.

Reviewer QWPG2024-08-13

Thank you for your response. **Regarding the first concern,** I disagree with your statement. I believe the second point you made is incorrect. For example, with a probability of 1/2, the algorithm selects $a_{t} = e_i$. Assuming both AA and CM have the same influence on the algorithm, and CM has a corruption level $C$, then, since AA only corrupts the algorithm with a probability of 1/2, the corruption level of AA would be $C/2$. I look forward to your reply.

Authorsrebuttal2024-08-13

We thank the reviewer for the feedback. Notice that in our previous response, we do not assume the type of algorithm the learner uses (i.e., whether it's deterministic or randomized). Therefore, the points we make still hold even when the learner uses a randomized algorithm. Below we demonstrate that in the example you gave (i.e., the learner chooses $a_t=e_i$ with probability $1/2$), the corruption levels are the same in the two viewpoints. **AA viewpoint:** Adversary strategy: If $a_t=e_i$ is observed, then set $\epsilon_t=c$. If $a_t\neq e_i$ is observed, then set $\epsilon_t=0$. Corruption level: $C=\sum_{t=1}^T |\epsilon_t|=\sum_{t=1}^T |c|\mathbf{1}\\{a_t=e_i\\}$. Thus, $\mathbb{E}[C]=\mathbb{E}\big[\sum_{t=1}^T |c|\mathbf{1}\\{a_t=e_i\\}\big]=\frac{T|c|}{2}$, where in the last equality we use $\Pr\\{a_t=e_i\\}=1/2$. **CM viewpoint:** Adversary strategy: Set $\epsilon_t(e_i)=c$, and $\epsilon_t(e_j)=0$ for $j\neq i$ before seeing $a_t$. Corruption level: $C=\sum_{t=1}^T |\epsilon_t(a_t)|=\sum_{t=1}^T |c|\mathbf{1}\\{a_t=e_i\\}$. Thus, $\mathbb{E}[C]=\mathbb{E}\big[\sum_{t=1}^T |c|\mathbf{1}\\{a_t=e_i\\}\big]=\frac{T|c|}{2}$, where in the last equality we use $\Pr\\{a_t=e_i\\}=1/2$. This shows that the corruption levels are the same in the two viewpoints. **Potential source of confusion:** We point out that in the CM viewpoint, although there could be ``non-zero corruption function'' in every round (e.g.., $\epsilon_t(e_i)=c$ for all $t$ in our previous example), the corruption measure we consider is $C = \sum_{t=1}^T |\epsilon_t(a_t)|$, which only depends on $a_t$. Thus, even though $\epsilon_t(e_i) = c$ for every round $t$, such corruption only contributes to $C$ when $a_t = e_i$, and it will not contribute to $C$ when $a_t\neq e_i$. This makes it equivalent to the strong AA adversary whose corruption adapts to the choice of the learner. We guess that the reviewer might be thinking about another corruption measure $C_\infty=\sum_{t=1}^T \max_a |\epsilon_t(a)|$, which accounts for the corruption for *every* (no matter chosen or unchosen) action. However, this is not what we want to show equivalence for with the strong AA adversary.

Reviewer QWPG2024-08-13

Thank you for the detailed response. I recommend focusing solely on the Corruption Measure (CM) Viewpoint, as the AA Viewpoint introduces some confusion. The concepts would be more clearly presented without it. This paper makes a valuable contribution to the understanding of corruption across various metrics and presents solid regret bounds. One potential weakness of this paper is that it focuses solely on a fixed action set. I will be increasing the score to 6. Good luck!

Authorsrebuttal2024-08-13

Thanks for your positive feedback and the adjustment for the score. We will make efforts to reduce the confusion to the reader.

Reviewer QWPG2024-08-07

Thanks for your rebuttal. --- Regarding A1: I disagree. In the AA perspective, the adversary can first observe the chosen arm $A_t$ and then decide whether $A_t$ is the arm it wants to corrupt or not. For instance, if the adversary aims to corrupt a specific dimension of $\theta^*$, it can choose to do so to maximize its profit, given that it knows $A_t$ beforehand. If $A_t$ is not known, the only option for the adversary is to add corruption to each arm and hope the algorithm picks the arm that has a high influence on the dimension it wants to corrupt. --- Regarding A2, the term "adversarial linear bandits" in the bandit literature refers to a different setting. Please consider changing the name of "adversarial linear bandits". --- Regarding A4, I believe that $\sqrt{dT \log k}$ is significantly better than $d\sqrt{T}$ regret. It might be beneficial to present the $\sqrt{dT \log k}$ results in your main theorem and clearly discuss the differences in settings and results compared with previous work.

Authorsrebuttal2024-08-12

We thank the reviewer for the time and effort spent reviewing our paper. As the discussion phase is about to end, we would like to make sure our responses have sufficiently addressed your concerns. We look forward to your feedback.

Reviewer TGCM2024-08-12

Thank you for your details. Now I understand the points of the authors. I will keep my score as it is.

Authorsrebuttal2024-08-12

We thank the reviewer for the time and effort spent reviewing our paper. As the discussion phase is about to end, we would like to make sure our responses have sufficiently addressed your concerns. We look forward to your feedback.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC