Summary
This paper proposed a new robust offline RL algorithm called the Corruption Robust PEVI (CR-PEVI). The CR-PEVI achieves smaller suboptimality error compared to existing methods. The key idea behind CR-PEVI is to utilize the uncertainty-weighting technique when solving the least-squares problem in PEVI. The weights in the proposed method do not have closed-form solution, thus the authors proposed an iterative approach and relies on the monotone convergence theorem to prove convergence to the target weights. Based on that and several other assumptions, e.g., dataset needs to be well-explored, the paper proved an upper bound on the suboptimality of the policy learned by the CR-PEVI algorithm. When specializing to the linear function approximation with dimension d, the proposed CR-PEVI achieves O(eta*d/n) convergence rate. Experiments demonstrate the effectiveness of the CR-PEVI method.
Strengths
(1). This paper studied robust offline RL, which is a hot and timely topic in recent years. Prior works mostly consider test-time attacks that are relatively easier to defend. However, less is understood about robustness against training-time attacks. This paper pushes the frontier in this domain and gained deeper understanding on how to design poisoning-resilient offline RL algorithms in the function approximation scenario, which is important and fascinating work.
(2). The theoretical analysis is intriguing. The authors combined the uncertainty-weighting technique and the PEVI algorithm to propose a new algorithm that is robust to offline poisoning attacks. The analysis is based on several critical theoretical results established in the paper, which are complete and solid. Overall, the paper has strong theory contributions.
(3). The empirical results are also convincing. On all the tasks, the CR-PEVI achieves much better performance compared to baselines.
Weaknesses
(1). It seems like the definition of cumulative corruption is not commonly used by prior works. It is defined as the gap between the Bellman operators induced by the clean dataset and corrupted dataset, which is less intuitive and harder to quantize. In contrast, it makes more sense to define corruption level as the real modification on the reward/state transition over the trajectories.
(2). In the experiment, the setup does not strictly conform with attacks under limited budget. The authors implemented different attacks without reporting the corruption level for every different attack and victim algorithm. Furthermore, there is not parameter study on the corruption level eta as well. This makes me wonder if the comparison to prior works, in particular, Zhang et al, is fair enough.
(3) Missing a few related references on training-time attacks in RL:
[1]. Policy Poisoning in Batch Reinforcement Learning and Control
[2]. Adaptive Reward-Poisoning Attacks against Reinforcement Learning
Questions
(1). Why the paper chooses to define the cumulative corruption based on the Bellman operator instead of the modification on the dataset? Is the goal to favor theoretical analysis? How does the theory changes if the corruption level is defined differently, e.g., total change on the rewards?
(2). It's important to report the corruption level induced by different attack and victim RL algorithms in the experiment section. Furthermore, I was wondering how the cumulative corruption level affects the policy learned by CR-PEVI in practical scenarios.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.