SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents

Summary

This paper proposes a new backdoor attack against Reinforcement Learning, termed SleeperNets. SleeperNets adopted dynamic reward poisoning to overcome the insufficiency of static reward poisoning proposed in previous works. The author provided a theoretical analysis of the advantages of dynamic adversarial poisoning and also conducted comprehensive evaluations to demonstrate the effectiveness of SleeperNets over previous backdoor attacks against DRL.

Strengths

* This paper shows the drawback of static reward poisoning adopted in the previous DRL backdoor attacks, which motivates dynamic reward poisoning. * The authors provide a theoretical analysis of the returned reward given the design of the dynamic reward, convincingly show that dynamic reward poisoning overcomes the drawbacks of static design. * Comprehensive evaluations including detailed ablation studies are conducted.

Weaknesses

NA

Questions

NA

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Summary

The SleepNets paper considers a new ("outer loop") threat model, more powerful than those typically considered in adversarial RL settings. The authors consider a stealthy attacker, aiming to both be successful (essentially tricking the learner into believing the underlying MDP is instead one of the attacker's choosing) and remain hidden (keeping the values of the original and corrupted policies similar). They provide theoretical results on the limits of the traditional, weaker threat model, and introduce a "Dynamic Reward Poisoning Attack Formulation". This yields their new attack "SleeperNets", which they empirically evaluate.

Strengths

1. The paper has an explicitly stated threat model -- a welcome sight in this area. 2. The paper provides a theoretical investigation of prior threat models, with an ultimately simple example demonstrating an impossibility result. 3. While it is too far out of my area for me to be sure of the coverage of related work, the paper does seem to well-situate its contributions in the broader body of literature. 4. There is a broad but not overly cumbersome set of empirical analyses. 5. The paper is well written and easy to follow.

Weaknesses

The main weakness I see is the applicability of the threat model. As the others state, the adversary is assumed to infiltrate the computer on which the agent is training. It's not clear to me what scenarios would exist where an attacker has that much access and can't perform a far deadlier attack (simply manipulating values directly). The paper would be improved if the authors gave examples of real settings where an attacker could act in this outer-loop way without having direct software access. I do believe such examples exist, they just need to be articulated. That is, described in detail with specifications about how Algorithm 1 could still be executed (and the assumptions about e.g., \beta hold). Some rough ideas for such settings: 1. The RL agent is acting on financial markets and the attacker is able to manipulate the reward signal by directly purchasing shares at an inflated cost from the agent. 2. The RL agent is acting in a physical environment and the attacker is able to manipulate that same environment (I'm picturing how humans train drug sniffing dogs by hiding toys). 3. The RL agent is flying, and the attacker has limited access to some of its instrumentation (e.g., can spoof its GPS location or jam certain signals).

Questions

Same as the weakness described above -- in what real world settings is the attacker powerful enough to perform the sleepernets attack, but not so powerful as to directly manipulate memory values on the training machine?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The primary limitation is the weakness described above. The other limitations are described in Section 7, and I agree with the authors that they are interesting areas for future work. This paper stands without a deeper investigation of them.

Summary

The authors introduce a novel framework for backdoor poisoning RL agents, SleeperNets. SleeperNets assumes that adversaries can inject adversarial perturbations into the agent's observations during policy training within some total budget. Unlike in prior frameworks, the adversary implements its attacks post-hoc on full episode rollouts. The authors show that their attack manages to be successful, while retaining the performance of the optimal unpoisoned setting. The authors implement their novel framework on four different environments and show that it works favourably.

Strengths

The paper has a number of strengths. First of all, I believe that the threat model innovations are sensible; it seems natural that the adversary could manipulate whole episodes and not just single steps. Equivalently, interpreting stealth as retaining policy performance seems sensible. The authors' insight that "dynamic" attacks can attain both success and performance while "static" attacks cannot is insightful. The empirical results seem to support the author's claims.

Weaknesses

* test-time defenders can still perform anomaly detection based on observations, actions or state transitions; I do think the authors should perform empirical investigations using out-of-distribution anomaly detection methods [2] to infer the information-theoretic detectability of their methods. * The idea of condition of increasing the adversaries' attack context beyond single steps is not entirely novel within the adversarial attack literature, see e.g. [1] who devise adversarial attacks that condition on the entire action-observation history. ### minor weaknesses line 264: "environment environment" [1] Franzmeyer et al., Illusory Attacks, https://openreview.net/forum?id=F5dhGCdyYh, ICLR 2024 [2] Nasvytis et al., DEXTER, https://arxiv.org/abs/2404.07099, AAMAS 2024

Questions

* you are not referencing adversarial cheap talk [3] - can you compare and contrast their setting against yours? * you are mentioning a adversarial perturbation budget - where does this budget come from, and why would a budget be justified in reality rather than say a constraint based on information-theoretic detectability as in [1]? [3] Lu et al., Adversarial Cheap Talk, ICML 2023

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors do not further investigate defenses against their novel attack, although they state that securing the training environment as well as developing test-time anomaly detectors would be suitable avenues. ## Update in Response to the Rebuttal The reviewer have successfully addressed my concerns, I therefore now recommend the paper for acceptance.

Response to rebuttal

I thank the authors for their additional comments. These have helped clarify the paper and its context for me. I recommend that more of the discussion the authors laid out be added to the manuscript as space constraints allow, and I have increased my rating of the paper.

Thanks for clarifying my concerns.

I thank the reviewers for clarifying my concerns; I have decided to increase my score in response.