Summary
Data poisoning attacks have been extensively studied in the offline setting, while poisoning attacks in the online setting have received little attention. In the online setting, the attacker is forced to craft attacks exploiting the data stream, taking into account the state of the model and the possible future data stream to divert the model. This work aims to study theoretically and empirically the learning dynamics of an online learner subject to a poisoning attack in the white-box setting, i.e., the attacker knows the architecture and the current parameters of the target model. The problem of finding the best attack policy is formalized as a stochastic optimal control problem and is theoretically studied in the teacher-student setup. From that, the authors were able to obtain results about the steady state of an attacked linear regression model, i.e., the model weights, the optimal action and the distance of the target model depending on the cost of the attack, supposing both infinite and finite size batches. Furthermore, the work shows analytically that a greedy strategy can perform quite optimally. The experimental evaluation confirms the theoretical results on synthetic data and explores the dynamics of online data poisoning attacks on real data and non-linear predictors, showing that the theoretical findings almost hold in more complex scenarios.
Strengths
This paper makes a valuable contribution to the literature on data poisoning attacks, where there has been limited work in online settings. In particular:
- The idea of analyzing the attacker’s control problem from the perspective of the teacher-student framework represents the strong main novelty of the paper, since the formulation of the attack problem as an optimal control problem has been discussed before. The analysis allows the authors to derive interesting results about the dynamics of the learning process of the model and the steady state of a linear regressor that have not previously been discussed in the literature. It is interesting to see that above a certain attack strength the accuracy of the attacked model may drop dramatically and perturbations above the threshold may not be sufficient to realize the attack.
- The set of considered attack strategies sufficiently covers the possible options available to the attacker. The idea of showing that a greedy attack can be as effective as an attack with full knowledge is valuable.
- The theoretical results are confirmed empirically and punctually. Moreover, the experimental results on non-linear architectures and real data suggest that the theoretical results may also hold for more complex scenarios, that’s an interesting finding.
- The derivation of the theoretical results is explained in the appendix and the well-organized code to reproduce the experimental results is available (even though more instructions to reproduce the results may be added, like the Python version used by the authors to run the experiments).
Weaknesses
- The theoretical analysis is limited to consider linear regression as the student model, the standard normal distribution as input distribution and an infinite time horizon, so the theoretical results are a bit limited. However, it is recognized by the authors that more general results are missing because deriving them in a more general scenario is hard. Furthermore, the authors tried to derive some intuitions about the results in a more complex scenario through the experimental evaluation. Covering the theoretical analysis in a more complex setting and with a finite time horizon is an interesting future work to explore.
- It is not clearly motivated why the attacker can only modify the labels instead of the entire inputs and why the attacker should be able to apply the same perturbation to the entire batch (which turns out to be a very strong assumption). Data poisoning attackers should assume little control over the data to be considered practical [P1, P2].
- The presentation of the theoretical results is a bit confusing since it is mixed with the presentation of part of the theoretical results.
- The presentation of the attack strategies and their implementation lack details that cannot be found in the appendix. See the “Questions” section for more details.
- From the paper, it is not evident what the take-home message is and what suggestions are provided to improve robustness in stream pipelines or make attackers more stealthy. The paper presents numerous useful analyses and extensive experimental settings but lacks a "Take-home message/Discussion" section that summarizes the overall findings and suggestions. While the authors have reported some of these in the conclusion section, I believe they can be further expanded with higher priority/attention.
[P1] Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Federated Learning. In IEEE Symposium on Security and Privacy (2022).
[P2] Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning. ACM Computing Surveys (2022).
Questions
Major questions:
- Why are the clairvoyant and RL attacks infeasible to deploy for complex models? Is it due to the time required for the calibration? This is not very clear from the text.
- In Eq. 1, please clarify why it should be important to model an attacker that “under/overshoots”. Why can the attacker obtain a perturbed label that resides outside the interval between the clean and target labels?
- I think that more details about the RL attack are needed. How is the problem of finding the optimal sequence of actions mapped to the RL problem? Which are the reasons for choosing to utilize TD3 (and a deep learning policy in general) instead of other RL approaches?
- Which is the utility of gamma^tilde in the objective function minimized by the greedy attack? Is it used to address the fact that the attacker should not try the most effective attack at the subsequent time step, but she has to take into account all the future streams of instances? How does the calibration of gamma^tilde work? I invite the authors to add more details about this.
- I suggest that the authors add a "Take-home-messages/Discussion” section that summarizes the theoretical and empirical results and highlights how the results can be useful for designing a poisoning attack or defending against it.
- The authors should clarify their threat model in the paper and where it may apply.
Minor questions:
- Are the results of Figures 1-B and 1-C obtained from the experimental evaluation? If yes, the setting used for obtaining these results should be indicated in the text. I may have missed it, so I kindly ask the authors to point me out the lines containing this information if it is included in the paper.
- Should g^nef be at the place of one of the g^pre in eq. 16?
Eventually, I kindly ask the authors to fix the links to the equations that appear in the text, since they don’t work at the moment.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors have briefly discussed some limitations of their work in the conclusion section and they have clearly stated that extending their analysis to non-linear architectures and different probability distributions of the inputs is challenging. However, I think that a discussion about the limitations of the threat model considered is missing and I invite the authors to cover this point, by discussing, for example, why the attacker should only modify the labels of the instances. Moreover, I ask the authors to include a “take-on-messages/discussion” section to summarize and clarify the obtained results and their implications.