We study indiscriminate poisoning for linear learners where an adversary injects a few crafted examples into the training data with the goal of forcing the induced model to incur higher test error. Inspired by the observation that linear learners on some datasets are able to resist the best known attacks even without any defenses, we further investigate whether datasets can be inherently robust to indiscriminate poisoning attacks for linear learners. For theoretical Gaussian distributions, we rigorously characterize the behavior of an optimal poisoning attack, defined as the poisoning strategy that attains the maximum risk of the induced model at a given poisoning budget. Our results prove that linear learners can indeed be robust to indiscriminate poisoning if the class-wise data distributions are well-separated with low variance and the size of the constraint set containing all permissible poisoning points is also small. These findings largely explain the drastic variation in empirical attack performance of the state-of-the-art poisoning attacks on linear learners across benchmark datasets, making an important initial step towards understanding the underlying reasons some learning tasks are vulnerable to data poisoning attacks.
Paper
Similar papers
Peer review
Summary
This paper explores poisoning attacks on linear classifiers. Specifically, the authors explore why poisoning attacks may have high efficacy on one dataset but low efficacy on another. The authors relate this phenomenon to two potential factors: (1) (projected) constraint set size and separability. The authors theoretically characterize an optimal poisoning attack w.r.t. Gaussian mixture data.
Strengths
Previous work has repeatedly shown that poisoning attacks represent a real and serious threat to machine learning models. Most work in the field focuses on how to design better attacks or better defenses. An improved theoretical understanding of what mechanisms allow an attack to work could accelerate research for both attacks and defenses. In my view, this represents a challenging, under-explored research topic. This paper makes an incremental step into improving understanding in this area.
Weaknesses
I rate the paper as borderline reject and explain my rationale below. I want to make clear that with a thoughtful rebuttal and changes to different aspects, I could definitely see my score increasing. While the paper has many things I like, it is my view that the paper still needs work. Perhaps the biggest problem is that the writing tends to overclaim results or draw conclusions that, as written, are not well supported. I discuss a couple of cases below under "Experimental Results." I also ask multiple questions about points I had concerns with in the "Questions" section below. * A more harmless example of what I perceived as overclaiming is on lines 42-43, which state, "we consider indiscriminate poisoning attacks for linear models, *the most commonly studied victim models in the literature* (emphasis added). It is not my assessment that linear models are the most commonly studied victim model, particularly recently. Reading this sentence, it felt to me that the authors were stretching to over-justify the paper's restriction to linear models. This is despite the subsequent sentences being more than adequate to justify the choice, e.g., "linear models are still not well understood" and "high interpretability." * A second example is on lines 35-36, which state "the reasons why attacks are sometimes ineffective have not been previously studied." I do not know this statement to be true and expect given other work (e.g., the lethal dose conjecture) that this is not true. **Experimental Results** My biggest concerns are around the experimental results and the conclusions drawn from them. * Lines 343-344 state, "Table 1 summarizes the results show that Sep/SD and Sep/Size can *largely* explain why datasets...are harder to poison than others." In my view, this statement is making primarily a causation claim. However, the results in the table provide primarily correlation results. To make a causation claim (which I think is the appropriate type of claim for this paper to try to make), the experiments should externally vary the separability and measure how the performance changes. I realize that is non-trivial, but I believe it is the right type of experiment here. * Lines 390-392 state, "Although limited in effectiveness, the defense still mitigates the poisoning to some degree *mostly by shrinking the projected constraint size*" I do not see how the empirical results as presented can support such a strong claim. As written, the text may barely support the claim that smaller projected constraint size *partially* helps (a weak, not that meaningful claim). To justify *mostly* a lot more is needed. * Section 7 Better Feature Representation: Lines 363-364 state, "We train LeNet and ResNet18 on the CIFAR10 dataset..." From this sentence, it is my understanding that your pretraining was done on all 10 classes. If so, your feature encoder goes beyond the transfer learning setting, and I believe this approach does *not* correspond with "the practical scenario where the victim has access to some *small number* of clean samples..." * On lines 360-361, we present two ideas -- using better features might improve separability and using data filtering might reduce the projected constraint size." I would expect data filtering would both improve separability and reduce the constraint size. The paper should explain how you are ensuring filtering is only affecting one criteria and not the other. * I understand why the authors chose to lay out Figure 1 as they did. However, I did not find the overlapping bars were the best format. I was initially confused by the formatting, and it took me some time to figure out exactly what was being visualized since I expected multiple attacks, but only saw two bars. **Miscellaneous Comments** * Overall, I found section 5.1 notationally cumbersome and lacking a clear structure and flow. I had to re-read it twice to extract all the main points. One possible fix is to provide more intuition about why each is being proposed and how the pieces relate. It felt as though you gave a lot of disparate pieces of information to the reader without walking them through how they fit together as you went alone * Lines 40-41 state, "An attack is considered ineffective if the increased risk from poisoning is roughly equal to or smaller than the injected poisoning rate." This perspective makes sense, but it took me a minute to understand the definition as stated. I think this sentence could be written more intuitively.
Questions
* Eq. (2) allows the two Gaussian distributions to have distinct variances (e.g., $\sigma_1$ and $\sigma_2$). In Theorem 5.3, you specify the two variances as equivalent. Is Theorem 5.3's equivariance specification for convenience in the proof or a necessity? * On Line 155, you wrote, "*We chose 3% as the maximum poisoning rate." I found this confusing wording. I understanding "maximum poisoning rate" is terminology you use in the theoretical discussion. When discussing experiments you perform, defining your poisoning rate w.r.t. a *maximum* implies to me you tested multiple poisoning rates. However, I do not get that indication elsewhere in the paragraph. Please be precise concerning your experimental setup. * Figure 1 uses the "Filtered Enron" dataset. I am not familiar with the filtered version of the Enron dataset. Is this filtering procedure something you performed or an existing dataset? How do the results change if you use the vanilla Enron dataset? If you filtered the dataset yourself, I believe the paper should explain why that filtering was done, not just how it was performed. * Your preliminaries section specifies a few design choices. For example, you restrict consideration to primarily hinge loss. Moreover, you specify an L2 regularizer (Eq. (1)). As a reader, I am somewhat left to guess why these specific choices are made. For example, I assume you chose hinge loss because you want max margin. Concerning regularizer, I was unsure if L2 was for convenience and whether a different convex regularizer (e.g., L1) could be used. If you are focusing on specific design choices, I believe it is better to explain why. * I do not recall any of the theorems requiring specifically the hinge loss as opposed to alternate convex losses. * Lines 183-184 state, "Definition 4.1 *suggests* that no poisoning strategy can achieve a better attack performance than that achieved by $\mathcal{A}_{opt}$" When reading this, I was confused by the choice of the word "suggests," as given $\epsilon$, I found Def. 4.1 clearly stated no alternate strategy achieves better attack performance. * One criterion for Theorem 4.5 to hold is that the support of $\mu_c$ should be a (proper) subset of $\mathcal{C}$. I believe this criterion does not hold in the optimal Gaussian case given the definition of $\mathcal{Q}(u)$. If so, please explain how that affects your results (including the proof). Ideally, this should be more explicit in the paper. * Searching through the paper, I did not see where $\mathcal{Q}(u)$ was used beyond line 246. Perhaps I missed it. If it is just for that one point, I think it is worth removing.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Better understanding what causes attacks to fail could lead to enable better attacks. At the same time, it can lead to better defenses. I see no serious ethical or societal risks for this work.
Summary
This paper delves into the impact and effectiveness of indiscriminate data poisoning attacks on machine learning models, particularly linear ones. It highlights significant variations in attack effectiveness across different datasets and seeks to understand whether this is due to inherent dataset robustness or shortcomings in current attack strategies. The paper presents optimal poisoning definitions, characterizes these attacks under a theoretical Gaussian mixture model, and identifies learning task properties that affect attack performance. It concludes with suggestions for designing better defenses against such attacks.
Strengths
- The question the paper is trying to answer, of whether some datasets are inherently robust to poisoning attacks or just resilient to state-of-the-art attacks, is original and interesting. - The paper flows smoothly and is decently written, but contains some grammatical mistakes and typos (some of which I mention below). A careful proofread can increase the quality of writing quite a bit. - The paper is sound, and the claims are proved on 1D toy Gaussian mixture model then extended to more complex data distributions. - The empirical results support the theoretical results. Specifically, on various datasets, the results show that the datasets that are less vulnerable to poisoning have higher values for metrics defined by the paper (Sep/SD and Sep/Size).
Weaknesses
- The contributions of the paper is not very significant as there is no clear practical implication of the findings of the paper. However, the paper investigates an interesting phenomenon of "inherent robustness" of some datasets which might inspire further investigations of this sort. - The paper analysis is restricted to linear models.
Questions
- The format of figure out is confusing. - Are the bars stacked? - Caption mentions “Datasets are sorted from lowest to highest base error rate”. Does this mean the error rate of the SVM trained on $S_c$? - Please change the color palette to more distinguishable colors. - The paper analysis is restricted to linear models. Would any of these insights hold for more complex models? Did the authors do any (empirical) investigation there? In particular, it would be interesting if the authors find that high Sep/SD and Sep/Size metrics are indicative of lower vulnerability to poisoning in more general neural networks and on datasets such as CIFAR-10 and ImageNet. - Are the percentage changes in the results paragraph in Line 157 absolute or relative? I believe it is absolute, but would be helpful to clarify that. ### Minor - Line 123: What is a “theoretic game”? - Line 278: an higher-> a higher
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Yes.
Summary
Inspired by the variant performance of indiscriminate data poisoning attacks against different datasets, this paper studies the inherent robustness of datasets against them. Using an optimal attack designed for a binary Gaussian dataset, the authors discover three properties, namely class separability, variation, and constraint size are important factors to determine the robustness of a dataset. Experiments further validate the theory on several datasets.
Strengths
(1) This paper studies an important problem, namely the dataset robustness against indiscriminate data poisoning attacks. (2) The authors provide initial but valuable evidence on understanding the vulnerability of datasets by explicating the three properties. Although these characteristics match our intuitive understanding of the problem, they have not been systematically analyzed or studied. (3) The empirical findings, especially the results in Section D, are nicely presented and very comprehensive. Moreover, I can find answers to most of my initial questions when reading the main paper in the appendix.
Weaknesses
(1) I don't find the title a nice summarization of the entire paper as it is probably too general. I would suggest mentioning the dataset robustness in the title. (2) I don't think comparing with LDC is necessary as I clearly see the drastic difference. I suggest the authors move this part to the appendix and present some more valuable content in the main paper, e.g., some experiments in Appendix D. (3) Although it has been mentioned in the appendix, I think it is important to mention what attacks you use to generate Table 1. The authors should at least provide a pointer to the appendix.
Questions
Questions are listed above.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
Limitations are well addressed.
Summary
This paper explores the issue of data poisoning attacks on machine learning models, focusing on linear models. The authors find that the effectiveness of state-of-the-art poisoning strategies varies across different datasets. They introduce definitions of optimal poisoning attacks for finite-sample and distributional settings and prove performance convergence under certain conditions. They also characterize optimal poisoning attacks under a theoretical Gaussian mixture model and find that a larger projected constraint size is linked with higher vulnerability, while larger separability and smaller standard deviation provide less vulnerability. These findings explain the performance differences of indiscriminate poisoning attacks on linear models across various datasets and suggest ways to improve robustness against poisoning.
Strengths
- The observation that the performance of state-of-the-art poisoning strategies significantly varies across different datasets is indeed insightful. It not only challenges the common understanding of indiscriminate poisoning attacks but also opens up new lines of inquiry into why these differences occur. - The theoretical analysis under specific data distributions is a major strength of this paper. It's robust, reliable, and underpins a comprehensive understanding of how and why different datasets might be more or less vulnerable to poisoning attacks. By tying the theoretical analysis back to empirical attack performance, the authors effectively bridge the gap between theory and practice, enhancing the practical relevance of their findings.
Weaknesses
- It's not explicitly clear from the paper whether the observation that "linear learners on some datasets are able to resist the best-known attacks even without any defenses" is novel or previously reported. The authors could benefit from addressing this explicitly. - The authors' focus on linear classifiers is indeed a limitation. While the paper provides valuable insights into data poisoning attacks on linear models, these findings cannot be directly extrapolated to non-linear classifiers such as neural networks. Moreover, the paper does not provide enough information to definitively conclude whether linear classifiers are more robust than non-linear classifiers without additional defense strategies. - The paper doesn't sufficiently distinguish its analysis from existing works on data poisoning that also provided theoretical analysis on Gaussian mixture distributions and linear classifiers, such as [1]. Could you elaborate on this? [1] Tao, Lue, et al. "Can Adversarial Training Be Manipulated By Non-Robust Features?." Advances in Neural Information Processing Systems 35 (2022): 26504-26518.
Questions
- Is the resistance of some linear learners to attacks, even without defenses, a new finding or reported in previous studies? - How applicable are these findings to non-linear classifiers like neural networks? Do they imply greater inherent robustness of linear classifiers against attacks?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
In summary, this paper makes a substantial contribution to the field. Although its primary limitation is its exclusive focus on linear classifiers, it effectively elucidates the observed phenomena within this scope.
Initial Reply
I have read the other reviews and the authors' rebuttals. I noticed that the authors may have felt constrained by the length limitations on their rebuttal. If the authors feel the length limitation prevented them from addressing all points they wanted to make, I am fine with them making multiple official comments under my review. I will read all of them. Given the amount of points I raised and the length of the rebuttal, I cannot respond to most points tonight. I will address some of the points below and plan to reply more over the weekend. Of course, the authors are welcome to rebut my partial response if they choose. > Why same variance in Theorem 5.3? Your explanation is reasonable. If there is not already a note explaining this choice, please consider adding one. > Is hinge loss specifically required? This updated explanation makes a lot more sense. I recommend updating the manuscript to reflect your clearer, more detailed rationale above. > We will move part of our results in Section D of the appendix to the main paper in the revised version. In my view, I would leave the lethal dose conjecture (LDC) discussion as is in the main paper. I understand the reviewer's statement "I don't think comparing with LDC is necessary as I clearly see the drastic difference." However, LDC is a recent important work in this area and I think the paper is stronger when the contrast is clear and explicit -- even if some readers may find the contrast obvious. > Figure 1 Would it be possible to add an updated Figure 1 to your one-page rebuttal for us to better visualize what you have in mind? I am not sure what is possible or if you could also link an anonymized separate document. > Experiments on better feature representation do not correspond to the practical scenario of In short, I consider the feature extractor used in this experiment particularly unrealistic. For example, since the feature extractor is trained on all of CIFAR10, the feature extractor's objective function specifically optimizes over the examples then used in your fine-tuning analysis. A more meaningful experiment setup would train the feature extractor only on 8 of 10 classes (e.g., everything but "Truck" and "Ship"). If the feature extractor is trained over all instances with the target labels, then, in my view, we are no longer in the "practical scenario where the victim has access to some small number of clean samples." Intuitively, an extractor that uses supervised pretraining to extract the best possible features will always be (far) better than when a victim who "has access to some small number of clean samples."
Additional response to unaddressed and new follow-up comments
We really appreciate your prompt response. Please find our additional responses to the follow-up comments and also the unaddressed comments in the initial rebuttal. > Same variance in Thm 5.3 We will definitely add clarification in the revised version to make the presentation of the results clearer. > Hinge loss requirement We will also clarify this in the revised version, with provided details in the initial response. This will greatly improve the readability of the paper. > About results on LDC We did not mean to exclude the comparison to LDC from the main paper. We will work around spaces in other parts of the paper to make sure the LDC content is there while also including some interesting results from Section D. We acknowledge the importance of LDC in the area and that is why we included a detailed comparison to LDC in Appendix E. > Figure 1 Please find the revised Figure 1 from this anonymous [link](https://drive.google.com/drive/folders/1KAwXQP-OtnDV89oF7UdbS2pi2TU0D_jk?usp=sharing). We were no longer allowed to update the attached PDF file in the global response. > Experiments on transfer learning experiment Thanks for the clarification. We now understand the concerns of the reviewer. We first note that the current experiment still shows the possibility of the victim selecting a better feature extractor free from poisoning to improve the resilience of the downstream classifiers to indiscriminate attacks. However, we agree with the reviewer that a more meaningful experimental setup should exclude the training data used in the downstream analysis when training the feature extractor. We are currently rerunning the experiments where the feature extractor is trained on the full CIFAR10 dataset except data points from classes of "Ship" and "Truck". We will post another response when the results are ready but decided to go with other comments first in case there are additional follow-up questions from the reviewer. Below, we provide responses to comments that are not addressed in the rebuttal. > linear models are not the most commonly studied victim against indiscriminate poisoning attacks, particularly recently Thanks for raising this point. In the original submission, we meant to say "most commonly" by the extensive number of concrete attack algorithms proposed for the indiscriminate setting that inject small amount of poisoning points, but realized this statement is indeed confusing and not general. We will simply remove this sentence in the revised version. Reviewer is also correct that, particularly recently, there are more focus towards using neural networks as the victim models and we also plan to perform more systematic investigation on non-linear models such as neural networks in our next step. > The statement of reasons on why attacks are sometimes ineffective have not been previously studied is not true In the original submission, we meant to say, for the previous works that propose concrete indiscriminate attacks, the reasons why sometimes their attacks are ineffective are not investigated. This is why this sentence appears right after the list of proposed indiscriminate poisoning attacks. We will make this statement clearer to avoid possible confusion. > Improve the presentation in Sec 5.1 Thanks for raising this, and our presentation will be much clearer if the connections of the presented definitions and theorems were introduced verbally at the beginning of Sec 5.1. The proof sketch under Theorem 5.3 attempted to serve for this purpose, but we will now revise our paper by making these connections more explicit at the beginning of the section. > Definition of vulnerability should be stated more intuitively We will restate this definition as "Following definitions given in the existing works, an attack is considered ineffective if the increased error from poisoning is equal to or less than the injected poisoning ratio". > Why stating 3\% as the maximum poisoning ratio? The reviewer is correct that, in the experiments, we only use 3% as the poisoning ratio, not as the maximum poisoning ratio. Although we included results of different poisoning ratios in Appendix D, these results are not related to the main claims made in the paper using the 3\% poisoning ratio and we will revise the statement to make it precise. > Definition 4.1 suggests? We will replace the word "suggests" with "states".
Further Reply
Thank you for providing the additional information I requested. Overall you have addressed most of my concerns. I look forward to your updated experimental results. I will reassess my score once those results are available.
New result on Figure 2
We are glad that our response addresses most of your concerns, and really appreciate your time and effort in reading our responses carefully. Below, we provide the new results on Figure 2 in the original submission. We repeated the experiments of Figure 2 in the paper with a new feature extractor whose training never witnesses data points from the classes of "Ship" and "Truck". The results are summarized in the updated figure available in this anonymized image: [link](https://drive.google.com/file/d/1umD2sv5fR9t1nK4Ed4N8fdX0YOsfaVrZ/view?usp=share_link). Each data point in the figure is marked with a text in the form of `F:(base,increase)`, where `F` denotes the feature extractor type. For the extractor types, similar to the description in the paper, `LeNet` means an extractor of a fully trained simple CNN model while `R-X` means the ResNet-18 model trained for `X` epochs. `base` denotes the base error (without poisoning) when we train a linear SVM model on the feature representations of "Ship" and "Truck" (these two classes are not used in the feature extractor training), and `increase` denotes the error increase resulting from the state-of-the-art poisoning attacks. Overall, the main message in the updated figure is similar to what we showed in Figure 2 in the original submission. In particular, when we adopt deep architecture and train it for more epochs, then the quality of the obtained feature extractor increases and so, it appears to improve the resistance of downstream classifiers to poisoning attacks. The increased quality of the feature extractor is in general correlated to increased Sep/SD and Sep/Size. An interesting (and somewhat surprising) observation is, the LeNet model, even though has the second-lowest base error, has the highest increased error among all tested datasets. We speculate one possible reason is that the search space of the poisoning points is relatively large, which makes the overall dataset more vulnerable, although further experiments would be necessary to confirm this. It would be interesting to perform further investigation and also to understand the differences between two model architectures (e.g., simple CNN and ResNet18). We note that there are also some differences compared to the results in Figure 2 in the original submission, where the feature extractor training additionally uses the downstream training data. First, because the feature extractor never sees images of "Ship" and "Truck", the base errors of the downstream linear models are high (the lowest base error in our new experiment is 12.8% while the lowest one reported in the original submission is only 1.0%). Second, the differences between adjacent epochs are no longer that distinguishable. For example, comparing R-90 and R-100, training for an additional 10 epochs does not appear to bring additional benefits. To conclude, our new results still show the possibility of the victim choosing a better feature extractor to improve the resistance of downstream classifiers to poisoning attacks. Based on our limited data points, one possible solution might be to adopt some deep architectures and train for a sufficient number of epochs. We leave a more systematic investigation along this direction for future work. We will also revise our statement to avoid drawing strong conclusions based on the preliminary experiments.
Updated Score
Below I attempt to briefly summarize the takeaways from the revised experimental setup. If you disagree with any parts of my summary, you are more than to point them out. 1. Your latest response writes, "*The increased quality of the feature extractor is in general correlated to increased Sep/SD and Sep/Size*." I agree with that general sentiment. I recommend including, at minimum, the Pearson correlation in your rebuttal and the paper to make "generally correlated" more concrete. 2. It seems the correlation is weaker than in the original manuscript's version of the figure (visually, I cannot determine if the results constitute "substantially weaker"). A comparison of the Pearson correlation with and without the change would probably be appropriate for discussion here. Unfortunately, it seems the modified experiments (somewhat) weaken the strength of your overall claims. If you have any thoughts on improving the experimental design to better support the claims, I would be interested in discussing more. For now, I will update my score based on the new results and discussion so far.
Follow-up discussion on new results
Thanks for updating the score and we are happy to address your follow-up comments. We agree that Pearson correlation coefficients are good metrics to demonstrate the correlation quantitatively between our identified factors and error increase. Below, we show the computed coefficients with respect to each individual factor (i.e., Sep/SD or Sep/Size) for both Figure 2 in our original submission and the updated Figure 2 in the new experiment: | Figure 2 | Sep/SD vs Error Increase | Sep/Size vs Error Increase | |:--- | :----: | :----: | | Original | -0.98 (-0.97) | -0.98 (-0.97) | | New | -0.66 (-0.90) | -0.90 (-0.99) | The value in the parentheses denotes the Pearson correlation when we exclude the `LeNet` structure and only consider `ResNet-18` models. From the table, if we directly measure the Pearson correlation with each individual factor, it is true that the correlation between the two factors and error increase becomes weaker when the feature extractor no longer uses the downstream training data. However, it is worth noting that the lowest (absolute) correlation coefficient `-0.66` still indicates a fairly strong negative correlation. And if we further restrict ourselves to ResNet-18 models only, then the (absolute) correlation coefficient is even higher, which again confirms the two factors are strongly correlated with error increase. Importantly, we want to clarify that the computed Pearson correlation coefficients in the new experiment being not as high as those of the original Figure 2 does _NOT_ weaken our overall claim. Recall for our main experiments in Section 6, the major claim is that the two identified factors (i.e., Sep/SD and Sep/Size) are strongly correlated to the effectiveness of indiscriminate data poisoning attacks across different datasets, which is well-supported through the high degree of correlation indicated by the Pearson correlation coefficients computed for the new Figure 2 in the new experiment above. In addition, we note that there is likely to be a joint impact of the two factors, and it is also possible for the correlation to be non-linear as suggested by our theoretical results (e.g., see Theorem 5.3 for Gaussian distributions), which may not be well captured by the Pearson correlation with each individual factor, since they are used to characterize the linear correlation between two variables. We believe it would be an interesting yet challenging future work to characterize such relationship as comprehensively as possible between the two identified factors and error increase under the transfer learning setup. Our initial thought to gain a better understanding of such relationship is to produce more dots in the new Figure 2 by varying the feature extractor (e.g., varying model architecture or using pretrained ImageNet encoders), which may potentially uncover interesting patterns. In addition to not weakening our main claim, the Pearson correlation coefficients calculated for the updated Figure 2 also suggest similar possible practical implications on how to build more robust defenses against indiscriminate poisoning attacks, as what we explained in Section 7 of the original submission. Recall that the implication drawn from the original Figure 2 is that using deeper architecture and training for more epochs on datasets free from poisoning can potentially be an effective approach for enhancing the downstream model robustness by increasing both factors simultaneously. Since the Pearson correlation computed for the updated Figure 2 confirms that each individual factor is negatively correlated with the error increase, we can still conclude that increasing both Sep/SD and Sep/Size regrading the feature extractor might be a potential way for improving the resilience of downstream classifiers. Nevertheless, there exists a slight difference when we consider feature extractors that are not trained using any downstream data, as explained in our last response. The identified two factors no longer strictly increase simultaneously (e.g., compare R-25 and R-50 in the new experiment, where R-50 has a reduced Sep/SD but increased Sep/Size) as we train for more epochs. But, if we compute the Pearson correlation between the number of epochs (for ResNet-18 model) and the error increase from poisoning, we observe a even stronger correlation compared to the original Figure 2, which again supports the claim that training the deep architecture for more epochs seems to be one possible approach to improve resistance to poisoning. The slight increase in correlation in the new experiment results from the slightly higher covariance between the error increase and the number of epochs, but this difference is not significant since we only consider a limited number of samples when computing the correlation. The comparison results are summarized below: | Figure 2 | Epoch Number vs Increased Error | |:--- | :----: | | Original | -0.95 | | New | -0.99 |
Continued follow-up discussion on new results
In summary, we believe the new results of Figure 2 still provide strong empirical evidence to support our main claim and suggest a similar future direction from the perspective of learning better feature representations to build better defenses against indiscriminate poisoning. Nevertheless, depending on different application scenarios (e.g., how much downstream data are used) and how the feature extractor is learned, the two factors may not consistently change in the same direction, as reflected in the new experiment. Therefore, developing general methods for learning meaningful feature extractors such that the two factors can be improved simultaneously in a controlled way would be an interesting research direction, which we plan to pursue in the future. However, we also note that this is a challenging task from the practical perspective since the goal is to extract useful feature representations for downstream applications, so the base error should also be considered in addition to error increase from poisoning, especially when no downstream data are used. Our initial thought is to consider an intermediate few-shot learning setup, where a small number of (clean) downstream training data are allowed to be incorporated in training the feature extractor. This few-shot learning setup might correspond to an intermediate setting between the original experiment with full knowledge of (clean) downstream training data and the new experiment with almost zero knowledge but still reflects certain real-world application scenarios. Thanks for your valuable comment. We will carefully discuss the implications with the computed quantitative metrics and also revise the statement in the original paper correspondingly.
Thanks for your constructive feedback
We greatly appreciate the thoughtful comments from the reviewers that have helped improve our paper. We will reflect on them carefully in the revised version of our paper. If there are any remaining questions or concerns, we are more than happy to address them before the rebuttal deadline.
Decision
Accept (poster)