Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models

Summary

The paper proposes a new attack called "privacy backdoors", which introduces backdoors into foundation models, making them more prone to leak fine-tuning data of a victim who is adapting the foundation model for their task. For this attack, the attacker collects a set of possible data points that might be used to fine-tune the model. The attacker then tries to inconspicuously alter the loss for the target data points such that they have an anomalous loss value. After the victim fine-tunes the model with private data, the loss of the target data points used for fine-tuning will be anomalous, allowing the attacker to tell whether they were used for training or not. The proposed approach is evaluated on vision models (CLIP) and on LLMs (GPT-Neo). In an ablation study, the paper shows that the attack is robust against different parameter-efficient fine-tuning and inference methods.

Strengths

- The paper is well-written and easy to understand - The paper is well organized, and a reader who is not an expert in privacy attacks can follow the paper easily - The proposed approach is novel - The approach is technically sound, and extensive experiments were conducted to show that the proposed approach is working with different models, fine-tuning methods, and inference methods.

Weaknesses

- My main concern is that the assumption that the attacker already has part of the training data (i.e., the target data points) is quite unrealistic. This setting assumes that the attacker has way more knowledge than in traditional membership inference attacks, where usually only similar but not the exact same data points are available. If the fine-tuning data is assumed to be the victim's private data, then it is not really private in the first place if the attacker can collect parts of this data set. As a result, it is not very surprising to me that after introducing the "backdoor", the model leaks more information about these data points the attacker had in the first place. - I am skeptical that the proposed approach is a "backdoor". Usually, backdoors in machine learning have a trigger and produce a predefined output. However, with the proposed approach, we are basically just changing the loss of specific samples. The way it is done is "stealthy", but the methodology does not fully align with the definition of a backdoor as it is currently defined in the literature. - I am not quite sure what the intention of section 2.2 is. At the beginning of the second paragraph, it is said that the presented method shares similarities with federated learning. But then only differences are brought up. So, for the reader, it is not clear what the similarities are to federated learning. - It is a bit hard to judge the performance of the LLMs based only on the log perplexity loss. It would be nice to have some kind of benchmark similar to the accuracy in the vision model experiments. - The authors state that maximizing the loss of target data points does not work for LLMs; however, no explanation or experimental results are given, which shows that it does not work.

Questions

*Q1:* What is the reasoning/intuition why LLMs have a problem reaching high losses? *Q2:* Minimizing the loss on the LLMs seems to improve the leakage way more than maximizing the loss for the vision model. Did you try to minimize the loss for the vision models? How is the minimization of target point losses working for CLIP? *Q3:* Are there experimental results showing that maximizing the loss of target data points for LLMs is not working? *Q4:* The target data points were from the same distribution. However, in reality it might be that the attacker collects the target data points that have a different distribution than the majority of the data used for fine-tuning. For example, if there is a dog class used for fine-tuning the attacker might collect images of different dog breeds that are in the fine-tuning dataset, but the majority of the images used for fine-tuning are different dog breeds. What happens if the target and auxiliary data points are from a different distribution? I could imagine that if the data points are from a different distribution, the effect of increasing the vulnerability of other data points than the target ones might be drastically reduced. *Q5:* Usually the models are not evaluated on the same data set they are fine-tuned on. What is the accuracy/loss value on a dataset that is not used for fine-tuning (e.g. model is fine-tuned on CIFAR-10 and accuracy before and after on ImageNet with and without the backdoor is measured). Is the backdoor still stealthy in this case? *Q6:* Why is the validation loss for the poisoned MIMIC-IV model so much lower than for the unpoisoned model? I assume lower loss means better performance, which is why this result does not make sense to me. Do you have any explanation on why this is the case? *Q7:* Do you have an explanation on why the OPT-350M model does not memorize the target dataset that well? *Q8:* Which model was used for the fine-tuning/inference method ablation study in table 3 and 4?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations are addressed. However, I would encourage the authors to discuss the potential impact of different data distributions on the proposed approach.

Summary

The authors focus on a new vulnerability that concerns pre-trained models which relies on an adversary modifying the pre-trained model in a way that increases the models vulnerability to membership inference attacks (MIAs). The attack is thoroughly evaluated on both vision-language models and LLMs when fine-tuning using different strategies and on different fine-tuning data sets. Furthermore, the authors conduct different ablations of the attack.

Strengths

- Originality: The work shows that SoTA MIAs (like LiRA) can yield better performance with an attacker that has not unrealistic amount of extra information or power. While it builds on-top of LiRA, it is quite clear that (loss-based) MIAs should benefit from this approach. Related work is appropriately discussed and cited. - Quality: The paper is technically sound and shows through an appropriate amount of experiments that their proposed attack works on multiple SoTA models. The threat model is carefully described and provides the reader with enough information about potential weaknesses of the method. Some minor points I'll mention in the Weaknesses. - Clarity: The paper is clearly written and I only have some minor suggestions under weaknesses that the authors can easily fix before a potential camera-ready version. - Significance: The work is highly significant as nearly all current SoTA approaches rely on fine-tuning pre-trained models. As mentioned by the authors, libraries like Huggingface make pre-trained models more accessible and lower the threshold for downloading pre-trained models. At the same time it seems very likely that malicious models could be downloaded (e.g., when searching for a CLIP like architecture). I think this work makes very apparent that the community must defend against these types of attacks.

Weaknesses

Major: - Stealthiness of the attack: The paper does not look at how, e.g. the zero-shot performance on unrelated data sets (e.g., NOT the target data set) changes once the model is poisoned. The original CLIP paper (Radford et al., 2019) considers many different data sets. I understood that the authors argue that poisoning for better MIA on CIFAR-10 is stealthy because the performance on CIFAR-10 doesn't change much. But how well is the model still performing on other data sets such as CLEVER, FGVC Aircraft or others? Is there forgetting regarding that data? I see that the scenario breaks a bit down if that is the case because the adversary would need to poison specifically for one victim while hoping that nobody else uses the model and wonders why it performs poor in zero-shot. Eventually this would lead to the model being flagged and the model being taken down. I am willing to increase my score if this point has been addressed by the authors or if they clarify why this is not relevant. Minor: - Defense and Detection: The paper would be better if it could provide some ideas regarding potential defenses against this attack or methods to detect that the model includes a backdoor. I don't expect experiments but it would be great to elaborate a bit more than "In the future, it may be necessary for those who make use of pre-trained models to perform as much (or more) validation of the pre-trained models that are being used as any other aspect of the training pipeline." - Table 3: It would be great to specify which model is being used in a separate column. It can be quite confusing as apparently the Linear Probing is only used with CLIP. - Tables: It would be great if the caption could be elaborating a bit more than just a heading. E.g., by mentioning the model that is under attack. - Pre-training data: It would be good to mention what pre-training data has been used for the pre-trained models. This can make quite a difference when replicating the results.

Questions

1) See major weakness: Have the authors investigated the stealthiness more? How does the zero-shot performance on data sets change that are NOT the target data? 2) Which pre-training data has been used? 3) Have you considered how effective your attack would be for fine-tuning under Differential Privacy? Could you speculate how your attack would perform there?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes

Summary

This paper introduces a so-called “privacy backdoor” attack. The attacker poisons a pre-trained model to make it susceptible to membership inference attacks (MIA) on an apriori known set of target examples. This poisoning is carried out by continually training the pre-trained model on the target examples and an auxiliary dataset (needed to preserve the base performance of the model), employing loss terms that later cause significant loss-contrast between target examples that are included in fine-tuning and that are not. The attack is empirically tested on vision and large language models, using (on a low level) opposite attack strategies. The evaluation presented in the paper shows that this privacy backdoor attack is effective at enhancing the performance of a prior MIA on both domains, and across various models, fine-tuning methods, and inference strategies.

Strengths

The paper studies an important and timely threat. The practice of downloading and fine-tuning open-sourced pre-trained models is currently wide-spread and understanding the associated safety and privacy risks is crucial. The poisoning attack appears to be highly effective in enhancing membership inference success under the examined setting. The experiments extend to various fine-tuning and inference schemes, which could be employed by the victim and cannot be influenced by the attacker. The attack is robust in the provided MIA improvement across these scenarios, highlighting the severity of the posed threat. Appendix C collecting negative results of failed attempts at constructing the attack is highly insightful and a refreshing sight given current publication practices.

Weaknesses

**Novelty** The paper claims in several places to introduce a “new” threat model and privacy attack, however, closely related [1] and virtually identical [2] settings have been proposed by other works. While the paper already briefly discusses [1], classifying it as concurrent work (available for slightly less than 2 months at submission time), it omits [2] (available for slightly more than 2 months at submission time). I believe that due to the large similarities in settings between these works, they warrant a longer discussion in similarities, differences, and concurrence in the paper; and the claim to unveiling a “new vulnerability” reassessed and tamed down in light of this discussion. **Strong assumptions** The paper reads currently as if the presented attack would be just a nice addition on top of any black-box MIA setting, and the presented game in Threat Model 2 makes it seem like it seamlessly integrates with the traditional MIA framework. However, I believe that this is misleading as the benefit from the introduced privacy backdoor attack is tied to assumptions that are stronger than those usually found in MIA literature. In MIA, online and offline attacks are usually distinguished [3]. In online MIA (weak setting), the attacker is assumed to be able to adjust the computationally heavy part of their attack (e.g., retrain their shadow models) when the challenger reveals a new target data point to them. In offline MIA (strong setting), the attacker prepares an attack once, before knowing specific target data points, and then this attack is employed (sometimes with minor adjustments without virtually any additional compute costs) once the challenger presents target data points. Also note that another standard assumption is that the challenger is allowed to continuously present new target data points to the attacker as long as they are samples from a distribution that is also available for the attacker for sampling. The paper currently does not introduce these standard elements and assumptions of MIA. Instead, the implicitly induced setting is in fact weaker than that of online MIA; the entirety of $D_{\text{target}}$ has to be known to the attacker when preparing the privacy backdoor. As such, if at MIA time the challenger presents a new target data point (which is allowed under usual MIA assumptions) the privacy backdoor has no “support” for it and adjusting the backdoor is not possible anymore, as the model on which the MIA is done is already released from the hands of the attacker and the fine-tuning has already happened. Further, the attack requires that the attacker possesses a dataset $D_{\text{aux}}$ that is disjoint from the target data points, which, while in many cases may be realistic, is a non-standard assumption in MIA once again. Standard MIA does not require that the dataset available to the attacker and the set of all potential target data points is disjoint. In summary, the paper makes several non-standard restrictive assumptions to the MIA setting, without discussing or motivating them explicitly and clearly; the assumptions are only stated in scattered places and not positioned in the context of MIA. **Only partially follows best practices in MIA result reporting** TPR@1%FPR is reporting at an order(s) of magnitude(s) higher FPR than suggested best reporting practices [3]. ROC-AUC score is included, while logarithmic full ROC curves are omitted, in stark contrast to suggested best practices [3]. As such, it is currently unclear if the attack provides as large benefits as currently perceived also at more relevant FPR regimes. **Concerns over the employed MIA** In the evaluation, the authors use the LiRA [3] attack with 16 shadow models. However, as it has been already shown in [3] and especially reinforced in [4], the standard LiRA attack is particularly weak for a low number of shadow models. While the version using global variance estimators is better in this regime, it is unclear which one was used for evaluation in this paper. As such, for this potentially weak attack the privacy backdoor provides a large benefit. However, it remains to be seen if the benefit is equally as large for stronger attacks, specifically tailored to perform well with just a low number of shadow models, such as [4] (available since 6th Dec 2023). **Weak justification of the different attack strategy choices** The paper currently employs two contrasting attack strategies for vision models and large language models. For vision models the loss of the target data points is increased in the backdooring phase (“maximization”), while for LLMs the target data points are encouraged to be heavily memorized (“minimization”). While both of these strategies are clear how they would encourage contrast at fine-tuning time between member and non-member target data points, the use of different strategies is currently weakly motivated, impacting the convincingness of the paper’s technical contribution. The differing choices could be better motivated by showing an experiment how the alternative strategies perform on the other domain, i.e., showing how minimization performs for vision, and how maximization performs for LLMs. In particular, the attacks on LLMs seem to be much stronger, which is currently unclear if this is due to the chosen attack strategy, the evaluation protocol and datasets, or due to some other factor. **Presentation, writing, and clarity** In several places there are certain inconsistencies, writing is not clear, or small errors in citation formatting or similar. This gives the overall impression that the paper was written hastily. In Threat model 2, point 3; to match with the generic setup of MIA, I assume that the idea what this point should stand for is (correct me if I am wrong) that the challenger randomly selects if it present a target data point from the target set which was also included in the fine-tuning set or one that was not. However, I think that the used notation currently does not reflect this: 1. “[i]f $c=\text{head}$, they randomly select a target data point $(x,y)$ from $D_{\text{target}}$” — this $(x,y)$ could still be both in the training set or not in it, this does not tell us anything about that; 2. “if $c=$tail, a target data point $(x,y)$ is randomly sampled from $(D_{\text{target}} \setminus D_{\text{train}})$” — this is confusing, as this would imply that $D_{\text{target}}$ is a superset of $D_{\text{train}}$, which is not only not stated anywhere nor followed in the experimental section, but would also mean another highly unrealistic assumption. I believe, sampling from $D_{\text{target}} \cap D_{\text{train}}$ when the coin is head, and sampling from $D_{\text{target}} \setminus (D_{\text{target}} \cap D_{\text{train}})$ when the coin is tail would be a correct notation/presentation. Section 3.2. is very confusing at the moment, as it gives a clear motivation for the maximization attack, but then presents the exact opposite of this idea for LLMs, solely because ‘maximization did not work’. In conjunction with the experiment justifying this choice and an adjustment in writing would make the presentation of the attack more compelling. While, as I already elaborated above, the assumptions made by the attack are rather strong, they could be justified by presenting a clear real-world example for the attack (but still have to be explicitly compared to the standard assumptions of MIA). While there is an attempt on this in l159-l162, I suggest to elaborate on this and present it more convincingly, given that the assumptions made are non-standard for MIA. I struggle to understand the paragraph from l267 to l271. I especially do not get the causal link between observed memorization and what is meant by similar format and similar types of personal information. What the supposed link between the results on Simple PII and MIMIC-IV is, is also unclear. I do not understand why these two results are compared, as to me, the outlier seems to be the result on Simple PII, with a base attack success of 0.242 TPR@1%FPR, compared to an order of magnitude lower TPRs on both ai4Privacy and MIMIC-IV. Further, if the statement is supposed to be that PII is memorized better by the LLMs than images by the vision models, then for this the experiment setup in my view is unfit, as the MIA scores on the unattacked models are comparable in each case (each time two datasets produce around 0.0X TPRs and once 0.1X-0.2X TPR), and in case of the attacked models, as the attacks are different, we cannot know if the difference stems from the data domain or the attack itself. As already elaborated in its own point, the setting of MIA and how the presented attack relates to it have to be presented clearer, as currently in parts it seems like the attack enables a more powerful setting; being an additive improvement over any MIA scenario. Citep and citet are mixed up in certain places, e.g., l68 or l115. **References** [1] S Feng and F Tramèr, Privacy Backdoors: Stealing Data with Corrupted Pretrained Models. http://www.arxiv.org/abs/2404.00473. [2] R Liu et al., PreCurious: How Innocent Pre-Trained Language Models Turn into Privacy Traps. https://arxiv.org/abs/2403.09562. [3] N Carlini et al., Membership Inference Attacks From First Principles. https://arxiv.org/abs/2112.03570. [4] S Zarifzadeh et al., Low-Cost High-Power Membership Inference by Boosting Relativity. https://arxiv.org/abs/2312.03262v3.

Questions

I find the paragraph “Results on Non-target Data Points” rather interesting, however I wonder how much this observation is related to the finding of [5]? What if $D_{\text{aux}}$ and $D_{\text{train}}$ are very different? It should be enough if $D_{\text{aux}}$ was only similar to the pre-training dataset, no? Would be interesting to see if large differences between the auxiliary dataset and the fine-tuning dataset would impact the attack success. Do the authors have any hypothesis why OPT-350M is an outlier in the model size trend in Figure 1? **References** [5] A Panda et al., Teach LLMs to Phish: Stealing Private Information from Language Models. https://arxiv.org/abs/2403.00871.

Rating

4

Confidence

4

Soundness

1

Presentation

2

Contribution

2

Limitations

Limitations are not prominently and explicitly discussed, only recognized in the checklist. In my view, the strong assumptions made in the threat model have to be discussed in the main paper. Note also other weaknesses pointed out in my review.

Summary

This paper proposed a new privacy attack for foundational models like CLIP and large language models (LLMs). The attack's key idea is to ''poison'' the target data (e.g., maximize loss) point into the pretrained models so that the victim's finetuned models uploaded on the open-source platform can reveal what target data points have been used for finetuning. In the end, the evaluation on both vision and language foundational models validate the attack effectiveness.

Strengths

+ Important research topic: data privacy for foundational models + New attack setting and method

Weaknesses

My main concerns are about the threat model and the evaluation of LLMs. - Although the paper studies a new threat setting, existing model ecosystem may not work like the described manner. Specifically, the paper assumes the adversary can firstly finetune (i.e., poison) the publicly available pretrained model $F_{pre}$ to $F_{adv}$, and release the F_adv to the platform. Then the victim downloads and finetunes the $F_{adv}$ with D_train to $F_{adv, ft}$, and releases $F_{adv, ft}$ to the platform so the adversary can infer membership privacy from $F_{adv, ft}$. But why would the victim finetunes $F_{adv}$ instead of $F_{pre}$? Typically the victim would choose the $F_{pre}$ for finetuning, just like the authors choose the CLIP for evaluation instead of a random finetuned CLIP on the Hugging Face. - Besides, the attack goal of maintaining a comparable level of performance on downstream tasks does not persuade the victim to use $F_{adv}$ instead of $F_{pre}$. Let's assume $F_{pre}$ and $F_{adv}$ have similar performance. As $F_{pre}$ is shared by organizations (e.g., Meta, OpenAI, etc.) that can pay the training cost, the downloads and likes of $F_{pre}$ is certainly high as what we have witnessed in the era of LLMs, and $F_{adv}$ can be just one of hundreds of finetuned $F_{pre}$, why would the victim prefer $F_{adv}$, with potentially not much downloads and likes? - The impact of finetuning to the LLM performance is also questionable. The results reported in Table 2 is lower validation loss after finetuning, but the loss cannot tell the model performance (i.e., low loss is not necessarily better). I would suggest the authors to provide more concrete LLM evaluation. - The evaluation on large language models is not thorough. The largest evaluated LLMs in this paper is GPT-Neo-2.7B while widely studied LLMs are above 7B such as LLaMA, Mistral, etc.

Questions

See my comments above. I also have a minor question about the target data points owned by the adversary. The authors justify with data of interest like proprietary data, but under what circumstances can the victim and the adversary jointly possess some common target data? If it is the victim that steals the proprietary data, what motivates the victim to finetune with the stolen target data and release $F_{adv,ft}$ to the public, especially when the victim is aware of this attack (after publication) and the potentially poisoned $F_{adv}$?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

None

Thank you

Thank you to the authors for their rebuttal. I appreciate the additional interesting experiments and the acknowledgement of some of my concerns. However, my main concerns are very much fundamental to how the paper is currently presented: **C1:** This backdoor operates with different assumptions than usual MIA---while I agree that this is fine, and the presented threat is relevant, this has to be openly and transparently presented in the paper. Nobody should get the impression that this is a plug-and-play sort of addition on top of any MIA. **C2:** The novelty claim certainly has to be toned down in the face of the presented related work. Once again, I do think that this work is valuable in itself, but I would appreciate an accurate and transparent positioning of the contributions in the related work. I also understand that this work may have been in resubmission cycles for longer, however, as for the acceptance at the conference, the paper has to be judged with respect to the time when it was submitted, as this is also the relative narrative that will later be represented at the conference. As the authors acknowledge, and seem to be in line with me on my main concerns, in the hope that they will adjust their presentation in the paper, I will slightly raise my score. However, ultimately, I can only hope that the changes will indeed be implemented, without which I would still vote for clear reject.

Thanks for the response

Thank you for your rebuttal and the additional evaluation. However, I still have concerns over the strong assumptions as noted by other reviewers. Take the third point in your rebuttal as an example, I don't believe the victim (patient or patient group) will really publish a poisoned model, and bet the hospital to finetune and publish their model to infer whether their privacy is leaked. The proposed attack is far more complicated than membership inference attack. For the first point, I agree the victim might choose the disguised model but they will probably not share their model again especially if there are privacy training data (we already know the model memorization). I searched "uncencored" and I noticed that these models are mainly for pretrained models (gemma, llama, etc.) but not the homemade ones. Given my personal experience playing with the open-source LLMs on Hugging Face, I'm not fully convinced by the justification and I still doubt the potential influence of this two-phase poisoning attack. Therefore, I would like to keep my scores.