We reveal the theoretical foundations of techniques for editing large language models, and present new methods which can do so without requiring retraining. Our theoretical insights show that a single metric (a measure of the intrinsic dimension of the model's features) can be used to assess a model's editability and reveals its previously unrecognised susceptibility to malicious stealth attacks. This metric is fundamental to predicting the success of a variety of editing approaches, and reveals new bridges between disparate families of editing methods. We collectively refer to these as stealth editing methods, because they directly update a model's weights to specify its response to specific known hallucinating prompts without affecting other model behaviour. By carefully applying our theoretical insights, we are able to introduce a new jet-pack network block which is optimised for highly selective model editing, uses only standard network operations, and can be inserted into existing networks. We also reveal the vulnerability of language models to stealth attacks: a small change to a model's weights which fixes its response to a single attacker-chosen prompt. Stealth attacks are computationally simple, do not require access to or knowledge of the model's training data, and therefore represent a potent yet previously unrecognised threat to redistributed foundation models. Extensive experimental results illustrate and support our methods and their theoretical underpinnings. Demos and source code are available at https://github.com/qinghua-zhou/stealth-edits.
Paper
Similar papers
Peer review
Summary
Introduces a new computationally efficient technique to edit facts in a LM. The proposed technique has nice theoretical properties that ensures the selectivity of the edits. The paper also discusses how the ability to edit a specific LM can be measured by its intrinsic dimension, which can be approximated with a data-driven approach.
Strengths
* The paper is nicely presented with crisp mathematical formulations and clear explanations. * Novel contribution in the intrinsic dimension based approach to measure the ability to edit a LM. * I liked the discussion on the selectivity of the edits and how the development of memory editing techniques leave the LMs vulnerable to *stealth attacks*.
Weaknesses
The only weakness I see in this paper is the lack of proper evaluations. Please see the detailed comments below. I am happy to revise my rating if the concerns are addressed.
Questions
* **Evaluation:** * **Generalizability:** I am concerned that the edits will not generalize well across different paraphrases of the prompt $p$. For example if you edit a fact `The Eiffel Tower is in -> Rome` and then query the edited model with the prompt `A great tourist attraction, the Eiffel Tower, which is located in the city of`, does the LM generate `Rome`? This paper works on the last token of the prompt and *hope* that the LMs semantic similarity of the latents would be high (same as GRACE). But this is not guaranteed and needs to be properly evaluated to claim generalizability. * **Performance comparison across different methods:** The paper does not compare the proposed method with other methods, such as ROME and GRACE. It would be nice to see how the proposed method compares with other methods in terms of the quality of the edits measured in *efficacy*, *generalizability* and also *specificity*. Although I found the discussions in Theorem 2, 3 and the corresponding figures in section 5 are convincing that the method will be highly specific to the edits. * **Scalability:** What is the maximum number of edits that can be made to a LM using this method? I'd assume, without jet-packs, you theoratically cannot edit more than $d$ facts, $d$ being the actual dimension of the latent space. Figure 1 suggests that the intrinsic dimension of the LM is even less. As a result, the LM might break even before $d$ edits. I think you will face a similar situation with the jet-packs as well, since with increasing number of edits will increase the chances that unintended detectors will get activated. What is your take on this? * **Detector Neuron:** * **Selecting the detector row in $W_1$:** For a single edit, why did you need to select the row with the smallest norm? Is the assumption that the row with the smallest norm has the least impact on the channel activations anyways and changing it will ensure minimal damage? * I also found it difficult to follow the discussion on Appendix B.1.3 as it introduces too many new variables without proper explanations of what they are. For example $\zeta$ was introduced in Eqn 12 but it is not clear what it exactly is. * **Difference with ROME**: The proposed method repurposes a specific column of $W_2$ and corresponding row in $W_1$ to rewrite one single fact. In contrast, my intuition on ROME is that it uses a rank-one update to distribute the load of the edit across *all* the columns in the $W_2$. Is this correct assumption? If so, then I'd be curious to see a comparison of the proposed approach vs ROME in terms of the quality and the scalability of the edits. **EDIT:** Bumped the rating to 6.
Rating
6
Confidence
4
Soundness
3
Presentation
4
Contribution
2
Limitations
* Was the efficacy measured on the successful prediction of only the first token? If so, it should be mentioned in the limitations.
Summary
This paper studies the problem of, when given a particular prompt, whether it is possible to surgically and efficiently edit a model’s parameters to produce a certain response in a way that does not otherwise change the behavior of the model. The paper provides an efficient technique for this in the form of both an in-place edit of a small amount of a model’s parameters, or alternatively as an additional block, “jet-pack”, inserted to encode the edits using the same framework. The key intuition of the technique comes from the perspective that the FFN block is a type of key-value memory (Geva et al. 2020), and we can rewrite a key that is not important to the overall model (smallest L1 norm, as used in this paper) to one that explicitly triggers on the desired input, and we can rewrite the corresponding value to a value that maximizes the likelihood of the desired output. The authors provide theoretical foundations that this change does not impact the general behavior of the model, as well as empirical evaluation of the technique over two factuality datasets to demonstrate the success rate of their technique, that it does not disturb the general ability of the model (on metrics like perplexity), and that the true false positive rate (FPR) is consistent with the FPR predicted by the theory.
Strengths
1. This is a very well written paper that clearly lays out the problem being studied, introduces the techniques, theoretical foundations, and empirically verifies the theory and edit success. 1. The contribution is significant and novel: to edit the output of a model so precisely by purely adjusting two rows/columns in a MLP transformation is an excellent finding. The framework developed here is important and will be of interest for many in the field. It may spark other research works that can apply or extend this technique to address other open problems in LLMs (e.g. perhaps continual learning, model merging) . 1. Within the specific setting that the work studies, the paper is rigorous in its analysis and evaluation: modern LLMs of different architectures (Llama, GPT-J, Mamba) are evaluated, where to add the edit within the model is thoroughly ablated and plotted, methods and experiments cover a variety of scenarios: in-place and jet-pack editing, editing under certain contexts: corrupted prompt, unexpected context.
Weaknesses
1. The motivation of the paper focuses on stealth, but this dimension could be better defined. The paper specifically studies "stealth" under the definition that 1) the edit is directly to the parameters and is somehow small 2) given a fixed set of prompts, only the edited prompts should show altered behavior. However, stealth could also be taken to mean: whether it is possible to detect in the weights that the model maliciously altered -- which this paper does not study. 1. It is unclear to what degree the jet-pack approach is "stealth" as it alters the parameters / layer stack used in the model. 1. Given the motivation of fixing hallucinations, the limitation of the approach is somewhat hidden by the set-up and assumptions. This may be perfectly fine if these limitations were clearly stated in the framing of the paper, however it does not seem to be and the results could be misinterpreted. Specifically: a. The mechanism of the edit is designed such that given a particular prompt, the model outputs a particular output. It seems like such an approach may only be effective under very specific prompts, which is not useful in the case of addressing hallucinations, which this paper uses as motivating example. If this is not true, it would be excellent to add additional empirical evidence that these edits can adjust concepts/facts rather than just the output given a specific prompt. b. If the edit is indeed specific and results replacing an entire row/column in the MLP, this would be limiting for real use cases. c. The edits are only done over the pre-trained models, and only for very short factual prompts. Perplexity ratio is only computed for 50 tokens, and over the datasets considered in this work. It is possible that the model may have compartmentalized Wikipedia / factual knowledge of this type in certain parts of the MLP, allowing for edits to be done in other parts of the MLP corresponding to other abilities. To demonstrate true "stealth" edit/attack that would deceive a standard user, would require evaluation that other capabilities are not impacted as well. Here, evaluation of perplexity over a general multi-domain dataset such as The Pile, or reporting performance a wide array of few-shot tasks would be much more convincing empirical evidence. d. In Section C.2, it is mentioned that edit success is defined as 1) the first token matches the first token of the expected output, and 2) the target output is contained within the generated text. This is different than Exact Match metrics typically used for close-book QA datasets like Natural Questions, TriviaQA, as it could allow for some degree of degeneration. Providing samples of outputs from the model could alleviate this concern. Minor typos: - Line 338
Questions
1. In some ways the techniques here are similar to other parameter-efficient fine-tuning approaches. e.g. with LoRA [1] a low rank adapter is trained, and during inference its weights are added to the base model's weights. This is somewhat similar to the MLP key-value row/column trained here and replaced in the base model's weights. One could imagine training a LoRA adapter to address hallucinations while otherwise maintaining general performance. To what degree is this a work a stealth edit and such a standard approach is not? Is this a reasonable naive baseline to try? 1. What is the scaling like for the number of edits a model is able to take before degenerating? [1] https://arxiv.org/abs/2106.09685
Rating
6
Confidence
3
Soundness
3
Presentation
4
Contribution
3
Limitations
See weaknesses response above. The authors do provide a limitation section that covers a reasonable degree of limitations otherwise.
Summary
This paper proposes a new algorithm and studies a family of methods it refers to as *stealth edits*, which modify a large language model to selectively correct a set of known hallucinations without otherwise affecting the responses. It also proposes *intrinsic dimension*, a pairwise separability-based metric to determine the ease of editing any given network block of an LLM. It provides theoretical guarantees for the selectivity of edits and demonstrates that the probability of edit activation decreases exponentially with increasing intrinsic dimensionality. It proposes two versions of the algorithm - one for in-place editing a network block to correct a given prompt and target output, and another for inserting an additional *jet-pack module* in the network to correct a series of hallucinations. Both of these algorithms optimize the rows/columns of the projection layers of the block with gradient descent to produce the expected target output(s). The *jet-pack block* controls the selectivity of an edit using a modified *RMSNorm* normalization layer that is re-centered to the mean normalized features of a set of Wikipedia prompts to maximize their intrinsic dimensionality. It also highlights that these methods expose a potential vulnerability in LLMs and can potentially be used by malicious actors to perform a *stealth attack* to produce targeted responses for specific trigger inputs of their choice. Specifically, it discusses two variants of attacks - a *corrupted prompt attack* in which the attacker inputs slightly different versions of the trigger prompt and an *unexpected context attack* in which the attacker mentions the target response for only a given prompt that is followed by "context" sentence. In both of these cases, the attacks are difficult to monitor or track through automated tests. It validates the effectiveness of their proposed algorithms by reporting the performance on MCF and ZsRE datasets using the metrics: edit success rate, perplexity ratio of attacked model and the original model, and actual/theoretical worse-case of detector false positive rates. It demonstrates a lower edit success rate and higher false positive rates in the earlier layers of the model in the case of stealth editing. It also shows a near-perfect detector false positive rate in the case of *jet-pack edits* and observes similar trends of selectivity in both variants of the stealth attack. The proposed algorithm's formulation unifies seemingly different related methods GRACE and Transformer-Patcher. It also notes that normalization functions in LLMs can help enable selective edits while also being a point of potential vulnerability in these models.
Strengths
- This paper introduces a novel formulation and theoretical framework to analyze methods addressing the important problem of selectively correcting hallucinations. The formulation generalizes to a wide variety of architectures and selective editing algorithms. - The metric *intrinsic dimensionality* seems particularly useful in understanding both the efficacy of selective edits and identifying potential vulnerabilities in an LLM. - The experiments reveal important flaws in prior work such as the sub-optimal choices of applying edits at the end of the model and find them to be much more effective near the middle layers.
Weaknesses
- The gap between theoretical expectations and empirical results w.r.t false positive rates in the layers towards the end isn't sufficiently addressed. The looser near-zero worst-case bounds for false-positive in this case also aren't that helpful. - Some of the important observable trends in the figures of the experiments section aren't justified. Specifically, the trends for the edit success rate metric in Figures 3, 4, and 5 seem to vary widely across all layers among different model families, while it is relatively more consistent for the other metrics.
Questions
1. Could you provide an intuitive or theoretical justification for why you observe the false positive rate slightly trend up towards the final layers of the model and find the optimal edit insertion location to be about half-way?
Rating
7
Confidence
4
Soundness
4
Presentation
3
Contribution
4
Limitations
The authors have generally addressed the limitations and potential societal impacts of their proposed algorithm. A few suggestions to improve: - In section [7], according to the guidelines they should provide further details on which error bars are used and state the corresponding confidence intervals in their experiments. - While the negative social impacts of stealth attacks are discussed in different sections of the paper, it should ideally be discussed again in the Discussion section, especially the practicality or limitations of such attacks in practice.
Summary
This paper focuses on stealth editing in large language models, presenting methodologies for making targeted, subtle changes to these models without retraining. The techniques, called "stealth edits," aim to correct specific issues like factual inaccuracies by directly updating the model's weights. The research also reveals that all modern language models are susceptible to stealth attacks, which are targeted and hard to detect, involving minimal changes to a model's weights to alter its response to a specific prompt. Experimental results demonstrate high success rates for both edits and attacks while maintaining low false positive rates. The findings have significant implications for AI system security and reliability, indicating that even extensively trained models can be vulnerable to targeted manipulation. The authors conclude by highlighting the broader impact of their work and the need for further research in this area.
Strengths
1. This work analyzes how the intrinsic dimension of data is a crucial factor: the higher the dimension the higher the probability that a given edit is successful. The findings here are novel and intriguing, which could guide future research in the field. It also combines the edit and attack together. 2. The experiments are sufficient. They considered different architecture and different model sizes.
Weaknesses
1. The paper's organization can be improved to make it easier to understand. Maybe the space is limited and I think some information can be moved from the Appendix to the main paper.
Questions
1. In the text around Line 187-190, it says it would add a new row to W1 and a column to W2, and in Algorithm 2 Step 9， it still uses the W1 as the updated model. I'm a bit confused about whether the W1 here has extra rows. I didn't get this jet-pack editing here.
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
N/A
Thanks for your clarification. I have no remaining concerns and will keep the score.
I thank the authors for their detailed response, which helped me to understand the motivation of the paper better. The authors have promised to include some further evaluations to address the concerns raised in the reviews. I am happy to revise my rating to a 6. I am looking forward to seeing the final version of the paper. Congratulations on your fine work! Why not an increased rating? - Although this paper makes novel contributions I don't see many practical applications as the edits lack generalization, which the authors have clarified that they were not aiming for. In my personal opinion (not meant to be a critique of the paper and the authors views), I think the authors are over-complicating the notion of generalization to some extent. Based on their example in the global response - the answer to *'Who is the Prime Minister of the UK?'* is a fact retrieval task for the LMs, the edit to which should indeed generalize well across different paraphrases of the prompt. For *'The year is 2015. Who is the Prime Minister of the UK?'* where the relation (according to ROME) can be considered as (s = "UK", r = "Prime Minister in year 2015") or (s = "Year 2015", r = "Prime Minister of UK"). And for the other example, *'Let's write a story about a world where everyone is called Blarg. .. '*, is about following cues from the in context prompt, which I think is a different task altogether.
Thank you for responding to our comments, and for increasing the score. We would like to take this opportunity to clarify some remaining points. Regarding the question of generalising edits, we undoubtedly see the potential use for edits which really can reliably generalise. Unfortunately, current editing methods do not seem to be able to promise this. To take an example, we tried editing the phrase 'Who is the Prime Minister of the UK?' in GPT-J-6B using ROME, to produce the output 'Oliver' (for example). Afterwards, the edited model responded with 'Oliver' to all three of the example prompts, even when this is not an appropriate response because of the wildly different contexts. Of course, changing models or parameters may produce different results, but this demonstrates the lack of control and robustness these techniques offer with regards to generalisation. The re-formulations of the edit proposed by the reviewer demonstrate the need for nuanced extra knowledge about the query in order to implement a generalising edit, not required for our edits. Moreover, for editing a general purpose language model (as we aim to do), it is difficult to filter prompts at run time which have a different context as being 'out of scope'. This is especially true in the common setting where models have user-modifiable instructions contextualising prompts. As our experiments in the additional PDF attached to the global response confirm, even a single jet-pack (many can be added to a single model) can support exponentially many edits (in the feature dimension) without degrading the performance of the model. This provides an opportunity for the designer to fine-tune the applicability of their edits in a controlled fashion, accounting for the fact that the truthfullness of a response depends on the context in which a question is asked.
Rebuttal response
Thanks for answering the concerns and questions. I will keep the score.
Response
"For each, we calculate the perplexity ratio between the modified and original model on text generated by the models up to 50 tokens. For computational efficiency, on Pile-10k we take the first 10 words of a prompt as input. These results show that little perplexity is produced by setting a neuron to zero." Does this mean that perplexity in this work is calculated over model generations? E.g. text is generated by the model up to 50 tokens, then the model is used itself to score the text? Rather than having the model score the original document in the eval dataset?
To clarify, the perplexity ratio metric aims to measure the extent to which the edits/attacks change the original behaviour of the model. This is in accordance with our overall aim of making targetted edits to a model, without impacting the model's broader baseline performance on a wide variety of tasks. For this reason, we measure the perplexity of the edited model to the outputs of the original model. We take the original model's perplexity to its own outputs as a baseline, which we use to produce the ratio. A perplexity ratio of 1 can therefore be used to indicate that the behaviour of the model has not changed for a given prompt, in combination with the range of complementary metrics given in Section 5. In the setting of this work, we feel that reporting raw model perplexities would not measure the ability of our edits to avoid disrupting the model. In particular, we use only public datasets and pre-trained models which have had their perplexity profiles thoroughly analysed elsewhere. Ultimately, it is therefore not the baseline performance of the model which is our focus here, but the ability of our edits to preserve it.
Updated rating
Thank you for your additional results and clarifications. In general I believe this work contains novel and valuable techniques that should be published, and that the right reading of this work is as a foundational one: it would set a valuable start for other works to potentially build on and improve. It seems that the authors are open to revising the framing as mentioned to be clearer and more specific about the aim of 'specific hallucinations' or 'specific edits'. I think being clearer upfront about the limitations of the work was one of my main concerns, as given the current framing it is easy to read the abstract and misconstrue the paper as a silver bullet for manipulating model parameters to fix hallucinations -- which it is not. I might argue that "hallucination" in general is an overloaded word that may imply some degree of generalization, that the authors might want to avoid in the abstract. Given the author's proposed revisions to the framing, I will raise my score to a 6. Why not higher? I think the author's extra experiments on ThePile-10k, and example outputs does add a little bit more confidence that the work does not adversely affect the model's abilities over various domains. However, when the authors define "stealth" as "stealth in the sense that the architecture is unchanged and/or performance on a large unknown validation set is unchanged," I think it is fair to empirically require that for most ways that a user might use or evaluate a checkpoint, that the performance / quality of the model is unchanged. For most practitioners this would mean performance on a wide set of few-shot tasks, or chat performance like MT-Bench, or at the very least, basic perplexity evaluation over documents, like the full ThePile-10k. The "stealth" part should from the user's perspective, rather than the author's perspective for the purposes of this research paper. For example, while it is reasonable for the paper to compare perplexity over the baseline model's outputs to see that the edited model still matches the baseline model -- it is much more likely for a user who is trying to verify that a model is the same to rerun normal perplexity evaluation to verify that it is reasonable, or even *identical*, in the strictest case, on the same hardware. This sort of comparison is a common sanity check when doing checkpoint conversion between platforms, or comparing hardware. The short sequence lengths considered in this work, also prevents us from understanding if the quality of the model would be affected at all for longer inputs / outputs.
Thank you for your feedback and for updating the rating. We stress that it is difficult for us to run these additional experiments requested by the reviewer less than 24 hours before the end of the discussion period. Despite this, we have performed some further experiments which are described below, using gpt-j-6b float16 and the standard LM Evaluation Harness. Overall, these results demonstrate that our edits do not adversely affect the model's abilities over various domains, as requested by the reviewer. We will be happy to include these results into the appendix of the revised version. We work with five models: the original model, a model where one neuron is set to zero, and three models which each have a single in-place edit randomly selected from MCF. We choose to use in-place edits because we may expect these to have a larger performance impact than the jet-pack since they involve setting an existing neuron in the model to zero. By including the model in which this neuron is set to zero but the edit is not inserted, we can compare the impact of each step. 1. We measure perplexity of the models on the original document of the full Pile-10k dataset: the original model, the zero neuron model, and all three edited models had the exact same word and byte perplexity to at least four decimal places. This is supported by the fact that the FPR rates, as shown in Table 1 (of the PDF in the global response) for Pile-10k are low. 2. We measure performance scores on the few-shot tasks of the TinyBenchmarks (including 25-shot tinyARC, 5-shot GSM8K, 10-shot tinyHellaswag, 5-shot tinyWinoGrande). TinyHellaswag and tinyWinoGrande returned same accuracy (to at least four decimal places) for all five models, tinyGSM8K returned same Exact-Match score for both flexible-extract and strict-match filters for all five models (to the same precision), tinyARC returned 48.90% ACC for the original model, and 48.44% ACC for all edited models. 3. We measure performance scores via 5-shot on MMLU-Pro: the benchmark returned $0.1256\pm 0.003$ for the original model and $0.126\pm 0.003$ for the zero-neuron model and the three edited models with in-place edits. Note that for tinyARC and MMLU-Pro the zero-neuron model and the in-placed edited models shows the same accuracies. Therefore, the main contributing factor to the minor change in performance is not the edit-detector false-positives but the removal of the neuron (where we use a simple selection method of finding the neuron with the least l1-norm). Overall, our methods showed minimal impact on the original behaviour of model. We hope this extra experimental evidence resolves the reviewer's outstanding questions.
Decision
Accept (poster)