Deployed language models decay over time due to shifting inputs, changing user needs, or emergent world-knowledge gaps. When such problems are identified, we want to make targeted edits while avoiding expensive retraining. However, current model editors, which modify such behaviors of pre-trained models, degrade model performance quickly across multiple, sequential edits. We propose GRACE, a lifelong model editing method, which implements spot-fixes on streaming errors of a deployed model, ensuring minimal impact on unrelated inputs. GRACE writes new mappings into a pre-trained model's latent space, creating a discrete, local codebook of edits without altering model weights. This is the first method enabling thousands of sequential edits using only streaming errors. Our experiments on T5, BERT, and GPT models show GRACE's state-of-the-art performance in making and retaining edits, while generalizing to unseen inputs. Our code is available at https://www.github.com/thartvigsen/grace}.
Paper
Similar papers
Peer review
Summary
The paper suggests a new method for continual model editing. To deploy a parameter-efficient model editing remedy, the authors introduce an adapter for a target layer, keeping the pre-trained model weights intact yet revising the model prediction correctly. The suggested adapter contains a codebook - key-value pairs and corresponding distance coefficient \epsilon. By training them via a fine-tuning mechanism, they can edit the original prediction accordingly.
Strengths
The proposed method introduces the interesting idea of continual model editing. The basic idea of codebook updating and introducing new entries is simple yet reasonable. And suggested method outperforms baselines by a significant margin. And the ablation study done in the supplementary file is also good.
Weaknesses
The most critical limitation is that the choice of editing layer is essential, but it is achieved via heuristic search. Recently released deep neural network architectures provide a depth (layerwise/blockwise) variety according to the scale, and heuristic search to find the best layer to edit is a nontrivial problem. Unfortunately, as shown in the analyses and the limitation section of the main paper, this limitation is the main obstacle to the practical usability of the suggested model. Also, most baselines are not targeted to the continual learning setting, so the benefit of the suggested method for continual model editing is a bit difficult to measure fairly. Simple combinations of existing (and recent) continual learning methods with model editing baselines can be better competitors to reveal the merits of GRACE more intuitively. And it will be also great to provide not only model performance like TRR, ERR, or ARR, and the change of additional memory overhead over long model edit sequences.
Questions
N/A
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Please see the weakness.
Gentle Reminder by AC
Dear Reviewer, Could you carefully read the authors' rebuttal as well as the others' reviews and their rebuttal, and make responses at your earliest convenience? The deadline for the discussion phase is fast approaching, which is due Aug 21st 1pm EDT, so your quick responses will be greatly appreciated. Best, AC
Summary
The authors propose GRACE, an approach to model editing that is tailored to many-edit scenarios. GRACE augments a hidden layer of a neural network with a key-value memory that overrides the output representations of the layer when the input representation falls near the stored key. The method is simple and computationally efficient, and compares favorably with existing editors. Closer analysis shows that GRACE ultimately provides a tradeoff between forward generalization of past edits and retention of pre-existing knowledge.
Strengths
The authors study a clearly-defined and very relevant problem, sequential model editing, which extends the setting of applying one or a small number of model edits to hundreds or thousands, which is much more realistic. The method they describe is simple and novel, and does not include the requirements for training data that some existing methods have. GRACE also leverages the internal representations of the pre-trained model itself to (in theory) enable generalization of the edit information to related inputs (however empirical validation of this capability is mixed). The analysis of the method's memorization and generalization in 4.2.2 is welcome. While the final results show that there is still a strong tradeoff between the ability to generalize edits forward (i.e., to future queries that ask about similar information to those queries that were already edited) and the retention of pre-existing information, I think the paper is a useful exploration of a straightforward and intuitive approach to sequential editing. The authors also include a clear discussion of the limitations of their work, which is welcome.
Weaknesses
The main weakness I see in the paper is the fact that GRACE might do little more than memorize the edits it is exposed to. The analysis in 4.2.2 (which is very welcome!) suggests that GRACE essentially has two regimes, one where we essentially memorize past edits and retain all past knowledge, but are not able to generalize edit information to future, related inputs (editing layers 4 or 6 in the $\epsilon=1.0$ plot) or where we are able to generalize edits to future, related edits, but suffer a (potentially dealbreaking) penalty in terms of the information we retain from the original task. In my view, the tradeoff as shown in the paper is not viable for real-world use cases. However, I still think the paper makes a useful and interesting contribution to model editing/sequential model editing. One other point about related work: I think L79 is wrong or misleading; MEND and ROME are only loosely based on PEFT. MEND only requires a single fine-tuning step, and neither ROME nor MEND is more prone to overfit than regular fine-tuning to my knowledge. Whether or not PEFT overfits more than regular fine-tuning isn’t relevant here, IMO.
Questions
L156: Does this expansion process ignore the fact that we can have multiple word senses? For example, if we have an edit for “the robber of the ___” and “the house by the river ___”, even though the labels for both edits might be “bank”, it seems like we’d want to represent these separately. L257: “unsurprisingly, MEND underperforms since it forecast edits for new, unobserved inputs” What does this mean? L87: what privileged information does MEMIT use? L139: Can we not have the case where the hidden state is not within the deferral radius of the closest GRACE key, but the hidden state is within the deferral radius for the n-th closest key? Considering only the closest key seems like it ignores this possibility. What, in the authors' view, is the primary cause of the difficult tradeoff between retention of old information and forward generalization of edits? Currently, GRACE either requires a problematic destruction of pre-trained knowledge or a failure to apply edits to future, related edits.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors include a clear discussion of relevant limitations of their work.
Gentle Reminder by AC
Dear Reviewer, Could you carefully read the authors' rebuttal as well as the others' reviews and their rebuttal, and make responses at your earliest convenience? The deadline for the discussion phase is fast approaching, which is due Aug 21st 1pm EDT, so your quick responses will be greatly appreciated. Best, AC
Summary
The paper presents a new method for sequential model editing (GRACE). The approach works by modifying the output of a pretrained model's given layer $l$ through the use of an external key : value storage. Whenever an edit is required on input $x$, the model stores the input representation $h^{l-1}$ as key, and learns a value $v$ via gradient descent which yields the edited output $y_{edited}$. In order to enable generalization, the authors propose to map all representations within an adjustable (edit-specific) $\epsilon$ to the stored value $v$. Crucially, to enable sequential editing, the authors propose to update the codebook (Alg. 1) the following way : whenever a new edit overlaps with the $\epsilon$-ball of a previous edit, this $\epsilon$ is either shrunk to reduce the overlap between edits, or expanded if the two edits share the same desired output. The authors evaluate their method according to three metrics, which look at whether the sequentially performed edits are memorized, and the model's ability to preserve its original input on non edited points. From the results, it is shown that the proposed method performs across several benchmarks. Finally, the authors perform an model analysis, looking at the impact of $\epsilon$ and the choice of layer to add the GRACE mechanism.
Strengths
Originality : - The proposed is, to the best of my knowledge, a novel approach of an external discrete memory for sequential model editing - Grace seems like a solution which could actually be deployed in realistic edit scenarios Quality / Clarity : - The paper is well written and easy to follow. - The intuition behind how the method works is well-built, and the model analysis section is interesting - The experimental section is overall well-designed
Weaknesses
1. Given that the choice of layer and value for $\epsilon$ seems to highly impact the behavior of the deployed system, it's unclear how practitioners would choose this value a priori. Given that this deployed in an online setting, standard cross-validation cannot be used directly. 2. The experimental section is somewhat limited, as the authors only look at NLP tasks. That being said, given that the authors evaluate different model scales, this is more a suggestion on how to make the paper more complete than a hard weakness.
Questions
1. Could the authors please explain how GRACE layer deals with multi-token inputs with autoregressive models ? What is implied by "replace only the final token of an input prompt" ? 2. How are rephrasings generated for the results shown in Table 2 ?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
limitations properly addressed.
Summary
The authors propose a method to editing a model’s behavior, making corrections or specifying the desired output for a particular input. Such a method is broadly applicable, but is especially relevant in the context of large pre-trained models, which are costly to fine-tune or re-train, and whose data is often not publicly available. Specifically, the authors propose General Retrieval Adaptors for Continual Editing (GRACE), which store activations (keys) for past edits into a codebook; if a particular input is close enough to one of these stored keys, then the output of the layer is overridden by the key’s corresponding value. A deferral radius for each key allows grouping of similar edits and provides some generalization behavior. For experiments, the authors compare with a number of relevant model editing baselines, editing large language models (LLMs) hundreds to thousands of times, which is significantly more than prior evaluations. Empirical comparisons look promising across a variety of editing metrics.
Strengths
S1. Relevance: This paper tackles a topic that is especially important given recent developments. Large models trained on massive data (e.g. language, vision) are seeing increased interest and more exposure to the general public with the releases of chat bots/search, image generators, etc. This is largely due to impressive results, but there have also been a concerning number of wrong, offensive, or misleading examples as well. As these models continue to be used by the public, some way to apply targeted patches is essential. S2. Assumptions: GRACE has relatively simple and realistic model assumptions. In particular, unlike some prior work, it does not require access to the pre-training data; this is important, as such data is often proprietary and massive in scale, as well as a potential data privacy risk. S3. Experiments: \ a) Realistic settings: While past methods tend to evaluate success of a single or a handful of edits, this paper evaluates on the ability to make hundreds to thousands of edits, sequentially. This is a significantly more realistic and utilitarian setup. \ b) Three text-editing settings: The authors include experiments on multiple settings (context-free question-answering on a T5, BERT classification on SCOTUS dataset, and correction of hallucinations on GPT-3 wiki bios). \ c) GRACE outperforms a good selection of competing methods on a number of editing and retention metrics, demonstrating strong empirical potential. S4. The paper is well-written and easy to understand. The figures and illustrative examples do an excellent job conveying the key concepts.
Weaknesses
W1. Editing examples vs concepts: From my understanding, the proposed method focuses on editing predictions of a model for a specific example. This contrasts with continual learning methods, which tend to focus on learning new concepts or classes. While the editing approach is good for correcting mispredictions or hallucinations, it doesn’t seem like it is capable of learning to make new predictions: for classification, this could represent new categories or new vocabulary. W2. Scalability: The number of GRACE layers required in the codebook mostly scales linearly (technically sublinear because edits can be grouped with previous GRACE layers, but from Figure 4, the overall behavior is still close to linear—converging to 50% cache rate is still O(n)). In addition to storing an increasing number of GRACE layers, this increasingly large codebook also requires an increased number of comparisons at inference time, for *every* forward pass, even inputs that are correct and otherwise unaffected by the edits. Experiments show up to 5K edits, but real-world deployments may (e.g. chatbot search engines) may require significantly more. W3. GRACE layers value storing vs override: My understanding is that the models being considered in this work are fully deterministic during inference. Thus, once a particular input triggers a particular GRACE layer anywhere in the model, then the retrieved activation $h^l$ should always result in the same output. This raises a couple questions: \ a) If the output from a GRACE layer deterministically maps to a particular output, then why wouldn’t we instead directly store the desired output and save the computation of the subsequent layers $l+1, …, L$? \ b) This also suggests a much simpler approach: maintain a dictionary of input-output corrections, and check if the input exists in the dictionary before running the model. The primary advantage of GRACE then would be the potential for better generalization, as matching to previous edits is done at the feature level, but this needs to be verified empirically. Less critical concerns: - Title: While the title is kind of cute, “Aging” is only partly relevant to the problem being addressed here. Edits are also necessary for examples for what the model gets wrong, even when it’s a “newborn”. - I think it’s debatable whether the proposed method should be considered a “continual” or “lifelong” learning method, as it’s more or less sidestepping the catastrophic forgetting problem altogether with a roughly linearly increasing codebook of overrides (see W2). With the $\epsilon$ radius, there is some potential for forward or backward transfer, but I would classify it as minimal and relatively rudimentary. On the flip side though, I can see similar arguments being made about some expansion-based CL methods, though generally their scaling characteristics aim to be better. - I was a little disappointed that there weren’t any experiments with large vision models, but I understand the desire for limiting scope. Miscellaneous: - Fig 2: “No near key” => “No nearby key” - Equation between Lines 139-140: The top line doesn’t quite make sense. I believe the authors meant to write something along the lines of “if $d(h^{l-1}, K_{i_*}) < \epsilon_{i_*}$” instead. Also please include an equation number. - Line 149: “one of two”? - Line 191: missing period “.” - Line 279: possibly a missing space after the “:” ======== Overall Assessment: I found this paper interesting and well presented; the proposed GRACE layers appear to be a reasonable, straightforward solution to editing. However, I’m concerned about the method’s overall scalability, especially when the method in many ways is similar to keeping a dictionary of input-output overrides (e.g. prefacing the model inference call with simple if-then statements). By operating in feature space, perhaps GRACE has better generalizability than such a rudimentary approach, but there is little empirical evidence of this. Also, it still isn’t clear to me why values are activations for the next layer as opposed to just jumping straight to the output. I’m willing to raise my score if my concerns are sufficiently answered.
Questions
Q1. What causes the fluctuations in ERR when applying GRACE to layer 6 with $\epsilon=1.0$ in Figure 4? Q2. How does the inference time change with increasing numbers of edits?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
2 fair
Limitations
The limitations section is well-written and frankly acknowledges some valid limitations and potential ethical concerns. While important, I don’t necessarily see these as blocking acceptance of the paper.
Rebuttal Acknowledged
I thank the reviewers for the responses. I read through the rebuttals as well as the other reviews, from which I see pretty good alignment with my original review. Reviewer wTwG shared my concern on GRACE’s memorization vs generalization, Reviewer Qd7q w.r.t. comparisons with continual learning, and Reviewer nd9W on experiments being limited to NLP tasks. Many of my initial concerns/questions were mostly addressed. As a result, I raise my score to 6. Some additional comments below: Inference time: I appreciate the table on edit times as a function of codebook size, which was quite informative, but it’s not quite what I was asking about. My question instead was how does the runtime for forward-pass inference (not making an edit) change with respect to code size? Essentially my question can be answered by adding another column with “Inference Time (s)”. My concern is that “fixing”/”editing” a model’s response to a what is collectively a tiny region of the model’s input space can lead to a disproportionate increase in running time. Lifelong solution: This reply wasn’t quite aligned with my original comment either. My original concern was the question of whether saving input-output overrides (in feature space or not) really qualifies as continual learning, philosophically. I do agree there are some commonalities, but not as much as the paper’s title and discussion seem to suggest. I also agree with Reviewer Qd7q that the comparison with EWC is not quite appropriate.
Thank you for your increased support, we're glad that our responses were helpful! Since there's not much time left for author--reviewer discussion, we plan to address your lingering concerns with updates to the paper and future experimental plans, as described below along with a preliminary experiment: **Inference Time**: Thanks for clarifying this point, we believe we now understand your question better. To answer it, we ran the following extra experiment that we'll extend and add to the paper. *Experimental setup*: * We made 3k edits to the QA T5 model's 4th layer with a small $\epsilon_\text{init}$ (the final codebook contained about 2k keys) * Before each edit, we timed the one forward pass through both the original (unedited) model, and the edited model on one singular input (batch size of 1). * We report and compare the inference time for the unedited and the edited models over the sequence of edits. *Caveats*: * **Our implementation is not optimized for inference time**, so any results may not be reflective of practical considerations. Inference speed will likely be improved by incorporating more advanced techniques like approximate similarity search. * Reporting inference for one input in isolation is not a complete picture. We'll replicate this experiment multiple times and show confidence intervals as a line graph. *Results and Discussion*: * **Inference with the unedited model took .022 seconds on average and remained stable throughout editing, as expected**. * **Inference with the edited model started at .022 seconds prior to editing and increased to .078 seconds after 3k edits (2060 keys)**. * While slow-down is often acceptable depending on the importance of the edits, we do see some slow-down. But we believe it is reasonable given it's not our focus and this is already more edits than prior works. For example, assuming these single-inference numbers are accurate and increase linearly, inference would slow to 1 second only once the codebook contains 35,977 keys. * A richer comparison would involve other editors, some of which require significant pre-computation and then underperform on this setting. | Num Inputs | Codebook Size | Unedited Model Inference (seconds) | Edited Model Inference (seconds) | :--------------: | :--------------: | :--------------: | :--------------: | | 0 | 0 | .022 | .022 | | 1 | 1 | .022 | .030 | | 50 | 50 | .022 | .028 | | 100 | 99 | .024 | .028 | | 1000 | 843 | .020 | .046 | | 2000 | 1518 | .018 | .063 | | 3000 | 2060 | .024 | .078 | **Lifelong solution**: Thanks for the extra details on this point, too, we will revisit this and circulate for feedback.
I thank the authors for the additional experiment. While it's true that some sort of slowdown is to be expected, I somewhat disagree with the authors on how acceptable the longer inference time is. 3.5x longer inference time after 3K edits is a pretty serious issue, and may not be practical in many use cases; to re-emphasize a comment in my initial review, this slowdown applies *every* time we want to use the model, as we don't know ahead of time if the input corresponds to a model edit or not, even if the portion of input space the edits apply to is small. The authors are correct that smarter similarity search techniques should partially alleviate this problem, but this reinforces my general concern about the method's scalability nonetheless. I still believe there are good contributions here, enough for warranting acceptance, but the method's scalability appears to be a clear direction for future work.
Thanks for the extra points, we wholeheartedly agree scalability is a clear direction for future work. When we add the extended version of this experiment to the paper, we will clearly state this is a limitation. Thanks for suggesting this experiment! It will directly improve the impact of our work and shine more light on where there's room for improvement.
Summary
The paper proposes GRACE, a method towards lifelong editing of models through the use of a codebook, a mechanism to write to/make use of the codebook without perturbing model weights. In essence GRACE can be viewed as an inserted adapter layer, which caches activations as keys, fine tunes values that the keys map to, and employs a so called deferral radius per key to decide when to activate GRACE. ---- Rating raised post-rebuttal from 6 to 7, as weaknesses were at least in parts clarified or addressed.
Strengths
The paper has many strengths, starting from a well-motivated introduction and scope that tackles a highly important problem. In general, the exposition of the paper is clear and it is well written and structured. The mechanisms are mostly explained in sufficient detailed and GRACE itself becomes intuitive through the added illustrations. The empirical investigation further investigates a set of metrics and includes a set of meaningful benchmarks. Within the given set of explored alternatives, GRACE can be viewed to provide consistent advantages, making GRACE a good contender for this kind of problem setting. The addition of a limitations section with some statements on potential (harmful) social impact is also appreciated.
Weaknesses
The paper seems to have two primary weaknesses. One lies in the exposition with respect to related work and its empirical comparison, the other is a potential concern with GRACE’s practical hyper parameters. With regard to related work, the empirical comparison to continual learning works seems interesting and it is not fully clear why it is meaningful. If one were to take a look at one of the plethora of continual learning reviews (take Hadsell et al, Mundt et al, de Lange et al), they typically categorize techniques into rehearsal (memory buffers), adding components to the architecture or regularizing parameters. The comparison in this paper primarily seems to compare to fine tuning and EWC, which fall into the regularization category. Yet, to the best of my understanding, GRACE implies the use of extra parameters and memory, which would in that sense be much fairer to compare to continual learning methods to do something similar. There are many of those, but think of storing attention masks, adaptively learned gates, or memory buffers that similarly store activations. It is presently not clear how GRACE differs conceptually from these prior arts (mentioned in e.g. the above surveys) and following in this line, what the “cost” of GRACE is with respect to growth/memory of the codebook. As mentioned above, the second concern lies in the choice of parameters, specifically how to pick appropriate values for epsilon and how to decide which layer to edit. From figure 4 it seem the system is not very robust to this choice necessarily and it becomes a trade-off with respect to what value is chosen for which layer. Whereas this may be clear, it will be important to clarify how to pick suitable hyper-parameters without being able to look into the future in lifelong editing (i.e. plotting figure 4 and deciding on how the curve develops on the x-axis); also see below question.
Questions
It is not really clear how to choose GRACE’s hyper-parameter epsilon initially. The intuition has been nicely explained, i.e. large values will apply edits with more influence, but it is not obvious how one can pick the value in advance before starting to do lifelong editing for an arbitrary dataset. In this scenario, hyper-parameter search does not seem possible because one cannot look into the future. Additionally, figure 4 also shows even more trade-offs with respect to choice of layer, which seems to be a second hyper-parameter in that sense. It is not clear how to decide on what layer to edit at which value, specifically if the values of epsilon for different layers need to be different for GRACE to work effectively in practice. It will be appreciated to get clarifications here in the rebuttal.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Limitations are mentioned appropriately and are appreciated. Divining deeper into the mentioned limitation of slowing down inference through similarity computation, along with potential other limitations on storage cost and codebook overhead, will be a further improvement to the paper and its limitations section.
Rebuttal is acknowledged
I thank the authors for the rebuttal. In particular, the clarifications, extra figures and discussion of hyper parameter selection is appreciated. I still think there can be a more suitable set of continual learning methods to be contrasted, but also acknowledge that having EWC/fine-tune is appropriate for some context. Ideally, both these simple baseline and even more related ones would exist, but I nevertheless raise my rating to accept, given the extended hyper parameter analysis and the other parts of concerns that were addressed.
Thank you for your increased support, we're glad that our responses were helpful! Given limited remaining time for author--reviewer discussion, we will address your remaining concerns by stating in the paper's experiments that EWC/fine-tune are fairly simple, preliminary comparisons and our aim is to provide this context. We do hope our work will inspire future such comparisons or adaptations of more-advanced methods.
Answer to Rebuttal
Thank you for the rebuttal. After reading the other reviewers' comments and your rebuttal, I am still of the opinion that this paper meets the bar for acceptance. I am therefore keeping my score as-is.
Kind reminder of our response
Thank you again for your feedback and for supporting our work! We'd like to kindly remind you that we've addressed your concerns. Since there's very little time remaining for the discussion period, we'd love to know if you have any lingering concerns that we can address within the timeframe.
Kind reminder of our response
Thank you again for your feedback on our work! We'd like to kindly remind you that we've addressed your concerns. Since there's very little time remaining for the discussion period, we'd love to know if you have any lingering concerns that we can address within the timeframe.
I appreciate the authors' comprehensive responses to the reviewers' feedback. The TRR is significantly improved in the revised results, and it does look like GRACE is learning something beyond pure memorization. I still feel that more focused analysis of edit generalization on something other than rephrases is crucial to make model editing useful in the real world. Nonetheless, I think the ideas and updated evaluations in this paper are sufficient to warrant acceptance. Therefore I will raise my score to 7.
Decision
Accept (poster)