Modern language models have the capacity to store and use immense amounts of knowledge about real-world entities, but it remains unclear how to update such knowledge stored in model parameters. While prior methods for updating knowledge in LMs successfully inject atomic facts, updated LMs fail to make inferences based on injected facts. In this work, we demonstrate that a context distillation-based approach can both impart knowledge about entities and propagate that knowledge to enable broader inferences. Our approach consists of two stages: transfer set generation and distillation on the transfer set. We first generate a transfer set by prompting a language model to generate continuations from the entity definition. Then, we update the model parameters so that the distribution of the LM (the student) matches the distribution of the LM conditioned on the definition (the teacher) on the transfer set. Our experiments demonstrate that this approach is more effective at propagating knowledge updates than fine-tuning and other gradient-based knowledge-editing methods. Moreover, it does not compromise performance in other contexts, even when injecting the definitions of up to 150 entities at once.
Paper
Similar papers
Peer review
Summary
This paper tackles updating the knowledge in LMs, focusing on allowing LMs to make new inferences consistent with the updated facts. To do this, the authors propose using the LM itself (or a teacher) to generate natural continuations for the "updated/new entity" definition. These continuations are used to update the LM. The update is conducted using a KL divergence loss between the LM conditioned on the definition and the LM that doesn't see the definition. The results show superiority to baselines in updates and in preserving old knowledge.
Strengths
The paper seems like an excellent contribution. It's well motivated, well presented, and the key idea is simple, novel, and effective. The evaluation is convincing.
Weaknesses
The method, like many others, is relatively opaque in terms of what it teaches the models and why/how it works precisely. However, it's well motivated and the analysis in Sec 7 begins to shed a little bit of light into this. More work is needed on that front, but I think it's fair to assume this will lie beyond the scope of this paper.
Questions
N/A
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
N/A
Summary
This paper studies the problem of injecting new entity knowledge in LLMs, such that these knowledge can be propagated and utilized when LLMs make inference on related queries. The paper proposes a context distillation method that consists of two steps to inject entity knowledge in a definition sentence: 1) Use a LLM to generate a set of continuations (a.k.a transfer set) for the definition sentence. 2) Fine-tune a student model such that its output distribution without conditioning on the definition sentence is close to the output distribution of a teacher model that conditions on the definition sentence. They conduct experiments on two datasets about entity knowledge and show that the proposed method outperforms several baselines including standard fine-tuning and previous knowledge editing methods.
Strengths
1. This paper studies an important question of knowledge injection and propagation of injected knowledge. The proposed method is novel in this context. 2. Some of the conducted analyses are insightful, such as the NLL with/without definition sentence for analyzing the supervision from the teacher model.
Weaknesses
1. On Entity Inferences dataset, the conclusion that the proposed method improves the model ability to make inference using the injected knowledge is suspected. The reported performance improvement might due to the overlap between the generated transfer set and the probe sentence in the evaluation set. Without reporting (1) the level of overlap, and (2) a baseline that simply fine-tunes on the transfer set, the possibility of this overlap cannot be ruled out. 2. How does the method perform compared to a baseline that simply prepends the transfer set to the query?
Questions
1. In Table 2, the Target for GPT2-XL should be 64.3 instead of 65.3 (based on the $\Delta$ value)? 2. In Table 2, why would using GPT3.5 to generate transfer set result in worse specificity for GPT2-XL? 3. I'm not sure why most of the analyses are done on the ECBD dataset, as I thought Entity Inferences dataset concerns more about injected knowledge propagation.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
Limitations are discussed.
Summary
This paper proposes a method to propagate knowledge update to LMs via training a student model through context distillation, such that the LM can make inference on an entity even though the relative context/knowledge of the entity is not given. The framework involves two steps: 1) create a transfer set that contains the knowledge that the student model will be learning from; 2) compute the distribution of the transfer set tokens for both the teacher model (while given context, i.e. a definitional sentence) and the student model and update the student model's parameters by minimizing the KL divergence of the two distributions. The paper evaluates the student model with two sets, Entity Inferences and ECBD to show that the knowledge has successfully propagated.
Strengths
This paper is more efficient with multi-entity editing and achieves competitive performance on the two evaluation set in terms of propagation success (accuracy and decrease in perplexity) while causing little impact on specificity.
Weaknesses
It seems like the paper is more focusing on new knowledge ingestion, either in Entity Inference (synthetic entities) or ECBD (introducing new entities after 2022). While this is an important aspect, a harder task is to update existing knowledge in the old model. One dimension could be temporal shifts, e.g. after a new election, population/economic changes (potentially resulting changes in superlative statements), factual changing official announcement (e.g. solar system has 9 planets before 2006 and Pluto was downgraded to dwarf planet in 2006 - solar system has 8 planets now). It is unclear whether the model can adapt to the new facts while maintain low specificity. Another baseline is to try prompting the LLMs with new knowledge and see how it propagates. If the existing LLMs can handle such knowledge updates well, it may be hard to justify why we need to train a separate student model.
Questions
On line 126, it states the distillation is done through updating $M_s$ parameters to minimize the KL divergence. Does it update all the parameters in $M_s$, or is it possible to combine the distillation with other network editing techniques to only a local set of parameters? How much would it negatively impact the performance if only a local edit is allowed? Asking since if we want to extend this framework to larger LLMs (as current good-quality LLMs usually have 100B+ parameters and updating all parameters seem to be impossible).
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
As mentioned by the authors, this work mainly uses relatively small size LMs for experiments and its generalizability to LLMs is unknown. While it may apply to LLM trainers/creators to adapt this method to update their models, it does not extend to end users/organizations of the LLMs who want to ingest or update knowledge, e.g. from specific domains or confidential sources, potentially through local edits.
Summary
The paper propose a context distillation-based approach that can both impart knowledge about entities and propagate that knowledge to enable broader inferences. This approach consists of two stages: transfer set generation and distillation on the transfer set. In the first stage, a transfer set is generated by prompting a language model to generate a continuation from the entity definition. In the second stage, the model parameters are updated so that the distribution of the LM (the student) matches the distribution of the LM conditioned on the definition (the teacher) on the transfer set. The authors' experiments demonstrate that this approach is more effective in propagating knowledge updates compared to fine-tuning and other gradient-based knowledge-editing methods without compromising performance in other contexts, even when injecting the definitions of up to 150 entities at once.
Strengths
1. A straightforward motivation that conditioning on information about the entity can lead to lower perplexities. 2. The authors' method of generating a transfer set by prompting an LM to generate a continuation from the entity definition is a unique contribution to the field. 3. The authors compare their method with other knowledge injection methods, including fine-tuning, and demonstrate the superiority of their approach. They also conduct an in-depth analysis of the types of continuations needed in the transfer set. 4. The authors' method provides a scalable and effective way to update the knowledge of LMs.
Weaknesses
1. As the authors concede in Section'Limitations', their proposed methodology has yet to be substantiated on models of a larger scale. For instance, LLaMA-65B may present a fitting candidate for such validation. 2. The experiments in the paper focus on a specific type of knowledge update: adding definitions for entities. It's unclear how well this method would work for other types of knowledge updates, such as knowledge revision. 3. The results of Finetuning on transfer set (full) are not shown in Table 2. 4. Writing content issues. (1) It would be clearer to add arrows in the table to show whether the larger or smaller values are better. (2) What are Finetuning on definition (full) and Finetuning on definition (last only)?
Questions
What are Finetuning on definition (full) and Finetuning on definition (last only)?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes.
Thank you. Following up on your discussion with reviewer ARok on prompting with updating definitions, have you experimented with that for the results in Figure 4? Are there any bottlenecks that would prevent scaling the x axis to, say, 10x its current max? I realize that the current datasets may not even be large enough for this (looking at your appendix). Can you include a note on whether/how you think test sets that are this small can support the results you report as significant? All three together add up to 1500 examples, which in aggregate is substantial (though small individually) but it's unclear to me how validation/development was conducted. In general, the main challenges I see for the work are: - Making the case that small-scale edits, from definitions, have a place when prompting works so well. I think the paper can make this case but it needs more explicit consideration. - Based on the discussions with other reviewers, the evaluation datasets ECBD and Entity Inferences are perhaps less satisfying that my original impressions of them. I am cognizant that this is, in part, plausibly a weakness of this area rather than this paper. That said, are these datasets large and meaningful enough to support the claims? What would a more ideal dataset look like, and do the authors believe that these two are the best that exists? - Can we see some representative free-form generations from the models after editing? Under edit-relevant prefixes and unrelated prefixes, ideally. I think that's different from the appendix's reported content in Table 8.
Thank you so much for your comment! > Following up on your discussion with reviewer ARok on prompting with updating definitions, have you experimented with that for the results in Figure 4? Great suggestion! We ran an additional experiment where we prompt the model with definitions of the correct entity and multiple other “distractor” entities (following the setting in Figure 4, as suggested). Because of context window limits, we could only append up to 30 entity definitions concatenated together. On GPT-Neo-1.3B, we find the model’s perplexity gets significantly worse, increasing from 31.0 to 36.7. Having irrelevant information (the 29 other entities) hurts in the prompting setting, but does not in the knowledge updating setting, which we view as a point in favor of knowledge updating. > Could we scale the x axis to, say, 10x its current max? We are not able to scale the x-axis significantly more due to limitations in our dataset. We created a 2023 version of ECBD following the procedure in the original work in an attempt to add more evaluation data. However, to be included, entities need to have emerged in 2023, be of certain entity types, and have sufficiently long Wikipedia articles (written in the first few months of 2023), which are collectively quite stringent criteria. At the time of the NeurIPS deadline, ECBD 2023 only had 19 entities. However, this is not a fundamental limitation of our method, and we plan to explore this more in future work. > Making the case that small-scale edits, from definitions, have a place when prompting works so well. Thank you for the suggestion! We will include more discussion on the benefits that editing has over prompting in the final version. We explicitly state some of the benefits here: * Prompting the model with irrelevant information hurts performance (as noted above). * Injecting the information allows the model to use it flexibly and whenever necessary. * Prompting will not be an efficient solution for injecting larger amounts of information; even when injecting little information, parameter updates are more efficient when there are *multiple* queries on the same piece of information. Injecting information only happens once, but prepending incurs computational cost for each. > On whether our datasets are large enough, and if we believe they are the best available It is true that the ECBD dataset, constructed to capture emerging entities, is modest in size, especially when we limit to entities that occurred in 2022-2023. Our main results are reported on the ECBD 2022 set, which has 1000 examples covering 153 entities. We performed a paired bootstrap test on the ECBD 2022 results (Table 3), drawing N=10000 samples. We find that, for each of the three base models we tested (GPT-Neo, GPT2-XL, and LLaMA), the gains of distillation over fine-tuning are statistically significant with p<0.05. This demonstrates that the datasets are at least large enough to substantiate performance differences on the order of what we report in this work. We will add these results in any future version. Comparing to other related datasets, we still believe that ECBD and Entity Inferences are the best datasets for our focus here. The majority of datasets used in knowledge editing works only study whether a specific fact is injected, and a few study very basic sorts of inferences such as paraphrases (such as CounterFact [1]). By contrast, Entity Inferences and ECBD require a broad range of inferences. Success on ECBD, in particular, requires a deep understanding of the entity; [2] found that existing parameter-updating methods only succeed on ECBD when there is a significant overlap between the definition and the cloze span in the evaluation sentence. In addition, it is structurally difficult to create benchmarks in this area with standard dataset collection methodology, as emerging language models will necessarily have been exposed to the changes they embody. > Can we see some representative free-form generations from the models after editing? Thank you for the suggestion! We include two tables, each with three randomly sampled entities and corresponding generations from GPT-Neo-1.3B. In each cell, the prefix provided to the model to condition upon during generation is in **bold**. In the first table, we provide the model with the entity name of the entity from ECBD 2022 we distill. In the second one, we use a prefix containing a manually chosen *related* entity that it should have knowledge of, to see whether distillation harms the model’s prior knowledge of that entity. In the first table, we see that our distillation procedure allows GPT-Neo-1.3B to gain an understanding of each entity; this is especially clear with the PitchCom example, where the pre-edit generations are some form of code while the post-edit generations indicate that the model understands what PitchCom is. In the second, we see that the model’s conception of related entities is unharmed.
References and Tables
### Distill and generate on the same entity | Entity | Definition Distilled | Pre-edit generation | Post-edit generation | |--------------|--------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| | Don't Pay UK | Don't Pay UK are a grassroots direct action campaign in the United Kingdom that urge collective non-payment of energy bills. | **Don't Pay UK**\n\nPay UK - The UK's national pay system for public sector workers\n\nOverview\n\nPay UK is one of the oldest and largest pay and benefit | **Don't Pay UK** are organising campaigns against the British Government's plans to impose charges of up to 40 percent of your household electricity bill | | PitchCom | PitchCom is a wireless communication system used in baseball that lets a player request pitches without using visible signals. | **PitchCom**posers.ComposerFactory.GetComposer();\n }\n\n public void | **PitchCom**, using an integrated set of transmitters and receivers to communicate through a series of alternating transmission signals with players in the area. | | Fastelavn | Fastelavn is a Carnival tradition in the Northern European, and historically Lutheran, nations of Denmark, Norway, Sweden, Iceland, Estonia, Faroe Islands, as well as Greenland. | **Fastelavn**, at første årsløn hadde dagen vært for en gruppe mange | **Fastelavn**\n\nTotalfjellet, or the Day of the Year of Totalfjellet (; in the Norwegian language: ), is a holiday| ### Distill and generate on the same type but different entity | Entity | Definition Distilled | Pre-edit generation | Post-edit generation | |------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| | Hurricane Grace | Hurricane Grace was a powerful hurricane which became the strongest landfalling tropical cyclone in the Mexican state of Veracruz ever recorded, as well as tying the record with Hurricane Karl of 2010 for the strongest hurricane ever recorded in the Bay of Campeche by maximum sustained winds. | **Hurricane Katrina** (2007)\n\nHurricane Katrina was a category four hurricane that struck the Gulf Coast, and a post-tropical cyclone that severely damaged the Texas | **Hurricane Katrina**\n\nHurricane Katrina was a storm that made landfall along the Gulf Coast of the U.S. state of Louisiana as a Category 5 storm | | Windows 11| Windows 11 is an upcoming major version of the Windows NT operating system developed by Microsoft. | **Apples MacBook Pro** (2018) is a very good laptop with lots of excellent features for the money. This laptop comes with the same 11.6 inch display as the previous model | **Apples MacBook Pro**\n\nThe MacBook Pro is a full-featured and powerful computer from Apple. The MacBook Pro is a powerful and powerful computer designed with professional users in mind | | The Dixie Fire | The Dixie Fire is an active wildfire in Butte, Plumas, Lassen, and Tehama Counties, California. | **Camp Fire (2018)** (season 5)\n\nThe fifth season of the Australian reality television series Camp Fire premiered on The Hub. | **Camp Fire (2018)**\n\nThe 2018 Camp Fire, also known as the Camp Fire, is an wildfires in the state of California, the largest wildfire in California's history | References: [1] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. [2] Yasumasa Onoe, Michael J.Q. Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowl- edge. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
Thanks for your response
Dear authors, Thanks for your work and the detailed response to the reviewers. As we are waiting for the reviewers to acknowledge your response and potentially raise follow-up questions, I would like to kindly ask a high-level question regarding your proposed approach. Isn't the fact that the method relies on distilling knowledge updates from one model (the teacher) to edit another model (the students) implies that it can be applied only to knowledge that is encoded in the teacher? Namely, if we want to inject a new fact that is more recent than the data used to train both the teacher and the student, then it is basically impossible to use your method in this case, right? Best regards, Your AC
Thanks very much for your comment! However, we think there may be a misunderstanding about our method. Our method is explicitly designed to inject information more recent than the data used to train the student and the teacher (which is a copy of the student). Indeed, our results (Table 2 and Table 3) use datasets which almost entirely consist of entities the base models we study have never seen. In our procedure (detailed in Figure 1), the teacher is not a different model, but merely conditions on additional input (an entity definition, such as “ChatGPT is an AI chatbot by OpenAI.“). By having access to this information in-context, it still achieves better predictions than the student model on the continuation sentence here, which serves as the signal for distillation. Please let us know if you have any further questions!
Thanks for the clarification!
Decision
Accept (poster)