Episodic Memories Generation and Evaluation Benchmark for Large Language Models

Answer to reviewer 1gS7 (events being realistic or unrealistic, 1/2)

> The events described in the book chapters are not filtered in terms of whether they are somewhat realistic or not. It seems very likely that the detail of the event does not align very well with the location. The fact that the dataset contains some non-realistic events might limit the capability of the LLMs to recall them, since the LLMs may use their common sense knowledge to think that it is not likely for this event to take place at the specified location. > Following my comment regarding the events being unrealistic in the "weaknesses" section, I would hope to get more insights on this from the authors, e.g. whether unrealistic pairings actually impact LLM performance differently than realistic ones? First, this is a well-thought consideration. We agree that the book chapters are not explicitly filtered for realism. While the event content is relatively generic (tech hackathons, jazz nights), some combinations may be unrealistic (such as fire dancing performances around the Statue of Liberty). However, we believe that both LLMs and humans should be capable of answering questions about past, future, and fictional episodic events. When prompted, the model is explicitly asked about episodic events within a particular fictional book. During this rebuttal period, we have created two new universes and associated books: one focused on far-future science fiction (less realistic) and another on world news events (more realistic). An example chapter can be found in the section "Illustration of a single world news fictional chapter." Nonetheless, the suggestion provided by the reviewer is sound and interesting because LLMs may treat differently realistic and unrealistic events. For that reason, we provide the analysis below. We classify the 196 events from our default large book according to their degree of realism (as judged by an LLM, which also provided an explanation for each event). Overall, we found the following : - Realistic events: 100/196 (Example: "This event is entirely plausible as it involves a common activity (photography exhibition) at a real location (Port Jefferson) with a reasonable future date. Photography exhibitions and workshops explaining post-processing techniques are regular occurrences in art communities, and the timeframe (2026) is in the near future.") - Moderately realistic event: 7/196 (Example: "This event is moderately realistic because karaoke nights are common social activities, and Chelsea Market is a real venue that could host such events. Performing songs in different languages is also common in karaoke. The specific date in the future and named person make it plausible, though we can't verify if this exact event will occur.") - Somewhat realistic event: 52/196 (Example: "While fashion shows in museums do occur occasionally, and the American Museum of Natural History has hosted special events, it's a relatively unusual venue for a fashion show. The specific date in the future and named individual makes it plausible, but museums focused on natural history aren't typical locations for fashion events compared to art museums or conventional fashion venues.") - Non-realistic event: 31/196 (Example: "This scenario is unlikely because Bethpage Black Course is a prestigious golf course that wouldn't typically allow parkour activities. Golf courses are carefully maintained for golfing and would not permit activities that could damage the turf or disturb golfers. Additionally, parkour typically requires urban structures or obstacles, which wouldn't be present on a golf course.") - Impossible event: 6 /196 (Example: "Fire performances are strictly prohibited at the Statue of Liberty as it's a protected national monument with strict security measures. Additionally, visitors are not allowed to perform any kind of shows or demonstrations inside or around the statue due to safety regulations and preservation concerns.") We observed that only a small number of events are non-realistic or impossible. But *even these events would be plausible within the context of fiction*. Next, we further categorized the chapters into two classes: - R: Realistic and Moderately realistic events - N: Somewhat realistic, Non-realistic, and Impossible events Based on this classification, each question is assigned to one of four groups: - Question related to empty events: No related chapter exists - Question related to realistic events: Question relates only to chapters in class R - Question related to non-realistic events: Question relates only to chapters in class N - Question related to mixed events: Question relates to multiple chapters, with at least one from each class (R and N) This binary classification (R/N) is necessary to achieve balanced groups, as allowing more granular combinations would lead to excessive fragmentation. Below, we present the results for in-context gpt-4o. This analysis will be extended to all other models.

Answer to reviewer 1gS7 (data and evaluation quality)

> As the authors acknowledged in the paper, there might be multiple episodic events that can be extracted from each chapter. In addition to the specified time and location their might be other times and locations referred to in the chapter. Are there any measures to filter out these chapters? Sorry for the lack of clarity. We have developed as part of our benchmark a verification system that incorporates two complementary quality control layers to address this concern: - First, we conduct exact verification checks to confirm that primary event details (time, location, entity, and content) appear verbatim in their designated paragraphs and nowhere else in the text (see appendix B.1.6). This establishes an unambiguous anchor point for the main event. - Second, we employ LLM-based verification (details are in appendix B.1.7) through four targeted boolean questions that validate whether the chapter maintains: 1) a single geographical focus, 2) a single temporal day, 3) a single main character, 4) a single main event. While we acknowledge that realistic narratives inherently contain multiple micro-events (e.g., having conversations), these details are subordinate to the primary event constructed. Our verification system ensures these supporting elements enrich the narrative without introducing competing main events, thus maintaining authenticity while preserving a single, clear "ground truth" event. > Also, does the fact that there are many other events (especially contents) in addition to the specified one make the F1-score metric less suitable, since false positives might not actually be false? This is the main concern that I would like to get addressed. Thank you for the depth of your understanding of our work. This is indeed a critical challenge that we faced. We think that we carefully addressed it in our paper. Unlike typical Q/A benchmarks, our ground truth answers comprise lists varying from 0 to over 10 elements. Furthermore, the LLM provides responses in a freeform format. As the reviewer correctly noted, predicted answer strings may contain additional details related to the main event. As illustrated in Listing 13, while the model correctly identifies the main event (Tech Hackathon), it also extracts related details, producing a list of identified_items: ['Tech Hackathon', 'developers gathered', 'collaborative projects', 'innovative solutions', 'presentations']. To address this challenge, when computing the F1-score, we adopted a lenient approach by estimating an upper bound for precision (while maintaining accurate recall calculations): rather than penalizing all additional details as false positives, we use #pred = min(#identified_items, #ground_truth) when computing precision. This provides an upper bound for precision since it effectively ignores excess predictions beyond the ground truth size. This means our reported F1 metrics actually represent an optimistic bound - the performance when considering stricter counting would be even lower. This strengthens our conclusion about current LLMs' limitations in episodic memory tasks, as they suffer despite our leniency. We will clarify this aspect in the text. > For the results in the main text, the entities and the books are generated by Claude 3.5 Sonnet. Does this possibly give Claude models an unfair advantage in the evaluation? Thank you for raising this interesting consideration. To verify it, we evaluated the overall performance on the gpt-4o-generated short book. The results are as follows: | Memory | Model | book | 0 (150) | 1 (150) | 2 (48 for claude, 47 for gpt) | 3-5 (18 for claude, 21 for gpt) | |--------|-------|---------|---------|---------|--------|----------| | in-context | gpt-4o-mini | Claude | 0.53±0.50 | 0.92±0.23 | 0.87±0.21 | 0.89±0.16 | | in-context | gpt-4o-mini | GPT* | 0.73±0.44 | 0.91±0.26 | 0.82±0.25 | 0.87±0.16 | | in-context | gpt-4o | Claude | 0.86±0.35 | 0.96±0.16 | 0.93±0.16 | 0.88±0.16 | | in-context | gpt-4o | GPT* | 0.88±0.33 | 0.92±0.24 | 0.87±0.20 | 0.82±0.18 | | in-context | claude-3-haiku | Claude | 0.81±0.39 | 0.74±0.43 | 0.59±0.31 | 0.65±0.20 | | in-context | claude-3-haiku | GPT* | 0.90±0.30 | 0.73±0.43 | 0.55±0.32 | 0.56±0.27 | | in-context | claude-3-5-sonnet | Claude | 0.98±0.14 | 0.94±0.23 | 0.73±0.22 | 0.73±0.20 | | in-context | claude-3-5-sonnet | GPT* | 0.97±0.18 | 0.77±0.41 | 0.65±0.25 | 0.61±0.15 | Overall, we observe *mixed performance patterns*: - Claude models seem to perform better on Claude books - GPT models show better performance on Claude books for all questions, except for hallucination questions. Recall that our main results use Claude books, which appear, here, to favor both model families To validate these observations, we will conduct statistical analyses comparing model pairs (gpt-4o-mini vs. claude-3-haiku, and gpt-4o vs. claude-3-5-sonnet) across both books (likely in the camera-ready version).

Answer to reviewer uWQ8 (Q1)

We appreciate the positive view on the importance of the challenge, the structured and rigorous approach, with systematic assessment tasks. We will answer to the different questions and weaknesses in the next paragraphs. > Q1.How does the synthetic data generation process ensure realistic temporal and causal relationships between events? > 2.The scope of the benchmark is unnecessarily limited. The current implementation: (1) only considers fictional narratives with human-like protagonists, (2) Uses oversimplified temporal representations, (3) Fails to address complex episodic memory scenarios involving interconnected events. The authors should expand the benchmark to include more diverse scenarios, complex temporal relationships, and interconnected event sequences that more accurately reflect real-world episodic memory challenges. First, thank you for offering us the opportunity to enhance our work, and we are sorry for the unclarity. The questions made us realize that we likely failed to correctly present our framework. To address this, we will include in the rebuttal the following [flowchart](https://figshare.com/s/863956f3e6592d3dad34?file=50683452) showing our end to end generation pipeline. Let us use it to clarify the following points. - **Shared Universe Structure**: While our events are generated independently, they exist within a shared universe with: - Common set of entities (e.g., "Jackson Ramos" later) - Common set of locations (various locations within New York in our default book) - Coherent timeline *This creates the opportunity to track entities across space and time, even without causal links.* - **Beyond Simple Retrieval**: Consider a question like "Where was Jackson Ramos seen?". Our benchmark: - Must track Jackson's appearances across multiple chapters - Identify for each appearance which places he was seen at To make this even more complex, this information is likely spread across different paragraphs within chapters. - **Illustration of entity state tracking** To better illustrate this, we will include [the following example representing the tracking of a single entity](https://figshare.com/s/863956f3e6592d3dad34?file=50682921) (here Jackson Ramos with red segments) over the chapters (other entities are the grayed segments), for the default book with 200 chapters (which happens in New York). So even without causality, our tasks require, a form of temporal reasoning (e.g. tracking the same entities across different dates, ordering their events), a form of spatial reasoning (e.g. tracking movements between locations), entity state tracking (what an entity was doing at different events, what an entity was doing *last* etc). All this requires the ability to integrate, beyond retrieval, information across chapters. With this, we believe that our benchmark is providing significant additions compared to the retrieval-oriented benchmarks. We definitely agree that causally linked events would further strenghen our work. However, it is much more challenging to create cause and effect between events, and still create million-token books that are coherent and consistent, a challenge we left for future work. > 3.While Section 3.1 emphasizes the importance of entity state tracking, the experimental results in Table 3 do not adequately measure this capability. The evaluation focuses on simple recall rather than complex state changes. The paper claims to test "understanding temporal sequences" but does not properly evaluate how models handle causally related state changes. The authors should design specific test cases for complex state tracking, evaluate models' ability to handle causally related state changes, and include metrics for measuring state tracking accuracy. Now that we clarified that entities have multiple states across the book, Table 3 focuses indeed only on simple recall tasks as you rightly mention. However, Table 4 is the one that answers your question since it evaluates (i) latest state recall (*Match latest* in the table) and (ii) chronological ordering (*Match all* and *Kendall tau*). The table show cases how poor is the performance of the LLM in such a difficult task. We are grateful for any suggestion that can help us better clarify this (would changing the names in the table be enough? for example to "latest state" and "chronological order"?)

Answer to reviewer YG9E

We appreciate the strengths provided by the reviewer, and corrected the citation format in line 123. We would like to first comment on the summary provided, then answer to the question. > While some models demonstrated near-perfect accuracy in chapters involving zero or one event per entity, their performance declined significantly as the number of events increased. We'd like to clarify the document generation process. In our framework, each chapter corresponds to a single defined event with known time, location, entity, and event content. While the events are generated independently, they all exist within the same static universe. To illustrate this, we provide an example tracking a single entity (Jackson Ramos, shown with red segments) across chapters (with other entities shown as gray segments) in the default 200-chapter book. [Illustration of an entity tracking example](https://figshare.com/s/863956f3e6592d3dad34?file=50682921) In this example, the question "at which date did events involving Jackson Ramos occur" links to 5 distinct events, yielding 5 dates. To evaluate recall performance, we assess the accuracy of answers to questions associated with varying numbers of events (ranging from 0 to 6+), while maintaining that each chapter corresponds to one ground truth event. We hope this clarifies the structure of our benchmark > It would be beneficial to explore whether different fine-tuning parameters, or fine-tuning applied to other models could enhance the performance of episodic memory tasks This is an excellent suggestion that touches on a fundamental challenge we uncovered in our work. Our experiments reveal an interesting phenomenon: even though models can learn individual facts through fine-tuning (e.g., "Jackson Ramos attended a jazz concert in Central Park on September 22, 2026", "Jackson Ramos gave a photography workshop at Ellis Island on April 09, 2026", "Jackson Ramos led a business meeting at One World Trade Center on August 24, 2026"), they struggle to synthesize information across chapters to build a complete picture (e.g., tracking Jackson's progression from teaching photography to attending cultural events to conducting business meetings across New York City over time). We are not aware of any fine-tuning strategy that can lead the models to perform such an integration. Finding such novel finetuning or learning strategies is a great challenge that we submit with this work to the community. Our episodic memory benchmark is a first step towards that goal. and we demonstrate in this paper that direct fine-tuning with question/answer pairs is inadequate. > **Weaknesses:** : line 123 has different citation format Given that all evaluation factors are rated as good or excellent (3/4 for soundness, 4/4 for presentation and contribution), we would appreciate any additional feedback about remaining concerns that led to the "marginally below acceptance threshold" rating. This would help us better understand how to strengthen our work for future versions.

Answer to reviewer PJrN

We thank the reviewer for the validation of the benchmark design and realization. We comment in the next paragraphs the different explicited weaknesses, and provide answers to the questions. > A limitation of the paper is that it only evaluates proprietary models like GPT-4o and Claude, rather than open-source models such as LLaMA 3. Including open-source models would make the findings more generalizable and accessible to a broader research community, enabling comparisons across a wider range of models and methods. We agree with the reviewer and plan to add the evaluation with one or several LLaMA 3.1 models. These models have a context window of 128k (contrary to LLaMA 3 that is limited to 7k). ~~We attempted to apply the LLaMA 3.1-405B model using a cloud API, but the service was still limiting the input token to 7k, preventing the application of our benchmarks on the small and large books. We welcome any suggestion of cloud API services that can effectively manage a 128k context window. To demonstrate our code's ability to integrate new models, we tested LLaMA 3 with an even smaller book (including only 10 chapters instead of 20 or 200), and these results will be uploaded in the reproducibility section.~~ During the rebuttal period, we have evaluated llama-3.1-405b-instruct and llama-3.2-3b-instruct on the default short book (for the camera ready version, we plan to add the results on the default long book with llama-3.1-405b too). The results are as follows (* for new experiments; adding the other models for reference): | Memory | Model | 0 (150) | 1 (150) | 2 (48) | 3-5 (18) | |--|--|--|--|--|-| |in-context| llama-3.1-405b-instruct*| 0.91±0.28 | 0.95±0.18 | 0.89±0.18 | 0.83±0.17 | |in-context| llama-3.2-3b-instruct*| 0.75±0.43 | 0.38±0.47 | 0.34±0.34 | 0.48±0.33 | |in-context| gpt-4o-mini| 0.53±0.50 | 0.92±0.23 | 0.87±0.21 | 0.89±0.16 | |in-context| gpt-4o| 0.86±0.35 | 0.96±0.16 | 0.93±0.16 | 0.88±0.16 | |in-context| claude-3-haiku| 0.81±0.39 | 0.74±0.43 | 0.59±0.31 | 0.65±0.20 | |in-context| claude-3-5-sonnet| 0.98±0.14 | 0.94±0.23 | 0.73±0.22 | 0.73±0.20 | |in-context| o1-mini| 0.97±0.16 | 0.94±0.21 | 0.90±0.18 | 0.93±0.11 | The llama-3.1-405b model performs comparably to GPT4o and outperforms Claude 3.5 Sonnet specifically when multiple events match the given cue (pending significance testing). However, the smaller llama-3.2-3b-instruct model underperforms, occasionally producing lengthy, irrelevant responses. The reproducibility notebook can be found in the main message. > The benchmark mainly uses clear cues, which, while providing consistency and control, may not capture the subtler cues common in natural language memory tasks. Adding more ambiguous time markers and indirect references could better simulate real-world memory challenges and lead to a stronger test of models' episodic memory abilities and their handling of less obvious retrieval cues. We fully agree with the reviewer and we actually even believe that it is an exciting direction for future work, i.e. to probe close locations and close dates. > Could you explain the "naive fine-tuning" approach mentioned in the paper? What datasets and methods were used, and how does this approach differ from other fine-tuning strategies for episodic memory tasks? Our naive fine-tuning experiment aims to incorporate the essential information needed to generalize answers across all benchmark questions. The question/answer pairs linked to individual events establish basic facts like "entity i was in location j at date k doing l" (for all items in the book), which enables deducing answers to questions involving multiple events, such as "where has entity i been seen?" For the fine-tuning process, we selected all 3,199 questions, each tied to one specific chapter (we cover all possible questions about each chapter). This set of 3,199 question/answer pairs forms our training dataset. We utilized the standard fine-tuning method provided by the OpenAI API. We are not aware of existing fine-tuning strategies specifically designed for episodic memory tasks, and our results demonstrate that direct fine-tuning with question/answer pairs is inadequate (while fine-tuning typically succeeds in modifying style, tone, or learning new tasks, we show it is not directly suitable for memory retention). Integrating new memories directly into model weights (rather than relying on context or external databases) represents one of the key research directions we propose for future work. > Just curious—does your benchmark have a specific name, or is it simply called "Short Book" and "Long Book"? We asked an LLM to suggest a name and title that capture the story's essence. The suggested title was "Synaptic Echoes 2026: The Neuro-Temporal Paradox of Episodic Precognition" (shown in Listing 17 in the appendix). We propose using "Synaptic Echoes" for the short version and "Synaptic Echoes (long)" for the extended version. Edit: adding llama3 results on the short book

Answer to reviewer prP7 (1/5)

We would like to first thank the reviewer for the constructive feedback on our paper. We appreciate your recognition of the importance of the research problem and of the positive comments about our approach. In the following, we answer the different questions by providing clarification and presenting additional experimental materials and results. ## Data generated > I think the biggest weakness of this paper is the quantity and quality of data generated as part of the short book and long book splits. This first stood out to me when reading Table 2. For this amount of content, it feels like even the use of automated means are not necessary i.e. a human can annotate an actual book with questions and it would be a lot higher quality than what is produced here. For this question, it would help to know the following. Does the perceived low quality of the book come from the assumption that humans can do a better job at (i) generating events/chapters? or instead at (ii) annotating existing books? Knowing this would help us better answer the question. That being said, our methodology offers unique advantages that human annotation, or even generation, cannot match easily: - Control and contamination: we generate contamination-free books with deterministic ground truth events, avoiding ambiguity that could exist in annotating real books (real books may be in LLM training data). - Systematic task design: we are the first to design a comprehensive episodic memory evaluation based on systematic (time, space, entity, details) cue-based recall tasks, grounded in cognitive science principles. This systematic design could, in principle, also benefit human annotated or even human-generated efforts. - Fine-grained control: our method further enables precise control over (i) the distribution of recurring entities, locations, dates and event content, (ii) question difficulty based on cue precision/specificity and the number of chapters needed for answer retrieval (see for instance the table 12 in the appendix showing the "widespreadness" of the selected questions, that would be difficult to obtain from a human, and prone to error). - Verifiable ground truth: unlike human annotation which may have subjective interpretations, our generated content has unambiguous ground truth since we inject, generate and verify the events. - Scalability: finally, a major strength of our synthetic approach is its *scalability*. We can generate benchmarks (book and question/answer pairs) of increasing length. To demonstrate it in this rebuttal, we further generate multiple books from different universes, even producing a 1-million-token book. Such level of scalability would be difficult to achieve with human annotation due to the sheer volume of content and the variability in narrative structures across different books. If this does not answer the question, it would be helpful if the reviewer could clarify which aspect (e.g. generation vs annotation) they find problematic to better address their concerns. > Given that the authors are specifying a general strategy for generating benchmarks in section 3, I thought at least a number of randomly generated books would be considered if not also significant diversity within these books. We agree with this comment. Our framework is indeed designed to generate diverse books by simply updating the universe components (dates, locations, entities, and events, as detailed in appendix B.1.1). To demonstrate this capability, we have now generated additional books* including: - World news collections (synthetic fictional news chapters) - Science fiction books (chapters set on different planets/moons in year 2200) While our initial evaluation focused on four similar books ({Claude, GPT4o} × {20, 200 chapters}), this was primarily driven by: - API cost considerations - The observation that models already struggle with 10k and 100k tokens, making larger contexts less informative at this stage For the revision, we will evaluate these new diverse books using GPT-4o (our best performing model per Figure 2) to explicitly demonstrate generalization across different domains. We already obtained the performance on recall tasks for the short books, as shown below (first row indicates the number of events matching the cues, with the count of questions between parentheses for respectively the default, the news, and the scifi books). | Memory | Model | Book | 0 (150) | 1 (150) | 2 (48, 33, 44) | 3-5 (18, 27, 12) | |--------|-------|---------|---------|---------|--------|----------| | in-context | gpt-4o | default | 0.86±0.35 | 0.96±0.16 | 0.93±0.16 | 0.88±0.16 | | in-context | gpt-4o | news | 0.91±0.29 | 0.99±0.06 | 0.89±0.18 | 0.86±0.12 | | in-context | gpt-4o | scifi | 0.85±0.36 | 0.99±0.06 | 0.94±0.14 | 0.92±0.15 | *One excerpt chapter is available in the common paragraph "Illustration of a single world news fictional chapter". Links to the complete books will be available in the reproducibility paragraph

Illustration of a single world news fictional chapter

## Illustration of a single world news fictional chapter We finally provide below a single chapter from one of the additionally synthetically generated book named "world news", following our methodology. This generated chapter is fictitious, and is generated given the event 'May 11, 2026', 'New South Wales', 'Benjamin Green', 'flash flood emergency', with meta-data information being 2 paragraphs, with positions {'location': 2, 'date': 1, 'entity': 1, 'content': 2}. >In a dramatic turn of events on May 11, 2026, Benjamin Green found himself documenting the rapid transformation of peaceful suburban streets into raging torrents of muddy water. The local meteorological station's emergency sirens blared through the rain-soaked air as Hamza Avila and Koa Berlin, emergency response coordinators, rushed to evacuate residents from the low-lying areas. Rising waters had already submerged vehicles to their windows, while the relentless downpour continued to intensify, creating treacherous conditions across the region. >As the situation in New South Wales deteriorated, Benjamin witnessed a flash flood emergency that would later be described as unprecedented in its ferocity. Water levels rose at an alarming rate of nearly one meter per hour, prompting Emilia Hooks, a veteran emergency services spokesperson, to declare it a "catastrophic event." The flood's destructive force was evident as debris-laden waters crashed through streets, uprooting trees and damaging infrastructure. Local authorities reported that over 300 residents were evacuated to emergency shelters, while rescue teams conducted more than 50 water rescues throughout the affected areas. The disaster response teams continue to monitor the situation as meteorologists predict additional rainfall in the coming hours.