7 threads · one per rubric
Real reviews from the record, one for every shape peer review arrives in. Open a thread to read it in full — venues do not ask the same questions, so the fields change from row to row.
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
The dataset and pilot study require human annotators but no ethics statement included.
Reviewer LhuP · 2024-10-23
details of ethics concerns
The dataset and pilot study require human annotators but no ethics statement included.
questions
Please address the concerns/questions in the above presentation (P1-P3), soundness (S1-S5), contribution (C1-C2), and the following other questions.
Other:
Do the authors include the Limitations section? If not, then is this work thoroughly studied? What are the future work?
What is the temperature used in this paper? What is the version of GPT-3.5 turbo tested in Figure 4-a?
In Figure 4-a, do the author have insights regarding why gpt-4o-mini is better than gpt-4o?
Do the authors touch upon the NDCG metric in the discussion section?
In Lines 419-420, are the timestamps sorted by ascending or descending order? Do the authors evaluate the performance when the timestamps are not sorted in the preliminary experiments?
In Table 2, why K = summary and K = V + summary experiments are not included in this paper? Is it because of the budget-aware manner? Moreover, why is the numeric value (0.590) in QA performance using Llama-3 (70B) in bold? It is smaller than the corresponding value when K = V (0.592).
The rounding is confusing.
In the third contribution (Line 99), the authors report improvement is 7% ~ 11%, but it's 6.7% ~ 11.4% in Lines 480-481. Moreover, the result is for GPT-4o LLM, and this improvement does not hold when Llama-3.1 (8B) is used, as stated in Lines 482 (We also find that the effectiveness of this method depends on using a strong LLM). Nevertheless, this strong prerequisite is not mentioned in Line 99, which could be a bit misleading regarding the "7% ~ 11%" improvement.
Can the author provide how to compute the numerical values in Line 467 ()? Specifically, how to get the result of a 5% improvement in Table 2? Is it $1/3 * [(0.714-0.670)/0.670 + (0.584-0.570)/0.570 + (0.490-0.464)/0.464] = 0.04875$, then 4.9% rounds to 5%? If so, why use Top-5 in GPT-4o and Top-10 in Llama-3.1 (70B and 8B)? Because I can get the 4% result in retrieval metrics if I average all the improvement in "Recall@5, NDCG@5, Recall@10, and NDCG@10": $1/4 * [(0.732-0.706)/0.706 + (0.620-0.617)/0.617 + (0.862-0.783)/0.783 + (0.652-0.638)/0.638] = 0.0411$ (4.1%); however, when I try to average all the top-5 and top-10 improvement in these LLMs (6 entries), it does not match 5%.
strengths
summary
This paper proposes a 500 LongMemEval dataset focusing on evaluating five crucial long-term capabilities of LLMs: (single-session) information extraction, multi-session reasoning, knowledge update, temporal reasoning, and abstention.
The authors define each conversation as a key-value pair, where the key is the "time index" and the value is the corresponding chat.
They propose a three-stage framework for indexing, retrieval, and reading.
The framework has four control points: key (in the indexing stage), value (in the indexing stage), query (in the retrieval stage), and reading strategy (in the reading stage).
They conduct experiments to test their framework using LongMemEval on GPT-4o and Llama-3 (70B and 8B).
weaknesses
This paper has several drawbacks regarding presentation, soundness, and contribution.
TL;DR, the major weaknesses are as follows (see Details below for further elaboration):
Weakness 1. [Presentation and Soundness] This paper lacks careful proofreading, as it contains an unusually high number of typos and inconsistent writing. The numerical rounding is also confusing. For example, in the third contribution, the authors report improvement is 7% ~ 11% in Line 99, but it's 6.7% ~ 11.4% in Lines 480-481 (should report the exact value or at least add the term "around" in Line 99). Additionally, the flow in each section is not strongly connected, requiring significant revision to reach a publishable standard. Many tables and figures need substantial revisions and adjustments. The authors also need to be crisp in their writing. For one thing, in the main content, the "result and discussion" of their proposed framework only spans two pages (Sections 5.2 to 5.5), including two tables and one figure. Without these, the text is roughly a page only in the main paper, which is insufficient for the ICLR community.
Weakness 2. [Presentation and Soundness] The LongMemEval dataset should be tested on other papers' existing baselines (with external systems like RAG). However, no "baseline section" is mentioned in this paper, and I cannot find the baseline mentioned in the "experimental setup" in Section 5.1 as well as in Tables 2 and 3 and their titles (e.g., "K = V is the baseline from xxx paper"). Hence, it is challenging to understand the prior works' baselines based on each individual's assumption. If non-familiar readers try to search for the "baseline" keyword, they will find it only appears once in the second contribution (in Line 95). After jumping to Section 5.3, they cannot find the recent papers conducting LLM with RAG baselines within this section either. If no comparable baselines are suitable for this task, see S2 below for suggestions.
Weakness 3. [Presentation and Soundness] The authors claim this paper presents a unified framework, as shown in Figure 5, but without a formal mathematical definition, it is difficult to justify whether this is indeed true. For example, how does the proposed framework encompass the existing RAG frameworks? Moreover, it is also hard for other future researchers to follow/revise/expand their ongoing work, particularly the four control points (CP) in Figure 5. Take the conducted CoN experiment (Section 5.5) as an example: The connection between the case when CoN is applied and Figure 5 is unclear. This figure is not illuminating, and the imprecise definition will let the readers interpret this paper on their own. Note that each separated component can be visualized and formalized in a better way (see Figure 1 in [0] and [1] for visualization; see also Section 3.1 in [1] for formalization). In addition, this paper only displays general terms without an explicit example to showcase their framework in Figure 5, whereas it is common for various RAG papers (see [2] and [3]).
AdanCA: Neural Cellular Automata As Adaptors For More Robust Vision Transformer
The authors have addressed most of the limitations of their approach, as shown in the Limitations section (Section 5) of the main manuscript and across various points of the main manuscript and appendix. I would like to point out that the potential negative impacts of their approach have not been considered, and so I recommend they comment on this in Section 6 (Broader Impact), even if there are none.
Reviewer h6zC · 2024-07-12
limitations
The authors have addressed most of the limitations of their approach, as shown in the Limitations section (Section 5) of the main manuscript and across various points of the main manuscript and appendix. I would like to point out that the potential negative impacts of their approach have not been considered, and so I recommend they comment on this in Section 6 (Broader Impact), even if there are none.
questions
Competition Dynamics Shape Algorithmic Phases of In-Context Learning
I collect various questions, minor concerns, or suggestions that occurred to me while reading the paper. Given the length of my review I don't expect the authors to respond to all of these questions (though I would be happy for them to do so). If the authors are interested in me revising my decision and have limited capacity in the discussion period I would recommend that they engage with me on the three weaknesses before the contents of this section. More importantly I hope that they might consider my questions and suggestions and consider revising the paper to improve the presentation as they find most appropriate. **Q1. The four algorithms have unclear and misleading names.** As I said, I am a believer in the importance of names. I felt strongly that the choice to use '-ICL' as a suffix in the names of 'Uni-ICL' and 'Bi-ICL' is a mistake that undermines the quality of the paper. By selecting these names the authors have at the same time created the following two problems. 1. They have conflated one half of their list of algorithms with the concept of in-context learning, entrenching a false connotation that these two algorithms are more exemplary instances of ICL algorithms than others, in fact that the Bayesian solutions are not classified as ICL. This contradicts the authors' stated message in the abstract and conclusion, that ICL is an umbrella concept that encompasses multiple concrete algorithmic instantiations. Recalling again the paper's title, all four phases are supposedly 'of in-context learning'. I do not think that the authors mean to hold up Uni-ICL and Bi-ICL as 'truer' examples of ICL than the 'Bayesian' algorithms, yet this is what their naming choice achieves. 2. They have missed an opportunity to communicate what is unique about these particular ICL algorithms. This is the role played by 'Bayes' in the names 'Uni-Bayes' and 'Bi-Bayes' (in my reading this helpfully conveys that the methods use Bayesian averaging). I leave it to the authors to decide what would be an appropriate analogue of 'Bayes', but I invite them to consider using the term 'frequencies' or 'induction' and I urge them to avoid anchoring on induction *heads* themselves (a feature of transformers rather than ideal algorithms) or using the misleading 'non-Bayesian' terminology of Raventos et al. I respect the right of the authors to name the algorithms. I can't say this concern alone would prevent me from recommending the paper's acceptance. However, in this case I feel strongly enough to register my protestation about the names given my fresh perspective on the algorithms and the authors' chosen takeaway message. **Q2. Inconsistent summary of algorithms between main text and figure 4 caption:** In the figure caption, the Uni-Bayes and Bi-Bayes descriptions talk about selecting a 'closest task' from the mixture. Based on the main text, my understanding is that they do not select a single task but rather they form a posterior distribution over all tasks and use the posterior predictive distribution to make their prediction. Using the 'closest distribution' sounds more akin to using a maximum likelihood distribution, rather than using the posterior predictive distribution. **Q3. Questionable choice to use a nonuniform task prior:** If I understand correctly from the setting description, the authors sample a prior vector from a uniform Dirichlet distribution over task priors. The resulting prior will be almost certainly non-uniform and with high likelihood for high task diversity it will have a small number of tasks with quite large probabilities and a large number of tasks with very small probabilities. It follows that after a reasonable amount of pre-training there may be some tasks that have barely been sampled at all. This seems to me to be a significant departure from the use of task diversity by of Raventos et al., for whom, if I remember correctly, the task prior is always uniform. I believe that having a skewed task distribution may confound experiments since what I would call the 'effective task diversity', meaning roughly the number of tasks the transformer has to 'memorise' in order to get good performance on in-distribution evaluation (assuming tasks are sampled from the same skewed prior for ID evaluation) will be smaller than the specified task diversity, since the transformer can get away with not remembering low-probability tasks. The inclusion of this additional complication appears to be unjustified by any particular argument in the manuscript. I would be curious if the authors have a strong reason for including this detail. Of course, it is a virtue of the setting that one can consider different distributions of tasks, since this might be an interesting direction for future work, but such work would surely involve *systematically* varying the prior rather than abdicating control over the prior by sampling it from a high-dimensional Dirichlet distribution. Finally I note that it is regrettable that this detail appears to be documented only in the appendix. **Q4. Convoluted phase isolation tests:** These test seems very intricate. It is not clear to me that they are the clearest ways of isolating phases, and I wonder if you have considered and ruled out simpler alternatives. 1. For the bigram utilisation test, it occurs to me that the predictions of a model paying attention exclusively to unigram patterns would not vary much depending on the current state (at least late in the sequence). On the other hand, bigram-based models would vary their prediction based on the state. Therefore, I wonder if you have considered a simpler test of somehow quantifying uniformity in the set of rows of the revealed transition matrix? 2. For the proximity test, I wasn't able to think of a simpler test, beyond the idea that perhaps something with generalisation performance could be used. **Q5. Missing details for proximity test:** I was left wondering about some details of the proximity test. Unless I missed something, I would recommend clarifying the following points, if possible in the main text or otherwise by expanding on the 'additional details' in the appendix. 1. How is distance to a set of tasks defined? Is it the distance to the closest member of the set? 2. Does the measurement of distance to the training task set account for the task prior at all? I am concerned that for example if the closest task in the training set happens to be a task with very low prior probability, then this task will not draw the model's posterior towards that closest task, and the posterior may be more likely to be falsely detected as closer to a random task than if the closest training task happened to be one with higher prior probability. 3. How is the chain that is not part of the training set or the control set sampled? 4. I think I can guess how you turn the procedure you outlined into a single number used to colour your phase diagram, perhaps it involves repeating this procedure several times, and estimating an empirical probability of the closest task being from the training set, giving a number between 0 and 1. If this is correct, I think it is worth spelling out in the text, as well as noting somewhere how many trials you take. If this is wrong then it's definitely worth spelling out in the text. **Q6. Missing details about figure colour schemes:** The phase isolation methodology and the LCA analysis are two distinct methods for colouring a point on a phase diagram. I realised that it is not always clear which of these methods you are using in each of the diagrams. It is clear in Figure 4 and figure 5 where these techniques are explicated. For the remaining figures, I am not sure which methodology you use, and I couldn't find it documented anywhere. Please consider clarifying this in each figure's caption. **Q7. Why is LCA formulated in terms of L2 distance for probabilities?** It would seem more natural to minimise cross entropy or KL as is used elsewhere in the paper, and this would allow a clearer comparison to other quantities such as model KL vs. LCA KL and so on. I don't think this is necessarily a major issue but I just wondered if the authors had a good reason for it. **Q8. What is the relationship between LCA and delta metrics from Raventos et al.?** Raventos et al. consider two setting-specific metrics, they denote them 'delta ridge' and 'delta dMMSE', measuring the L2 distance between the predictions of their pre-trained transformer and those of their idealised linear regression algorithms (ridge regression and dMMSE). These are essentially measures of how close in function space the transformer is to one of the algorithms. Have you thought about the relationship between these metrics and LCA? **Q9. How close is the LCA fit?** LCA weights are defined via a least squares optimisation problem. In figures 6 and 7 the authors plot the argmin weights. What is the min? In other words, how large is the irreducible component of the least squares loss representing the distance of the transformer from the simplex spanned by the four algorithms in function space? It is important that this metric remains low in order to believe that the LCA has captured something meaningful about the behaviour, rather than a very lossy projection of the behaviour. Therefore I believe the authors should report this metric in the paper, if not in the main text then at least in an appendix. A somewhat related metric appears to be the comparison between the LCA KL and the Model KL in figure 6. However, if two models have similar KL from the ground truth sequence that does not necessarily imply that they have low KL between them. It would be informative to add the KL between the LCA and the model to the two lines in these figures. This would play a similar role to the residual. **Q10. Non-ICL algorithms early in training:** Figure 6 shows shifts in the model's development across training. One lesson from Hoogland et al. (a paper cited by the authors in the appendix) is that the choice of algorithms early in training might be even more unsophisticated than those that eventually arise at convergence for a given task diversity and sequence length configuration. For example, early in training, I hypothesise that some of these transformers might behave in a way that is well-described by an algorithm that does not involve any in-context learning at all. Some particular algorithms that I would consider searching for include the following: * **Unigram prior:** Learning the average stationary distribution from all tasks and predicting tokens based on this distribution without looking at context. * **Bigram prior:** Learning the average transition matrix and predicting based on this without looking at context. Have the authors considered adding such non-ICL algorithms into the LCA analysis? I would be curious whether doing so reduces the residual at all. **Q11. Questions about evaluation:** Two small questions about the methodology for evaluation. 1. During ID evaluation, what prior do you use for sampling the tasks from the set of tasks? Do you use the training prior or a uniform prior? You just say you 'choose one from a set.' 2. During evaluation (for both ID evaluation and OOD evaluation), how many tasks do you sample? **Q12. Sample implementation of data generating process:** Could the authors please clarify the relationship between the sample implementation of the data generating process and the actual code used in experiments? I take the description of the implementation to imply that this is not the implementation used in the experiments. This opens the possibility that it may actually differ in important ways from the actual methodology used in the experiments. There are certain details, such as the fact that the sequences drawn from each chain are initialised with a sample from the stationary distribution of the chain, do not appear to be noted in the manuscript in any form other than in the reference implementation. If the authors intend to open source their codebase after the peer review process as noted in appendix J then I wonder if they intend to keep this reference implementation in the paper? Have the authors considered mentioning any details such as the initialisation of sequences in text form as well as in code form? **Q13. Paper structure:** On first read I found it slightly difficult to follow the paper's first few sections. There is a lot going on in the paper, between the problem, the phenomena, the phases, and the explanations. I personally found understanding the phases helpful to my understanding of the remainder of the paper. I wonder if the authors have considered promoting section 4 to come before section 3? Of course, this is up to the authors. **Q14. Terminology and notation:** A small number of minor notes. 1. Have you thought about whether the 'phases' are indeed phases in the sense of physics? 2. A bold "1" is overloaded as both a vector of ones (when describing the configuration of the Dirichlet distribution) and also an indicator function (line 232). I wonder if the authors have considered for example using blackboard bold for indicators, to avoid any potential confusion, not that I think the risk of confusion is particularly severe. **Q15. Typos:** (just the ones I happened to notice): 1. Line 186/187: I think there is a stray closing parenthesis. 2. Line 422: "more experiments on this sorts". 3. Line 870--: The variables in this list look like they should be typeset in math mode. 4. Line 1638: unfinished sentence. 5. Line 2024: "BICL", is this meant to be Bi-ICL?
Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
N/A
Reviewer benk · 2024-05-21
ethics concerns details
N/A
questions to authors
The expert choice with segment level routing model (EC, segment-level in Figure 5) is barely better than the dense model (Figure 3). What is the intuition on the poor performance ?
For the first segment routing, why not use a BOS segment of 1 token length, and use that for priming the mixture weights for all subsequent generations ? Otherwise, you might be limiting the model from learning from the first segment. For example, if the model used absolute position embeddings, then if I understand correctly, the embeddings for the first segment would never get trained.
[Minor] It would be good to have the notation for e_{i} to be consistent between Equation (1), (2) and (3). Maybe consider changing Eqn (1) to be Top-k over (e_i) ?
Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images
The method in this paper can be used to plagiarize copy-righted artworks. The authors should discuss the potential impact of using this method.
Reviewer 1ZQM · 2024-05-10
ethics concerns details
The method in this paper can be used to plagiarize copy-righted artworks. The authors should discuss the potential impact of using this method.
reasons to accept
The method is technically sound and could have broad applications.
reasons to reject
This is the main problem for this paper, and important baselines are missing for a fair comparison to evaluate the effectiveness of the proposed method. This paper only chooses two off-the-shelf captioning models as baselines, while the proposed model is trained on (2M samples for fine-tuning CLIP and 300k examples for modifier classification). The is also not comparable. The proposed model uses the backbone ViT- G/14 pretrained on LAION-2b. Do the BLIP2 and CLIP Interrogator use the same backbone and pretrained on a similar amount of data? Moreover, the proposed system uses GPT-4V, which is a big advantage.
Do Membership Inference Attacks Work on Large Language Models?
- How do the authors ensure that their MIA evaluation setting (non-members from the Pile test data) does not suffer from distribution shift ? For instance, is there any reference stating that the Pile data was randomly shuffled before being split into train and test ? - Other works have already challenged the reliability of MIA predictions, see [A] and [B]. For instance, [A] shows that neighboring non-member samples would be classified as members which is related to the point made in this paper. Therefore the insight that MIAs are brittle is not new (although demonstrating it empirically in the LLM setting is). The paper does not position itself with respect to these works. Can the authors elaborate on their differences with [A] and [B] and add a discussion of previous works challenging MIA reliability ? [A] Rezaei, S., & Liu, X. (2022). On the Discredibility of Membership Inference Attacks. arXiv preprint arXiv:2212.02701. [B] Hintersdorf, D., Struppek, L., & Kersting, K. (2021). To trust or not to trust prediction scores for membership inference attacks. arXiv preprint arXiv:2111.09076. - How does the state-of-the-art MIA method of Meeus et al, 2023 perform compared to the other approaches? - Figure 1: How are the confidence intervals computed, e.g.., over how many samples? - Sec 3.2.2 – Resampling of non-members to have low overlap. This step seems over-engineered and, as the authors notice, results in distribution shift. A simpler approach would be to rank the non-members by the MIA scores and check if MIA confidence is negatively correlated with n-gram overlap. Did the authors consider this approach ? - Benchmarking of MIA approaches : Are the results statistically significant ? - ArXiv temporal shift experiment: there is a difference in the pipeline used to construct members and non-members (S3 vs arXiv API), which could introduce a distribution shift beyond the temporal shift. To substantiate the claim that « both arXiv bulk access and API should yield the preprints in the same manner regardless », can the author apply the API to retrieve some of the data before July 2020 and check that after pre-processing the documents match ? Other comments: - Please consider adding the definition of a sample (200-length documents) in the main paper. - Appendix A.4: I disagree with the statement that the reference-based attack is largely similar to the offline LiRA attack of Carlini et al., 2022. Two key features of LiRA, which don't feature in the reference-based attack, are its use of the logit loss (log(p/(1-p)) and fitting a Gaussian distribution to its distribution, which allows extrapolating to the high-confidence regime. The reference-based attack would be more similar if the log(1-p) terms were substracted from the loss. - Reference-based attack: The authors acknowledge the limitation of LLM reference models: although they are not supposed to be trained on the target sample (for the one-sided hypothesis test to be valid), they likely are. I agree with the authors that this seems hard to fix. However, because the attack’s assumptions are invalid, it’s hard to interpret why and when it works. Some analysis is provided in Appendix B.2; however it would be better to explain the first time the attack is presented (in the main paper) when the attack is likely to work in spite of the invalid assumptions and why. Perhaps this intuition can also guide the choice of the reference model. Minor : - First paragraph of Sec. 3.2.1 – the argument needs to be completed by explaining that MIA success if linked to overfitted (Yeom et al, 2018). - Figure 2 : the nuances of blue on the left are hard to distinguish in print, please consider increasing the contrast and the font size. - n-gram overlap formula: Please consider adding 1-2 sentences explaining what the formula is meant to capture. - Figure 3 – Github: is my understanding correct that on Github 40 % of the samples have 100 % n-gram overlap with the training dataset ? Please consider discussing this in the main paper and explaining that this does not mean that these non-members are members. - Please consider fully defining the loss in the main paper using the negative log-likelihood of tokens.
Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models - A Survey
1. The paper addresses a topic at the very forefront of modern AI: LLM reasoning capabilities. 2. The paper surveys an impressive number of recent papers and summarizes them clearly. 3. The paper also presents a few insights on top of presenting a summary. Insights include: focus on outcome evaluation of most of prior work and general lack of robustness of LLM reasoning.
Reviewer AG6y · 2024-05-22
reasons to accept
reasons to reject
Similar to the previous question, can the authors list the steps when computing the average metric in Lines 480-481? How can we get the 11.4% and 6.7% results from Table 3? Is there any additional rounding method involved after the computation?
Weakness 4. [Contribution] As for the dataset comparison, LongMemEval needs justification to significantly distinguish it from other related works, such as task-oriented dialogues (as stated in Line 50) and other QA-based datasets (see C2 below). Specifically, several question types are highly similar to (that is, can be leveraged by) those existing datasets, especially the single-session scenarios. Nonetheless, they are missing in the references. Hence, further analysis is necessary regarding why not to leverage their datasets.
[0] Tree of Thoughts: Deliberate Problem Solving with Large Language Models, Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan, NIPS 2023
[1] Memory-Based Model Editing at Scale, Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, Chelsea Finn, ICML 2022
[2] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, Hannaneh Hajishirzi, ICLR 2024
[3] REPLUG: Retrieval-Augmented Black-Box Language Models, Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih, NAACL 2024
Details
Presentation (P)
P1. The contexts need to be reorganized to provide a clear flow.
As shown in Figure 2, the dataset construction pipeline is (a) Question Construction $\rightarrow$ (b) Evidence Session Construction $\rightarrow$ (c) History Construction. However, the (a) and (b) steps are merged into a single "paragraph" in Line 241 (within Section 3.2: LongMemEval: Benchmark Curation), while the (c) step has its own single "subsection" (Section 3.3: History Compilation).
In the main paper, the results and discussion of their proposed framework only have two pages (Sections 5.2 to 5.5), including the two tables and a figure. More analysis should be moved to the main content. For instance, I find Table 7 in Appendix C informative, so perhaps the authors could consider moving this to the main content.
There is a discrepancy between Section 4.2 and Figure 5. In Section 4.2, the authors use "CP 1: Value" and "CP 2: Key" as the paragraph title. However, it is "CP 1: Key" and "CP 2: Value" in Figure 5.
What is NL in Figure 7? I think it is natural language (as opposed to JSON format), but "NL" is not a common abbreviation and is not mentioned in the main content either. Moreover, this figure reports the result on LongMemEval_s, yet it is not mentioned in Figure 7 and Section 5.5.
P2. The figures/tables do not follow the order or ICLR format. Many of them require adjustment in format and location in this paper for better reference.
LongMemEval challenges chat assistants through its diverse question types (a), emphases on multi-session reasoning (b),
and diverse evidence locations within sessions (c).
It should be:
LongMemEval challenges chat assistants through its (a) diverse question types, (b) emphases on multi-session reasoning,
and (c) diverse evidence locations within sessions.
P3. Too much emphasis on many terms and two (or more) adjectives to describe certain words. Specifically, the overuse of "\textit" and compound words (excluding common words in NLP like long-term). While I could understand that the authors want to emphasize the scenario they aim to test and why not conduct more experiments etc., it further adds the difficulty to grasp the main point in this paper. The texts in italics are already overwhelming even without these. Below are several sentences that may distract the readers:
We introduce LongMemEval, a comprehensive, challenging, and scalable benchmark designed to assess the long-term memory capabilities of chat assistants. LongMemEval consists of 500 human-curated, high-quality questions to test five core memory abilities: (...). Each question requires recalling information hidden within one or more multi-turn task-oriented user-AI dialogues that are LLM-simulated and human-edited. Inspired by the “needle-in-a-haystack” test (Kamradt, 2023), we design an attribute-controlled pipeline to compile a coherent, extensible, and timestamped chat history for each question.
Soundness (S)
S1. The proposed unified framework is not formalized. See Weakness 3 above.
S2. The authors do not compare their proposed framework with other similar baselines such as LLM with RAG or external systems. While the authors do report 97 data using proprietary LLMs (ChatGPT and Coze) in the pilot study, this subset of data is 5x smaller than the original size of the data (500), not to mention the conversation sessions are extremely short, which is mentioned in Lines 317-318: approximately 10x shorter than LongMemEval_s. What's more, the distribution does not even really match the original LongMemEval_s dataset (see Lines 1030-1034 in Appendix B). Is there at least an implicit baseline comparison between your and other works in this paper in some way (see Weakness 2 above)? Are the works mentioned in Line 393 the baselines?
On the other hand, if the nature of this dataset is that no prior baselines are suitable for comparison, then the authors should test their framework using more LLMs to benchmark the LongMemEval dataset for future researchers and demonstrate its effectiveness across various LLMs.
In this scenario, to strengthen the method, it would be necessary to (a) run the experiments multiple times (see [4]) or (b) enlarge the existing LongMemEval dataset. This paper only has 500 data and experiments on two LLMs in its current state: GPT-4o and Llama 3 (70B and 8B). If the budget is an issue, testing other small LLMs (e.g., below 7B) is also a welcome contribution.
S3. Regarding the soundness in Figure 4-a, the authors only test the closed-source LLMs with a memory system once in a small (97) data. While the authors already mention (multiple times) that they are interested in testing "online interactions with chatbot", this topic further narrows down to a very specific setting. Moreover, as these LLM-based memory systems do not have a snapshot (nor do they release a technical report demonstrating the robustness of their memory systems on various reasoning tasks), including this potentially immature result in the main content could be a problem for future reference (despite the evaluation time mentioned in the footnote), because these systems would be constantly improved over time without any notification, increasing the difficulty in terms of reproducibility.
S4. The dataset contains only 500, which is rather small. Moreover, many questions focus on single-turn sessions (31%; see Figure 3-a). As these settings are constantly tested in previous long-term/short-term datasets (see Contribution C2 below), the authors should create a dataset with more "multi-session, knowledge update, and temporal reasoning" for current LLMs.
S5. In Figure 1, the definition of those question types is unclear. Specifically, in Figure 1, why not treat the "knowledge-update" as a "multi-session" example, and vice versa? Moreover, after adding the time and date in each session, they can also be generalized to the "temporal-reasoning." The authors do not define these types of questions clearly and only use "The other types of questions are multi-session (MR), knowledge-update (KU), and temporal-reasoning (TR)" in Lines 236-237, and yet the distribution of question types is distinctly shown in the pie chart in Figure 3-a.
Contribution (C)
C1. Could the authors explain why it is necessary to differentiate "human-AI" and "human-human" conversations? Am I missing something in this paper? If so, could you kindly explain why you are interested in the user-AI setting and further differentiate this in Table 1? For instance, is there a significant difference between them when training an LLM or in some tasks that prefer user-AI datasets over human-human ones?
C2. Prior works of task-oriented dialogues (TOD) are missing, such as the MultiWoZ dataset [5]. On the other hand, the AirDialogue dataset [6] has a more narrower scope. As a result, there may be an issue in Lines 48-50: Many datasets focus solely on human-human conversations (Xu et al., 2022a; Maharana et al., 2024; Kim et al., 2024), while others omit task-oriented dialogues, which represent a significant portion of chat assistant usage. As for the Related Work, there is another issue in Lines 142-144 (Despite these advancements, existing QA-based benchmarks overlook several memory capabilities critical to long-term user-assistant interactions: synthesizing information across numerous sessions, recalling assistant-side information, ...) because the CoQA dataset [7] encompasses these two issues. The model needs to refer back to the conversation history. Lastly, the SituatedQA dataset [8] $-$ related to temporal context and retrieval $-$ is missing in the Related Work.
[4] Self-consistency improves chain of thought reasoning in language models, Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou, ICLR 2023
[5] MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines, Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, Jindong Chen, NLP4ConvAI 2020
[6] AirDialogue: An Environment for Goal-Oriented Dialogue Research, Wei Wei, Quoc Le, Andrew Dai, Jia Li, EMNLP 2018
[7] CoQA: A Conversational Question Answering Challenge, Siva Reddy, Danqi Chen, Christopher D. Manning, TACL 2019
[8] SituatedQA: Incorporating Extra-Linguistic Contexts into QA, Michael J.Q. Zhang, Eunsol Choi, EMNLP 2021
I would like to see my questions and suggestions above addressed to the best of the authors' abilities. My largest concerns lie with the lack of a clear and detailed comparison with the only other NCA model that operates under a ViT setting (ViTCA from Tesfaldet et al.), the relatively insufficient evidence of the usefulness of Dynamic Interaction and its multi-scale counterpart, and the lack of commentary on the ablation study.
strengths
I just want to say that my favourite part of the experimentation in the main manuscript and the appendix was the layer similarity analyses. Everything from the motivation for focusing on layer similarity, to introducing the Set Cohesion Index, to the figures showing the similarity structure before and after AdaNCA is applied, to the plots and commentary proving a relationship between layer redundancy and robustness, to the dynamic programming algorithm for placing AdaNCA in the best spot based on this proven relationship, and to comparing it with a no-prior approach, were all well done. Also, I'd like to commend the authors for providing the following:
summary
This paper proposes a strategy for improving the image classification robustness of Vision Transformers (ViT) through the use of specialized networks that are inserted at strategically placed layers within the ViT model. These networks are called Adapter Neural Cellular Automata (AdaNCA) and are intentionally chosen due to the proven robustness characteristics of NCA on various tasks such as image generation and classification. Although there has already been a connection made between NCA and ViT models (ViTCA from Tesfaldet et al.), AdaNCA differentiates itself by not trying to be a ViT in and of itself (as ViTCA does), but by acting as an adapter that can be placed at various layers within a ViT to improve its robustness. Through exhaustive experimentation, the authors prove the viability of using NCA in a much larger scale setting than ever before (as far as I know) through this adapter-type approach, thus providing a new pathway for the NCA community to consider when it comes to practical applications. Another difference between ViTCA and AdaNCA worth mentioning is the downstream task at hand: small-scale image denoising vs. large(r)-scale image classification.
A quick summary of what NCA are: they're a fairly recent (circa 2019-20) computational paradigm that build upon the much older model of Cellular Automata (CA). Both NCA and CA consist of a connected lattice of stateful cells whose states are recurrently updated through the repeated application of an update rule. The update rule consists of two stages: an interaction stage, where for each cell, information is gathered from its neighbouring cells; and an update stage, where this information is processed to produce a cell update, which can be applied in a residual manner or be directly treated as the new cell state. CA use a handcrafted update rule, with a popular one being Conway's Game of Life, while NCA use a learned update rule in the form of a neural net with convolutions in the interaction stage and an MLP in the update stage. The NCA update rule is trained via some downstream task, where the cell grid / lattice is evaluated against some target state after a certain number of cell updates. Due to the way NCA are trained (stochastic application of cell updates, repeated evaluation against target state at various points of cell lifetime via pool-based training, hidden states to facilitate cell message passing, etc.), they end up being fairly robust models, able to correct themselves in the presence of adversarial attacks (e.g., various types of structured or unstructured noise) and adapt to OOD situations.
The three main contributions of this paper are as follows:
There are some smaller things I'm leaving out of this summary due to how extensive the paper is in some areas, but this is the overall approach.
weaknesses
Reviewer medY · 2024-11-02
questions
I collect various questions, minor concerns, or suggestions that occurred to
me while reading the paper. Given the length of my review I don't expect the
authors to respond to all of these questions (though I would be happy for
them to do so). If the authors are interested in me revising my decision and
have limited capacity in the discussion period I would recommend that they
engage with me on the three weaknesses before the contents of this section.
More importantly I hope that they might consider my questions and suggestions
and consider revising the paper to improve the presentation as they find most
appropriate.
Q1. The four algorithms have unclear and misleading names.
As I said, I am a believer in the importance of names. I felt strongly that
the choice to use '-ICL' as a suffix in the names of 'Uni-ICL' and 'Bi-ICL'
is a mistake that undermines the quality of the paper. By selecting these
names the authors have at the same time created the following two problems.
They have conflated one half of their list of algorithms with the concept
of in-context learning, entrenching a false connotation that these two
algorithms are more exemplary instances of ICL algorithms than others, in
fact that the Bayesian solutions are not classified as ICL.
This contradicts the authors' stated message in the abstract and
conclusion, that ICL is an umbrella concept that encompasses multiple
concrete algorithmic instantiations. Recalling again the paper's title,
all four phases are supposedly 'of in-context learning'.
I do not think that the authors mean to hold up Uni-ICL and Bi-ICL as
'truer' examples of ICL than the 'Bayesian' algorithms, yet this is what
their naming choice achieves.
They have missed an opportunity to communicate what is unique about these
particular ICL algorithms. This is the role played by 'Bayes' in the names
'Uni-Bayes' and 'Bi-Bayes' (in my reading this helpfully conveys that the
methods use Bayesian averaging). I leave it to the authors to decide what
would be an appropriate analogue of 'Bayes', but I invite them to consider
using the term 'frequencies' or 'induction' and I urge them to avoid
anchoring on induction heads themselves (a feature of transformers
rather than ideal algorithms) or using the misleading 'non-Bayesian'
terminology of Raventos et al.
I respect the right of the authors to name the algorithms. I can't say this
concern alone would prevent me from recommending the paper's acceptance.
However, in this case I feel strongly enough to register my protestation
about the names given my fresh perspective on the algorithms and the authors'
chosen takeaway message.
Q2. Inconsistent summary of algorithms between main text and figure 4
caption: In the figure caption, the Uni-Bayes and Bi-Bayes descriptions
talk about selecting a 'closest task' from the mixture. Based on the main
text, my understanding is that they do not select a single task but rather
they form a posterior distribution over all tasks and use the posterior
predictive distribution to make their prediction. Using the 'closest
distribution' sounds more akin to using a maximum likelihood distribution,
rather than using the posterior predictive distribution.
Q3. Questionable choice to use a nonuniform task prior: If I understand
correctly from the setting description, the authors sample a prior vector
from a uniform Dirichlet distribution over task priors. The resulting prior
will be almost certainly non-uniform and with high likelihood for high task
diversity it will have a small number of tasks with quite large
probabilities and a large number of tasks with very small probabilities. It
follows that after a reasonable amount of pre-training there may be some
tasks that have barely been sampled at all.
This seems to me to be a significant departure from the use of task diversity
by of Raventos et al., for whom, if I remember correctly, the task prior is
always uniform. I believe that having a skewed task distribution may confound
experiments since what I would call the 'effective task diversity', meaning
roughly the number of tasks the transformer has to 'memorise' in order to get
good performance on in-distribution evaluation (assuming tasks are sampled
from the same skewed prior for ID evaluation) will be smaller than the
specified task diversity, since the transformer can get away with not
remembering low-probability tasks.
The inclusion of this additional complication appears to be unjustified by
any particular argument in the manuscript. I would be curious if the authors
have a strong reason for including this detail. Of course, it is a virtue of
the setting that one can consider different distributions of tasks, since
this might be an interesting direction for future work, but such work would
surely involve systematically varying the prior rather than abdicating
control over the prior by sampling it from a high-dimensional Dirichlet
distribution.
Finally I note that it is regrettable that this detail appears to be
documented only in the appendix.
Q4. Convoluted phase isolation tests: These test seems very intricate. It
is not clear to me that they are the clearest ways of isolating phases, and I
wonder if you have considered and ruled out simpler alternatives.
Q5. Missing details for proximity test: I was left wondering about some
details of the proximity test. Unless I missed something, I would recommend
clarifying the following points, if possible in the main text or otherwise by
expanding on the 'additional details' in the appendix.
Q6. Missing details about figure colour schemes:
The phase isolation methodology and the LCA analysis are two distinct methods
for colouring a point on a phase diagram. I realised that it is not always
clear which of these methods you are using in each of the diagrams. It is
clear in Figure 4 and figure 5 where these techniques are explicated. For the
remaining figures, I am not sure which methodology you use, and I couldn't
find it documented anywhere. Please consider clarifying this in each figure's
caption.
Q7. Why is LCA formulated in terms of L2 distance for probabilities? It
would seem more natural to minimise cross entropy or KL as is used elsewhere
in the paper, and this would allow a clearer comparison to other quantities
such as model KL vs. LCA KL and so on. I don't think this is necessarily a
major issue but I just wondered if the authors had a good reason for it.
Q8. What is the relationship between LCA and delta metrics from Raventos et
al.? Raventos et al. consider two setting-specific metrics, they denote
them 'delta ridge' and 'delta dMMSE', measuring the L2 distance between the
predictions of their pre-trained transformer and those of their idealised
linear regression algorithms (ridge regression and dMMSE). These are
essentially measures of how close in function space the transformer is to one
of the algorithms. Have you thought about the relationship between these
metrics and LCA?
Q9. How close is the LCA fit? LCA weights are defined via a least squares
optimisation problem. In figures 6 and 7 the authors plot the argmin weights.
What is the min? In other words, how large is the irreducible component of
the least squares loss representing the distance of the transformer from the
simplex spanned by the four algorithms in function space?
It is important that this metric remains low in order to believe that the LCA
has captured something meaningful about the behaviour, rather than a very
lossy projection of the behaviour. Therefore I believe the authors should
report this metric in the paper, if not in the main text then at least in an
appendix.
A somewhat related metric appears to be the comparison between the LCA KL and
the Model KL in figure 6. However, if two models have similar KL from the
ground truth sequence that does not necessarily imply that they have low KL
between them. It would be informative to add the KL between the LCA and the
model to the two lines in these figures. This would play a similar role to
the residual.
Q10. Non-ICL algorithms early in training: Figure 6 shows shifts in the
model's development across training. One lesson from Hoogland et al. (a paper
cited by the authors in the appendix) is that the choice of algorithms early
in training might be even more unsophisticated than those that eventually
arise at convergence for a given task diversity and sequence length
configuration.
For example, early in training, I hypothesise that some of these transformers
might behave in a way that is well-described by an algorithm that does not
involve any in-context learning at all. Some particular algorithms that I
would consider searching for include the following:
Have the authors considered adding such non-ICL algorithms into the LCA
analysis? I would be curious whether doing so reduces the residual at all.
Q11. Questions about evaluation: Two small questions about the methodology
for evaluation.
Q12. Sample implementation of data generating process:
Could the authors please clarify the relationship between the sample
implementation of the data generating process and the actual code used in
experiments?
I take the description of the implementation to imply that this is not the
implementation used in the experiments. This opens the possibility that it
may actually differ in important ways from the actual methodology used in the
experiments. There are certain details, such as the fact that the sequences
drawn from each chain are initialised with a sample from the stationary
distribution of the chain, do not appear to be noted in the
manuscript in any form other than in the reference implementation.
If the authors intend to open source their codebase after the peer review
process as noted in appendix J then I wonder if they intend to keep this
reference implementation in the paper?
Have the authors considered mentioning any details such as the initialisation
of sequences in text form as well as in code form?
Q13. Paper structure: On first read I found it slightly difficult to
follow the paper's first few sections. There is a lot going on in the paper,
between the problem, the phenomena, the phases, and the explanations.
I personally found understanding the phases helpful to my understanding of
the remainder of the paper. I wonder if the authors have considered promoting
section 4 to come before section 3? Of course, this is up to the authors.
Q14. Terminology and notation: A small number of minor notes.
Q15. Typos: (just the ones I happened to notice):
strengths
As I said I think this is a strong paper. I note at least the following
strengths.
The paper is well-motivated by the importance of of understanding the
emergence of algorithmic structure inside transformer sequence models,
which is a priority for the science of deep learning.
To this end the authors contribute a neat setting with a clean and flexible
data generating process and an interesting and rich collection of idealised
solutions.
The authors have also conducted a comprehensive study of the numerous axes
of variation in this setting.
The LCA technique, carefully interpreted, is an elegant idea for
behavioural analysis of models in general, when plausible candidate
algorithms are known. I like it and I think I would use it (I have noticed
several opportunities to use this in my research since seeing it in this
paper.)
Overall I think the paper makes a valuable contribution that enriches our
understanding of ICL phenomenology and creates a rich framework for future
research that can continue exploring this important topic.
summary
With this paper the authors contribute the following:
The authors propose a new sequence modelling setting, namely learning to
predict sequences sampled from a Markov chain sampled from a finite mixture
of Markov chains.
The authors list four idealised algorithms for solving this sequence
modelling setting, along two axes of variation, namely:
The authors devise a pair of tests for distinguishing a learned predictor
along these two axes, and in doing so reveal that depending on the degree
of task diversity, the sequence length, and the number of training steps,
small transformers trained for this task will behaviourally resemble each
of these four algorithms. This creates the titular 'algorithmic phases'.
The authors propose a behavioural analysis method, "Linear Combination of
Algorithms (LCA)" whereby they decompose a transformer's outputs (given
in-distribution inputs) into a mixture of the outputs of the four idealised
algorithms. The decomposition is made by projecting the outputs onto the
probability simplex spanned by the four algorithms in function space.
The authors track this decomposition over training and observe a close
alignment between trends in the weight of certain algorithms and trends in
out-of-distribution generalisation including the "transience" phenomenon.
This phase isolation methodology and auxiliary metrics replicates a number
of phenomena reported in prior works on ICL in similar and disparate
settings, and some new phenomena. In the main text, the emphasis is on the
following three phenomena:
weaknesses
W1. Insufficient evidence for confidence in algorithmic phase identification
The authors chose to name their paper 'algorithmic phases of in-context
learning'. I am a believer in the importance of names and I think this gives
me grounds to assume that the authors view their labelled phase diagrams as a
major element of their contribution.
I consider this labelled phase diagram to have two distinct parts. The first
of which is the division of the hyperparameter space into regions where a
trained transformer displays distinct modes of behaviour as quantified by the
behavioural classification metrics outlined in section 4.2 'isolating
algorithmic phases'. I am impressed by this part of the contribution.
I am concerned about the second part, namely the labelling of phases with
specific algorithms. The authors confidently association of each of the four
behavioural phases with one of the four algorithms described in section 4.1
'the Bayesian and non-Bayesian solutions of finite Markov mixtures'. I am
concerned that the authors have not secured sufficient ground from which to
confidently present these algorithms as uniquely accurate descriptions of the
transformer's behaviour in each phase.
Phase isolation is not exhaustive:
The phase identification methodology appears to assume a priori that these
are the only plausible algorithms, as if that were the case, then it would be
sufficient to associate these algorithms to each phase. However, these
algorithms do not appear to me to be uniquely principled ideal models of
how a sequence model might solve the Markovian prediction task, so I do not
think it appropriate to rule out a priori the many other possible
algorithms that could explain similar behavioural patterns and would be a
better 'label' for each phase.
A short list of other algorithms I can think of is as follows:
Unigram- or Bigram-likelihood Bayesian Averaging with a different
prior: An algorithm that functions like Uni-Bayes or Bi-Bayes, but uses a
different task prior probability vector as a starting point for formulating
its posterior predictive distribution.
reasons to accept
The proposed method allows the previously proposed SMEAR approach to be used for autoregressive modeling with the segment based routing approach, potentially allowing for fully differentiable MoE models to be scaled.
The discussions on parallelism, especially Appendix H is very informative, and in my opinion forms for a good discussion point for scaling fully differentiable MoE models compared to the hard decision based token routed MoE models
The authors demonstrate the utility of ICL pre-training for MoE model training, showing improved gains compared to random batching. While the results are preliminary, that forms for a good avenue of exploration for training MoE models
reasons to reject
My concerns with the paper are as follows:
1.1 Controlling for prompt-length: By averaging the hidden representations for generating the FFN weights, intuitively there would be a diffusion of information. If that is indeed the case, then a generative needle in a haystack ([1]) or a multi-key NIAH ([2]) would be interesting to test if this diffusion indeed causes issues in learning the routing weights
1.2 Another controlled experiment that can help measure this is via perplexity measures: specifically, by fixing a prompt length and completion length to a fixed value, and then measuring the perplexity on the completion as a function of the segment length for a trained model. The difference in perplexity should be a good indication on how much impact does the train-test mismatch actually have.
1.3 Another test that is important to understand the (potential) limitations for this approach would be to have generative tasks that are long (i.e the generations are >> segment length). Intuitively, with a larger generation, the model should progressively adapt to use different experts based on the prompt + generation content, but that would not happen based on the proposed inference methodology. It would be good to quantify this limitation if possible.
2.1 If the hypothesis is that token based routing requires advanced training techniques like Expert Choice to be better than the proposed method, it would be good to demonstrate the proposed methods superiority compared to a vanilla top-k based routing approach
2.2 If the hypothesis is that the proposed routing achieves better OOD generalization because of expert specialization, it would be good to demonstrate that. Concretely, in the current paper, it is hard to disentangle if the expert specialization comes from the sim-batch data approach or from the proposed routing (or rather, if the token routed models were also trained in the sim-batch setup, would they also result in expert specialization).
[1] Kamradt, G. "Needle in a Haystack–pressure testing LLMs." (2023).
[2] Hsieh, Cheng-Ping, et al. "RULER: What's the Real Context Size of Your Long-Context Language Models?." arXiv preprint arXiv:2404.06654 (2024).
summary
1.1 A segment level routing tasks that computes the expert mixing weights for a segment based on the (aggregate) hidden representation from the previous segment, and uses that to softly mix the experts in parameter space
1.2 Leveraging similarity based data batching, similar to In-context pretraining [1] to avoid expert under-utilization, that subsequently results in better domain specialization of experts
The authors demonstrate superior performance on both perplexity as well as downstream tasks compared to an (active) parameter matched dense model setup across a series of compute budgets. They also demonstrate competitive performance compared to token level hard-routing baselines (Expert Choice [2])
Furthermore, they also demonstrate that the proposed use of similarity based batching benefits MoE training more than it's random sampling counterpart, and that strong performance gains are observed even for dense models (with the dense sim-batch model outperforming the MoE random batch baseline).
The paper also demonstrates that the proposed method achieves good domain level expert specialization without any domain level supervision.
[1] Shi, Weijia, et al. "In-Context Pretraining: Language Modeling Beyond Document Boundaries." arXiv preprint arXiv:2310.10638 (2023).
[2] Zhou, Yanqi, et al. "Mixture-of-experts with expert choice routing." Advances in Neural Information Processing Systems 35 (2022): 7103-7114.
To make a fair comparison, the following baselines are suggested:
This is also important to evaluate the effectiveness of the model design. The proposed pipeline involves multiple steps: fine-tuning, multi-label classifier, and GPT-4V to summarize based on the entities, keywords, etc. Are all those steps necessary? Which step is more important to the final performance? This version of the paper did not carefully ablate those components. The performance gain over the baselines could be just because of using more data and a bigger model, as mentioned before.
Overall, this paper has problems in baseline comparison and lacks ablations, making it difficult to evaluate the method's effectiveness.
summary
This paper proposed a method to recover the prompt used to generate the input image and use the prompt the reproduce the image. The method is basically a captioning system consisting of (1) a fine-tuned CLIP model for constructing a pool of image and text embeddings, which will be used to retrieve relevant information for the input image, (2) a multilabel classifier to predict the modifiers given the image, and (3) use GPT-4V to generate and refine the final prompt given the information from (1) and (2) with the image to generate the prompt. The proposed method performs better than two off-the-shelf captioning models.
Reviewer sAnK · 2024-04-28
questions to authors
How do the authors ensure that their MIA evaluation setting (non-members from the Pile test data) does not suffer from distribution shift ? For instance, is there any reference stating that the Pile data was randomly shuffled before being split into train and test ?
Other works have already challenged the reliability of MIA predictions, see [A] and [B]. For instance, [A] shows that neighboring non-member samples would be classified as members which is related to the point made in this paper. Therefore the insight that MIAs are brittle is not new (although demonstrating it empirically in the LLM setting is). The paper does not position itself with respect to these works. Can the authors elaborate on their differences with [A] and [B] and add a discussion of previous works challenging MIA reliability ?
[A] Rezaei, S., & Liu, X. (2022). On the Discredibility of Membership Inference Attacks. arXiv preprint arXiv:2212.02701.
[B] Hintersdorf, D., Struppek, L., & Kersting, K. (2021). To trust or not to trust prediction scores for membership inference attacks. arXiv preprint arXiv:2111.09076.
How does the state-of-the-art MIA method of Meeus et al, 2023 perform compared to the other approaches?
Figure 1: How are the confidence intervals computed, e.g.., over how many samples?
Sec 3.2.2 – Resampling of non-members to have low overlap. This step seems over-engineered and, as the authors notice, results in distribution shift. A simpler approach would be to rank the non-members by the MIA scores and check if MIA confidence is negatively correlated with n-gram overlap. Did the authors consider this approach ?
Benchmarking of MIA approaches : Are the results statistically significant ?
ArXiv temporal shift experiment: there is a difference in the pipeline used to construct members and non-members (S3 vs arXiv API), which could introduce a distribution shift beyond the temporal shift. To substantiate the claim that « both arXiv bulk access and API should yield the preprints in the same manner regardless », can the author apply the API to retrieve some of the data before July 2020 and check that after pre-processing the documents match ?
Other comments:
Please consider adding the definition of a sample (200-length documents) in the main paper.
Appendix A.4: I disagree with the statement that the reference-based attack is largely similar to the offline LiRA attack of Carlini et al., 2022. Two key features of LiRA, which don't feature in the reference-based attack, are its use of the logit loss (log(p/(1-p)) and fitting a Gaussian distribution to its distribution, which allows extrapolating to the high-confidence regime. The reference-based attack would be more similar if the log(1-p) terms were substracted from the loss.
Reference-based attack: The authors acknowledge the limitation of LLM reference models: although they are not supposed to be trained on the target sample (for the one-sided hypothesis test to be valid), they likely are. I agree with the authors that this seems hard to fix. However, because the attack’s assumptions are invalid, it’s hard to interpret why and when it works. Some analysis is provided in Appendix B.2; however it would be better to explain the first time the attack is presented (in the main paper) when the attack is likely to work in spite of the invalid assumptions and why. Perhaps this intuition can also guide the choice of the reference model.
Minor :
First paragraph of Sec. 3.2.1 – the argument needs to be completed by explaining that MIA success if linked to overfitted (Yeom et al, 2018).
Figure 2 : the nuances of blue on the left are hard to distinguish in print, please consider increasing the contrast and the font size.
n-gram overlap formula: Please consider adding 1-2 sentences explaining what the formula is meant to capture.
Figure 3 – Github: is my understanding correct that on Github 40 % of the samples have 100 % n-gram overlap with the training dataset ? Please consider discussing this in the main paper and explaining that this does not mean that these non-members are members.
Please consider fully defining the loss in the main paper using the negative log-likelihood of tokens.
reasons to accept
The authors perform an extensive evaluation of several MIA approaches applied to LLMs, on several datasets and models. I think that this benchmarking of methods is valuable for the community. The authors also make their code available.
The evaluation reveals the intringuing finding that current MIA perform quite poorly. As this finding directly challenges results from some of the previous works, and a plausible explanation is offerred, I believe that publishing the paper can be beneficial to the community.
An important finding is that the MIA decision boundary is not robust to small changes in the text. The authors show that a slightly modified version of a member text that would be considered as a member by a human (but is technically a non-member as per the strict MIA definition) would be classified as a non-member. This challenges the current definition of MIAs and could motivate interesting developments in the field.
reasons to reject
Potential methodological flaw: the Reference-based and min-k prob attacks seem to be tuned on the test data. I re-read Appendices A.3.1. and A.4 several times looking for a mention of validation data and I found only one, for the DATABLATIONS model, while the Reference-based tuning is done on PYTHIA-DEDUP models (for which there is no mention of validation data). Tuning the methods on the same data on which final performances are reported is a methodological flaw because it biases the results. It also gives an unfair advantage to the methods being tuned compared the ones which are not tuned. This is especially problematic here since the extensively tuned Reference-based attack is found to perform « best » in the end. This can be fixed by using a validation dataset for tuning the methods which is independent from the test set.
Unclear exactly how the author’s analysis of temporal shift relates to the previous works of Shi et al, 2023 and Meeus et al, 2023. The paper should clarify whether the same experiment settings (LLMs, data domains, temporal ranges) as the two papers are reproduced, or if the author only speculate that these two papers suffer from temporal shift. Making this very precise is needed to support the abstract claim that « this apparent success can be attributed to a distribution shift », else the claim should be more nuanced. For instance, going through these two papers, I understand that in the arXiv domain, Shi et al uses members and non-members from before 2016 and after 2023, respectively, Meeus et al uses members before February 2023 and non-members after February 2023, and the current paper uses members from July 2020 and non-members from later months. Since the temporal shift is slow (Figure 6), it likely affects the former paper more than the latter. Furthermore, Meeus et al. also uses a second dataset where there is no temporal shift and uses longer documents for which MIA risk is likely to be higher. These differences are not discussed in the paper.
Limited evaluation: the paper focuses on 200-word texts. When it comes to copyright infringment, longer documents such as news articles and books pose a bigger concern. As mentioned by the authors and shown by Shi et al, 2023, MIA risk increases with text length. Therefore, it is likely that MIAs might still work for, e.g., books. If this is true, the paper’s core claim would not hold. There is no discussion of this in the paper.
A point is made that the MIA definition is not aligned with how we reason about privacy, but no better definition is proposed. The link between this part and the rest of the paper is not very clear. The usefulness of experimenting with large edit distances of 10 and 25 is unclear to me: if a large fraction of the text is replaced with random tokens, would the text still be considered to be a member by a human? Even for an edit distance of 1, replacing a token with a random token does not guarantee that the text still makes sense (which would be necessary in any notion of neighborhood). The empirical part of this section seems insufficiently developed.
summary
This paper questions the effectiveness of MIA approaches which are currently used to evaluate the leakage of large langage model (LLM) pre-training data. The authors evaluate several MIA methods from previous works. They find that the performance of MIAs is quite poor on several LLMs trained on the Pile when evaluated on subsets of the Pile from different domains. This finding seems to contradict previous works which reported much better MIA performance. The authors explain this difference by arguing that the previous works used member and non-member data from different time ranges, resulting in distribution shift of non-members with respect to members. Therefore, previous MIAs likely performed distribution inference rather than membership inference, with distribution inference being an easier task. The authors also question the relevance of the task’s definition for natural lanauage where it is possible to construct different but similar samples (which are technically non-members), but should reasonably be viewed as members. MIAs are shown not to be robust to such samples, which are wrongly classified as non-members in the high-confidence regime.
I think that this work is timely and tackles an important question, which is relevant for privacy auditing and for copyright infringment. The paper does not propose a new method, so the technical novelty is very limited. However, it performs a large-scale evaluation of several state-of-the-art MIA approaches, whose results challenge the strong performances of MIAs reported in some of the previous works (Shi et al, 2023 and Meeus et al, 2023). This is an intriguing finding and the authors propose a plausible explanation for these results. The authors further challenge the definition of MIAs in the context of LLMs, showing that MIAs are not robust to small changes in the sample.
However, the paper in its current form does not seem ready for publication, as it is imprecise on several important points neccessary to substantiate its claims and the benchmarking of MIAs seems incomplete and to suffer from a methodological flaw. Most of these points seem fixable and I hope that the authors will address them in their rebuttal.
Some claims don't feel well-supported.
I’m not sure I’m buying the distinction between reasoning behavior vs final task performance. You can still study reasoning behavior by tracking final task performance if you design your tasks really well such that they track particular kind of error or behavior. For instance, one can measure the compositionality gap [1] (the accuracy difference between two one-hop and a single two-hop question) to study a sophisticated reasoning behavior (two-hop reasoning) while relying on metrics tracking just an outcome of a well-designed task.
How do authors motivate defining core reasoning tasks a logical, mathematical or causal reasoning? Why are they non-overlapping (e.g. doesn’t math reasoning pre-supposed logical reasoning)? Why do they exhaust the space of reasoning types? Should this typology be thought as listing natural kinds, carving nature at its joints or is it just a useful albeit fuzzy set of labels that LMs might not might not respect?
Authors frequently hint at the brittleness of reasoning capabilities being a result of memorization, but don’t discuss this in detail and more rigorously. The strong reading of this claim is clearly false: LLMs do generalize to unseen problems, e.g. the gsm1k paper [2] finds that only some models (Mistral) show overfitting to gsm8k that could explain their good performance. I wonder what’s the more nuanced reading of the claim authors hint at. I’d be great to see it discussed in more detail.
The survey seems to be missing some important thread of recent work and is somewhat shallow when discussing some others.
Some more specific comments:
skeptics contend that the models’ performance is a mere reflection of the extensive training data and their vast number of parameters (Razeghi et al., 2022; Wu et al., 2023; Berglund et al., 2024).
I don’t think Berglund et al., 2024 lends evidence to this claim. The reversal curse is a failure of knowledge storage/retrieval not reasoning.
Moreover, I’d rephrase the claim to make it more precise. That reasoning capability is a result of increased scale is not a skeptical take at all. We all know that larger models and models trained on more data generally tend to get better results. Probably what the authors wanted to say is something about memorization vs generalization?
Furthermore, theoretical findings suggest that due to the models’ autoregressive nature, mistakes in early stages of the reasoning process can lead to compounding errors that exponentially diminish the likelihood of arriving at the correct solution in subsequent steps.
Citation needed. Maybe [3], section 4?
At the same time, McKenzie et al. (2023) demonstrate a form of goal misgeneralization
I’d avoid the term “goal misgeneralization” as it has its own technical meaning [4]
extensive training data, rather than genuine reasoning capabilities
This strikes me as a false dichotomy. We don’t have any reason to think that genuine reasoning capabilities cannot be learned from extensive training data. What the authors mean is perhaps memorization vs generalization again.
Similarly, McKenzie et al. (2023) indicate that LLMs tend to replicate
human-like logical errors when engaging with the logical principle of modus tollens, suggesting an imitation of flawed reasoning patterns from their training data. Notably, this trend becomes more pronounced with increasing model size, a phenomenon denoted as inverse scaling.
This is too strong Some tasks show inverse scaling, but it’s a minority and tasks in McKenzie et al. (2023) were selected for exhibiting inverse scaling. For the vast majority of reasoning tasks, performance improves with scale.
summary
The paper is a review of recent literature of evaluating LLM reasoning. Authors focus specifically three families of core reasoning tasks: mathematical, logical and causal reasoning. After reviewing recent work, they discuss general trends in reasoning evaluation, arguing for paradigms that go beyond measuring the correctness of the outcome of a reasoning process and that also evaluate the reasoning process itself. They suggest reasoning capabilities of modern LLMs are not robust yet.
The authors argue that their finite mixture of Markov chains setting
offers a unified setting in which to study the emergence of in-context
learning, which has previously been studied in disparate settings.
Summary of my review:
I was impelled to write a long review. So, I also include a summary of my
review here.
I think the authors have made a strong contribution in an important area of
the science of deep learning. Their setting is elegant, expressive, and
permits an interesting variety of idealised solutions. The results from
their phase isolation and LCA analyses are interesting and informative.
However, I think the claims made in the paper at times overstate the
results or are overly confident given the limitations of the methodology,
which are not adequately discussed. In particular:
I believe that the framing is sufficiently misleading that I cannot
recommend the paper for acceptance in its current state. However, if the
authors are able to back up their confidence or if they commit to tempering
their claims then I would be happy to recommend the paper for acceptance
because I didn't note any major technical flaws and I think the work is
important and interesting.
In addition, I note a number of additional questions and more minor
concerns including about the naming of the "-ICL" algorithms and some
details of the setting and results, detailed in the questions section of
this review.
I look forward to the discussion period.
EDIT TO ADD: Summary of discussion:
Unigram- or Bigram-likelihood MLE or MAP: An algorithm that functions
like Uni-Bayes or Bi-Bayes in weighing the likelihood of each task for
describing the sequence in front of it, but then rather than performing
Bayesian averaging to make predictions, simply predicts based on the
most likely task (possibly accounting for the task prior).
Slight modifications of frequency counting approaches: An algorithm
that functions like Uni-ICL or Bi-ICL but uses slight modifications to
frequency counting such as starting the count for each unigram from epsilon
instead of zero or similar for bigrams, or increasing the predicted
proportions non-linearly in the counts.
These algorithms are close in function space to the proposed algorithms so
that it were the case that these were more accurate descriptions of a
transformer's behaviour, this would not show up in the phase isolation
metrics (nor in the LCA analysis which is based on projections in function
space---the above algorithms would be nearby in function space and so
projections would be similar). The only ways to rule out these alternatives,
along with every other algorithm, would be to devise specific tests to probe
the behavioural differences between these algorithms and the proposed
algorithms, or to probe the internal computational structure of the
transformers to seek to distinguish the mechanisms underlying their
behaviour.
On appendix E and mechanistic analysis: I did notice that the authors
also include additional experiments in appendix E that investigate the
activation patterns in various locations inside the transformer. However,
these experiments are insufficient to rule out the alternatives to Bi-Bayes
and Uni-ICL I have listed above, since for example counting transitions is
also consistent with these alternative algorithms. There are no mechanistic
experiments into Uni-Bayes. It appears the authors have more evidence
consistent with Bi-ICL than the other algorithms including the presence of
statistical induction heads, but the broader point stands.
Overall:
To be clear, I believe that the algorithms proposed by the authors and their
experiments are novel and significant. Given the infeasibility of uniquely
associating a particular principled algorithm with a phase, it seems fine to
me to start with approximate behavioural resemblance. I think this is what
the authors have grounds to claim they have contributed---a 'fuzzy' label for
each phase that serves as an initial proxy for what the transformer is going,
and this is worth sharing with the research community.
However, I also strongly believe that the authors have an essential
responsibility to clearly articulate the status of their labels. I am
concerned that in the current form the reader will draw the as-yet
unwarranted conclusion that, for example, transformers perform Bayesian model
averaging (in the appropriate phases). While the algorithmic variations I am
proposing are also deviations from 'optimality' as the authors have framed
the problem, it seems an open question to what extent transformers implement
'optimal' algorithms and to me, modelling the specific variations from
optimality in the pre-trained transformer is an important direction for
future work and it is not yet time to declare that the algorithms have been
definitively identified even in this simple setting.
Requested revisions:
I would be welcome the authors to correct my understanding of their
methodology and its limitations, or correct my reading of their confidence.
Otherwise, until this framing issue has been addressed the paper is unfit for
publication in my judgement. I would suggest that the authors commit to the
following revisions.
The authors should amend their presentation to refrain from claiming that
the transformer implements these particular algorithms. Some examples of
language I think is unwarranted include (not exhaustive):
The authors should note prominently and explicitly in the paper (for
example when introducing their methodology, in the conclusion, or in a
dedicated 'limitations' section) that their methodology is insufficient to
rule out other algorithms as being a more accurate description of each
phase.
W2. LCA is not mechanistic and does not offer a new explanation
The proposed LCA techniques is described in the abstract, introduction and
figure 1 caption as a "mechanistic analysis" offering a "new" "causal
explanation" of transient OOD generalisation. The terms "mechanistic" and
"explanation" are repeated throughout section 5. I do not see grounds for any
of the terms "mechanistic", "causal explanation", or "new explanation" to be
used in describing this methodology.
On "mechanistic": It is inappropriate to call LCA a mechanistic
decomposition. The tool is clearly based on analysing (in-distribution)
behaviour, and it does not reliably reveal internal mechanisms (though it may
do so, or approximately so, in this case). Accordingly, this analysis should
be called "behavioural" rather than "mechanistic".
I suppose the authors are using the word "mechanistic" because they believe
that LCA has identified the algorithm implemented by the transformer, and
that this is confirmed by the fact that the in-distribution behavioural
decomposition is predictive of out-of-distribution behaviour. It may very
well be the case that the transformer implements a coherent mechanism and
that LCA has, in this case, identified it (or approximately identified it).
However, LCA is incapable of revealing the mechanistic nature of a general
transformer, and so this reasoning is not sound.
As I argue in the previous section I do not believe they have grounds for
this, and have only roughly categorised the behaviour rather than
identified the precise algorithm. LCA will find a decomposition in terms
of any list of algorithms given to it. If the true mechanisms inside the
transformer are not represented, it will identify the behaviourally
closest algorithms as comprising the behaviour.
LCA actually makes it clear that the transformer in some sense implements
multiple algorithms, not one algorithm, at many points during training,
since the mixtures are not concentrated. While the authors call this a
'competition' between algorithms, they have not given any mechanistic
model of this apparent competition. There are many possible internal
mechanisms that would lead to such an apparent 'competition' appearing in
the LCA decomposition, some of which would not even correspond to a proper
competition but would rather be artefacts of the projection in function
space.
In this case that the transformer's (lack of) reliance on pre-training
task statistics is recoverable from in-distribution behaviour and then
this prediction is validated on out of distribution behaviour, this does
indicate that the tool happens to have uncovered part of the transformer's
mechanism in this case. However, LCA would make the same predictions for a
transformer constructed or trained to follow one mechanism in-distribution
while following another mechanism out of distribution.
Because it only measures behaviour, LCA should not be called mechanistic. The
predictive power in this case is not coming from LCA but rather from the
apparent fact that the transformer happens to have roughly the mechanism the
authors expect.
On "new explanation": The authors also claim that tracking the LCA
decomposition over training reveals an explanation of the phenomenon of the
transience of out-of-distribution generalisation posed by Singh et al. This
explanation is described in the abstract as a "new insight".
As far as I can tell (the authors can correct me here if I have missed
something), by "explanation", the authors refer to their observation that
the shift in generalisation coincides with a shift in the transformer's
algorithm from predominantly resembling Bi-ICL to gradually increasingly
resembling Bi-Bayes later in training. The authors also observe that Bi-Bayes
is a better solution according to the training distribution, so this shift is
in turn explained by the learning algorithm pursuing better on-distribution
performance.
I think there are a number of problems with the framing of these observations
as a new explanation of Singh et al.'s transience derived from LCA.
First, it is not clear to me that the observations go beyond what was
already reached in Singh et al.'s original paper. In that paper, they
already talk about one algorithm, ICL, being replaced with a different
algorithm, "in weight learning" (IWL) with worse generalisation
performance. The authors of the present paper are pointing to a similar
trend arising in their replication, but they now have slightly more
specific models of the algorithms.
Second, note that actually there is an important difference between this
setting and that of Singh et al.'s transience. Namely, in Singh et al.,
the ICL and IWL algorithms by construction are equally performant on the
training distribution. This is a crucial difference in this case. As the
authors note, in this case, Bi-ICL giving way to Bi-Bayes is even less
mysterious than ICL giving way to IWL in Singh et al., beacause this is
driven by the in-distribution performance.
A closer analogy to the setting proposed in the present paper comes from
related work on in-context linear regression. Hoogland et al. (cited by
the authors) and (slightly earlier and in more detail under the name of
"forgetting") in the paper by Panwar et al. "In-context learning through
the Bayesian prism" (from 2023, on arXiv before Singh et al. was published
actually). These works observe that in the Raventos et al. setting, a
generalising algorithm gives way in favour of a training-task-dependent
algorithm, driven by improved in-distribution performance and with a
concomitant drop in out-of-distribution generalisation performance.
Third, given this prior work, upon reading the abstract I was expecting
perhaps some deeper explanation for what actually drives this phenomenon.
For example, some insight into why the transition occurs at a given period
during training or at a given rate, or why the generalising performance
was preferred in the first place. I could not find any such deeper result
in this paper.
Finally, I do not see a connection between LCA specifically and this
explanation. The transition from Bi-ICL to Bi-Bayes would also show up as
a shift in the proximity to Bayesian solution metric from the phase
isolation methodology in section 4. I see no reason to attribute this
explanation to LCA.
I certainly believe LCA offers a particularly crisp behavioural perspective
on this transition. But given all this, I think a more appropriate framing
for the authors' contribution would be to say that in addition to replicating
the transience phenomenon, the authors have used LCA to offer a new and
detailed perspective on the (behavioural) competition between algorithms
already thought to underlie the phenomenon of transient generalisation.
On "causal explanation": As far as I can tell, "causal" is only mentioned
in the introduction. I don't see any grounds for its inclusion here at all. I
suggest it should be removed.
Overall: I believe LCA is an interesting and useful behavioural
technique, but unless the authors can persuade me otherwise I am strongly
opposed to the framing in the introduction and throughout section 5 that the
technique is capable of offering mechanistic insights or that it has offered
a new explanation of the phenomenon of transient generalisation in in-context
learning.
W3. The motivation in terms of unifying and generalising phenomena is inaccurate
Upon first reading the abstract and introduction I understood this paper to
be claiming that there is a need for:
The authors don't seem to be positioning this as a motivating vision toward
which they are contributing a 'first step' or something 'in this direction'.
Rather, they say for example in the abstract that their work "enables a
unified framework for studying" ICL. In the introduction they claim that
findings from prior work may be "disparate findings that manifest in specific
scenarios".
I think this motivating story is not an accurate description of the state of
the field and I think the degree of unification achieved in the paper is
strong but not as strong as the authors claim on this first page.
On generalisation: It appears to me that the authors have indeed provided
a novel contribution by 'generalising' several prior phenomena by exhibiting
them in a new setting. This is indeed a valuable contribution. However, it
appears that most of these phenomena have already been exhibited in multiple
settings, so it is inaccurate to claim that the phenomena were previously
only known to hold in isolated settings.
In the case of the task diversity threshold, the authors have generalised
this to the Markovian sequence modelling setting. Previously, it was shown
for in-context linear regression (Raventos et al.), along with image
classification (Kirsch et al.) as cited by the authors. The same phenomenon
has recently been shown in a multi-task modular addition setting by He at
al. in a recent preprint "Learning to Grok: Emergence of in-context
learning and skill composition in modular arithmetic tasks".
In the case of the emergence of statistical induction heads, there is only
a very minor generalisation taking place since statistical induction heads
were already shown by Edelman et al. to arise in Markovian sequence
modelling with infinite mixtures of Markov chains. The generalisation to
finite mixtures of Markov chains is, in my opinion, valuable but not very
surprising especially given that the original exhibition of induction heads
by Elhage et al. and Olsson et al. was in the completely distinct setting
of language modelling.
Moreover, I am not aware of induction heads being found in other settings
such as in-context linear regression. If any ICL phenomenon is isolated to
certain settings, this suggests that induction heads represent such a
phenomenon.
In the case of transience, it's true that Singh et al. study a specific
scenario that is designed specifically to isolate this phenomenon, but it's
not true that their findings have not been replicated in other settings.
In the previous section I already noted work noting transient
generalisation for in-context linear regression including Hoogland et al.
(already cited by the authors) and the earlier work by Panwar et al.
Beyond in-context linear regression, He et al. "Learning to grok"
(mentioned above) have also demonstrated transience in in-context modular
arithmetic.
These examples actually seem more relevant to the present work than Singh
et al., as they concern transitions between two different ICL algorithms,
rather than between ICL and pure memorisation as in Singh et al.
(I have not evaluated the novelty of generalising the other phenomena, as I
am less familiar with these parts of the literature, and moreover the above
three phenomena are the ones the authors discuss in the main text.)
Once again, I believe the authors have made a strong contribution, but my
concern is that they have not accurately described it in their abstract and
introduction, and they need to reframe their contribution in a more accurate
manner in my opinion.
On unification: Moreover, I did not find the motivation in terms of
unification compelling.
To me it seems like the extent of a 'unification' achieved by the authors is
that they have combined multiple interesting axes of experimental variation
(e.g. studying models at varying task diversity and training time) previously
varied individually in prior work. They have created one setting rich enough
to include these axes of variation, and they have demonstrated that along
each of these axes previously studied phenomena are replicated in their
experiments. I will repeat that I find this comprehensive investigation of
the axes and their combinations is a strong contribution.
However, in describing this contribution, 'unification' seems too strong of a
word. When I read the word unification, I think the prior settings should be
recoverable as special cases of a more general setting. Operationally, the
field should be able to continue forward by discarding previous settings in
favour of using a truly 'unified' setting. I think this standard has not been
met by the proposed setting. Rather, if the field universally adopted the
proposed setting, at least the following research directions would be
precluded.
Different settings encourage different mechanistic solutions in the
transformer's internals. For example, while the Markovian setting allows
one to study statistical induction heads, one does not have the ability to
study the particular mechanisms that emerge in order to perform in-context
linear regression or modular arithmetic. For the field to make progress on
mechanistic analysis of in-context learning in transformers, it seems
useful to be able to take advantage of the various constructions that have
been proposed for specific implementations of in-context linear
regression, for example. I don't see why we shouldn't try to keep our
range of interesting synthetic settings as broad as possible.
Another example comes from the quite specialised setting studied by Singh
et al., where, by construction, ICL and IWL are equally performant on the
training distribution. Nevertheless, there is still an algorithmic
transition between these algorithms at some point in training. This gives
rise to questions that can't be asked or answered in a setting where the
main four algorithms achieve quite different performance in-distribution,
such as what drives the transition even in this case (when the need for
better in-distribution performance is ruled out as an explanation for
driving this transition).
The authors have included appendix G with some discussion on the perceived
benefits of studying Markovian sequence modelling tasks rather than modular
arithmetic or linear regression due to the lack of "sequence space structure"
in these alternative settings.
I must admit unfortunately I did not follow the discussion despite trying to
see their point of view. It is not immediately clear to me what sequence
space structure means. But if the authors refer to the fact that in Markovian
sequence modelling in order to count bigrams the transformer must look at
pairs of tokens, I note that even though modular arithmetic and linear
regression are usually formulated in-context using an i.i.d. sequence of
inputs, it is still necessary for example in Raventos et al.'s setting for
the transformer to look at pairs of sequence items (one containing the x and
the next the corresponding y). It is not clear to me how this is less rich
than looking at bigrams.
Overall: I can't emphasise enough, I really like the setting and the
comprehensive analysis along multiple axes. I am only concerned that the
introduction does not provide an accurate motivation for the work, and I
would like to respectfully challenge the authors to lay out a stronger case
for their contributions in their introduction. Doing so, in my opinion,
should not be too hard, because the authors have made some strong
contributions on an important topic.