Unveiling LLMs: The Evolution of Latent Representations in a Dynamic Knowledge Graph

Summary

This paper proposes a framework to understand how Large Language Models (LLMs) recall and use extensive factual knowledge. This paper uses the activation patching method to perform interpretability analysis on two datasets, FEVER and CLIMATE-FEVER. The local interpretability analysis exposes different latent errors from representation to multi-hop reasoning errors. The global analysis uncovered patterns in the underlying evolution of the model's factual knowledge.

Rating

7

Confidence

2

Ethics flag

1

Reasons to accept

1. This paper analyzes the factual knowledge encoded in the latent representations of Large Language Models (LLMs), conducting a detailed analysis. 2. The framework proposed in this paper appears to have scalability for research on interpretability.

Reasons to reject

1. The method in this paper is somewhat limited, and how to expand it to a broader range of factual knowledge research is not yet clear.

Questions to authors

1. Can the method described in this paper be extended to other tasks involving factual knowledge in Large Language Models (LLMs), especially in AI-assistant dialogue scenarios?

Summary

This paper presents a method for probing latent representations of (local) knowledge graphs in an LLM, in a way that is amenable to both local understanding – the representation of a particular statement – and global analysis – looking at how groups of layers behave across examples. The method builds on previous activation patching work from Ghandeharioun et al. 2024 and Zhang and Nanda 2023, but uses more complex prompts to decode full relational triples from the representations of a factual (or false) statement. The authors validate both their base model inference (M on S) and the patching (M on T) on the FEVER and CLIMATE-FEVER datasets, measuring both standard accuracy as well as self-consistency as a way of validating the decoded facts. They present a variety of interesting qualitative findings, both at the local level - tracing the evolution of a model’s knowledge graph around a single claim - and globally by clustering graphs across layers and identifying common patterns, such as a transition from entity resolution in earlier layers to representing factual knowledge about these entities in the middle layers.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

This paper presents an elegant extension of the methods of Ghandeharioun et al. 2024 to decode full relations, rather than just specific attributes. This enables richer analysis not easily accessible with other probing techniques: an entire (local) knowledge graph can be decoded from the probe output and the evolution of this graph studied over the layers of the model. There are some flaws - see below - but I think this paper presents a unique and novel method that advances the scope of what we can learn about internal representations and layer-wise processing in LLMs.

Reasons to reject

I have some reservations about the complexity of the measurement technique, as the complex prompts (particularly on the target side) and relatively long-form output leave room for potential bias or skew in the results that may be due to the measurement technique rather than inherent properties of the model. This has been discussed in the literature w.r.t. traditional probing models (e.g. https://arxiv.org/abs/1909.03368), and I worry about the same issues appearing here - for example, the authors discuss at least one case of this where there appears to be leakage from a few-shot example in the target prompt. I would encourage the authors to discuss this more, as this can often be addressed by experimental design - such as including appropriate controls or baselines, marginalizing over different prompts, or corroborating specific findings using simpler methods. The global analysis also may suffer from this, as the graph node embedding and clustering makes the findings a few more steps removed from the underlying source-side model representations. I wonder if some observations such as the shift from “Robin” to “The Joker” and back to “Robin” (2nd paragraph on page 8) could be artifacts of the discretization steps involved here. That being said, I think this is at least somewhat orthogonal to the main contributions, and would like to still see this paper published.

Questions to authors

- It's not super clear until later in the paper (3.3) what the “temporal” dimension was that the title refers to. Maybe mention in the abstract and introduction that this means over layers of the model? - I’d be curious to see ablations on the pooling across tokens here. On one hand, it seems like the noun/verb last-token approach gets most of the content words in the input sentence, and so I wonder if it would be much different from just pooling across all tokens. On the other hand, since the model processes left to right, I wonder if it would work to just take the representations of the last token in the claim? - Did you experiment with any alternative decoding strategies, such as the model to decode the SPO triples directly rather than the relation form? I wonder if these would be easier to generate, as they seem closer to natural language. - “In the form of ground predicates and without relying on external models or training processes” - I would hesitate a bit to claim there is no training involved, since the method relies on few-shot examples in the prompts. But I don’t think this detracts from the main contributions here.

Summary

Authors propose an approach to understand the inner workings of Large Language Models. To do so, they chose the task of factual knowledge verification. The idea behind factual knowledge verification is to utilize claims from verification datasets like FEVER and CLIMATE FEVER, converting them into a restricted first-order logic form. They create a prompt and pass it through the model to predict the logical form as well as the label.

Rating

3

Confidence

5

Ethics flag

1

Reasons to accept

Overall, the approach is novel for understanding the internal workings of LLMs.

Reasons to reject

* Evaluation: While the work is novel, the performance of the proposed approach is nearly similar to or slightly better than a coin toss (50%) since the model's ROC AUC performance is close to 0.5 or 0.6. Given this, it's difficult to determine if the internal workings of LLMs are appropriately evaluated with proposed approach. * Missing design choices: Throughout the paper, there are several instances where authors have made design choices without explanation. For example, why the study is limited to LLama and not other kinds of models. Is the observation visible only in Decoder-type models or can it also be seen in Encoder-Decoder type models like FLAN? Additionally, what was the intuition behind removing "Not Enough Info"? Moreover, what is the reason to consider only FEVER or CLIMATE FEVER ? * The approach uses restricted First Order Logic to represent the claim. The language used to represent the claim is restricted as the relations are binary and not n-ary. Hence, they do not accurately represent each of the facts mentioned in the section. The datasets used in FEVER and CLIMATE FEVER are full of sentences containing n-ary relationships among the entities. Therefore, a better representation would benefit the work.

Questions to authors

Suggestion * Remove the words "Temporal" and "knowledge graph." The term "temporal knowledge graph" is misleading as it triggers the reader to think in a different angle, making it hard to relate to the present approach. Also, the term "temporal" is used loosely, as the layers of LLM; it's better to consider three tiers of layers in LLM: Bottom, Middle, and Top.

Summary

**Paper Summary** This work describes a new method for attempting to provide some interpretability in the context of LLMs when verifying factual claims. More specifically, the work describes a way to construct a temporal knowledge graph, derived from LLM hidden representations, to better visualize working state and reasoning of the LLM during inference. The mechanism uses a few ideas including asking the model to generate a chain of predicates as rationale for claim verification, along with using activation patching to translate hidden representations of different layers of the LLM into knowledge triplets. The work describes mostly qualitative analysis of constructed temporal knowledge graphs when evaluating LLaMA-2 model on the FEVER and CLIMATE-FEVER datasets. **Review Overview** Overall, it is not possible to recommend this work for publication in its current form for two main reasons: (1) I have several concerns about the soundness of the techniques employed (that is, the interpretability techniques can be trusted to correctly reflect internal state of the model) and (2) the analysis done using the proposed techniques on FEVER and CLIMATE-FEVER are a bit unclear or redundant given prior work. In terms of originality, given that this is a heavily explored area, this proposal in this paper adds only a small delta to recent interpretability techniques of LLMs. The “temporal” (layer-wise) aspect of the work is not novel (e.g., https://aclanthology.org/P19-1452/ from 2019 although not explicitly focused on factuality). Augmenting LLM rationales with first-order-logic is also interesting but not quite new (e.g., https://aclanthology.org/2023.findings-emnlp.416/ from EMNLP 2023 and https://openreview.net/forum?id=qFVVBzXxR2V from ICLR 2023). In terms of clarity of presentation, I found the paper generally clear and well written. A few potential useful references that are missing: Dissecting Recall of Factual Associations in Auto-Regressive Language Models - https://aclanthology.org/2023.emnlp-main.751/ https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

- The paper is well written and presented making it clear what the problem setting is, the proposed technique and results so far, - Interpretability of factual knowledge in LLMs, specifically debugging mistakes, is still an open problem and practical tools and results are beneficial to the community.

Reasons to reject

As mentioned in the summary, the results of the work are still a bit immature. The paper is not thorough in convincing that the techniques used are faithfully representing the internal workings of the LLM and are not, for example, post-hoc explanations. Specifically: - One important component of the proposed system is the first-order-logic derivation that the model produces when also producing a verdict of whether a statement is factual. There was never any analysis of whether the logic produced was faithful to the end result. What evidence do we have that the first-order-logic produced is not a hallucination or post-hoc explanation by the model? How faithful is it to its response? - It is also unclear how faithful the patching mechanism works. It would seem that averaging the output of many tokens (all first-order-logic tokens?) into a single representation would constitute a potential loss of information (as described in Section 3.2). If the first-order-logic contains several entities and relations can we assume that the averaged representation of the tokens can recover the information? Presumably this can be tested in some way (e.g. just for the last/output layer where you know the ground truth first-order-logic and the graph that could be constructed from it), - The analysis in Section 4 is a bit limited. Aside from general trends in layer-depth representation (which has been studied previously), there are no new insights derived from the results. It makes me wonder if this may be a reflection of how useful temporal KGs are in practice. How would they be useful to debug specific factuality problems in practice?

Questions to authors

Besides the comments above, I few more minor questions: - There are two prompts used in the work (source S, and target T) that are almost the same. I would have expected them to be identical since the patching exercise is attempting to reconstruct the internals of the original inference. Why are these prompts not the same? - In Section 3.2, the way to construct a summary representation uses some heuristics (averaging, weighting certain terms more or less). Did you try different representations to transfer from inference to patching? What was the impact of those choices? Why not take the entire sequence instead of trying to summarize into a single representation? - In the results, there is a mention that “ILM” and “Industrial Light and Magic” are different nodes in the knowledge graph. This seems to be just an aliasing problem (that is, the model is likely internally representing the two surface forms almost the same). Why did authors think that this aliasing is causing a multi-hop reasoning problem? How do we know the model internally is not representing these concepts sufficiently similarly?

Authors' Response to Reviewers' Feedback and Proposed Improvements

We appreciate the thoughtful and positive comments from all reviewers. You have recognised that our framework is *an elegant extension* of the recent work by Ghandeharioun et al. (ICML 2024, XVV9), *enables richer analysis not easily accessible with other probing techniques* (XVV9); *advances the scope of better understanding internal workings* (GvEw), *representations* (XVV9), *factual knowledge* (3hbG), and *layer-wise processing* (XVV9) in LLMs; provides *scalability for research on interpretability* (yHaY); and the paper is *well-written* and presents the problem setting, the proposed technique and results clearly (3hbG). We believe that addressing the reviewers’ feedback and questions has significantly enhanced the quality of our manuscript. In the final version, exploiting the additional page, we will improve the following aspects: ### Clarification of the work's objective, the roles of the logic representation and the NLP task We will provide the following clearer explanation of the objective of our work as well as the purpose of the NLP task and logical representation within our framework (GvEw, 3hbG) in the abstract, introduction, and methodology sections: - Using latent representations of LLMs, our work uses the NLP task of factual knowledge verification to elicit a model behaviour for probing: exposing the factual information that an LLM holds internally when evaluating the truthfulness of an input; with no ground truth available. We thus neither predict the logical form of claims (GvEw) nor the model outputs (3hbG). - On the other hand, we mainly use the logical representation to consistently format the factual information, which supports the evaluation of the input claim, for the subsequent graph generation. Ground predicates, in first-order logic, format the facts related to claims rather than converting the input claims (GvEw). For example, the factual information and the ground predicate “AuthorOf(Hamlet, William Shakespeare)” for the input claim “Edgar Allan Poe wrote Hamlet” (Section 3.1). ### Graph utility Sections 1 and 4 will better present the advantages of our graph representation, from highlighting entity and fact centrality in LLM reasoning via local analysis to revealing trends in factual knowledge resolution in global one (3hbG, XVV9). Specifically, we will better introduce the result section by further stressing its showcase purpose, presenting use cases of actual outputs of our framework, rather than as our main contribution. Further works can leverage our framework to conduct other graph-related analyses on LLM’s latent representations. ### Clarification concerning the chosen datasets Section 4.1 will clarify that the FEVER and CLIMATE-FEVER datasets are used to collect factual and common-sense claims for eliciting a behaviour to analyse (GvEw), as in conventional mechanistic interpretability workflow (Conmy et al., 2023). These input claims, already verified with publicly available information (GvEw), are exploited to trigger a language model to recall factual knowledge that supports or debunks them. ### Clarification concerning the classification performance Section 4.2 will better explain the purpose of analysing the classification performance of model inferences. We will rename the section and remove the latent representations’ performance (GvEw) to reduce misunderstandings for readers. ### Confusion with the term “temporal” - In Sections 1 and 3, we will better introduce and rationalise the temporal dimension of our graph representations. The sequential nature of hidden layers in a transformer-based model allows segmentation within a discrete temporal dimension during inference: layer *t* at time *t*, layer *t+1* at time *t+1*, and so on (GvEw). - However, based on reviewers' feedback about the confusion and non-conventional use of this term (GvEw, XVV9), we will adopt the more general term “dynamic” in the final version of our manuscript as Harary and Gupt (1996). ### Explanation and ablation concerning the token strategy for the summary representation We acknowledge the reviewers' questions about the rationality, effects, and alternatives of our strategy for the summary representation (3hbG, XVV9). Section 3.2 of the final manuscript will detail the experimental results that led us to propose such a token pooling strategy: - We will explain the need for pooling (3hbG) and the unsatisfactory results of using single tokens, such as meaningless texts from punctuation or model’s special tokens, and single-word information for the input’s last token. - An appendix will include an ablation study with different multi-token strategies, showing outputs when considering all claim tokens or complete words. ### Integration with a preliminary comparison with other models We will include a visual comparison of outputs generated by other instruction-tuned LLMs in a tabular appendix, testing the latest LLaMA3 (8-billion) as well as LLaMA2 (13-billion) and Falcon (7-billion).

Authors' additions

*Layerwise aspect is not novel*: ... For example, previous studies have analysed simpler objects than factual knowledge such as *factual associations* (Meng et al., 2022 and Geva et al., 2023), *semantic concepts* (Nanda et al., 2023 and Anthropic, 2024) and *entity resolution* (Ghandeharioun et al., 2024). *Patching and prompts*: During the inference on prompt T, the patching technique replaces the vector representation of the placeholder token with the input claim's summary vector representation from the original inference (Figure 4). This patched inference interprets the semantics within this injected vector representation. Essentially, it translates the unknown vector space used internally by the LLM (and altered by merging multiple token representations) into human-understandable semantics. This process is more similar to translation rather than reconstructing the original inference.