Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data

Summary

The paper proposes a formal probing framework to test whether LMs have learned the right laten variables to behave as a “structural causal model” (SCM). An SCM forms a directed graph of token representations as a causal pathway from an initial state to a final state. The proposed methods tries to separate what is learned during pretraining (LM) from the training (calibration) for probing (SCM) by differentiating free (outside the calibration) and bound (seen during pretraining and calibration) variables Their method is tested using a Code-generation model that is trained on a corpus of specification-program pairs. Their empirical test shows that the LM representation positively contribute to the causal predictive accuracy.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- a probing frame work that goes beyond testing solely on the elicited output but tries to trace the causal relations between implicit latent variables that lead to such output - provides evidence that the LM learns representations that support such causal structures

Reasons to reject

- the code generation “language” is very constrained (5 actions) and it is not clear how this can be used for LLMs that learn far more complex and diverse representations that may interfere in many different ways. - the section on the results is only 9 lines and way to short. Figure-4 is too small to read and is hardly addressed in the results section. It is not clear what the figure a - h stand for and what the different graph patterns show. - critical discussion lacks on how this method can be extended to more divers and complex LLMs.

Summary

This paper proposes latent causal probing, a new probing framework that leverages latent causal structure to design auxiliary tasks. In particular, the task is to assess whether the language model has learned to represent the latent variables of the structural causal model (SCM). A method is proposed to disentangle the causal effects using the causal mediation analysis framework. Theoretical justifications are provided, and empirical results also support the claims of the paper. Overall, the proposed idea is novel, and the paper is well motivated and clearly written.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. This paper presents a new probing framework from the perspective of causality. The idea is well motivated. The proposed latent probing framework helps address some limitations in existing work. 2. The authors introduced the background of structural causal model in Section 2. Some motivating examples and use cases are discussed and explained. 3. The authors proposed a very interesting idea, i.e., causal mediation analysis of probing.

Reasons to reject

1. This work only focuses on causal structure of programming languages. It is unclear whether it is easy to extend the proposed latent probing framework to natural language. Additional discussions should be provided in the paper. 2. Experiments only involve an LM trained from scratch. It would be helpful to include evaluations on pretrained LMs. 3. There are some typos in the paper, such as "we refer the ready to" in Section 2.1.

Summary

The authors propose a probing framework based on Structural Causal Models (SCMs) in order to uncover the latent variables LMs learn. As their main contributions, the authors argue that they show that LMs learn the "underlying semantics of the language" (page 2), and they can learn to generalize to unseen data (page 2). While I agree with the authors that there is much to be explored when it comes to developing robust and reliable probing mechanisms and I appreciate the presentation of SCMs, I fail to see how this work contributes on top of already existing work and what its main audience is. [edit of the review during discussion period: I adjusted my overall score from 4 to 6.]

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

The overall topic is relevant to the NLP community. The paper is clearly structured and SCMs as well as probing mechanisms are intuitively introduced.

Reasons to reject

My main concern is that the overall goal of the paper is unclear to me. If the paper is about investigating whether LMs learn "the underlying semantics of the language" (page 2), then this is very well-established already. It is a direct consequence of the good performance of LMs on relational similarity prediction (king-man+woman=queen; see e.g., Vylomova, Rimell, Cohn, Baldwin 2016), using semantic representations for semantic cluster analyses, and vision-language representation space alignment work. If the paper aims to establish overall that LMs latently encode a variety of semantic variables (which I'm again not sure adds much to existing work), this should be evaluated in a suite of experiments. It is my intuition that this isn't the main goal. However, especially the framing of the results, and also the narrative in the abstract and introduction appear to put a significant focus here instead of using this task as a "proof-of-concept". If the paper is intended as an argument for using this particular SCM-based framework as a probing mechanism, the paper needs to focus on establishing it against alternative probing methods, elaborating on how to use it for a variety of potential downstream tasks, as well as establishing its effectiveness. For example, in the results, the paper solely focuses on the implications for the model under investigation but under this framing, I would instead expect a focus on establishing the method. Lastly, I'm confused about the way in which the authors present this work as "interven[ing] on the model's training data" [page 9], as opposed to other work that intervenes on model representations. However, intervening on training data to understand the "causal effect of dataset statistics" [page 9] is the general idea of traditional data augmentation and model generalizability setups (e.g., Perez and Wang 2017; Liu, Kusner, and Blunsom 2021). How does this setup situate within the established data augmentation and generalizability frameworks?

Questions to authors

Very minor note: There are multiple cases where the text says "casual" where it's supposed to say "causal" (I assume). The authors might find the following paper interesting to make connections to given the parallels to SCMs, a focus on the training data, the navigational instruction task, and connections to generalizability and data augmentations (Inducing Causal Structure for Interpretable Neural Networks; Geiger, Wu, Lu et al.; ICML 2022).

Summary

The paper proposes integrating Structural Causal Models (SCM) with the existing probing techniques in NLP. This method aims to refine how we interpret probing results, providing insights into the information that language models (LMs) encode from their training data. One key contribution of this paper is its focus on the probe's capacity, addressing a common shortcoming in traditional probing studies. By using causal mediation analysis, the paper effectively distinguishes the information genuinely encoded by the LM from what is learned independently by the probe. This approach is crucial as it aims to ensure that high probing accuracy reflects the LM's capabilities rather than the probe's. The paper also advances beyond merely associating high probing accuracy with an LM's ability to understand causal structures. Instead, it uses causal models to rigorously test hypotheses about the causal information that LMs might capture, providing a more nuanced interpretation of probing results. However, the reliance on SCMs assumes clear causal relationships within the training data, which may not always be evident, especially in natural language settings where such relationships can be complex and less defined. This assumption is a significant limitation and could restrict the method's application to more naturalistic datasets where causal relationships are not explicitly outlined. The paper demonstrates its approach in controlled experimental setups. However, extending this framework to broader, real-world datasets and demonstrating its effectiveness in these contexts could further validate its practicality and impact.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. **Methods**: Integrating Structural Causal Models (SCM) with language model probing is both novel and interesting. This methodology introduces a new way to interpret what language models learn from data, focusing on understanding causal relationships rather than accuracy metrics. 2. **Tackling limitations of probing**: The paper tackles the limitations of current probing practices by focusing on the probe's capacity to learn independently of the language model. This is important for obtaining more accurate interpretations of model behaviors and capabilities, addressing a gap in how probes are currently used.

Reasons to reject

1. **Assumptions on causal relationships**: The method assumes clear causal relationships within the training data for language models. In practice, especially with complex real-world data, these relationships are often unclear or confounded by other factors. 2. **Limited experiments**: The paper primarily validates its approach in controlled settings. To strengthen the case for publication, it would need to demonstrate effectiveness with real-world datasets, which are generally less structured. 3. **Overfitting**: There is a concern that the causal models could be overfitted to specific characteristics of the training data, given the high dimensionality of language model embeddings. This risk could lead to incorrect conclusions about what the language models are learning.