Summary
This paper investigates whether it is possible for an agent to learn how to perform and make use of active causal interventions, despite being trained on purely passive / observational data (e.g. supervised learning, behavioral cloning, etc). The authors first argue that this is possible in principle, if the observed data contains behavioral traces of an expert who performs causal interventions in an environment to learn about its causal structure, and then exploits the learned information to achieve a goal. They then perform experiments in a causal DAG environment, a visual "odd-one-out" environment, and linguistic version of the "odd-one-out" environment (in order to test pretrained LLMs). In the casual DAG environment, they find that trained agents can learn to generalize causal exploration and exploitation strategies to unseen DAGs, just by passive learning of expert strategies on the training set. They find similar results in the odd-one-out environment, with stronger performance achieved when the training data includes natural language explanations of the interventions performed by the expert. Finally, they find that LLMs prompted with few-shot examples of causal experimentation can generalize to new settings where the latent cause of reward is absent from the few-shot examples, provided that explanation and reasoning traces are also provided in the few-shot examples.
Strengths
This was a well-written paper that addresses an important topic of debate that has arisen about the capabilities of models trained purely on observational data, including pretrained (large) language models. I have several questions about the details of the experiments, and about the generality of causal experimentation strategies that are learned from observations of an expert, but overall I think the paper presents clear theoretical arguments and solid empirical demonstrations for its claims, showing that learning causal intervention strategies from offline/data is possible in principle, and is exhibited in practice to at least some degree of generality in actual ML models. The DAG experiments are especially well designed, showing that passively learned behavior is not well explained by approximate heuristics, and better explained by optimal causal reasoning. The odd-one-out and LLM experiments are more open to interpretation due to the absence of heuristic or correlational baselines, and not knowing whether the LLM was trained on similar data, but still provide evidence that passive learners may acquire generalizable causal intervention strategies in non-trivial settings (insofar as it is unlikely for the high performance to be achieved by a learned heuristic).
Weaknesses
While the experiments have been designed carefully enough to show that observed casual strategies are generalized to some extent, I think they could be extended in the number of ways to probe the limits of this generalization.
In the DAG experiments for example, it is clear that the agent must have learned some kind of permutation invariant strategy for experimentation and intervention, given that the training data entirely lacks causal links that are present in the test episode. However, I think the experiments still leave open exactly how abstract the learned strategy is -- does it generalize to DAGs with 6 nodes, or exhibit size generalization more generally? (I expect not, due to the network architecture.) Can it generalize to DAGs with non-linear causal relationships? And can it adapt its experimentation strategy when it is *not* cued about which variables are relevant, perhaps by learning to imitate an expert that uses an optimal experimental design that minimizes the number of exploration queries before exploiting? (I found the explicit cues about which variables are relevant to be a fairly strong simplifying assumption -- it would be good to at least vary the number of variables which are cued as relevant to a wider range of numbers.) Similar size generalization tests would be interesting to test in the "odd-one-out" and LLM experiments -- e.g. having a test trial with 4 distinct objects, not just 3. For the LLM experiments in particular, it would also be good to say something about whether similar prompts are likely or unlikely to have appeared in Chinchilla 70B training data.
Generalizability is related to a more general question I have about what exactly "counts" as having learned an active causal strategy. In its strongest form, this might require verifying that a learned model actually implements a causal discovery algorithm and causal exploitation algorithm that is optimal, or at least sound and complete. Of course, this is very hard to verify without strong mechanistic interpretability techniques, and in the absence of that, heuristic baselines are helpful to compare against. Because heuristic baselines were included in the DAG experiments, I'm willing to believe that the model has successfully learned a close approximation to the optimal exploitation strategy in the n = 5 case, which presumably requires the agent to simulate the effect of a causal intervention on an internal representation of the DAG structure that was identified from the experimentation phase. But because similar baselines aren't included in the "odd-one-out" or LLM experiments, it could be that the learned strategy in those cases is just some "clever trick" that happens to do well most of the time without "true" causal reasoning. As such, it'd strengthen the paper to show some heuristic or correlational baselines for those experiments, if only to demonstrate that there are no obvious heuristics that can easily achieve the ~90% success rates in the odd-one-out and LLM experiments. For example, one baseline might be to always transform the color of the first/nearest object during experimentation (and to pick the unique colored object in the test trial if that transformation is successful). Another heuristic might be to always pick the object based on the feature that was most often unique in the training data. In addition, I think it'd good to note the limitations of what we can conclude about a model's learned strategies based on observational equivalence alone.
Regarding the need to predict/include linguistic explanations in the "odd-one-out" experiments, it was unclear to me whether the results shown in Figure 3 (and in the corrected version, Figure 13) are from a model that was also trained to predict natural language explanations. My guess is that they are, but I think the presentation clarity of this aspect could be improved, e.g. by stating in the caption of Figure 3 that the results shown involve predicting the explanation, by giving an example in the Appendix of what a full training episode (including the natural language explanations) looks like, and by explaining how the natural language explanations are generated (I had to consult the Lampinen et al (2022) paper in order to determine this).
Finally, I think it's somewhat misleading to describe the passive data in section 5.2.1 as "perfectly confounded" -- after all, the explanations constitute observational data that makes the setting no longer confounded! After all, it is only in worlds where some feature X causes reward R that choosing a object unique in feature X would result in the explanation E = e_X+ ("In this game, I was rewarded for unique X", and choosing an object unique in some other feature would result in the explanation E = e_X- ("The rewarding dimension must not be Y." where Y != X). Hence, the relevant part of the causal DAG is:
R <- X -> E
Once the agent observes E = e_X, it can reliably infer that there is causal link from X -> R in the true causal graph, since X is a common cause of both R and e_X. So the fact that explanations "deconfound" is no surprise at all.
Questions
See above for various questions and suggestions I have about additional baselines, additional generalization tests, clarifying the role of explanations in the odd-one-out experiments, whether these models can be said to have "truly learned" causal reasoning, and whether the term "perfectly confounded" is apt.
In addition, I'm curious whether the authors still endorse the following analysis, given that their post-submission corrections show much better performance of passive learners in the odd-one-out experiment:
> Indeed, although our passive agents performed far above chance in the odd-one-out environments, they did not achieve as robust of performance as the online RL agents in earlier work [30]. This gap may be due to irreversible choices in these settings, which are difficult to learn from purely positive examples, without some active correction of off-optimal-policy behavior [e.g. 48, 11].
Post-Rebuttal Response:
The authors have addressed my questions and suggestions, and I continue to recommend acceptance.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Some limitations of their findings are discussed (see "What this paper does not imply:"), and the authors fairly summarize their findings as showing that passive learning of causal strategies is restricted to the case where expert intervention policies are actually present in the data. I think it'd be valuable to add discussion of exactly how much we can expect the learned causal strategies to generalize, and whether it is possible to establish if they actually constitute sound and complete causal discovery algorithms.