Robust agents learn causal world models

Summary

The paper shows that any agent that could effectively "learn" the optimal decision under distribution shifts MUST have learned the (approximate) causal model of the data generating process. The implications of this result on the related research areas such as transfer learning and causal inference have been discussed.

Strengths

1. The problem considered is fundamental. 2. The idea is cute and clean.

Weaknesses

Only necessary condition is proved but not the sufficient condition. It will be stronger to prove something like, if the agent has learned some "approximate" causal relationship, it can efficiently learn the optimal decision under distribution shift.

Questions

How to identify and prove the sufficient condition for learning the causal model for learning the optimal decision making under distribution shift?

Rating

8: accept, good paper

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Summary

This paper proposes some theoretical results about decision making tasks that, if the environment is generated by a Bayesian network, and an agent is able to learn a low regret strategy for all mixture of local interventions on the environment, including hard intervention and randomized experiments on any number of variables, then we can recover the causal structure of the environment from the optimal decision learned by the agent. Therefore, if we want to obtain such an agent, it is necessary to learn the causal structure.

Strengths

1. They propose theoretical results connecting decision making and causal structure learning. As suggested by their results, a robust enough agent should always learn the causal structure. 2. The limitation for learning causal structure can be transferred to limitation of robust decision making by their results. 3. Their result gives an example about inferring causal structure when only one variable is observed under each intervention.

Weaknesses

1. They do not conduct an experiment for justifying their results. 2. Their results can only be applied to a small range of scenarios, where we need to reach small regret for all mixture of local interventions. However, most applicable tasks, such as transfer learning, only consider interventions on a subset of variables. 3. There are some spelling mistakes in their text, and some usage of notations are unclear in their text and proof.

Questions

see in Weaknesses

Rating

6: marginally above the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Summary

The paper presents theoretical results showing that any agent that learns well under distributional shifts, must have learned the causal structure of the environment. Here distributional shifts that are most important are shifts of the latent causal variables. That is, if one can generalize across the set of possible changes in these variables, one has learned the causal structure. The result is both deep and intuitive, and has widespread implications. Although theoretical in important senses, e.g. it assumes some unspecified learning method, the result is no less powerful in arguing that to transfer one must learn causal structure.

Strengths

This paper is a gem. The theoretical analysis is simple and clear, the implications are broad and powerful.

Weaknesses

The only weakness, in my opinion, is that the statement of the result in the introduction felt pretty slippery. (See detailed comments below.) All of this was satisfyingly resolved, but I do think the paper would benefit from an effort to sharpen that first section. Details comments: - Please define these: "distributional shifts" "distributionally shifted environments" "target domains" "causal modelling and transfer learning" - " used to derive out results" typo - "Our analysis focuses on distributional shifts that involve changes to the causal data generating process, and hence can be modelled as interventions (Schölkopf et al., 2021)" This would have been nice in the intro. - "This does not assume that all shifts an agent will encounter can be modelled as interventions, but requires that the agent is at least capable of adapting to these shifts." I don't know that I understand this sentence. - "By cCreftheorem: main,theorem: main approx agents" typo?

Questions

I would like to hear what changes to the introduction might look like.

Rating

10: strong accept, should be highlighted at the conference

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Summary

This paper shows a formal connection between generalizing under distribution shits and learning causal models, a connection that has been expressed as hypothesis before (e.g. in Schölkopf 2021). Specifically, they show that if the agent performs well under distribution shifts (bounded regret), then it must have learned a representation that captures the causal structure of the world - in this case, the conditional independencies and causal relationships in the true causal bayesian network.

Strengths

This paper makes an original and significant theoretical contribution by formally establishing a fundamental connection between causal learning and generalisation under distribution shifts. ## Originality: * They provide a proof for showing that an agent that is sufficiently adaptive has learned a causal model of the environment. This is an impressive achievement and a stronger statement than the one stated by good regulator theorem (which as the authors have cited, has been misunderstood and misrepresented in the past) ## Quality: * The theoretical results are technically strong, with detailed proofs provided in the appendices. * The assumptions are clearly stated and well-motivated. The analysis meaningfully relaxes the assumption of optimality. * The writing is clear, well-structured, and accessible given the technical nature of the work. ## Clarity: * The paper is well written and easy to read. ## Significance: * The results have important implications in safety and robustness under distribution shifts. * The proof is non-trivial and provides a great stepping stone for extending to richer settings (e.g. mediated decision tasks)

Weaknesses

- As the authors acknowledge, the results are mainly theoretical. Even a minimal empirical validation of the key insights would strengthen the paper. For example it would be great even if you turn the informal overview (appendix C) into a simple simulation example rather than remain a thought experiment. - The scope is currently limited to unmediated decision tasks. Extending the results to broader RL settings would increase applicability (although I acknowledge that seems significantly more challenging task and out of scope of this work - it’s just a personal curiosity at this point and would be excited to see the next paper already). - The proof is still quite challenging to understand and I believe that there a more informal / simplified sketch that can be introduced to help the reader before dive into the more formal proof. - On a similar note, the implications of the assumptions are not discussed. (e.g. I’d like to see things like, “assumption 2 implies that there exist distribution shifts for which the optimal policies are different”.

Questions

(apologies for repetition from weaknesses) - The implications of the assumptions are not discussed. (e.g. I’d like to see things like, “assumption 2 implies that there exist distribution shifts for which the optimal policies are different”. - "The environment is described by a set of random variables C..." this sentence belongs to the main text since it you don't explain C random variable although it's heavily used. - Although discussed in the appendix, i'd like to see the description of what squares, circles and diamonds mean in the CID - In assumption 1 you stated $\text{Desc}_D \cap \text{Anc}_U = \emptyset$ but this doesn't exclude the trivial setting (which you state in the appendix is not of focus). Can you either extend the assumption or comment in the main text that it's not of interest the trivial setting? (i know it's a nitpick but got me wondered while reading it in the main text and i feel that since you thought about it you could have mentioned it earlier in the text). - Definition 6 in the appendix: Shouldn't it be $\mathbb{E}^{\pi_\sigma}$ (subscript on policy)? Also, $\delta \geq 0$ is missing. - Can you please give a simple sketch of the proof? This would help the readability significantly. Also i feel there is a simple sentence that can be written on each theorem that explains its implications

Rating

8: accept, good paper

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Global response to reviewers and AC

We thank the reviewers for their invaluable feedback, and are encouraged that they describe the paper as "an impressive achievement", " an original and significant theoretical contribution" that has "important implications in safety and robustness" (Reviewer gS4b), and that "The problem considered is fundamental. The idea is cute and clean" (Reviewer VrNg), and "This paper is a gem. The theoretical analysis is simple and clear, the implications are broad and powerful." (Reviewer 4TT8). **main changes to updated manuscript** (details and additional changes in individual replies) 1. Empirical evaluation of results, learning the underlying CBN from agent's policy for randomly generated CIDs (Appendix F.1). Application to causal discovery including pseudocode for Theorem 1. Exploration of how the regret bound effects the accuracy of the learned model. 2. Simplified and shortened overview of proof (Appendix F). 3. Extended sufficiency theorem to cover approximate causal models (Theorem 3). 4. Updated introduction, interpretation and discussion sections. Updated figures.