Do Finetti: On Causal Effects for Exchangeable Data

Summary

The paper generalizes the traditional iid settings in casual inference to exchangeability settings by de Finetti theorem, and proposes a new model named the casual Polya urn model to illustrate the new scheme and to catch more relationship. The experiments show when the number of environment is less than 5000, the new schemes performs well.

Strengths

First of all, I am sorry that I do not know much about the casual inference. But the paper uses exchangeability instead of iid settings, which seems an improvement.

Weaknesses

1. Aldous 1985 shows many (not all) conclusions in iid can be naturally transformed into those under exchangeability. So the theoretical improvement seems not much. 2. Only simulated experiments for the casual inference problems. 3. De Finetti Theorem is ‘iff’. So it is inappropriate to use methods based on iid settings on the exchangeable but not iid data to compare. 4. For the experiment, what about the larger number of environment? It seems the original one performs better.

Questions

Besides above, 5. In casual inference problems, is it easy to identify the exchangeability, especially for the real data?

Rating

7

Confidence

2

Soundness

4

Presentation

4

Contribution

2

Limitations

Besides above, 6. Some typos even in reference; for example, the first reference is not well-written. 7. Could see more complicated structure between theta psi and X. In the paper, psi is independent of X. 8. Typos, like ‘Nature’ should be ‘nature’.

Summary

The paper studies causal effect identification and estimation in exchangeable data. The main result here is theorem 1, which shows that causal effects are identifiable in ICM generative processes.

Strengths

- The paper provides a great framework to think about interventions in exchangeable data. Starting from what interventions should be considered (Definition 3) to identifying a procedure for computing the post-interventional distributions. - The paper presentation, at least in the first part, was simple and intuitive. I always found myself asking a question and then find it being answered in the next paragraph. However, probably due to space constraints, this did change in the latter parts of the paper.

Weaknesses

- The latter parts of the paper is rushed and left me confused. For example, it is unclear how causal de Finetti theorems apply to the Causal Pólya Urn Model, Theorem 2, and the entirety of section 4 is very rushed and I have struggled to understand what theorem 2 say exactly. - I have felt that the algorithm could have taken more of real-estate in the presentation of the paper. Also, it is unclear how the graph structure is learned in the algorithm - This is more of a nit pick, but the appendix contains a few typos and is in a worse state in general than the main text. For example, the use of index i in equation 51, 53, ...

Questions

- In the experiments, it seems to me that the model generating the synthetic dataset is different from that described in section 3.2. In particular, in the experiments, X_i is sampled from a Ber(theta) and hence P(X_i = 1) = P(X_2 =1) = ... = theta, whereas if I understood the model in 3.2, then the probability P(X_n =1) will be much greater than P(X_1 =1) if for example all X_m =1 for all m < n. Are they actually different? Or did I misunderstood? And how can the model described in 3.2 be represented by equation 4? (I read F.2 but it seems to me that equation 51 follows the model in Section 5). - In the experiments, can the authors elaborate on the IID baseline? Do you run the algorithm on the "full" graph G which have nodes X_1 Y_1 X_2 Y_2? I assume this is what's been done as it is the fairest baseline, but I'm not sure. Appendix K seems to imply that and the main paper did not make it clear. - In the description of ICMs, the author mention the expression: > Causal mechanisms are independent of each other in the sense that a change in one mechanism P(Xi | PAi) does not inform or influence any of the other mechanisms P(Xj | PAj) What would be a concrete example where such condition is violated?

Rating

7

Confidence

2

Soundness

4

Presentation

3

Contribution

3

Limitations

It has been addressed.

Summary

The paper formalizes the observational and interventional distribution under the ICM generative process, of which iid is the special case. It provides an identifiability result for the causal effect given that the causal graph is known. Then, it shows that both the causal graph and the causal effect can be identified simultaneously.

Strengths

1. Problem: The problem is important as it will bring the causal effect estimation literature closer to real-world scenarios. 2. Theory: The theoretical results are strong, especially Theorem 2, which shows that both the causal graph and the effect can be estimated simultaneously. I have not checked the proofs, though. 3. Experiment: The experiment on the simulated data verify the theoretical claim. 4. Presentation: The paper is well-written and easy to follow. All the notation and definitions are clear.

Weaknesses

1. Experiments: I understand the main purpose of the work is to establish the theoretical foundation of causal effect estimation for exchangeable data, but it would be interesting to apply the method to some real-world datasets (not necessary for the rebuttal).

Questions

1. Definition 3: We should also break the edge from the de-finnetti parameters to the intervened variable, right? Or do we not need any graphical operations? 2. I am slightly confused by the statements in Lines 153-154 and Line 197. They seem to contradict each other. Is it due to conditioning on x1 and x2 that Eq10 and eq11 are not equal since, in ICM, they are not iid?

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

Yes, the authors have addressed the limitation in the Conclusion section and Appendix L.