Causal normalizing flows: from theory to practice

Summary

The authors propose a normalizing flow (NF) model that incorporates causal information, either via a partial order or complete DAG. They derive identifiability conditions for causal estimands and implement their algorithm on a range of simulated and real-world datasets. The resulting NF can be used to compute treatment effects and counterfactual probabilities, under some reasonable assumptions on the data generating process.

Strengths

This is a strong contribution on an important topic. The manuscript is well-motivated and thoroughly researched. The writing is clear and the experimental validation is convincing. Despite a few minor questions/comments (see below), I am generally supportive of this work.

Weaknesses

There are a few minor points that left me somewhat confused. I suspect these could be cleared up with a few brief lines. -I’m unsure how to interpret $\sum_{n=1}^\infty \mathbf{G}^n$. Why the infinite sum? The examples in Fig. 4 all appear finite? -It appears that the empirical evaluation only considers causal chains of the form $X_1 \rightarrow \dots \rightarrow X_d$. What about more complicated/realistic causal structures? -If I understand Fig. 5 correctly, the ordering model (green curves) outperforms the graph model (orange curves) by KL-divergence and causal effect estimation in generative models when $L < 3$, while the ordering model dominates effect estimation across all numbers of layers in the abductive model. This echoes the results summarized in Table 1. I feel I must be missing something, however – why should a model do better with a causal ordering than it does with the entire graph? Surely the latter is strictly more informative?

Questions

See above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

n this paper, the authors derive and demonstrate the usefulness of normalizing flows (NF) in causal inference. As a first building block of their approach, the authors show that structural causal models (SCMs) can be expressed as triangular monotonic increasing maps. By this reduction, they show that SCMs can be approximated by autoregressive NFs and that those are causally consistent. They proceed by discussing different architectures to best model the SCM with NFs and conclude that an abductive setup for the NF might be the most suitable architecture. In order to perform interventions on an NF they define the do-operator on those. Due to the recursive representation of their approach, the authors resort to defining an intervention by fixing the exogenous variables accordingly. In empirical evaluations, they show that their approach compares favourably to the SOTA on representing the true data distribution and causal inference tasks w.r.t. to both performance and time efficiency. Lastly, they show on real-world data that their approach can be used for fairer, but still precise classification.

Strengths

- The elaboration of the connections between SCMs, TMIs, and ANFs is well executed and can find additional applications in other research. - The implementation of interventions as performing them on the exogenous variables is non-trivial and opens up further paths of investigation. - Figures 2-4 are very valuable for presentation and understanding. - Clear and sound experimental setup. - Only mild and standard assumptions. - Showcasing the usefulness of causal NFs in ML w.r.t. fairness.

Weaknesses

- The paper would benefit of an example of how a learned causal NF behaves, how the interventions influence the output, and how it matches an SCM. - Consider pushing some of the information from the appendix to the main text. E.g.: - 152: A sketch of the proof would strengthen the line of argumentation of this work. - 327: At least a short sentence on what this data is about. - Some background on TMI maps could be introduced to make it easier to follow the content. - No resources such as codebase, trained models, and analysis implementation are provided.

Questions

No questions

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

The authors address the limitations of the work clearly and describe how their approach relies on their assumptions. Furthermore, they describe how violating each of their assumptions would affect the overall approach. These observations seem to be complete w.r.t. the possible limitations of this approach.

Summary

This paper proposes causal normalizing flows, a method to learn a structural causal model (SCM) using normalizing flows (NFs), when the causal order (and potentially more information about the graph) is available. They consider a general SCM where we have observed variables X which follow the equations X_i = f_i(X_pa(i), u_i) where pa(i) are causal parents of i and u_i is endogenous noise. Then, by unrolling the recurisve definition, the authors propose to model this distribution via NFs. In fact, because of acyclicity, the NFs are actually Triangular Monotonic Increasing (TMI) maps. Under the following assumptions - The functions f_i are diffeomorphic - the causal graph is acyclic - the causal ordering is known (this can be a strong assumption, see below) - causal sufficiency, i.e. the endogenous variables u_i are mutually independent it follows froms prior works that the model is identifiable (means there is a unique product distribution and TMI maps given the observational dataset). The authors restate this result and use it as a basis for their experimental explorations. The main selling point of the work is that with approproate design choices of causal normalizing flows, the causal task of do-operations can be performed efficiently. Different design choices for causal normalizing flows are proposed, including generative (modeling the mixing directly) and abductive (modeling the inverse map) methods. To handle do-operations, the authors propose modifying the endogenous variable directly (since the recursive SCM form is lost). They also show how to adapt their framework when the data is discrete or partial knowledge of the graph is available. For experiments, ablation studies compare the design choices, and the framework is also compared to baselines CAREFL and VACA, showing that CNFs perform better on a few SCM tasks. Finally, a fairness use-case is shown with the German credit dataset. The target audience are people interested in causal inference.

Strengths

- Both structural causal modeling and normalizing flows (by now) are well-studied topics and this work is a nice interplay between these two frameworks. - The application to deduce credit risk while being unbiased on sex is an interesting and a bit unusual application of their framework.

Weaknesses

- Assuming knowledge of causal ordering is usually a very strong requirements in applications, since if we have the knowledge of causal ordering, we can use traditional nonlinear regression techniques, e.g. with spline functions, to learn the underlying SCMs. Therefore, this may potentially be a very limiting assumption in experiments and reduce the usefulness of the proposed method. - The identifiability results are not novel and therefore the thrust of this work is experimental. As the authors clarify, theorem 1 is taken from Xi and Bloem-Reddy.

Questions

Please comment on the issues above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitations have been discussed.

Summary

This work explores the use of normalizing flows for causal reasoning. The authors demonstrate that causal models can be identified from observational data using autoregressive normalizing flows. They discuss design choices, learning strategies, and the implementation of the do-operator to handle interventional and counterfactual questions. Through experiments, they validate their approach, compare it to alternative methods, and show its effectiveness in addressing real-world problems with mixed discrete-continuous data and partial causal graph knowledge.

Strengths

Novelty: This work introduces a novel approach by employing causal normalizing flows to identify the underlying causal ordering and effectively address interventional and counterfactual queries using the do-operator. To the best of my knowledge, this methodology is both novel and reasonable. Significance: The ability to predict the interventional effect of the causal data-generating process is a highly important problem with practical implications. Contribution: The paper presents a clear and comprehensive method, accompanied by necessary conditions that support the theoretical aspects. The overall technical contribution is commendable. The writing quality of this paper is good, and it provides ample experimental results to validate its effectiveness. I thoroughly enjoyed reading this paper.

Weaknesses

Challenge: From my understanding, the previous work has already achieved satisfactory results regarding the identifiability of causal normalizing flows. This paper extends these findings to include counterfactual and interventional reasoning. While the theoretical contribution and methodology are commendable, I think this extension is relatively straightforward, and the challenge presented may not be substantial. Consequently, this could be considered the primary weakness of the paper.

Questions

I don't have any specific questions about the main paper, but I recommend that the authors consider transferring some crucial conclusions from the supplementary material to the main paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

.

Summary

This work explores the use of normalizing flows (NFs) in causal inference. The authors demonstrate that causal models can be identified from observational data using autoregressive NFs. They investigate design choices and implement the do-operator in causal NFs to answer interventional and counterfactual questions. The experiments validate their approach and show that causal NFs can effectively address real-world problems with mixed data types and partial knowledge of the causal graph.

Strengths

The authors clearly stated the problem of interest as well as compare with relevant previously showed approaches. They also conducts extensive experiments, including ablation analysis. Finally, they discussed practical limitations and present possible lines of investigation as future work.

Weaknesses

The number and characteristics of the given dataset are limited when compared to practical applications. The ablation analysis miss hyperparameter tuning.

Questions

The authors should give the intuition of presenting Fig. 1 in a more clearly way. It is well-known that computational complexity is a challenge when using normalizing flows, especially for high-dimensional data. In this matter, I believe that would be difficult to use the proposed method with such data (e.g. images). Is it limited to tabular data? Is it possible to use it with high-dimensional data? Any pre-processing steps are necessary? It would be viable to learn the causal graph (e.g. PC) and use it as input the proposed method? My concern is whether it would be wast of time, given that the information is available in the data. In other words, is there a necessity of domain knowledge to use the proposed method?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The datasets used are not strictly representative of practical applications. I believe that there is an avenue of possible developments here turn the proposed method largely applicable.

response to authors

Thank you for the detailed responses.

Re: rebuttal

Many thanks to the authors for their comments and clarifications. I still believe my original score of 7 is fair and will keep it as such. Great work on this paper!

Response to rebuttal

I thank the authors for their response. With their clarifications, I upgraded my score.