Joint Bayesian Inference of Graphical Structure and Parameters with a Single Generative Flow Network

Generative Flow Networks (GFlowNets), a class of generative models over discrete and structured sample spaces, have been previously applied to the problem of inferring the marginal posterior distribution over the directed acyclic graph (DAG) of a Bayesian Network, given a dataset of observations. Based on recent advances extending this framework to non-discrete sample spaces, we propose in this paper to approximate the joint posterior over not only the structure of a Bayesian Network, but also the parameters of its conditional probability distributions. We use a single GFlowNet whose sampling policy follows a two-phase process: the DAG is first generated sequentially one edge at a time, and then the corresponding parameters are picked once the full structure is known. Since the parameters are included in the posterior distribution, this leaves more flexibility for the local probability models of the Bayesian Network, making our approach applicable even to non-linear models parametrized by neural networks. We show that our method, called JSP-GFN, offers an accurate approximation of the joint posterior, while comparing favorably against existing methods on both simulated and real data.

Paper

References (56)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer 9gh36/10 · confidence 3/52023-06-21

Summary

In a recent line of research, Generative Flow Networks (GNFs) are used for structure learning by drawing DAGs (that represent BN structures) from an implicit DAG distribution that fits the data (Deleu et al 2022). The contribution of the present paper is extending this line of research by drawing joint DAG and the parameters (of the factors associated with the DAG's nodes). To be more precise, firstly they construct a graph progressively, by adding new links between nodes (as in Deleu et al 2022 paper), and then the parameters are chosen to form a pair of (structure, parameter).

Strengths

1. The problem that this paper addresses i.e. "structure learning" is an important task. 2. The paper is overall well written and the literature review is thorough. 3. The reported results are good. 4. This is an interesting paper but unfortunately, I believe its theoretical flaws are serious and have to be addressed before publishing the work.

Weaknesses

1. The core theory behind this work, i.e. equation (6), seems fundamentally incorrect and has measure theoretic problems: Detailed balance is sufficient for "fixed point condition" --which in this case means, the convergence of the probability of the terminal structure/parameters (G, theta) to a posterior probability that is proportional to its reward R(G, theta)-- only if the dimensionality is fixed. When the sub-spaces have different dimensions, measure-theoretic problems rise due to mapping a single point (in a low-dimensional subspace) to infinite points (in a high-dimensional subspace). The authors correctly mention (in line 242) that MCMC encounters trans-dimensional move problems. However, this (as discussed in Green's Reversible Jump MCMC paper) is due to the detailed balance condition which is the foundation of the present work as well. As such, as far as I can say, the same issue should be addressed here too otherwise the correct convergence is not guaranteed. 2. The code is not provided. As such, the reported results are not reproducable.

Questions

1. I would appreciate it if the authors could comment on the mentioned measure theoretic problem. 2. In lines 176-179, there is a confusing statement that I interpret as follows: "Although there is no guarantee that equation (6) guarantees the convergence but if we assume that (6) holds for all pairs (which I suppose, means if we assume that the dimensionality is fixed which is not (?)) then the convergence is guaranteed." Could you please clarify these lines?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Reviewer jHMM5/10 · confidence 3/52023-07-06

Summary

The authors present a method for learning a posterior distribution over Bayesian networks using an extended version of GFlowNets that can jointly construct the underlying DAG and associated parameters in a two-stage process. They use an objective based on a generalization of detailed balance to train their flow network and provide a theoretical justification that optimizing this generalized objective leads to the desired distribution. The method is evaluated on synthetic problems and gene regulatory networks, outperforming recent methods based on MCMC.

Strengths

The problem of learning Bayesian networks is relevant to the community. The authors provide a deep theoretical framework for the presented problem and derive it thoroughly. The strength of the method is that it provides not just a point estimate, but a full posterior distribution over Bayesian networks and parameters in an amortized fashion.

Weaknesses

- While the theoretical framework is explained in much detail, the actual implementation is quite opaque to me. The authors could spend more time in explaining the architecture and implementation details. - “You are strongly encouraged to submit code that produces this result.” [NeurIPS guidelines]. - A simple educational example that is dissected in detail could help to better understand the framework. - The experimental results are restricted to some performance metrics, but especially in the biological datasets, it would be interesting to dig deeper. Could JSP-GFN discover known gene regulatory networks? How do posterior predictives look compared to the real data? What role does interventional vs. observational data play? - The experiments provide little intuition about the regime on which this method can be expected to work in terms of the number of training data, the number of nodes, and the number of parameters per node.

Questions

- Fig. D.3c: is this the negative LL (lower=better) or just LL? Depending on the answer the interpretation of the results would change. - What is the relationship between accuracy vs. number of training data? - How does the method scale with the number of nodes? Can the authors provide some intuition about the size of Bayesian networks one might be able to learn with this method, also in relation to the number of training data? - Why is the additional use of a graph neural network necessary compared to the architecture used in DAG-GFlowNet? - The authors discuss how $P_{\phi}(\theta_i|G,stop)$ can be parameterized by a Normal/diagonal-Normal distribution, or a 2-layer MLP for each $\theta_i$. How do the authors ensure that correlations between $\theta$s of different $G_i$ can be learned if different MLPs are used for each $\theta_i$?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors only touch on some limitations and a more detailed discussion would strengthen the paper.

Reviewer c3AM6/10 · confidence 2/52023-07-19

Summary

The authors utilize a generative flow network to jointly model the structure of the DAQ and its parameters.

Strengths

The authors utilize GFlowNets to jointly model the structure of the DAQ and its parameters. The experiments on simulated data and real-world data are promising.

Weaknesses

I am not an expert on this area and I did not bid this paper. So I will listen to other reviewers' opinion on weakness.

Questions

None

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did not discuss this which I think they should.

Reviewer 5gCu5/10 · confidence 4/52023-07-26

Summary

This paper presents a new method for Bayesian structure learning from observational data, based on the framework of GFlowNets. GFlowNets have been used for Bayesian structure learning, but previous approaches view the distributions over the parameters and structures of a model modular and may learn them with separate methods. The paper introduces a joint method that learns them with a single GFlowNet. To do so, the paper extends the previous approach (Deleu et al., 2022) by introducing a new reward function, deriving the subtrajectory balance for the new reward function, and introducing a few adaptations to the parametrization of the forward transition probabilities of GFlowNet. The paper conducts experiments on small synthetic graphs to evaluate how well the posterior of the model of the graph and parameters matches the ground-truth. Moreover, experiments with data generated by linear/non-linear Gaussian Bayesian networks are also conducted, with a focus on evaluating different models' performance of predicting held-out data (likelihood).

Strengths

1. Structure learning is an important area in machine learning. GFlowNets is a new and emerging framework for structure learning. There are only a few studies in this direction, which adds to the novelty of the paper. 2. The paper studies the problem of how to jointly learn parameters and structures in Bayesian structure learning with a single GFlowNet. The proposed solution to the problem is conceptually intuitive and technically sound, including the introduction of subtrajectory balance conditions and the parametrization of the forward transition probabilities. 3. The experiments of the paper are quite comprehensive. Benchmark datasets and recent advances are included. Multiple metrics are compared. 4. The quality of writing of the paper is good. The detailed derivations and settings are given. The paper is easy to follow.

Weaknesses

1. Jointly learn parameters and structures in Bayesian structure learning with a single GFlowNet seems to be an interesting idea. But the benefit of doing so is less convincing in the paper. We have not seen theoretical or analytical study on why jointly learning them with one GFlowNet is useful. Empirically from the experiments of the paper, it is a bit hard to see that the joint learning scheme can help the quality of both the parameters and the structure. Specifically, the proposed method shows advantages on small graphs in Section 5.1, while its performance gain over others in Section 5.2 is a bit marginal in terms of the likelihood of held-out data. More importantly, one of the main purpose of structure learning is to reveal the true graph of the data, where the proposed method has not shown advantage over VBG (the method learns graphs with GFlowNet and parameters with variational inference, respectively) in Figure D.2. Therefore, it is a bit unclear whether people should learn them jointly. 2. The experiments in Figure 3 are about predicting held-out data. In the linear Gaussian case, BCD Nets shows very competitive performance. Given that BCD Nets and VBG are linear models, one can still use them to model data generated from non-linear Gaussian models and predict the held-out data, although it might be hard to use them to compare SHD or AUROC. These two methods may serve as informative baselines for the non-linear case (Figure 3.b).

Questions

1. Is it possible to add the results of BCD Nets and VBG in the non-linear case of Figure 3.b? 2. Is it possible to provide complexity analysis between the proposed method and VBG?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/a

Reviewer rtyV8/10 · confidence 2/52023-07-26

Summary

The present paper extends upon prior literature in making use of GFlowNets for structure learning. The extension is concerned with finding an adequate approach to learning a complete Bayesian network, that is, the graph's structure and its parameters. Theory to support the consistency of the representation and empirical results comparing to state of the art.

Strengths

_Note to the authors: I apologize in advance for being unable to provide an extensive, constructive feedback for your manuscript due to capacity issues on my end. Therefore, the confidence is being kept low. Also, there is an OpenReview bug regarding "First Time Reviewer," which sates 'Yes' although it is 'No.'_ * GFlowNets are an interesting alternative approach to sampling for various tasks of interest * Learning a complete Graphical Structure is the optimum for structure learning * Wonderful presentation, both theory and visualizations/schematics * Self-enclosed work showcasing most/all relevant aspects to answering this research question

Weaknesses

* 'Just' learning the parameters brings a completion to the structure learning approach for GFlowNets but IMHO poses a more marginal/incrimental improvement

Questions

* What are the key assumptions (or 'sacrifices') one has to take for the real world example?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

---

Reviewer MmTz5/10 · confidence 2/52023-07-27

Summary

This paper uses GFlowNets to learn a generative model that samples from the posterior distribution of both the graphical structure and corresponding parameterisation of Bayesian networks given some observation. The paper demonstrates that sub-trajectory balance conditions suffice to ensure that the GFlowNet induce a distribution that is well proportional to the reward function associated with a final sample. Experimental results demonstrate that this approach is able to recover that follows closely the true posterior distribution. The method outperform alternative strategies for linear gaussian CPD and is competitive with alternative approach for non-linear Gaussian.

Strengths

- The problem statement and extension of Gflownets proposed to solve it are very sound. Jointly learning discrete structures and continuous parameterisation has many promising applications in various domains - Although merging continuous and discrete GFlownets does not seem very original in itself, it requires to adapt the balancing conditions appropriately and making it work is a strong results in itself. - Experimental results are very promising.

Weaknesses

Although I have found the idea presented in the paper of learning a generative model over the structure and continuous parameters very interesting, I also find the paper hard to read and understand technically for a reader non-familiar with GFlowNets. For instance, it is unclear exactly how the policy is parameterised and optimised. Although section 3.3 discusses the learning objective it does not provide a clear explanation on how exactly the training process happens, which makes it hard for me to argue for more than a weak accept. I consider important for a paper at a top venue to be clear by itself and to not require to check all supplementary materials and or related work to be understood by someone with the sufficient mathematical background. I would suggest authors to merge sections 2 and 3 in order to save space and give a more precise description of the parameterisation of the policy and training algorithms used. Moreover, I find strange that 5.3 is also so weakly developed in the main materials whereas it hints very important capacity of the proposed method, which might handle well interventional data.

Questions

- Why did not you try more expressive parameterisation of the CPD, such as with normalizing flows? - Why did you only consider N=100 in 5.2? - If the edge feature is a probability why do you used MSE, rather than a better metric for comparing distributions such as the KL?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

/

Reviewer rtyV2023-08-10

Reponse to Author Rebuttal

Thank you for the response. I've read "Advantage of joint posterior" and the argument convinces me more so than before of the benefits of having a joint posterior. Furthermore, thank you for giving your perspective on necessary assumptions on the statsitics of the data, when the data is real world collected. I've also read the argument by Reviewer 9gh3 on the "measure theoretic flaw" and I disagree, that is, I'm siding with your detailled argument that resolves the alleged problem.

Authorsrebuttal2023-08-14

Proof of Theorem 3.1 from a measure theoretical perspective

*(In this comment, we give some details on how to prove Theorem 3.1 of our submission from a purely measure theoretical perspective, using the condition above)* The joint distribution over graphs & parameters is defined over the measurable union space $\mathcal{X} \triangleq \bigcup_{G\in\mathbf{G}}(\{G\}\times \Theta_{G})$, where $\mathbf{G}$ represents the (finite) set of all DAGs over $d$ nodes. We first can prove that if we have transitions $G \rightarrow G' \rightarrow G''$ in the GFlowNet (i.e., $G'$ is the result of adding one edge to $G$, and $G''$ is the result of adding one edge to $G'$), then for any bounded measurable function $h: \Theta_{G}\times \Theta_{G''}\rightarrow \mathbb{R}$, we have: $$ \iint_{\Theta_{G}\times \Theta_{G''}}h(\theta, \theta'')R(G'', d\theta'')P_{B}(G\mid G')P_{B}(G'\mid G'')P_{\phi}(d\theta\mid G) = \iint_{\Theta_{G}\times \Theta_{G''}}h(\theta, \theta'')R(G, d\theta)P_{\phi}(G''\mid G')P_{\phi}(G'\mid G)P_{\phi}(d\theta''\mid G'') $$ We can use the SubTB condition above, observing that $h(\theta, \theta'')$ is a bounded measurable function of $\Theta_{G'}\times \Theta_{G''}$ (constant on $\Theta_{G'}$ for a fixed $\theta$), and using Fubini-Tonelli’s theorem. $$\Bigg[\iint_{\Theta_{G}\times \Theta_{G''}} h(\theta, \theta'')R(G'', d\theta'')P_{B}(G\mid G')P_{B}(G'\mid G'')P_{\phi}(d\theta\mid G)\Bigg]\times\Bigg[\int_{\Theta_{G'}}P_{\phi}(d\theta'\mid G')\Bigg]$$ $$= \int_{\Theta_{G}}\Bigg[\iint_{\Theta_{G'}\times\Theta_{G''}}h(\theta, \theta'')R(G'', d\theta'')P_{B}(G'\mid G'')P_{\phi}(d\theta'\mid G')\Bigg]P_{B}(G\mid G')P_{\phi}(d\theta\mid G)$$ $$= \int_{\Theta_{G}}\Bigg[\iint_{\Theta_{G'}\times\Theta_{G''}}h(\theta, \theta'')R(G', d\theta')P_{\phi}(G''\mid G')P_{\phi}(d\theta''\mid G'')\Bigg]P_{B}(G\mid G')P_{\phi}(d\theta\mid G)$$ $$= \iint_{\Theta_{G}\times \Theta_{G'}}\Bigg[\int_{\Theta_{G''}}h(\theta, \theta'')P_{\phi}(G''\mid G')P_{\phi}(d\theta''\mid G'')\Bigg]R(G', d\theta')P_{B}(G\mid G')P_{\phi}(d\theta\mid G)$$ The quantity inside the brackets is a bounded measurable function of $\Theta_{G}\times \Theta_{G'}$. We can therefore apply the SubTB condition above. $$= \iint_{\Theta_{G}\times\Theta_{G'}}\Bigg[\int_{\Theta_{G''}}h(\theta, \theta'')P_{\phi}(G''\mid G')P_{\phi}(d\theta''\mid G'')\Bigg]R(G, d\theta)P_{\phi}(G'\mid G)P_{\phi}(d\theta'\mid G')$$ $$= \Bigg[\iint_{\Theta_{G}\times\Theta_{G''}}h(\theta, \theta'')R(G, d\theta)P_{\phi}(G'\mid G)P_{\phi}(G''\mid G')P_{\phi}(d\theta''\mid G'')\Bigg]\times\Bigg[\int_{\Theta_{G'}}P_{\phi}(d\theta'\mid G')\Bigg]$$ Which proves the equality above on the product space $\Theta_{G}\times\Theta_{G''}$. By induction, we can also prove that for a partial trajectory $G_{0} \rightarrow G_{1}\rightarrow \ldots \rightarrow G_{T}$ in the GFlowNet, we have for any bounded measurable function $h: \Theta_{G_{0}}\times \Theta_{G_{T}}\rightarrow \mathbb{R}$: $$ \iint_{\Theta_{G_{0}}\times\Theta_{G_{T}}}h(\theta_{0}, \theta_{T})R(G_{T}, d\theta_{T})\prod_{t=0}^{T-1}P_{B}(G_{t}\mid G_{t+1})P_{\phi}(d\theta_{0}\mid G_{0}) = \iint_{\Theta_{G_{0}}\times\Theta_{G_{T}}}h(\theta_{0}, \theta_{T})R(G_{0}, d\theta_{0})\prod_{t=0}^{T-1}P_{\phi}(G_{t+1}\mid G_{t})P_{\phi}(d\theta_{T}\mid G_{T}) $$ Theorem 3.1 of our submission can be rewritten in measure theoretical terms as > If the SubTB conditions above are satisfied for all undirected paths of length 3 between any $(G, \theta)$ and $G', \theta')$ of the form $(G, \theta) \leftarrow (G, \cdot) \rightarrow (G', \cdot) \rightarrow (G', \theta')$, then we have for any bounded measurable function $h: \Theta_{G}\rightarrow \mathbb{R}$ > $\displaystyle \int_{\Theta_{G}}h(\theta)P_{\phi}^{\top}(G, d\theta) \triangleq \int_{\Theta_{G}}h(\theta) \sum_{G_{0}\rightsquigarrow G}\prod_{t=0}^{T-1}P_{\phi}(G_{t+1}\mid G_{t})P_{\phi}(d\theta\mid G)\propto \int_{\Theta_{G}}h(\theta)R(G, d\theta)$ And we can prove this using the lemma above. Since any bounded measurable function $h:\Theta_{G}\rightarrow \mathbb{R}$ is also a measurable function over $\Theta_{G_{0}}\times \Theta_{G}$ (it is constant wrt. $\theta_{0}$), we can directly apply the lemma above (using the notation $G = G_{T}$) $$\Bigg[\int_{\Theta_{G_{0}}}R(G_{0}, d\theta_{0})\Bigg]\times \Bigg[\int_{\Theta_{G}}h(\theta)P_{\phi}^{\top}(G,d\theta)\Bigg]$$ $$= \sum_{G_{0}\rightsquigarrow G}\iint_{\Theta_{G_{0}}\times\Theta_{G}}h(\theta)R(G_{0}, d\theta_{0})\prod_{t=0}^{T-1}P_{\phi}(G_{t+1}\mid G_{t})P_{\phi}(d\theta\mid G)$$ $$= \sum_{G_{0}\rightsquigarrow G}\iint_{\Theta_{G_{0}}\times\Theta_{G}}h(\theta)R(G, d\theta)\prod_{t=0}^{T-1}P_{B}(G_{t}\mid G_{t+1})P_{\phi}(d\theta_{0}\mid G_{0})$$ $$= \Bigg[\int_{\Theta_{G_{0}}}P_{\phi}(d\theta_{0}\mid G_{0})\Bigg]\times \Bigg[\int_{\Theta_{G}}h(\theta)R(G, d\theta)\sum_{G_{0}\rightsquigarrow G}\prod_{t=0}^{T-1}P_{B}(G_{t}\mid G_{t+1})\Bigg]$$ $$= \int_{\Theta_{G}}h(\theta)R(G, d\theta)$$

Authorsrebuttal2023-08-14

Additional response regarding complexity analysis

*(This comment contains an addendum to our rebuttal, due to the limited space)* > *Is it possible to provide complexity analysis between the proposed method and VBG?* Complexity analysis might be difficult to derive since both of these methods involve training neural networks. However as described above, JSP-GFN is conceptually simpler than VBG as it only involves training a neural network with stochastic gradient methods, as opposed to VBG which requires a more complex inner-loop/outer-loop procedure, each inner-loop itself consisting in training a neural network.

Reviewer 5gCu2023-08-17

Thanks for the response and additional experiments.

I've read the authors rebuttal and other reviewers' reviews. I saw a reviewer questioned the core theory of the detailed balance of the paper. But I'm not an expert in the particular theory so I look forward to seeing more discussions between the reviewer and the authors. Apart from that, I think the paper studies GFlowNet for causal discovery, which is an emerging and less studied area, although the idea is slightly incremental and the performance gain is a bit marginal. So I will keep my score.

Authorsrebuttal2023-08-14

Additional responses

*(This comment contains additional responses to the review, due to the limited space)* > *While the theoretical framework is explained in much detail, the actual implementation is quite opaque to me. The authors could spend more time in explaining the architecture and implementation details.* > We invite you to read the general comment (**Practical implementation**). For the details of the neural network architecture, we could add some additional details in the appendix too, but we believe that they would not contribute to the overall description in the main text. > *A simple educational example that is dissected in detail could help to better understand the framework.* > We were hoping that the illustration in Figure 1 (along with the caption, and the main text in Section 3.1) would be sufficient for understanding how sampling from the GFlowNet works; we plan to add pseudo-code to describe the training process of JSP-GFN (see also the general comment, **Practical implementation**). A more detailed example would require either (1) specifying the whole state space, which could be too cluttered, or (2) specify real values of the forward transition probability $P_{\phi}$ to illustrate the process, both of which would be challenging to illustrate easily. However we are open to any suggestion to make this part as clear as possible. > *“You are strongly encouraged to submit code that produces this result.” [NeurIPS guidelines].* > We invite you to read the general comment (**Code**). > *The experimental results are restricted to some performance metrics, but especially in the biological datasets, it would be interesting to dig deeper. Could JSP-GFN discover known gene regulatory networks? How do posterior predictives look compared to the real data? What role does interventional vs. observational data play?* > We completely agree with you, and an analysis of the networks found by JSP-GFN against known regulation pathways would be extremely interesting. We are using the negative log-likelihood on held-out interventions to follow the evaluation of (Sethuraman et al., 2023), which introduced the dataset we are using here (specifically, the subset of genes). We would be cautious with interpreting the gene regulatory networks found by our method since they assume that the graphs are acyclic, thus ignoring potential feedbacks existing in real biological systems; see also Appendix B.2 for a discussion of these limitations for biological data. > *The authors only touch on some limitations and a more detailed discussion would strengthen the paper.* > We can suggest adding the limitations highlighted above in terms of challenges for scaling the size of the graphs and size of datasets, to complement the limitations already mentioned in Appendix B.2. --- (Sethuraman et al., 2023) Muralikrishnna G. Sethuraman, Romain Lopez, Rahul Mohan, Faramarz Fekri, Tommaso Biancalani, Jan-Christian Hütter. NODAGS-Flow: Nonlinear Cyclic Causal Structure Learning. AISTATS 2023.

Reviewer jHMM2023-08-15

I thank the authors for their response and I appreciate their additional explanations (although ignoring the NeurIPS guidelines of 6000 characters several times). As pointed out by several reviewers, the current manuscript needs further clarifications not only in terms of the practical implementation but also on the overall comprehensibility. If the authors don’t dare to interpret the results of the experiments on the real data, I don’t see any practical use in showing that we learned some abstract object which is not (?) useful to interpret. As such a (meaningful) application to real-world data is missing. Further, I find the (implicit) assumptions of modularity and thus independence of the corresponding posteriors rather strong and a major weakness. The authors should discuss this more clearly in their work. Given these comments and unsatisfactory responses on scalability, I will decrease my score but increase my confidence as I invested a reasonable amount of time to read the related literature and I am quite familiar with other approaches in this domain. *PS: I did not take into account Reviewer 9gh3 ‘s comment, and assume that there are no major theoretical flaws.*

Authorsrebuttal2023-08-15

Thank you for your response. We are quite surprised to hear that you are decreasing your score given our rebuttal. > *As pointed out by several reviewers, the current manuscript needs further clarifications not only in terms of the practical implementation but also on the overall comprehensibility.* We provided further explanations of the practical implementation in the general comment (**Practical implementation**), and suggested to include this in the final version of the paper. We would like to emphasize once again that all the components are already described in the paper. In terms of comprehensibility, we have addressed the points raised by Reviewer MmTz, and we welcome more precise aspects that you think may be updated to improve comprehensibility. > *Further, I find the (implicit) assumptions of modularity and thus independence of the corresponding posteriors rather strong and a major weakness. The authors should discuss this more clearly in their work.* The assumption of modularity is a widely used assumption in structure learning. Most of the score-based methods for structure learning make this exact assumption, under the name of *score decomposability*. For example in (Koller & Friedman, 2009): > Score decomposability has important ramifications when we search for structures that maximize the scores. The high-level intuition is that if we have a decomposable score, then a local change in the structure (such as adding an edge) does not change the score of other parts of the structure that remained the same. As we will see, the search algorithms we consider can exploit decomposability to reduce dramatically the computational overhead of evaluating different structures during search. *(Section 18.3.6.2)* > > Let $G$ be a network structure, let $P(G)$ be a structure prior satisfying structure modularity, and let $P(\theta_{G} | G)$ be a parameter prior satisfying global parameter independence and parameter modularity. Then, the Bayesian score over network structures is decomposable. *(Proposition 18.3)* And this assumption is made in an overwhelming majority of the score-based methods for structure learning. For example in the classic GES algorithm (Chickering, 2002): > For any scoring criterion $S(G, D)$, we say that $S$ is decomposable if it can be written as a sum of measures, each of which is a function only of one node and its parents. [...] Most scoring criteria derived in the literature are decomposable. [...] [W]e show how using a decomposable scoring criterion leads to an efficient implementation of the two-phase greedy search algorithm. As far as we know, there is no well-established algorithm using a scoring function that is not satisfying this assumption. However, since you are quite familiar with the other approaches in this domain, it would be very helpful if you could share any work *not* making this modularity assumption. > *Given these comments and unsatisfactory responses on scalability* Can you give some details about what in our response regarding scalability is unsatisfactory? We have been as precise as possible in our response, showing the range of application of our method (backed up by our experiments), and giving insights as to why scaling even further would be challenging (both in terms of size of graphs and size of datasets). As far as we know, JSP-GFN scales to the same sizes of graphs and datasets as any other existing Bayesian structure learning method. --- (Koller & Friedman, 2009) Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques. (Chickering, 2002) David Maxwell Chickering. Optimal Structure Identification With Greedy Search. JMLR.

Reviewer jHMM2023-08-15

Thank you for your fast response. I agree with the authors that they gave some explanation on the scalability, however, I have not seen any experimental evidence on the comments raised in the initial review. If I understood it correctly, JSP-GFN “offers an accurate approximation of the joint posterior […] on both simulated and real data.” While the power of JSP-GFN needs to be shown, on real data, I suspect that for many real-world problems the marginal posterior $P(\theta|G, D)$ might have non-neglectable correlations. Learning this marginal posterior can be seen as a classical inverse problem for a data-generating process $G$, which might be composed of several components $G_i$. For such a problem, all kinds of methods exist to learn non-modular (correlated) posterior distributions, starting from standard MCMC approaches for models with known likelihood, over ABC (Sisson et al. 2018) or more recent SBI approaches (Cranmer et al. 2020) fueled by normalizing flows (Papamakarios et al. 2021). While the authors highlight the to learn the joint posterior, they focus on the perspective of structural network learning. --- Sisson et al. Handbook of approximate Bayesian computation. CRC Press. (2018) Cranmer et al. The frontier of simulation-based inference. PNAS. (2020) Papamakarios et al. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research. (2021)

Authorsrebuttal2023-08-16

We are indeed focusing on structure learning of Bayesian networks, and accordingly we are making standard assumptions in this field. Our goal with JSP-GFN is not to approximate the joint posterior between any data generating process $G$ and its parameters, but to approximate the joint posterior between the structure of a Bayesian network and the parameters of its CPDs (lines 5-7 in our submission: *"we propose in this paper to approximate the joint posterior over not only the structure of a Bayesian Network, but also the parameters of its conditional probability distributions."*). We understand that the methods linked above may be used for approximating non-modular posteriors, but the problems considered are *not* structure learning of Bayesian network. Once again, if you know any existing work *specifically on structure learning of Bayesian networks* that is not making this modular assumption, we would be happy to hear more about it. However, we would like to highlight that our framework *could* model correlations between different $\theta_{i}$s: we can model $P(\theta\mid G, stop)$ using any distribution, including non-factorized ones. That being said, just from a practical point of view this is not desirable: in our experiment on gene expression data, we have a total of $|\theta| = 61, 671$ (line 909), and having a model approximating $P(\theta\mid G, \mathcal{D})$ with a full $|\theta| \times |\theta|$ covariance matrix (e.g., if we approximate the conditional posterior over the parameters using a Normal distribution) means that the neural network predicting this covariance matrix would have to return $\approx 10^{9}$ outputs, which would be unreasonable. That's why the literature on Bayesian Neural Networks has focused on fully factorized posterior approximations. This is true for *any model* approximating $P(\theta\mid G, \mathcal{D})$, not specifically JSP-GFN.

Reviewer jHMM2023-08-18

I apologize for my harsh critique about the modularity. I don’t know any structure learning method for Bayesian networks not making this assumption. But as the authors highlight, their method is more than structure learning. All other comments still hold.

Area Chair xZwM2023-08-15

Reviewer 9gh3: Reply to Potential Theoretical Flaw

Dear Reviewer 9gh3, The authors have replied to your main concern regarding a potential theoretical flaw (with measure-theoretic and trans-dimensional issues). Have their reply and their additional comments addressed this concern? Do these change your assessment of the paper?

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC