Characteristic Circuits

Summary

I am not qualified to review this paper.

Strengths

I am not qualified to review this paper.

Weaknesses

I am not qualified to review this paper.

Questions

I am not qualified to review this paper.

Rating

10: Award quality: Technically flawless paper with groundbreaking impact, with exceptionally strong evaluation, reproducibility, and resources, and no unaddressed ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

I am not qualified to review this paper.

Summary

This work proposes a new tractable probabilistic model called characteristic circuits or CC. The CC is defined in a similar way to probabilistic circuits (PCs) but with leave nodes defined as characteristic functions instead of the distributions as in PCs. The authors further show the computation of marginals and the learning algorithms for CC. Empirical evaluations on both synthetic datasets and UCI datasets are presented.

Strengths

- This paper is generally well-written, with sufficient backgrounds provided to help readers understand the proposed new model. - The authors show that CC shares the same efficient marginal computations as in PCs. - CC provides a unified view for mixed continuous and discrete distributions. Also, the \alpha-stable distributions are less explored in the previous literature of tractable probabilistic models to my knowledge and the use of these seems help deliver good performance in the UCI experiments.

Weaknesses

- From what is presented, it seems that the authors simply rewrite the distribution nodes and computations of PCs into their characteristic function duals. It is unclear what are the key differences between CC and PCs and when would one prefer one over the other. I don't think the unified view of the discrete and continuous distributions serves as a strong motivation since mixed SPN can also handle the mixed distributions. - To further illustrate the previous point, one would expect to see investigations on expressiveness, such as are there any distributions that can be tractably represented by CCs but not PCs, or investigations on tractable operations, such as what probabilistic queries would be intractable for PC but tractable for CC, while none of these are discussed in this work. - A proof for the validity of the marginal computations seems to be missing.

Questions

- Can the authors elaborate on the motivation for CCs, that is, when one would prefer CC over PC? - Can the authors provide some discussions on the expressiveness and tractable queries of CC? - For the empirical results on UCI datasets, I wonder if the improvements are from CC itself or from the alpha-stable distributions. What would happen if the continuous leaves in MSPN are defined as alpha-stable distributions? Such an ablation study might make the results more convincing.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

Yes.

Summary

The paper introduces characteristic circuits (CCs), a new family of tractable probabilistic models (TPMs) that leverages univariate characteristic functions as leaves of probabilistic circuits (PCs) for modelling a tractable joint of heterogeneous data distributions (i.e. with both continuous and discrete variables). CCs model the characteristic function of the data distribution in the continuous spectral domain (cf. Equation 1), thus providing a unified framework for discrete and continuous random variables. As a consequence, one of the main advantage of CCs is that they can model distributions that do not have closed-form probability density functions, such as $\alpha$-stable distributions. Importantly, authors also show that CCs allow exact and efficient computation of joint and marginal probabilistic queries. CCs are evaluated on two synthetic datasets and 12 heterogeneous real-world tabular datasets.

Strengths

- The research is definitely original as it proposes a new class of TPMs with many (novel) benefits - The model naturally lends itself to modelling heterogeneous data - The model allows to use distributions that do not have closed-form expressions, such as $\alpha$-stable distributions, something that is not possible in current PCs - Despite having input units with no closed-form expressions, CCs can still deliver exact marginalisation Overall, a solid contribution.

Weaknesses

- From experiments, it looks like inappropriate structure can limit the modelling power of CCs. This is can prevent using CCs when a good structure is not available. - It's unclear how reliable/precise numerical integration can be (lines 128-129-130) - Sampling it's an important inference routine of PCs yet it is not discussed at all, and it's unclear if CCs can provide it

Questions

- Is sampling possible from CCs? If yes, can we know how CC samples compare with the ones of standard PCs? If not, what are the challenges? - Can you elaborate a bit more on what precisely you mean with "unified view for discrete and continuous random variables" (line 57)? While, to some extent, I understand what author mean, one may think "but even standard PCs can handle heterogeneous data". I think being more precise here can improve this major selling point of the paper. - Can you elaborate a bit more on lines 230-231-232? Why MLE is not tractable? How does Eq. 14 relate to MLE? (Also lines 248-249-250 are unclear to me)

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are not explicitly addressed. It looks like sampling can be one of these. Minors: - In Lemma 4.2, I think there's no explanation of what $\tau$ and $E(\mathcal{T}_i)$ represent. I know it's notation related to induced trees, but it is not introduced in the text. - In line 175, there's no $x$ occuring in the definition of $\phi_{L_\text{Normal}}(t)$, why?

Summary

This manuscript proposes a framework for directly representing characteristic function of random variables by a probabilistic circuit-like structure. Unlike the ordinal probabilistic networks, the proposed framework, characteristic circuits, can treat distributions that do not have closed-form expressions for the density, or even marginals of discrete and continuous random variables. This is because the characteristic function provides a unified view for both discrete and continuous random variables. As a result, the characteristic circuit can treat broader class of distributions than the ordinal probabilistic circuits. Despite this, it is proved that the characteristic circuit keeps tractability of computing densities and marginals, which is an important query for probabilistic inference. Also, the parameter and circuit-structure learning algorithms for characteristic circuits are given. The experiments showed that the proposed characteristic circuits perform well for density estimation task on heterogeneous data sets, which includes both discrete and continuous variables.

Strengths

I think that representing a characteristic function by a circuit is a simple yet strong idea for representing broader class of distributions. Also, proposing an algorithm for computing marginals on characteristic circuits is really good, because it inherits the strengths of the original probabilistic circuits that some rich inference queries are tractable in time proportional to the size of the circuit; although it only proves the marginal query, I think this query is one of the most important one for probabilistic inference. The experimental results truly support the usefulness of characteristic circuits when applied to density estimation task where the evaluation metric is the test log-likelihood. Since the original probabilistic circuits also show their strengths in this task, I think the selection of tasks is appropriate.

Weaknesses

It is not the first attempt to represent a generating function regarding random variables directly; the first one (as far as I know) is: Probabilistic Generating Functions https://arxiv.org/abs/2102.09768 (published in ICML 2021). This represents the probability generating function directly, thus I think this framework is for discrete variables only. However, to clearly show the standing position of this paper, the comparison with this work should be clearly done in the main article.

Questions

Within the queries that are tractable for the ordinal probabilistic circuits, are there any other tractable queries than marginal for characteristic circuits? Or, are there any queries that is proven to be hard (e.g., NP-complete) for characteristic circuits?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

I think the authors adequately addressed the limitations.

Summary

This paper studies the use of characteristic functions as probabilistic models for heterogeneous data and proposes characteristic circuits (CC) for their representations. The authors propose efficient algorithms for computing (marginal) densities with CCs and show that parameters of CCs can be learned by minimizing the CFD between CC and ECF. The authors also show that CCs achieve strong performance not only on synthetic data but also on some commonly used density estimation benchmarks.

Strengths

To the best of my knowledge, the use of characteristic functions as a new language for probabilistic modeling, especially as a unified framework for heterogeneous data, is very novel. The empirical results is also very strong. I believe this work opens a brand new avenue for density estimation.

Weaknesses

As a non-expert, I spent a lot of time on the background section; despite the uniqueness part in the inversion theorem, it is not completely intuitive how a probability measure is encoded as its characteristic function. It would be helpful if the authors could provide at least one simple example. Authors use too many acronyms throughout the paper, especially in Section 5; it is not easy for me as a reader to distinguish between CC-N, RS, SL, CFD and etc. For non-expert readers like me, more details on the empirical evaluations (in the main paper) could be helpful: e.g. how are the likelihoods measured for the heterogeneous datasets? are they computed via numerical integrations? etc.

Questions

In Section 4.2, the authors proposes to do parameter learning by minimizing the CFD between CC and ECF because the likelihood of CC is not guaranteed to be tractable. Yet I wonder if it is possible to discuss the relationship between CFD and likelihood.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The structure learning algorithm seems more of an adaptation of the existing structure learning algorithms for SPNs. This is not a major issue as structure learning of circuits has been a very challenging problem and probably not a main focus of this work.