Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits

Summary

This paper extends the work of PIC with continuous LVs, proposes to tensorize and to use neural functional sharing to scale the model. The results show that the proposed approach decreases the trainable parameters as well as the memory usage in training, and also reduces the running time. The approach also works well in density estimation tasks.

Strengths

This paper extends the work of PIC for better scalability and efficiency, and thus has its novelty. The paper is well written and not difficult to understand. e.g., the illustration of Algorithm 1 in Fig.2 (a-b) is a very nice visualisation. With the claimed contribution, the paper has a high potential to contribute to the community.

Weaknesses

It is not clear to me what data set is used in the experiment of "Scaling PICs", and how many RVs are modelled. I compared the density estimation results in PIC[18] on the MNIST family and the results in this paper outperforms the ones in PIC[18]. As far as I understand, the PIC (F, C) in this paper has reduced its trainable parameters and therefore should in principle has smaller model capacity. I am wondering why the density estimation results of the proposed model outperform the original PIC? Similarly, it would be nice if the drop (or increase, though unlikely) of density estimation performance can be shown between PIC (F, N) and PIC (F, C) so readers will know how much they can lose/gain w.r.t. performance when applying the functional sharing.

Questions

- Could you elaborate why the trainable parameters in PIC (F, C) have reduced significantly compared with the PC (F, N), since there is no functional sharing in PCs as I understand. That is, why the PIC (F, C) can have much fewer trainable parameters than PC (F, N)? Is this comparison fare? - In the left most plots in Fig 5, I would infer that the single orange triangles in upper left are PIC(F,N) with QG-TK, but without a dotted line connected it is a bit unclear. - is the ```and``` before $||$ in line 156 a typo?

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

The limitations are well discussed in the end of the paper.

Summary

This paper presents a pipeline to build probabilistic integral circuits (PICs) more generally as directed acyclic graphs, as opposed to the tree-shaped structure they were limited to before. This significantly increases the expressiveness of the PICs, improving their representation of distributions. The authors present a procedure for turning a region graph into a PIC, into a QPC that approximates the PIC, and finally into a folded QPC, which are effectively tensorized structures that significantly speed up the inference and learning process. From the tensorized PIC the authors then make further improvements on QPC training, using neural functional sharing techniques over groupings of the input and integral units to result in greater materialization of QPCs and far fewer parameters.

Strengths

+ The proposed approaches significantly increase the expressivity and learning efficiency of PICs, as showcased by the experiments. + The paper is technically sound with clear arguments presented for each step. + The paper is overall well structured and easy to follow. + The authors also address the current limitations of PICs with a clear goal for future improvements.

Weaknesses

Not a major weakness, but as the authors also point out, it is currently not possible to sample from PICs, limiting their impact as a generative model.

Questions

1. Among the techniques discussed in the paper, how significant is the tree-shape vs DAG-shape for empirical expressiveness of PICs? It appears that DAG-shaped PICs achieve slightly better bpds, but tree-shaped ones scale much better. 2. Section 3.2 describes a numerical quadrature rule as consisting of integration points and weights that minimize the integration error, and the authors later say that the same quadrature rule is used for each approximation. Does this raise any issues in approximation? 3. How crucial/limiting is the assumption that every integral unit integrates out all incoming latent variables? 4. Do the QPCs approximating PICs still support tractable inference (e.g. marginal probabilities) similar to traditional PCs? It would be great to clarify the probabilistic reasoning capabilities mentioned in the introduction. Minor comments: - Lines 133-134: do you also need $Z_{u_3}\neq Z_{u_4}$ for CP-merge? - Table 1: what does LVD-PG refer to?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors acknowledge the limitation of PICs that, despite being a continuous latent variable model like many generative models, they do not support sampling. This is left as future work.

Summary

This paper introduces a new approach for building probabilistic integral circuits (PICs), a recently introduced probabilistic model that extends probabilistic circuits (PC) with continuous functions of latent variables (in addition to discrete latent variables), with inference performed using a numerical quadrature procedure. In particular, while the previous implementation was restricted to tree-structured region graphs/variable decompositions, it is shown how to construct PICs for arbitrary region graphs. Additionally, sharing of functions across different regions is used to improve parameter efficiency. Empirical results show that the architectural improvements enable scaling to complex image datasets such as ImageNet64, outperforming comparable PCs trained using maximum likelihood or EM.

Strengths

The paper proposes a number of innovations on top of the recently proposed PICs, such as allowing for arbitrary region graphs and functional sharing. Taken together, this constitutes a well-engineered solution that significantly improves the scalability and flexibility of PICs. The experiments are well-executed with strong results on a range of image datasets in comparison to comparable PCs learned directly. Many experimental details are reported, such as memory usage, time taken and parameter counts/hyperparameters, which should make the results reproducible. The paper was a pleasure to read, with clear and consistent notation throughout explaining the method and accompanying pictorial illustrations that make the idea of the paper transparent and easy to understand.

Weaknesses

The paper is arguably a little weak in terms of novelty as most of the new components are adaptations of existing ideas in the literature (e.g. tucker/cp layers, parameter sharing) to probabilistic integral circuits.

Questions

- What numerical quadrature rule was used? - In the conclusion, it is remarked that differentiable sampling from PICs is not possible. What is meant by differentiable here, and is it possible to sample from the materialized PC if one is not interested in gradients (e.g. for visualization)? - To what extent do the authors consider their method a continuous latent variable model, as opposed to a means of effectively constructing PCs? For instance, if the same quadrature points are used throughout training and evaluation, can one expect the MLPs to learn meaningful function values between evaluation points? - Did the authors investigate learning PICs according to a HCLT region graph? This would give a better idea of whether a DAG (as opposed to tree-structured) region graph is necessary. Typos: - modes -> nodes 190 - capitalization of 'functional' 234 - 'pixels are categorical' -> 'pixels as categorical' 286

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The limitations of the method are appropriately addressed in the paper.

Summary

This work extends the probabilistic integral circuits (PICs) from tree-shaped region graphs (RGs) based ones to DAG-shaped ones. While constructing PICs from DAG RGs can lead to more expressive models, it comes with the concern of scalability since the circuit sizes might be increased a lot. To address this concern, this work proposes tensorized circuits to approximate the resulting PICs by a PIC to QPC reduction. This is further combined with functional sharing and materialization to achieve efficiency and scalability, which is validated by experimental results on several image datasets.

Strengths

- The extension from tree-shaped PICs to DAG-shaped ones is a solid contribution as it greatly improves the expressiveness of the resulting PICs and is an important step towards PICs constructed from more complex RGs for complicated tasks. - The functional sharing and materialization techniques proposed in this work are shown to be very useful by the empirical results. From Fig 5, learning PICs consumes similar resources to PCs which are efficient models and it requires much less trainable parameters than the ones without functional sharing, making the training of large-scale PICs feasible.

Weaknesses

I appreciate that this work aims to solve an important problem and the proposed solution is shown to be novel and efficient. Still, it's heavy in technical details and thus I don't think I fully understand the algorithmic details. Specifically, - I'm confused by the definition of the integral unit from Line 58 to Line 62, especially by the inconsistency in the input of g_i between Line 61 and Line 62 (the g_i inside the integral). Also, the integral in Line 62 contains the function g_i while Figure 1(b) does not, which is also confusing. It might be helpful to put a numeric example there. - This also applies to the other technical sections of this work given that this work is heavy in techniques. It would be very helpful to provide a toy example to showcase the reduction of PICs to QPCs to improve accessibility and reproducibility for readers who are not well-versed in these areas. - I'm curious to see some theoretical analysis on the circuit sizes characterized. I understand that the bounds might be loose and not that informative. Still, it might be helpful to understand how the PIC to QPC reduction affects the circuit sizes.

Questions

- Can you provide some examples to help understand the integral units? - From the definition of g_u in Line 71, the integral is a function of both variables X_i and Z_u. Can you explain the claim "In this way, the output LVs of any unit u will simply be Zu" that directly follows the definition of g_u? - Can you characterize the circuit sizes of the resulting PICs and QPCs?

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

Yes.