Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration

Summary

The paper introduces hierarchical semi-implicit variational inference (HISIVI) as an extension of semi-implicit variational inference (SIVI). HISIVI incorporates interpolating distributions between prior and target data to train conditional layers progressively, resulting in accelerated sampling in diffusion models.

Strengths

The paper is well-structured and easy to follow.

Weaknesses

- The novelty of HISIVI compared to related work is not clearly demonstrated. It is recommended to include more related works and comparison: cascaded diffusion models[1], diffusion schrodinger bridge[2], flow matching[3], or rectified flow[4], etc. - The experiment section is relatively weak. Comparisons could strengthen the evaluation (aforementioned references and more [5],[6], etc.) [1] Ho, J., Saharia, C., Chan, W., Fleet, D.J., Norouzi, M. and Salimans, T., 2022. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1), pp.2249-2281. [2] De Bortoli, V., Thornton, J., Heng, J. and Doucet, A., 2021. Diffusion Schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34, pp.17695-17709. [3] Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M. and Le, M., 2022. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. [4] Liu, X., Gong, C. and Liu, Q., 2022. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003. [5] Xiao, Z., Kreis, K. and Vahdat, A., 2021. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.07804. [6] Zheng, H., Nie, W., Vahdat, A., Azizzadenesheli, K. and Anandkumar, A., 2022. Fast sampling of diffusion models via operator learning. arXiv preprint arXiv:2211.13449.

Questions

How does the proposed optimization for score function in HISIVI outperform or differ from the conventional implicit score matching or denoising score matching[7]? [7] Vincent, P., 2011. A connection between score matching and denoising autoencoders. Neural computation, 23(7), pp.1661-1674.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

1 poor

Limitations

This work only deals with the geometric interpolation of distributions without providing justification. Exploring the impact of choice of interpolation on generation would enhance the quality of this work.

Summary

Semi-implicit variational inference increases the expressiveness of variational posteriors but introduces intractabilities to their inference. This work adds to the semi-implicit variational inference (SIVI) work of Yu and Zhang (https://openreview.net/forum?id=sd90a2ytrt) which uses an alternative objective (the Fisher divergence) that, combined with the hierarchical nature of semi-implicit variational distributions, can be approximated as the solution of a mini-max optimization problem. This work takes inspiration from simulated annealing and denoising-diffusion models by proposing to expand the single-layer semi-implicit variational family into multiple-layer variants by specifying hierarchical auxiliary distributions to guide the semi-implicit variational distribution towards the target distribution. The authors denote this method Hierarchical Semi-Implicit Variational Inference (HSIVI). The auxiliary distributions are specified either by assuming the marginal densities of a geometric interpolation between a target and base distribution are available analytically or by using pre-trained scores from a diffusion bridge. The main contribution to this paper is stated in the author's claim that "when used for diffusion model acceleration, we show that HSIVI can produce high quality samples comparable to or better than the existing fast diffusion model based samplers with small number of function evaluations on various datasets." This claim is tested empirically on the CIFAR-10 (32x32) and CelebA (64x64) datasets with results showing comparable or better FID scores to existing fast diffusion model samplers with few (between 5 and 15) function evaluations. As well as the main contribution, an algorithm for training is provided by noting that the conditional layers can be trained sequentially until convergence by exploiting the hierarchical structure of the semi-implicit distribution. Also, an efficient parameterization of the neural networks is provided to make the algorithm computationally feasible.

Strengths

Quality The work is a novel combination of well-known, peer-reviewed techniques including simulated annealing and SIVI. It is clear how this work differs from the work of Yu and Zhang (https://openreview.net/forum?id=sd90a2ytrt) using the hierarchical semi-implicit variational posteriors, and the value of this contribution is demonstrated on a multi-modal problem in Figure 2. The methods used seem to work well judging by the experimental results. Clarity The submission is well-organized and clearly written.

Weaknesses

Originality There is limited technical novelty in this paper as the technical content on the work is almost entirely covered by the work of Yu and Zhang (https://openreview.net/forum?id=sd90a2ytrt) that rewrites the Fisher divergence leading to a form that does not require computing the score of the hiearchical variational posterior. The paper extends this by sequentially score matching to noised versions of the target distribution, which have been obtained analytically or approximated from pretrained denoising diffusion models. The main novelty in this work is instead an experimental one, showing that HSIVI can produce high quality samples comparable to or better than the existing fast diffusion model based samplers with a small number of function evaluations on two datasets by comparing FID scores (CIFAR-10 (32x32) and CelebA (64x64)) and one dataset by comparing samples visually (MNIST). The experiments were made computationally feasible by a trick to parameterize the neural networks that is detailed in Proposition 1. Quality of results Relatively simple experiments are provided that on one hand demonstrate the contribution well, but on the other hand do not cover a very diverse range of cases. No empirical failure case is provided for a complicated distribution that requires many function evaluations to sample from, which may provide an interesting avenue for future work. On the reproducibility of the results, whilst the experimental setups are well detailed, no code is provided with the paper, nor indication it will become available upon publication. The reviewer would like to see that the code is made available since this is crucial for reproducibility. Furthermore, there may not be enough information to replicate the experiments. For example, as far as I can see, there is no discussion on the parameterization/values of the positive weighting function, beta(t) for joint HSIVI-SM training on CIFAR-10 and MNIST, which makes it impossible to reproduce without any code provided. Significance The work's significance would be improved if it addressed why we should expect improvements in sampling efficiency over denoising-diffusion sampling and where these improvements can be derived from (the particular form of variational family, or something else?).

Questions

1. It would be interesting if the authors could demonstrate a failure case for their hierarchical variational posterior approximating the denoising-diffusion model, what properties of the score in the backwards process (equation 9) would cause the proposed variational posterior to fail with a small number of function evaluations? 2. No interpretation of the neural network f_t(x_t) is given, apart from that it provides "guidance". Could the authors please clarify what is being guided, and to what? 3. Comparing the proposed sampling algorithm to denoising-diffusion: the sampling algorithm for denoising-diffusion uses a reverse-time Markov chain that has specified isotropic variance that appears due to the assumed forward-time transition densities. Whereas the proposed HIVI sampling algorithm in this work is a Markov chain with transition densities that are the conditional variational posterior distributions (called the conditional layer in this work). The difference between this Markov chain and the denoising-diffusion one is the trainable non-isotropic variance of the conditional layer q_t(\cdot|x_{t+1}; \phi). If the conditional layer is instead modelled as a Gaussian distribution with mean and an isotropic scale-Identity matrix, then do you expect that there is no improvement over denoising-diffusion with the same number of steps? It would be good to see an experiment that showed the trade off between sample quality and number of function evaluations using this simpler variational family, which is on par with the denoising-diffusion sampling. Typos: Fisher divergence is not symmetric in its arguments and has been incorrectly written with arguments transposed (the expectation is taken over the distribution of the first argument). See for example how the Fisher divergence is written in the papers that this work has cited https://arxiv.org/pdf/1602.03253.pdf or https://arxiv.org/pdf/1810.03545.pdf Line 510, 511, 512 there are typos in the norm regularizing the f network (written as f_{t}(x; \psi_t) and should be f_{t}(x_{t}; \psi_t))

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

4 excellent

Contribution

2 fair

Limitations

The limitation discussion picks up on the important points, but it would be good to know if higher dimensional problems were attempted, or problems with more parameters (such as the "huge VP deep continuous-time model (Song et al., 2020b) that has more channels and layers"), and did the authors run into memory issues that were discussed in Appendix F? An ablation of the variational family for the conditional layers (as described in question 3) comparing directly to the denoising-diffusion sampling remains unexplored, and would be useful to see.

Summary

Authors propose a hierarchical semi-implicit variational inference framework by extending the existing SIVI. Authors showed that the proposed HSIVI, given pre-trained score networks, can be used to accelerate the sampling process of diffusion models with the score matching objective. The numerical results show more enhanced performance in faithful modeling of complex distributions and diffusion model acceleration.

Strengths

The paper is fairly well-written, clearly motivated and generally addresses an important problem. The extension of SIVI to a hierarchical model, although not very novel, is intuitive and makes sense. I haven't checked the proofs but the overall approach seems to hold up.

Weaknesses

- Extending SIVI to a hierarchical model, by itself, is not very novel. However, the application to diffusion acceleration is interesting. - The experiments and presented numerical results could be improved. (check next section for more)

Questions

- It is mentioned in the manuscript that HSIVI requires more memory. How much more memory (with respect to data dimension) does it need compared to baselines? - ImageNet (64x64) is a common benchmark dataset for diffusion models. I would be nice to have the results for that as well. - How does the proposed model performs compared to Diffusion Normalizing Flows [1] both in terms of target distribution approximation and sample generation. - [A suggestion] It would be interesting to extend HSIVI on graphical models (in a setup like [2]). [1] Diffusion Normalizing Flow, Qinsheng Zhang and Yongxin Chen, NeurIPS 202. [2] Efficient Inference Amortization in Graphical Models using Structured Continuous Conditional Normalizing Flows, Christian Weilbach, 2019.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitations to some extent have been discussed but more discussion (specially around memory usage) is needed.

Summary

The authors extend semi-implicit variational inference to have multiple layers of latent variables, vastly increasing the. expressiveness of the model. A nice formulation of the asymptotic lower bound plus a training scheme is introduced as well.

Strengths

This paper is an extension that on the surface seems straight forward. But the authors demonstrate that this is far from the case and introduce a novel formulation of the semi-vi ELBO. The paper was also written well and easy to follow. The experiments section was great too.

Weaknesses

My only qualm is the flow between the first half of the paper and the second half where the focus is on diffusion models. Previously, the authors argued that training using a sequence of distributions would lead to better marginal distributions. But for speeding up sampling of diffusion models, it seems like training the joint distribution leads to impressive results. This opens up the question of whether the sequence of distributions is even needed at akk.

Questions

I don't have any questions.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

It seems like the main text is missing a limitations section.

Summary

The paper introduces a hierarchical semi-implicit variational inference that stacks multiple semi-implicit layers to construct a flexible generative distribution. The paper introduces a sequence of distributions that interpolate between the target and a base distribution. Each pair of intermediate inference and target distributions are matched by optimizing a SIVI bound. The method is applied in the diffusion model for sampling acceleration. ##Post-rebuttal Thanks for the authors' response. Most of my questions have been addressed and I will maintain my current score.

Strengths

- The idea of using semi-implicit distribution for the progressive approximation is interesting and intuitive. - The empirical results show HSIVI improves the distribution approximation and sample quality significantly on several tasks. - Though the modeling and training of HSIVI consists of many components, the paper explains the ideas concisely and clearly.

Weaknesses

- The validity of the training procedure needs more concrete explanations or theoretical results - The empirical study needs more evidence for the improvement of acceleration. - Some closely related works need more discussions of similarity and improvement. Please see Questions for more details.

Questions

- To train HSIVI sequentially, the paper mentions, “Let the parameters ϕ_t in the t-th conditional layer be independent across different ts.” But since q_t(x_t; ϕ_t) depends on x_{t-1} that depends on ϕ_{t-1}, It is understandable that the graph detaching in the algorithm benefits computation efficiency. Does it have any downside for not training ϕs jointly? For example, will there be convergence issues, or will some ϕs overfit while others underfit? - In Appendix C.3., the author shows that the detaching operation keeps the optimality of \phi unchanged in the joint training. Does a similar conclusion apply to the sequential training in the main paper? - Table 1 shows with the same layers/steps, HSIVI achieves better approximation than DDPM and DDIM. Would the computation with one layer of HSIVI be more than one step of DDPM? If so, it might not be a fair comparison. Can the author discuss the computational complexity and provide wall clock time? - [1] and [2] design a hierarchical model using semi-implicit distributions, which seem to be closely related to HSIVI. The paper might need more detailed discussions of these papers. [1] Importance weighted hierarchical variational inference (NeurIPS 2019) [2] Structured Semi-Implicit Variational Inference (AABI 2019)

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Summary

This paper presents a new method called Hierarchical Semi-Implicit Variational Inference (HSIVI) that enhances the expressiveness of Semi-Implicit Variational Inference (SIVI) on complex target distributions. HSIVI works by applying SIVI to a hierarchy of latent variables, and using the variational distribution of the previous step as the implicit prior for the later step. The authors apply HSIVI for sampling from diffusion models, learned on a variety of synthetic and real-world datasets.

Strengths

1. Novelty: The paper presents a new method called Hierarchical Semi-Implicit Variational Inference (HSIVI) that enhances the expressiveness of Semi-Implicit Variational Inference (SIVI) on complicated target distributions. This is a novel approach that has not been explored before, to the best of my knowledge. 2. Effectiveness: The authors demonstrate the effectiveness of HSIVI on a variety of synthetic and real-world datasets, including accelerating the sampling process of diffusion models. The results show that HSIVI outperforms existing methods in terms of accuracy and efficiency. 3. Clarity: The paper provides a detailed explanation of the HSIVI method, including its mathematical formulation and implementation. The authors also provide clear and concise descriptions of the experiments and results.

Weaknesses

The main weakness of this work is that the method seem fairly complicated to implement in a practical setting. The method is presented in a general way, with an algorithm box. However, later another objective function is presented with some parameter sharing. Then, in the larger-scale experiments, HSIVI-SM is trained by fune-tuning a larger model. Either the method is not extremely stable / usable, or there are too many details but those are necessary for the experiment to be setup properly? (in the latter case, perhaps one experiment should go to the supplements?).

Questions

(1) Why isn't the performance of HSIVI-LB reported on the other experiments? (2) In terms of computational efficiency, it wasn't clear to me that if two compared methods had the same "T", then their computational burden were comparable. Are the computational complexities exactly the same for sampling from HSIVI-SM and SIVI, for equal "T"? (3) In principle, if HSIVI was used to train the model, we could reach a better data likelihood? Why aren't the authors restricting the application of HSIVI to sampling, and not learning the score function as well?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Thank you for the answers

I thank the authors for answering my questions!