Moment Matching Denoising Gibbs Sampling

Summary

This paper proposes a new denoising scoring matching (DSM) technique to train energy-based models. They address the well-known inconsistency problem of DSM objectives by training a "noisy" model and then sampling "clean" samples from the "noisy" model by learning a denoising distribution. The contributions of the paper are as follows: 1) Theoretically, they show that a "clean" model is uniquely identified by an EBM learning a noisy data distribution. 2) A sampling strategy inspired from Gibbs sampling to approximate the denoising distribution, which allows to sample from the underlying clean model without the need to retrain the noisy model. 3) Practically, they show how to scale their approach to high-dimensional data by overcoming the quadratic cost of evaluating the Hessian of the DSM for every $\tilde{x}$.

Strengths

- Well-written paper, easy to follow and most claims are justified either by citing previous work or in appendix / math proofs - Prior work is duly cited - Experiments on synthetic and real-world (MNIST/CIFAR) datasets are appreciated to assess the quality of the technique proposed

Weaknesses

I'm not an expert in this field and I have a hard time evaluating the overall contribution of this paper to the field of "Generative Models". In particular, it feels like this denoising techniques introduce a scalability problem (although partly discussed and solved by contribution 3) in order to solve the inconsistency problem of score-based matching training. A discussion on the quality of the approximation and comparing the full Hessian with its approximation could help answer my doubts. Besides, I think that describing the full sampling algorithm could help the reader get a better picture of the whole training and sampling procedure. This would clarify what has been made in the experiments section. I list below a few comments / typos I have found while reviewing the paper: 1) l.33: there is a missing $\nabla_x$ in the definition of the score matching function. Otherwise, I don't see how $\nabla_x s_{p_{\theta}}(x) = -\nabla^2_x f_{\theta}(x)$. 2) l.26: "it is equivalent to MLE". A reference is welcome. 3) l.34: "under certain regularity conditions, the Fisher divergence is equivalent to SM". Reference?

Questions

As stated above in the weaknesses section, my main doubt revolves around how does the Hessian approximation affect the training stability and overall output samples quality.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have partly discussed the limitations in section 3, but an experiment comparing the quality of the approximation of the generated samples with the full Hessian and the approximated Hessian is welcome to strengthen the demonstration. No potential negative societal impact at this stage.

Summary

This paper proposed a sampling scheme called (pseudo)-Gibbs sampling with moment machine. It can be applicable to the sampling problem from from the underlying clean model when a noisy model hat has been well-trained via DSM. The proposed sampling technique is used to sample images for image generation and showed good efficiency on toy dataset.

Strengths

1. The paper is well written 2. The problem is a basic problem with many applications 3. The idea is new and seems working well.

Weaknesses

The improvement in image generation seems minor.

Questions

NIL

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NIL

Summary

This paper addressed the inconsistency problem in training energy-based models (EBMs) using denoising score matching (DSM). An efficient sampling framework, called (pseudo)-Gibbs sampling with moment matching, was proposed to enable effective sampling from the underlying clean model when given a well-trained 'noisy' model obtained through DSM. The presence of an underlying clean model within the noisy EBM was identified, and a pseudo-Gibbs sampling scheme incorporating analytical moment matching approximation was introduced for sampling from the clean model without requiring additional training. The effectiveness of the method was demonstrated in generating high-quality images for both single and multi-level noise settings. Additionally, the scalability of the method to high-dimensional datasets was illustrated. Comparisons with related methods highlighted the advantages of the proposed approach.

Strengths

The theoretical derivation and presentation are clear and well-defined.

Weaknesses

The limited number of experiments conducted to validate the proposed method. While the paper provides some demonstration of the effectiveness of the approach, further experiments across a wider range of scenarios and datasets would enhance the robustness and generalizability of the findings. The authors should provide more details for those experiments in Supplementary Material. The authors failed to demonstrate the wide applications of the proposed method.

Questions

In experiments, the authors used a noise-conditioned score network trained in [32], does the proposed model also excel in other settings?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

None

Summary

This work proposes a sampling method from implicitly defined clean model given a noisy model without retraining and demonstrates its applicability to high-dimensional data / images.

Strengths

- this work presents interesting theoretical results such as the uniqueness of underlying clean model given noisy data distribution, analytical covariance identity, and optimal Gaussian approx. - this work presents interesting corrections to prior works such as analytical DDPM, KL training Gibbs sampling.

Weaknesses

- It is unclear if the proposed method (e.g., Eq. (12)) is effective in general. Some of the toy examples look promising, but some image examples may not look good (see Table 2 for large performance gaps with DDPM and NCSN++ / possibly more with recent methods). While this work emphasized that it will work well with a single level of fixed noise, eventually the proposed method with multi-level noise was used for more complex image cases without the issue of mode collapse. This inconsistency is my major concern. - Weak experimental results: some of the experiments may not effectively demonstrate the performance of the proposed scheme well. For example, it is unclear what this work wanted to show in Figures 5, 6 and 8. Comparison experiments may be needed.

Questions

Please address the comments in the Weaknesses section. - What if log scale is used in Fig 3(a)? Then, it may show similar curves like Fig 3(b) unlike the claims in this work. - It is unclear why the comparable results of NCSNv2 with Langevin / the proposed method is ok. Are they using the same multi-level noise? Then, what is the advantage of the proposed method over the conventional Langevin dynamics?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes

Summary

This paper proposes a sampling framework, i.e., (pseudo)-Gibbs sampling with moment matching, which aims at sampling from the implicitly-defined clean model given the noisy energy-based model well-trained by the denoising score matching method. It first provides a theoretical justification of the existence of the underlying clean model for a well-trained energy-based model. It then introduces the gibbs sampling framework based on the proposed analytical Gaussian moment matching approximation method for the denoising distribution. Experiments on several toy datasets and MNIST and CIFAR10 datasets are conducted to demonstrate the effectiveness of the proposed approximation and sampling method.

Strengths

- The paper provides theoretical justification of the existence of the underlying clean model of a well-trained noisy EBM, and discusses the existence of the clean model for an imperfect EBM. - It derives an analytical Gaussian moment matching approximation method for the denoising distribution and validates the design on several toy datasets, and discusses some scalable implementations of the proposed method for image data. - The presentation in general is clear.

Weaknesses

- The full covariance moment matching requires calculating the Hessian matrix, which could be computationally expensive even with diagonal approximation methods. - The experiments are conducted on toy datasets and MNIST and CIFAR-10 datasets. More results on other image dataset could provide better intuitive understanding of the proposed method. - The performance of the proposed sampling method is close to the existing methods. The significance of the proposed sampling method is not very clear.

Questions

Please see the weaknesses section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have mentioned the limitations in Sec 3. Scalable Implementations for Image Data and Sec 4. Image Generation with a Single Noise Level.

Thanks for Your Response

I thank the authors for their response and the effort they put in resolving my concerns. I strongly recommend the authors to incorporate the new results in the rebuttal especially those on new datasets into the next version of the paper, as I believe these will further strengthen the paper. In light of the new results and the responses from the authors, I will increase my score to 6.