Sparse Bayesian Generative Modeling for Compressive Sensing

Summary

This paper introduces a new type of sparsity inducing generative prior for the inverse problem. The authors theoretically underpin our approach by proving that its training maximizes a variational lower bound of a sparsity inducing log-evidence.

Strengths

This work can learn from a few corrupted data samples and, thus, requires no ground-truth information in its training phase.

Weaknesses

1. The comparison methods include Lasso, CKSVD, and CSGAN, all of which are presented before 2018. The comparison methods are too old. 2. I think the author should clarify the specific application scenarios for this research direction. 3. In Figure 7, compared with CSGAN, the results of CSVAE and CSGMM are not very clear.

Questions

1. In Table 2: Resources for simulations on MNIST (M = 200, Nt = 20000, Fig. 2 a)), CSGAN has much less parameters than the proposed CSVAE , why does CSGAN take much more training time? 2. Could the reconstruction results be influenced by the amount of noise?

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

I think the application scenes of this work is not very clear. The datasets used for comparison are very naive, colorful images with more textures should be used. And the authors should include more recent comparison methods, existing comparison methods are all presented before 2018.

Summary

The authors present an elegant new approach for dictionary based compressive sensing wherein the sparsity inducing prior is tuned from data by maximizing a lower bound on the evidence. This is a new paradigm for compressive sensing which appears to improve on reconstruction error over standard approaches and does not require solving an optimization problem at inference time.

Strengths

- Approaching dictionary based compressive sensing in this way is novel as far as I'm aware. Not having to solve an optimization problem at inference time has the potential to be impactful. - I found the paper to be very well-written. The authors did a good job of motivating their approach by relating it back to relevant background literature and discussing necessary theory. - Their results appear to be promising and should hopefully motivate future work in this paradigm.

Weaknesses

- Since their approach offers a new perspective on sparse Bayesian learning, it would have been helpful to compare the sparsity of predictions in addition to the nMSE. - I understand that space is limited, but I found the description of the experiments to be a bit terse. - It wasn't clear to me when a practitioner should choose CSVAE vs. CSGMM

Questions

- How does your approach compare to methods for sparse Bayesian learning with proper priors; for example Louizos et. al. "Bayesian Compression for Deep Learning." NeurIPS 2017? - Have you tried evaluating the quality of error bars from the predictive posterior so that your approach might be used in the context of UQ? - Can you explain the reason for the numerical instabilities in the noise free simulations for some of the image datasets? - Have you analyzed the histogram of the posterior over $s$? Can you suggest how the practitioner who is interested in a sparse $s$ might $0$ entries of the vector?

Rating

7

Confidence

2

Soundness

3

Presentation

4

Contribution

3

Limitations

I think the authors have done a good job addressing the potential limitations of their approach in section 3.4. It would be interesting to analyze how well generative priors trained on a particular dataset transfer to similar datasets.

Summary

The paper introduces a novel training algorithm for generative models used as priors in linear inverse problems, with a specific focus on compressed sensing. The authors propose a training principle that regularizes the prior to learn a sparse representation of the signal of interest, implemented in Variational Autoencoders (VAEs) and Gaussian Mixture Models (GMMs). This is achieved by learning two posterior distributions p(y∣s) and p(s∣z) over a set of parameters through optimization of the Evidence Lower Bound (ELBO) derived in Equation 15. During the inversion phase, the true signal is estimated by sampling from the posterior distribution E[x∣y] as given in Equation 9. Unlike many contemporary generative priors, this method does not require solving an optimization algorithm to achieve the estimate. This approach simplifies the inversion process, making it more computationally efficient. The performance of these implementations is validated on datasets containing different types of compressible signals. The results demonstrate the effectiveness of the proposed method in accurately reconstructing signals. Additionally, the paper provides theoretical support on the tightness of the lower bound of the estimate, further validating the robustness and reliability of the proposed approach.

Strengths

The proposed method appears to be novel and aligns with an intriguing line of work that focuses on training generative priors with the intention of using them for inverse problems. Section 2 of the paper is well-crafted, effectively elucidating the connection between the proposed method and existing literature. For the given baselines, the proposed method yields better reconstructions of the true signal, as verified by commonly used metrics and visual inspection.

Weaknesses

The authors provide theoretical motivation; however, they do not address how the potential estimation of the measurement y impacts the estimation of the true signal x. Additionally, building on this point, the CSVAE may experience representation error, as noted in [5], which is not mentioned as a limitation. While the proposed method does improve upon [5], this remains a source of error. Section 4 is limited in the following ways: * The only metric reported for natural images is the normalized mean square error, while other commonly used metrics, such as PSNR and SSIM, are not included. * Given the computational focus of this paper, it is concerning that all experiments were conducted on low-resolution grayscale images, especially considering the A40 GPU budget. * The baselines appear to be less contemporary compared to current literature on compressed sensing inversion algorithms. [Normalizing Flow](https://proceedings.mlr.press/v119/asim20a/asim20a.pdf) and [Diffusion Models](https://openreview.net/pdf/4f6f0e2347a3d6f9a88b39e445f77c1e7503064e.pdf) have largely replaced GANs due to issues with representation error. Similar to the proposed method, Diffusion Generative Priors do not require gradient-based approaches to estimate the posterior. There are hyperlinks to the work in the responses. * Considering this paper is training generative before be used downstream, there seems to be no reported metrics about training other than computational resources.

Questions

* Could the authors please provide perceptual metrics (i.e. SSIM, LPIPS) for the reconstruction comparison experiments in Figures 2 and 3? * Could the authors elaborate on the computational overhead provided in footnote 4 and explain the reason for utilizing only grayscale images when reporting having a computational budget of NVIDIA A40 GPU for CSVAE? * Could the authors please explain the advantages the proposed method has over contemporary generative prior such as normalizing flow and diffusion-based priors? * One concept I am unclear about is how do you check if the proposed method is overly biased towards the measurements y? This issue might arise because the conditional distribution P(y|s) has equally likely probabilities when Y=AS, but does not guarantee S will be consistent with training samples depending on how sparse s is in practice.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

Please refer to the weakness portion of the review.

Summary

The paper proposes a set of methods to learn a generative model over the sparse representations of a signal. The dictionary basis is fixed and given, and the signal of interest is assumed to be sparse in this basis. Therefore, the generative model learns to provide such sparse representations. The method learns the generative prior from the noisy and compressed observations. The authors leverage the notion of conditional Gaussianity and propose VAE and GMM based models for learning such priors. The choice enables the inference with only forward operation.

Strengths

The idea of using purely forward operations for inference is quite nice, since most of the solutions based on generative priors require gradient descent on the latent space and suffer from latency among other problems. Furthermore, learning the prior from compressed observations makes the approach more practical as it does not need ground truth signal.

Weaknesses

Overall, the paper combines some of the existing ideas from prior works (for example posterior updates from SBL, VAE ELBO derivations), and therefore, the novelty seems marginal. To be more concrete, it *seems* straightforward that if one wants to do variational Bayes, then the VAE style decomposition is the standard approach. If there are any additional challenges, the authors should clarify. The idea of forward pass only inference has been tried before (see for example the paper “Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models” NeurIPS 2023). Learning from the compressed samples can be compelling although not new as mentioned by the authors themselves (for example CSGAN). The proposed method requires further elaboration on the loss with respect to learning from uncompressed measurements substantiated with more experiments. The experiment section is a bit thin given the limited technical novelty, particularly with respect to the baselines. Given a lot of prior works in the literature on learning based solvers for inverse problems (for example, all variations of generative priors based on flows and diffusions, unfolded algorithms like lista and its variations), the authors have a more difficult task of placing their contribution within the prior work. The paper is not very convincing in showing what is the unique direction in which the field is moved forward.

Questions

Some of my questions about the novelty was implicit in my comments above. - Can authors provide some comment on the performance loss with respect to uncompressed generative prior? - How many MC samples are used at inference for (9)? - Generative priors generally suffer from convergence issues and require multiple restarts. Could the authors comment if their forward-only operation has a similar issue? - It would be good to incorporate error bars in the plots and numerical results. Given the MC step, I expect a higher variance for the proposed approach.

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

See comments above.

Please review the rebuttal and respond

Dear Reviewers, Thank you once again for your time and effort in reviewing for NeurIPS 2024. The authors have already submitted their rebuttal to your comments, and the discussion phase has now begun. For those who have not yet responded, please review the authors' responses carefully at your earliest convenience. Your timely feedback on whether your previous concerns have been adequately addressed is crucial to the reviewing process. Thank you very much! Best regards, Area Chair

Follow-up on the answers

I would like to thank the authors for taking time to answer my questions and comments. I will discuss some of the responses a bit further, and that, with the intention of building more confidence and increasing my score, so hopefully the authors can bear with the process, and their time is well spent at the end. * To begin, I agree with the authors that “the capability to learn from compressed data” is not minor, as I mentioned in my original review. * I am also satisfied with the answers to my questions regarding MC samples, error bars, and convergence issues. * **Regarding deriving ELBO from SBL:** As far as I can see, the inequality in (7) is the standard step in ELBO bounds. The decomposition in (13) follows from the basic probability decomposition for $\mathbf{z}, \mathbf{s}$ and $\mathbf{y}$ (given in Appendix B). Then, the computation of the bound follows from MC sampling like classical VAE (also referenced by the authors themselves, namely ref. 36), and the tractable computation of some terms in Appendix D. The derivations of Appendix D (eq. 26-29) are not exactly the same as VAE paper but builds very similarly on Gaussian likelihood assumption and the linearity of expectation. So, unfortunately, I still feel that these derivations are based on standard linear algebra and probability techniques, very similar to the standard VAE. We can therefore focus on other comments regarding the novelty. * **Regarding “1. One technical contribution from our side is to extend the statistical model…”:** I agree with the reviewer that the proposed method enables “the learning of a nontrivial sparsity inducing prior with training data”, however the actual contribution is “the learning of a nontrivial sparsity inducing prior” from **compressed measurements**, as the authors emphasize (although it is not so clear from the title and not so pronounced in the abstract). So, the question is: does the method perform equally better from uncompressed data? If not, what makes the method better in case of learning from compressed data? I feel combining two contributions (namely learning a nontrivial sparsity prior AND learning from compressed data) and selecting the baselines accordingly makes the message convoluted: is the paper about a better generative prior or about learning from compressed data? Answering such questions is important for the paper, which is proposing a general method for solving a particular problem.