In ill-posed imaging inverse problems, there can exist many hypotheses that fit both the observed measurements and prior knowledge of the true image. Rather than returning just one hypothesis of that image, posterior samplers aim to explore the full solution space by generating many probable hypotheses, which can later be used to quantify uncertainty or construct recoveries that appropriately navigate the perception/distortion trade-off. In this work, we propose a fast and accurate posterior-sampling conditional generative adversarial network (cGAN) that, through a novel form of regularization, aims for correctness in the posterior mean as well as the trace and K principal components of the posterior covariance matrix. Numerical experiments demonstrate that our method outperforms contemporary cGANs and diffusion models in imaging inverse problems like denoising, large-scale inpainting, and accelerated MRI recovery. The code for our model can be found here: https://github.com/matt-bendel/pcaGAN.
Paper
References (59)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
To solve image inverse problems (denoising, super resolution, etc.), many works attempt to sample from the posterior distribution of ground truth images given a degraded measurement. One line of works propose to train a stochastic neural network as a CGAN (conditioned on the degraded image and an additional random seed as inputs), while regularizing the conditional mean and the conditional covariance matrix of the outputs. This paper proposes yet another, improved regularization term for such a CGAN approach. Specifically, the authors propose to correct (regularize) the K principal components of the conditional covariance matrix in additional to the conditional mean and the trace of the covariance.
Strengths
The paper is mostly written well. The proposed method is demonstrated on toy examples (such as Gaussian random signals and MNIST images), which I find appealing to gain intuition into the advantages of the proposed approach. The paper shows results on MRI reconstruction, which is not very common to report but an incredibly important inverse problem to solve.
Weaknesses
Even though I personally like this type of work and the proposed approach, unfortunately I see several major weaknesses in this paper, starting even with wrong/inaccurate claims in the introduction & motivation. I'll list some of the weaknesses below. 1) I disagree with most of the claims made in L17 onwards regarding the shortcomings of point estimators (introduction). The perception-distortion tradeoff theorem in [6] does not prove anything specific about point estimators (a claim made in L20-L21). In fact, this theorem proves that distortion can only be at odds with perceptual quality (but now always), regardless of whether the estimator is "deterministic" or not. Moreover, [6] defines perceptual quality as the statistical distance (e.g., KL) between $p_{\hat{X}}$ (the distribution of the reconstructions) and $p_{X}$ (the distribution of the source images), and not as the distance between $\hat{x}$ and the manifold of natural images (as claimed in L22). What is the definition of such a distance? Lastly, the perception-distortion tradeoff is not always a strict one, as hinted in L21. This tradeoff depends on the distortion measure, as discussed by the authors of [6]. Some distortion measures (e.g., "perceptual" losses in the deep image features space) virtually do not contradict with perceptual quality (as demonstrated experimentally in [6]). I would be pleased to see a complete revision of this paragraph in which the errors are corrected, or otherwise a clear explanation for why am I wrong. This is the entry point of the paper, it should at least be accurate. 2) I also do not agree with L27: "to address these shortcomings, posterior-sampling-based image recovery has been proposed". Posterior sampling as a solution for inverse problems has been proposed and suggested, e.g., with Langevin dynamics (though, it didn't work properly for high-dimensional data) much before we knew anything about the perception-distortion tradeoff or fairness issues in image reconstruction. As I understand it, the true motivation for the rise of posterior-sampling is the following: We are often dealing with ill-posed inverse problems, and performing posterior sampling allows us to conduct uncertainty quantification of the inverse task at hand, explore the space of solutions visually, and make more informed decisions in down-stream tasks. It has also been shown that stochastic estimators are more robust to adversarial attacks [1]. I don't see why the rise of posterior sampling is linked to the perception-distortion tradeoff and/or to fairness. Please provide evidence (citations and explanation) if that's the case. 3) L30: "posterior samples will display attributes that reflect minority and majority populations in the training data". As far as I know, this is often incorrect. In fact, we may often require to produce an intractable amount of posterior samples to observe an outcome for a minority. Please see [2] as an example. 4) L43: You are missing citations (and comparison) with highly related work that proposed to use posterior sampling using a CGAN with conditional mean and conditional covariance regularizations, prior to the work of Bendel et al. For example, [3] and [4]. In fact, the work of Bendel et al. is based on the idea proposed in [2] (to regularize the neural network by enforcing agreement of the average posterior sample with the posterior mean), and in [3] (to use a second-moment penalty of the covariance matrix as well), as you suggest in L75. As far as I understand, your algorithm is a different (and maybe improved) version of [4], but I don't know which one is better since a comparison with [3] and [4] is not reported in the paper. Please address the differences in your work so I could better understand why it's not appropriate to cite such papers and/or include them in your analysis. 5) I would suggest explaining L76-L79 better, by adding appropriate definitions for all the terminology used. For instance, the definition of a $P_{rc}$-sample supervised-$l_{1}$ loss is missing and unclear. I would be happy to see a more self-contained background section. Make sure every notation you are using is clearly defined before using it. 6) Some results seem wrong to me. For example, the results of DPS in Figure 4. I have been using DPS in many different image inverse problems, and never observed such inconsistencies between the outputs and the inputs (see, e.g., Figure 14 in the DPS paper appendices, which shows almost perfect consistency with the measurements for severe inpainting tasks). The FID results of DPS also seem odd to me. I find it very hard to believe that a CGAN algorithm would outperform DPS, given that the hyper-parameters of DPS are properly tuned. Can you please provide the code you used to execute these experiments so I could verify them? Maybe the hyperparemeters you used to run DPS are suboptimal? 7) The contribution of this work is quite limited in my opinion. The proposed algorithm only slightly improves its predecessor (Badler et al.), and the experimental evaluation is limited. A comparison with more works is missing, such as [3] and [4], and other, more recent state-of-the-art stochastic image restoration methods that produce better results than DPS. What about DDNM, DDRM, etc.? Why are these not included in the analysis? (besides the fact that they don't "promise" posterior sampling). I would be happy if you could clarify the contribution of your work and its impact on the field, as it seems quite incremental to me. [1] Reasons for the Superiority of Stochastic Estimators over Deterministic Ones: Robustness, Consistency and Perceptual Quality, Guy Ohayon et al., ICML 2023 [2] From Posterior Sampling to Meaningful Diversity in Image Restoration, Noa Cohen et al, ICLR 2024. [3] High Perceptual Quality Image Denoising with a Posterior Sampling CGAN, Guy Ohayon et al., ICCV workshops 2021. [4] High-Perceptual Quality JPEG Decoding via Posterior Sampling, Sean Man et al, CVPR workshops 2023.
Questions
Please address each of the weaknesses.
Rating
4
Confidence
5
Soundness
2
Presentation
1
Contribution
3
Limitations
The authors do provide a dedicated limitations section, but I observe additional limitations in this paper which are not addressed, such as incremental novelty of the proposed algorithm over previous work, limited experimental section (comparing only to one predecessor), and more, as explained in the weaknesses section.
Summary
The authors address inverse problems and aim to improve GAN-based posterior samplers by adopting regularization to correct the principal components of the covariance matrix of the approximated posterior distribution.
Strengths
1. The paper is well written and easy to follow. 2. The idea presented in the paper is novel and clear, especially given the tendency of GAN-based posterior samplers to experience mode collapse. This serves as a strong motivation for the proposed method in the paper. 3. The authors conducted empirical study with a variety of datasets.
Weaknesses
1. For each dataset in the empirical study, a different set of baseline methods was chosen, possibly due to the requirement of pretraining. However, for a fair comparison, this is necessary for a complete research. 2. The authors claim that the proposed method outperforms existing diffusion models (line 52). However, the experiments conducted with diffusion models are not comparable due to the much smaller test size (line 210 in MRI experiments and line 235 in inpainting experiments). 3. Figure 4 presents inpainting posterior samples, but for the diffusion model referred to as DPS, the masking was not applied, which results in incorrect and misleading visualization. 4. Some minor errors: - In line 28, “There the goal" should be revised. - In lines 33-34, there is a grammar mistake.
Questions
1. While diffusion models are slower compared to GANs, they have demonstrated success over GAN-based models in enhancing posterior richness for conditional tasks. A fair comparison with diffusion models would be valuable, as it remains uncertain whether the proposed method outperforms diffusion models. For faster sampling approaches, the authors might consider DDIM[1], which accelerates the diffusion process. References: [1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising diffusion implicit models." (2020).
Rating
5
Confidence
4
Soundness
2
Presentation
4
Contribution
2
Limitations
The authors explained the limitations of their work.
Summary
The paper focuses on learning posterior samplers for inverse problems, specifically aiming to accurately estimate the principal components of the posterior covariance matrix. To achieve this, the authors consider training a conditional generative adversarial network (cGAN), building upon previous work [21]. They introduce two new regularization terms: one for the eigenvalues and another for the eigenvectors. Recognizing the challenge of estimating these statistics from a single ground truth solution per input measurement, the authors propose gradually adding these terms during training, while utilizing intermediate results to estimate the statistics. The authors present a series of experiments to demonstrate their method, ranging from toy examples to real-world MRI data. Comparisons are primarily made with rcGAN and NPPC.
Strengths
* The paper is well-written, with clear motivation, method presentation, and background (mostly). * The detailed algorithm description, beyond pseudocode, significantly enhances understanding and reproducibility. * The authors nicely addressed the problem of estimating the desired statistics from a single ground truth vector. * The authors provide sufficient evidence to demonstrate their method (but additional evidence could further strengthen their claims).
Weaknesses
Major Issues: * The paper lacks an ablation study which is crucial. While the authors followed [21] and set $P_{rc}=2$, I still expect an ablation study on this parameter, especially since using 2 samples seems insufficient to estimate statistics properly. Furthermore, there is no ablation study regarding the different regularization terms and their relative weights. Given the newly regularization terms, it is possible that other terms, introduced in [21], are redundant, especially when $K=d$. Since accurately recovering the eigenvalues implies perfect recovery of the trace, for example. * I find the comparison to competing methods limited. I suggest adding comparisons to additional recent posterior sampling techniques. For example, the authors of NPPC presented comparison to DDRM, DDNM, RePaint, and MAT. * Unlike work [21], the paper present no proposition or theoretical guarantees regarding the estimated statistics, limiting the novelty of the paper. * The description of work [21] is imprecise. Specifically, the usage of the notations $P$ and $P_{rc}$ is incorrect, leading to confusion. * The authors state in lines 103-104 that they apply the regularization term (11) only after allowing the generated posterior mean to converge near the true mean. This raises the question of whether (11) is necessary or if its impact is marginal, as it only leads to small variations in the generated posterior mean. The same question applies to the generated eigenvalues. I suggest adding plots to illustrate the variation in generated statistics over training epochs, comparing them to their final values or their values before applying the new regularization term during training. * While the relevance of NPPC to this work is clear, the experimental comparison seems unfair as NPPC is not designed for generation tasks. Furthermore, the authors demonstrate a speedup against DPS, but neglect to discuss the computational load of their method compared to NPPC. Minor: * The authors in lines 24-26 correctly point out that bias can be an issue in single-sample recovery. However, this is also a potential concern for recovery methods that yield multiple solutions.
Questions
* Please address the weaknesses above. * I suggest conducting an ablation study to assess the impact of key priors in the method, such as the different terms and $P_{rc}$. * I suggest expanding the experiments to include comparisons with more recent posterior samplers in standard image inverse problems, like super-resolution and inpainting. * Please provide plots illustrating the variation in the generated statistics over the training epochs. * Do the additional regularization terms theoretically change the optimal solution, or does it remain the same?
Rating
5
Confidence
5
Soundness
4
Presentation
4
Contribution
2
Limitations
See weaknesses.
Summary
This work considers using posterior sampling to solve inverse problems via conditional GANs. Previous work has shown that cGANs are competitive in terms of reconstruction quality as compared to other generative models to sample from the posterior (e.g., diffusion models), while also having the advantage that, once trained, sampling is fast as it simply requires a forward pass. In this paper, the authors build on such approaches by adding regularization terms to encourage the cGAN to learn K principal components of the posterior covariance matrix, where K is user-defined quantity. Due to the computational burden of such an approach, a training scheme is devised where learning of the eigenvectors and eigenvalues is done after optimizing a regularized cGAN objective. The method is validated on a variety of linear inverse problems (denoising, accelerated MRI, inpainting) and datasets (mnist, fastmri, ffhq).
Strengths
- The paper is well-written and easy to read. In particular, the idea of adding regularization to fit to principal components of the posterior distribution is a simple and natural idea. It is also one of importance, as being able to visualize uncertainty in reconstructions is crucial in understanding which regions of the reconstruction can be “trusted” - The method is validated on a nice range of problem types and datasets. - Using a conditional cGAN allows for fast sample generation, especially as compared to diffusion models. - The additional use of the pca regularization term appears to improve performance of rcGAN, while also allowing for visualization of directions of uncertainty.
Weaknesses
- The proposed approach is a slight, potentially somewhat incremental, improvement to the prior rcGAN approach of [21]. In particular, rcGAN also aims to better model the posterior distribution by matching certain statistics of the posterior mean and covariance. This work takes this reasoning a step further by aiming to fit the principle components of the covariance. This has also been done (in a different way) in prior work [29], that the authors compare to. The bonus for this work, however, is that the authors can sample quickly, whereas in [29] sampling is not possible. - Coupled with the concern regarding novelty, the additional regularization terms are computationally expensive to optimize, especially for high-dimensional problems. This is evidenced by, for example, only being able to set K = 1 in the MRI experiments and K = 2 in the large-scale inpainting experiments, and is discussed by the authors in their limitations. Moreover, the training strategy becomes a bit more complicated in order to aid in efficiency, as one needs to first optimize the rcGAN objective and then integrate the PCA terms into the learning process.
Questions
- As the parameter $K$ is fixed prior to training with knowledge of the inverse problem, it would be good to discuss how the choice of $K$ relates to the particular inverse problem itself. For example, how does the choice of $K$ relate to how challenging the inverse problem is? I presume that ill-posedness plays a role here (e.g., comparing a linear inverse problem with a non-linear inverse problem, such as phase retrieval).
Rating
6
Confidence
4
Soundness
3
Presentation
4
Contribution
3
Limitations
The authors discuss some of the above weaknesses I mentioned near the end of their paper, which I appreciate. They also discuss other issues regarding how to properly incorporate this approach for rigorous uncertainty quantification.
I'd like to thank the authors for their thorough response. 1. I disagree regarding your explanation about the perception-distortion tradeoff. First of all, the paper does not only consider *single* draw estimators. In fact, the definition of distortion used in the paper is a mean taken over all Y's, and for each Y the mean is taken over all the random draws from the posterior and all the random draws from the estimator: $$\mathbb{E}[\Delta(X,\hat{X})]=\int\int\int \Delta(x,\hat{x})p_{X|Y}(x|y)p_{\hat{X}|Y}(\hat{x}|y)p_{Y}(y)dxd\hat{x}dy$$ Namely, when talking about stochastic estimators (e.g., the posterior sampler), distortion (as defined in this paper) actually considers all possible draws from the estimator, for each Y. So, I still don't see why the perception-distortion tradeoff tells us anything "bad" about point estimators. Moreover, the perception-distortion tradeoff proves that the perception-distortion function, $$P(D)=\min_{p_{\hat{X}|Y}}d(p_{X},p_{\hat{X}})\quad s.t.\quad \mathbb{E}[\Delta(X,\hat{X})]\leq D$$ is monotonically non-increasing in $D$, but it was never proven that this general function is *strictly* decreasing. For example, if we take the trivial distortion $\forall x,\hat{x}:\Delta(x,\hat{x})=0$, then $P(D)$ is constant in $D$ (it always equals zero). Thus, perception and distortion (as defined in the perception-distortion tradeoff paper) are not always at odds with each other. Lastly, I am sorry, but I don't understand what you mean by "with large $P$, there is essentially no PD tradeoff." 2. I don't see why would you remove the discussion about the perception-distortion tradeoff from your paper. It is central to motivating special types of regularizations when training with GANs/cGANS and not just a "regular" MSE loss. So actually, in my opinion, without discussing the perception-distortion tradeoff, the motivation of such work (e.g., pcaGAN) would be unclear. 3. Could you please point out to me where in the cGAN paper by Isola et al. "Image-to-Image Translation with Conditional Adversarial Networks" I can find the posterior mean regularization? Thank you!
We thank the reviewer for the continued discussion and opportunity to improve our submission. 1. We appreciate the reviewer's invitation to further our discussion around the PD tradeoff and its connection to point estimates and posterior sampling. Our main point is that an estimator that returns $P\\gg 1$ posterior samples lets us trade perception with distortion in a way that a point estimator (which returns a single sample, such as a single posterior sample) cannot. Regardless of whether the reviewer agrees with us on the specifics of this point, we would like to emphasize that **this discussion is immaterial to the main ideas of our paper**, which revolve around enforcing cGAN consistency with the principal components of the true posterior covariance matrix. To avoid directly tying point estimation to PD or the motivation for posterior sampling to PD, we have revised our Introduction; please see our response to your next point. 2. We are removing the discussion of the PD tradeoff from our Introduction (not our entire paper) to definitively address the first and second critiques in your original review. In particular, we propose to revise the second and third paragraphs of our Introduction as follows. > In much of the literature, image recovery is posed as point estimation, where the goal is to return a single best estimate $\\hat{\\boldsymbol{x}}$. But there are shortcomings to this approach. For example, it’s not clear how to define “best,” since L2- or L1-minimizing $\\hat{\\boldsymbol{x}}$ are often regarded as too blurry, while perceptually pleasing $\\hat{\\boldsymbol{x}}$ may suffer from hallucinations [1–5]. Also, many applications demand not only a recovery $\\hat{\\boldsymbol{x}}$ but also some quantification of the uncertainty in that recovery [7,8]. > > As an alternative to point estimation, posterior-sampling-based image recovery [10–28] has been proposed. There the goal is to generate $P$ samples $\\{\\hat{\\boldsymbol{x}}\_i\\}\_{i=1}^P$ from the posterior distribution $p\_{x|y}(·|\\boldsymbol{y})$. Posterior sampling facilitates numerous strategies to quantify the uncertainty in estimating $\\boldsymbol{x}$, or any function of $\\boldsymbol{x}$, from $\\boldsymbol{y}$ [7,8]. It also can help visualize uncertainty, and it can increase robustness to adversarial attacks [Ohayon,ICML'23]. That said, the design of accurate and computationally-efficient posterior samplers remains an open problem. We would be interested to hear your feedback. 3. We should clarify what we wrote in our rebuttal. The rcGAN paper [21] defines the "supervised-L1 loss" as $E\\{\\|\\boldsymbol{x}-\hat{\\boldsymbol{x}}\_{(P)}\\|\_1\\}$ with $\\boldsymbol{x}$ the true (training) image and $\hat{\\boldsymbol{x}}\_{(P)}$ the $P$-sample output average. On page 4, [21] says "regularizing a cGAN with supervised-L1 loss alone is not new; see, e.g., [Isola]". But looking closely at [Isola], it seems that only the $P=1$ case is described there (i.e., no averaging), and so perhaps [21] is the first to propose this "supervised-L1 loss". The paper [Ohayon'21] considers the squared-L2 version $E\\{\\|\\boldsymbol{x}-\hat{\\boldsymbol{x}}\_{(P)}\\|\_2^2\\}$, for which only the $P=1$ case is described in [Isola]. In any case, if accepted, our final paper will cite [Ohayon'21] for what appears to be the first cGAN regularization involving $\hat{\\boldsymbol{x}}\_{(P)}$ with $P>1$.
1. That sounds better and more accurate to me. There are still some things I would personally rephrase, which could be misleading to some readers. For example: "while perceptually pleasing $\hat{x}$ may suffer from hallucinations." The notion "hallucinations" carries a heavy load in several fields, so this could mislead readers about the meaning of producing perceptually pleasing $\hat{x}$. For example, researchers from the world of LLM would think that image reconstruction models that produce perceptually pleasing outputs often produce very wrong outputs, but any sample from the posterior distribution is actually not "wrong". 2. Note that you also need to revise L43 onwards, where you start talking about the posterior mean regularization. 3. In the rebuttal, you wrote: "We don't think that rcGAN is "based on" [Ohayon'21] or [Man'24], though. The rcGAN paper [21] cites the 2017 Isola cGAN paper [17] for posterior-mean regularization." But now you are saying that the Isola cGAN paper did not propose posterior-mean regularization? While your work may have good results, I think it is equally important to write a proper (and correct) introduction section where this work is properly positioned compared to previous/related literature. Could you please try to include here the entire new introduction so I could make sure it all makes sense to me?
We thank the reviewer for the valuable feedback and continued discussion. Due to length constraints, we'll need to split our answer in two. Below is our proposed revision to the Introduction. > In image recovery, the goal is to recover the true image $\\boldsymbol{x}$ from noisy/distorted/incomplete measurements $\\boldsymbol{y} = \\mathcal{M}(\\boldsymbol{x})$. > This arises in linear inverse problems such as denoising, deblurring, inpainting, and magnetic resonance imaging (MRI), as well as in non-linear inverse problems like phase-retrieval and image-to-image translation. For all such problems, it is impossible to perfectly recover $\\boldsymbol{x}$ from $\\boldsymbol{y}$. > > In much of the literature, image recovery is posed as point estimation, where the goal is to return a single best estimate $\\hat{\\boldsymbol{x}}$. But there are shortcomings to this approach. For example, it’s not clear how to define “best,” since L2- or L1-minimizing $\\hat{\\boldsymbol{x}}$ are often regarded as too blurry, while efforts to make $\\hat{\\boldsymbol{x}}$ perceptually pleasing can sacrifice agreement with the true image $\\boldsymbol{x}$ and cause hallucinations [1–5], as we show later. Also, many applications demand not only a recovery $\\hat{\\boldsymbol{x}}$ but also some quantification of the uncertainty in that recovery [7,8]. > > As an alternative to point estimation, there is posterior-sampling-based image recovery [10–28], which aims to generate $P$ samples $\\{\\hat{\\boldsymbol{x}}\_i\\}\_{i=1}^P$ from the posterior distribution $p\_{x|y}(\\cdot|\\boldsymbol{y})$. Posterior sampling facilitates numerous strategies to quantify the uncertainty in estimating $\\boldsymbol{x}$, or any function of $\\boldsymbol{x}$, from $\\boldsymbol{y}$ [7,8]. > It also can help with visualizing uncertainty and increasing robustness to adversarial attacks [Ohayon,ICML'23]. That said, the design of accurate and computationally-efficient posterior samplers remains an open problem. > > Given a training dataset of image/measurement pairs $\{(\\boldsymbol{x}\_t, \\boldsymbol{y}\_t )\}_{t=1}^T$, our goal is to build a generator $G\_{\\boldsymbol{\\theta}}$ that, for a given $\\boldsymbol{y}$, maps random code vectors $\\boldsymbol{z}\\sim\mathcal{N}(\\boldsymbol{0}, \\boldsymbol{I})$ to samples of the posterior, i.e., $\\hat{\\boldsymbol{x}} = G\_{\\boldsymbol{\\theta}}(\\boldsymbol{z}, \\boldsymbol{y}) \\sim p\_{x|y}(·|\\boldsymbol{y})$. > There are many ways to approximate the ideal generator. > The recent literature has focused on conditional variational autoencoders (cVAEs) [14–16], conditional normalizing flows (cNFs) [10–13], conditional generative adversarial networks (cGANs) [17–21,Ohayon'21,Man'24], and diffusion posterior samplers [22–28,Wang'23]. Although diffusion samplers have garnered a great share of recent attention, they tend to generate samples several orders-of-magnitude slower than their cNF, cVAE, and cGAN counterparts. > > With the goal of fast and accurate posterior sampling, recent progress in cGAN training has been made through regularization. > For example, Ohayon et al. [Ohayon'21] proposed to enforce correctness in the generated $\\boldsymbol{y}$-conditional mean using a novel form of L2 regularization. > Later, Bendel et al. [21] proposed to enforce correctness in the generated $\\boldsymbol{y}$-conditional mean *and* trace-covariance using L1 regularization plus a correctly weighted standard-deviation (SD) reward, and gave evidence that their "rcGAN" competes with contemporary diffusion samplers on MRI and inpainting tasks. > Soon after, Man et al. [Man'24] showed that L2 regularization and a variance reward are effective when training cGANs for JPEG decoding. > More details are given in Section 2. > > In a separate line of work, Nehme et al. [29] trained a Neural Posterior Principal Components (NPPC) network to directly estimate the eigenvectors and eigenvalues of the $\\boldsymbol{y}$-conditional covariance matrix, which allows powerful insights into the nature of uncertainty in an inverse problem. > > **Our contributions.** > Inspired by regularized cGANs and NPPC, we propose a novel “pcaGAN” that encourages correctness in the $K$ principal components of the $\\boldsymbol{y}$-conditional covariance matrix, as well as the $\\boldsymbol{y}$-conditional mean and trace-covariance, when sampling from the posterior. > We demonstrate that pcaGAN outperforms existing cGANs [18, 19, 21, Ohayon'21] and diffusion methods [24, 27, 28, Wang'23] on posterior sampling tasks like denoising, large-scale inpainting, and accelerated MRI recovery, while sampling orders-of-magnitude faster than [24, 27, 28, Wang'23]. > We also demonstrate that pcaGAN recovers principal components more accurately than NPPC using approximately the same runtime.
[This is a continuation of our previous comment.] Regarding hallucinations, we do find that, relative to posterior-mean recoveries, posterior samples tend to hallucinate fine details that are not present in the true image, and we show evidence of that in Figures 3, C.1, E.4, E.5, E.6, and E.7. While a posterior sample as a whole may not be "wrong," it can fabricate fine details. Note that individual posterior samples are farther from the truth than the posterior mean recovery (in terms of average L2 distance) since they have 3dB worse MSE. In addition to revising our Introduction, we plan to add the following after L79 in Section 2: > We note that the $\\mathcal{L}\_{1,P}(\\theta)$ regularization proposed in [21] is closely related to the regularization $\\mathcal{L}\_{2,P}(\\theta) \\triangleq E\_{x,z1,...,zP,y}\\{\\|\\boldsymbol{x}-\\hat{\\boldsymbol{x}}\_{(P)}\\|_2^2\\}$ proposed earlier by Ohayon et al. in [Ohayon'21]. > A detailed discussion of the advantages and disadvantages of various cGAN regularizations can be found in [21].
I sincerely thank the authors for the thorough discussion. I have given this a lot of thought. I reviewed the paper again and took the following into account: 1) Our discussion (including the revised introduction). 2) The additional experiments and results. 3) The reviews made by other reviewers, as well as the authors' responses. While I think the results are appealing, I believe the paper is not 100% ready for publication due to insufficient soundness and presentation (not only in the introduction). Therefore, I am: 1) Decreasing my scores for soundness and presentation. 2) Increasing my score for contribution. 3) Increasing my overall score to borderline reject (*given that the authors will incorporate the new introduction section*). Despite my opinion about the paper's soundness and presentation, I would like to emphasize that the paper's contribution is, in my opinion, worthy (given that all the results are 100% correct, which I have not confirmed). However, unfortunately, I can't confidently recommend acceptance for the current state of the paper.
Thank you to the authors for their detailed rebuttal and for responding to my comments. I appreciate that it is still a non-trivial task to extend rcGAN to fit to the principal components and singular values and there are certainly empirical benefits to the proposed approach. I was hoping to ask further about the choice of $K$. It is mentioned that the choice of $K$ behaves similarly to PCA on the posterior covariance, which makes sense. But I think I was asking more specifically about if there is insight from the form of the inverse problem $y = \mathcal{M}(x)$ that can give insight into the conditioning of the posterior covariance. For example, do ill-conditioned linear forward models induce posterior covariance matrices that have slower decaying singular values? Similarly, if the inverse problem is non-linear, inducing a multi-modal posterior, is it necessarily the case that we should expect the posterior covariance to also have slower decaying singular values (as compared to a less, multi-modal posterior)? I ask these questions from the perspective of giving the reader some insight into this choice of $K$.
Thanks, this is a great question. In general, the structure of the posterior covariance matrix will be governed by the interaction between the likelihood (defined by $\\boldsymbol{y}=\\mathcal{M}(\\boldsymbol{x})$ if we think of $\\mathcal{M}(\\cdot)$ as stochastic) and the prior on $\\boldsymbol{x}$. One can construct toy problems where the posterior covariance has an intuitive structure, but generally speaking the likelihood, prior, and their interaction will be complicated and non-intuitive. As a toy example, consider a noiseless inpainting problem, where the first half of the entries in $\\boldsymbol{x}$ are directly observed. Then the posterior distribution over those first-half entries would be a Dirac delta, while the posterior distribution over the last-half entries would be the *prior* distribution of the last-half entries conditioned on the first-half entries. For some priors, knowing the first-half entries tells you everything about the last-half entries, while for other priors it tells you nothing, and for most cases it gives you partial knowledge. So, little can be said without a good understanding the prior and, in particular, its interaction with the likelihood. As far as multi-modality is concerned, it does not directly impact the covariance structure, since covariance is based only on second-order statistics. Again, consider a noiseless inpainting problem. If the prior is a high-variance white Gaussian, then the posterior will be a high-variance white Gaussian in the unobserved dimensions. If the prior is instead a Gaussian-mixture with many randomly located component means, the posterior will also be a Gaussian mixture and thus will generally be multi-modal in the unobserved dimensions. But the posterior *covariance* will just measure the uncertainty *energy* along different dimensions, which might be very similar in the uni-modal and multi-modal cases.
Thank you for delving deeper into this. Indeed, it seems like there is not a direct mapping between the inverse problem structure and the posterior covariance, as the (unknown) prior plays a heavy role in determining the structure of the posterior. After carefully considering the author's comments and the other reviewer's discussions, I believe that my score still accurately reflects my view of the paper. Hence I will keep my score.
I thank the author for addressing my concerns by evaluating several diffusion algorithms against their method. It appears that the DPS algorithm yields results nearly identical to the proposed approach, however, DPS heavily relies on a specific scale factor. Additionally, while the DPS algorithm is useful, it is not currently the SOTA for posterior sampling. Although I recognize the novel and valuable contribution of this idea for GANs, I am not convinced that it outperforms the performance of current diffusion models. Therefore, I remain my score as it is.
We thank the reviewer for their valuable feedback and continued discussion. - **Reviewer**: I thank the author for addressing my concerns by evaluating several diffusion algorithms against their method. It appears that the DPS algorithm yields results nearly identical to the proposed approach, however, DPS heavily relies on a specific scale factor. Additionally, while the DPS algorithm is useful, it is not currently the SOTA for posterior sampling. Although I recognize the novel and valuable contribution of this idea for GANs, I am not convinced that it outperforms the performance of current diffusion models. Therefore, I remain my score as it is. **Response**: Yes, our method and DPS had comparable performance after the latter was finely tuned for the FFHQ inpainting application. While we agree with the reviewer that diffusion models are powerful and hold significant promise for solving inverse problems, we emphasize that sampling speed is critical for many applications. Even with accelerated sampling methods, such as those based on DDIM, diffusion methods are still orders of magnitude slower than cGANs. For example, in most clinical setups, MR images are expected to be available immediately after the data acquisition. This level of latency cannot be achieved with current diffusion models. Therefore, posterior sampling methods that can compete with existing diffusion models while offering orders of magnitude faster image generation hold great practical value.
Post-Rebuttal
I thank the authors for their rebuttal and clarifications, which have been very helpful in addressing most of my concerns. However, to further strengthen the paper, I still encourage an ablation study on the $P_{rc}$ parameter, or at least a more detailed discussion of its applicability to the proposed method, as the analysis in [21] may not directly transfer. While I acknowledge the novelty of pcaGAN in aiming for better posterior sampling, I believe theoretical guarantees would make the contribution even more impactful, similar to rcGAN. Additionally, an ablation study on the different regularization terms is recommended, as some original rcGAN terms might be redundant with the new ones and could potentially be removed to facilitate the training process. Currently, I believe my score accurately reflects the merits of the work.
Decision
Accept (poster)