PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging

Summary

This paper addresses the challenge of spatially varying Point Spread Function (PSF) in lensless imaging and introduces a two-stage approach for reconstructing lensless images.

Strengths

This paper is well-written, presenting a detailed statement of the problem, the physical imaging model, an innovative reconstruction method, and thorough experimental results. I believe this paper will significantly impact the fields of both lensless imaging and the broader area of spatially-varying computational imaging, given its contributions to both the imaging model and reconstruction methods. The demonstrated reconstruction results are impressive, showing significant improvements over existing methods. I believe this paper will make a significant contribution to both lensless imaging and the broader field of spatially varying imaging reconstruction. The proposed learnable spatially varying PSF will provide valuable insights for many other applications.

Weaknesses

The only two weaknesses of this paper are: (1) the lack of real-world experiments, but this is mitigated by the use of a standard lensless imaging dataset for evaluation, and (2) the paper would benefit from incorporating more knowledge and discussion from the optics field to strengthen its soundness. There are two suggestions to make this paper more robust: (1) Expand Fig. 4(b) and Fig. A1 to illustrate how PSFs change with different fields of view. (2) Expand Fig. 2 and Fig. 4(d) to better demonstrate the significance of considering spatially varying PSFs, possibly including pixel-wise error maps of the reconstruction results.

Questions

1. Fig. 4(a) should be revised. Light coming from an off-axis angle should not be shown as being ‘refracted’ to produce parallel output light, as this is inconsistent with the proposed spatially-varying PSF model. Instead, I suggest that the author illustrate two point sources: one originating from the axis with output also parallel to the axis (in cyan), and another from an off-axis point with output also at an off-axis angle (in blue). 2. In Fig. 1(1), vignetting is observed, while the reconstructed images show no signs of vignetting. It would be better to explain in the paper how the vignetting effect was removed. 3. It would be beneficial to provide more explanations from the optical perspective in Section 3.2 to strengthen the paper's soundness in optics. The paraxial imaging model (spatially invariant convolution) is commonly used to simplify the forward imaging model, but it is inherently inaccurate, particularly for large field-of-view imaging. It would be good to add references to corresponding off-axis wave propagation models, such as “Shifted angular spectrum method for off-axis numerical propagation” by Matsushima, “Modeling Off-Axis Diffraction with the Least-Sampling Angular Spectrum Method” by Wei et al., and “Shifted band-extended angular spectrum method for off-axis diffraction calculation” by Zhang et al. 4. Spatially varying PSFs have also been used in other applications with improved results. Therefore, I think it is important to add references to these works. For example, “Aberration-aware depth-from-focus” by Yang et al. demonstrated improvement in depth estimation when considering spatially varying PSFs. “Correcting Optical Aberration via Depth-Aware Point Spread Functions” by Luo et al. proposed spatially varying PSFs for optical aberration correction and depth estimation. Additionally, “High-Quality Computational Imaging through Simple Lenses” by Heide et al. proposed spatially varying PSF imaging for simple lenses.

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The limitations of this paper are well discussed.

Summary

This paper proposes a method for reconstructing photorealistic images in lensless imaging. The reconstruction process, which aims to be consistent with observations while achieving photorealism, is based on range-null space decomposition. To accommodate realistic cameras, the method introduces SVDeconv, which learns the deconvolution process with a spatially-varying PSF simultaneously. Additionally, the reconstruction in the null space is performed using a pretrained diffusion model. Quantitative and qualitative evaluations are conducted using two datasets, PhlatCam and DiffuserCam.

Strengths

+ For the generally challenging task of reconstructing high-frequency details in lensless imaging, introducing the concept of range-null space decomposition to achieve photorealistic and measurement-consistent image reconstruction is a very rational and technically sound approach. + The idea of using a generative approach solely for reconstruction in the null space, rather than relying entirely on generative priors for restoration, addresses the issue of hallucination. This approach showcases originality. + The effectiveness is also commendable as it achieves generally good results both quantitatively and qualitatively when compared with various other methods.

Weaknesses

1. There is a lack of detail regarding the training process, making it difficult to understand correctly. It is unclear whether fine-tuning is performed using the input images at the test time or the PSF and parameters are freezed using a pretrained network. 1. The method of dividing the Spatially-Varying PSF (SV-PSF) into a 3×3 grid is somewhat naive. Particularly, I doubt that the spatial dependency of the PSF is also influenced by the target scene depth, which does not appear to be considered. 1. The analysis of the results is also insufficient. It is unclear how close the estimated PSF is to the accurately calibrated SV-PSF. The comparison between the reconstructed images using the accurate SV-PSF and the proposed method is not discussed. Additionally, for the dataset used in the evaluation, the inference results of the range-null content are not provided (as shown in Figure A2). The data fidelity in the reconstructed images is also not confirmed. + Comment: In eq(4), $N\times N$ is used, while previously $K\times K$ was used.

Questions

If there are any misunderstandings in the weaknesses pointed out, please clarify them.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations have been addressed well.

Summary

This paper proposed a deep learning-based approach for lensless imaging. To address the problem of model mismatch that simple convolutional models cannot accurately describe the lensless imaging process, this paper introduced a spatially-varying devolution module that reweights the deconvolution results from multiple kernels using spaitally-varying weights. In addition, a two-stage model based on range-null space decomposition. In the first stage, a spatially-varying deconvolution network is designed to reconstruct the low-frequency content; The second stage uses the output of the previous stage as a condition to guide the pre-trained diffusion model in reconstructing fine high-frequency image details associated to the null space of the measurement operator. The experiments were conducted on two datasets: PhlatCam and DiffuserCam.

Strengths

1. Introducing diffusion model to reconstruct fine details of high-frequency content associated to the null space the measurement operator. 2. To some extent, the spatially-varying deconvolution solved the problem of model mismatch.

Weaknesses

1. Using diffusion model to recover the null-space related image information is not new. E,g,, [a] Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model. ICLR 2023. 2. Lack of experimental or deductive analysis processes to demonstrate the effectiveness or accuracy of using range-null space decomposition to describe the process of lensless image restoration. 3. The PSFs in lenless imaging are usually having a very large size (even larger than the image). Accordingly, the deconvolution kernels should be large. However, the deconvoluton kernel size seems small in the first stage.

Questions

1 . Why to simulate the PSF that has already been calibrated in the dataset, and what effect does the simulated PSF have in the algorithm? 2. Section 3 mentions that model mismatch is mainly caused by the presence of the incident angle θ. However, in the first stage of weight calculation, the error caused by only considering distance. Why? 3. What exactly does spatially-varying represent? How is the FoV center of the learnable kernel determined in weight calculation? What does the distance from point (u, v) to the FoV center exactly mean? Illustration with one figure could facilitate the understanding. 4. How to set the size and quantity of the learnable kernels?

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

The paper mentioned several limitations, such as computational cost due to introducing diffusion-based sampling and two-stage processing, as well as false details introduced by the diffusion model. One possible limitation that is not mentioned in the paper is, the two-stage framework may have the robustness issue, the error from the first stage may affect the accuracy in the second stage, compared to an iterative framework. I suggest the authors to have a discussion on this issue.

Summary

This paper proposes a novel two-stage approach for lensless image reconstruction. The approach ensures data consistency with a spatially varying deconvolution method and enhances photorealism using pre-trained diffusion models. This method outperforms existing techniques in data fidelity and visual quality on PhlatCam and DiffuserCam systems.

Strengths

1) The biggest strength of this paper lies in the quality of results displayed. The reconstructed images closely resemble the ground truth and are structurally very similar, unlike some of the other comparable methods. 2) Extensive evaluation has been performed. The method has been tested on two different lensless imaging datasets, which helps build confidence. The method outperforms peers on 6 difference metrics, 3 for quality and 3 for photorealism. The evaluation has been conducted on both range space recovery method and null space recovery method, demonstrating the merit of this approach. 3) Great clarity has been provided around the range space - null space decomposition, including mathematical derivations. 4) The method seems reproducible because of the great details provided in the paper, for both range space and null space reconstruction.

Weaknesses

1) The paper mentions usage of 3X3 PSF kernels for Deconvolution. They arrive at this conclusion using experiments with different sizes. However, no mathematical reasoning or intuition has been provided as to why this is a good choice, and under what circumstances, it will break down. Without this explanation, it is very difficult to reuse the same model for a dataset captured with a different lenseless system. More work/reasoning is needed to support this choice of sampling.

Questions

1) This is minor, but why is Wiener Doconvolution referred to as WinnerDeconv everywhere in the paper? 2) Although the paper compares the proposed approach with several other methods for qualitative evaluation, it is unclear how these approaches compare when it comes to computational complexity. Typically, there is a trade-off between quality and compute. Therefore I would like to see a comparison of the computational complexity involved in reconstruction. 3) I would recommend citing the following paper: V. Boominathan, J. T. Robinson, L. Waller, and A. Veeraraghavan, “Recent advances in lensless imaging,” Optica 9(1), 1 (2022). This paper gives an overview of the common modulation schemes used for lenseless imaging, and I believe your approach is only applicable for phase modulation masks and not for amplitude modulation masks?

Rating

9

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

1) Although the authors mention that real-time reconstruction is not possible, it is unclear how much time reconstruction actually takes. It would be great if this is explained as well.

Response to the rebuttal

Thanks for the rebuttal which has addressed most of my comments. I would like to raise the score to Weak Accept. One limitation is the time complexity shown in the response to Reviewer FAra, mainly caused by the use of diffusion models and much higher than CNN-based methods. I understand this is the common limitation in existing diffusion-based methods, and hope that it could be addressed in future work.

Reponse to the rebuttal

### 1. Details of the training process Thanks to the authors for the clarification. ### 2. Variation of spatially-varying PSF related to the depth I expect the authors to address in the final version that the spatial dependency of the PSF is negligible in the target application and is out of scope for the paper. ### 3. Comparison of reconstruction using accurate SV-PSF and our method The additional results are not convincing for me. Can the authors clarify why the proposed method outperforms the SV-deconv methods utilizing accurately calibrated SV-PSFs? It might be inadequate to simply compare the PSNR of the final results. I believe that some analyses such as checking and compare infromative paring the range- and null-space contents."を"I believe that additional analyses, such as checking and comparing the range- and null-space contents, would be more informative. ### 4. Data fidelity in the reconstructed range-space contents The description for Tab.2 in the original paper is insufficient. I appreciate the authors for providing the range space content in the rebuttal. It helps to clearly see the improvement achieved by the method.