Untrained Neural Nets for Snapshot Compressive Imaging: Theory and Algorithms

Summary

Please see Strengths and Weaknesses.

Strengths

1. Rigorous theoretical analysis of the proposed formulation 2. Detailed empirical evaluation.

Weaknesses

1. Formatting in Figure 2 can be improved. The text near smaller cubes is not readable.

Questions

NA

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Summary

This work gives some theoretical analysis for mask optimization and DIP-based SCI recovery methods. The work claims that the proposed SCI-BDVP achieves SOTA performance among UNN methods.

Strengths

1. This work provides some theoretical analysis, compared with the conventional work is rare. 2. Bagged DIP is introduced to develop SCI iterative methods and the proposed method claims to achieve SOTA performance among UNN methods.

Weaknesses

1. While this work does some theoretical analysis by introducing DIP's theory, it cannot be directly generalized to all untrained networks. On the one hand, it seems impossible that all untrained networks satisfy the DIP hypothesis in the paper, and on the other hand, how can the UNNs used in this work be guaranteed to satisfy the DIP hypothesis and Lipschitz's condition? As far as I know, networks that satisfy Lipschitz's condition require a specific design, and this work does not seem to give some related introduction. 2. The core of this work is to prove theoretical results for untrained networks, but the actual contribution is less than stated. When the properties of certain untrained networks are demonstrated by DIP, the work does not propose any new hypotheses in snapshot compressive imaging but simply adopts the original one, which decreases the contribution of the paper. 3. When the DIP hypothesis is used to simplify the UNNs as a minimization operator, the work is merely proving a boundary based on the minimization operator. Similar theoretical results based on the minimization operator can also be found in the theoretical analysis section of GAP-Net[1]. 4. The DIP hypothesis demonstrated in this paper seems to ignore some conditions in the original DIP paper and doesn't capture the unique characteristics of untrained networks, what if the same is true of pre-trained networks? 5. The comparison algorithm needs to be perfected. The current comparison algorithms seem to be some traditional optimization algorithms and your methods, and we even can't distinguish which of them is previous UNN works. So what does it mean that your method achieves SOTA performance in UNNs？ 6. Considering this work as a deep network-based approach, such comparisons are unfair. Why not add some recent self-supervised methods to demonstrate the effectiveness of the algorithm? In addition, despite claiming to be compared with the supervised algorithms in noisy scenarios, some recent algorithms, such as SCI3D[2] and EfficientSCI[3] have not been included. [1]Deep Unfolding for Snapshot Compressive Imaging, in IJCV2023. [2]Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot CompressiveImaging, in ICCV2021. [3]EfficientSCI: Densely Connected Network with Space-time Factorization for Large-scale Video Snapshot Compressive Imaging, in CVPR2023.

Questions

How can the UNNs used in this work be guaranteed to satisfy the DIP hypothesis and Lipschitz's condition?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors provide some limitations, which is their future research.

Summary

The focus of this paper is on developing recovery algorithms of snapshot compressive imaging (SCI) using untrained neural networks (UNNs). Besides, the paper introduces the concept of bagged-deep-image-prior (bagged-DIP) to create SCI Bagged Deep Video Prior (SCI-BDVP) algorithms, which are designed to address common challenges faced by standard UNN solutions in SCI recovery. Extensive experiments demonstrate the effectiveness of the proposed method in video SCI recovery. In scenarios with noisy measurements, this untrained network even outperforms supervised methods.

Strengths

- This paper is well structured and has clear logic. - The experimental results on the performance are convincing. - The proposed method is based on untrained neural network, making it more flexible to be applied in various scenes.

Weaknesses

- For untrained methods, runtime may be a crucial metric [1,2]. But this paper doesn't provide analysis in comparison to existing methods. - To improve clarity, consider adding clear notations for the symbols used. For example, what do different colored arrows represent in Figure 2, and the color of "Fusion" and "MSE" are too similar to be distinguished. [1] Rui, Xiangyu, et al. "Unsupervised hyperspectral pansharpening via low-rank diffusion model." Information Fusion 107 (2024): 102325. [2] Pang, Li, et al. "HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Questions

- SCI is also widely used in hyperspectral image, can your method be used in hyperspectral reconstruction task? If so, what's the prominent advantage of your method? - How does the method handle different noise models beyond additive Gaussian noise?

Rating

7

Confidence

3

Soundness

4

Presentation

4

Contribution

4

Limitations

- Exploring the application on additional datasets from different domains or varing conditions in the future may increase persuasiveness, e.g. hyperspectral data. And exploring the performance of the proposed method under different noise models would provide a more comprehensive evaluation of its robustness and general applicability.

Summary

This paper leverages untrained neural networks UNN (deep image priors DIP or deep decoder DD) as a prior to solve Snapshot Compressive Imaging (SCI), a technique used in ($n_1$ x $n_2$ x B)-dimensional 3-D imaging where the captured measurements lie in a 2-D plane ($n_1$ x $n_2$). The application of UNNs in the context of SCI itself is not novel; this paper's main contributions are: 1) Theoretical recovery guarantees for SCI (i.e. existence of a minimizer to the reconstruction problem) denoting number of 2-D frames B that can be recovered using 1 2-D measurement as a function of UNN model complexity (assuming original signal is close to range of UNN by $\delta$) under noise-free and additive gaussian noise settings. 2) Use bagged-DIP as algorithm for signal recovery, called SCI Bagged Deep Video Prior (BDVP). 3) Optimize binary-valued masks used in the measurement process and show empirical analysis.

Strengths

The main contribution of this paper are theoretical reconstruction guarantees for SCI using UNNs under both noisy and noiseless measurements. They derive a bound on reconstruction error in terms of signal parameter B, Bernoulli sampling pattern mask p, measurement and signal parameter n, and $\sigma_z$ noise. Authors further validate their reconstruction error bound and its dependence on sampling parameter p, on empirical datasets on 6 videos: Drop, Runner, Aerial, Crash, Kobe, Traffic. They propose SCI Bagged Deep Video Prior method as an algorithmic framework for solving SCI. On the empirical data, they show improved SNR and SSIM against baseline untrained methods.

Weaknesses

Main comments: Computational complexity: Deep Video Prior/Untrained Network Prior setups have high computational complexity; on top of this a bagging approach requires solving K such problems simultaneously. This requires a discussion on how computational complexity compares to baselines. Theoretical claims are summarized in main paper; however I was not able to find longer version of paper in supplementary zip folder to validate the proofs. Theorem 3.1: Are parameters $p$ in $u [0,1]^p$ and Bern($p$) for $D_{i,j}$ the same? Minor comments: line 47-48: "Existing UNN-based SCI solutions either recover the image 48 end-to-end in one shot or employ iterative methods akin to projected gradient descent (PGD)." (relevant citations missing) line 88-97: Literature review of DIP + UNN: Add relevant citation to "Qiao, M., Liu, X., & Yuan, X. (2021). Snapshot temporal compressive microscopy using an iterative algorithm with untrained neural networks. Optics Letters, 46(8), 1888-1891." Theorem 3.1. reconstruction error bound - $\rho$ not defined.

Questions

Theoretical claims are summarized in main paper; however I was not able to find longer version of paper in supplementary zip folder to validate the proofs - this would help consolidate the contents.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Limitations have been discussed; they should additionally discuss time/computational complexity of the proposed BDVP algorithm.

Acknowledging the rebuttal

Thank you for the rebuttal. I have no further questions.