PUCA: Patch-Unshuffle and Channel Attention for Enhanced Self-Supervised Image Denoising

Summary

This paper propose a novel network architecture for self-supervised image denoising. The network is build with foundation blocks of NAFNet, which combines mask and dilated convolutions to implement the blind-spot network. Patch-unshuffle is introduced as downsample/upsample operations with enables multi-scale design of the network while maintaining the blind-spot mechanism. Experimental results show the effectiveness of proposed method on real-world denoising datasets.

Strengths

1. The idea of searching for flexible network architecture is reasonable for improving the self-supervised denoising performance. 2. Patch-unshuffle is well designed to enable downsample/upsample operations for implementing multi-scale network architecture. 3. Experimental results show the effectiveness of proposed method in self-supervised image denoising on real-world images. 4. The paper is well written and easy to follow.

Weaknesses

1. The overall novelty is limited. The training process is the same as AP-BSN, and the network architechture is assembled with existing DBSN and NAFNet. The only contribution is the idea of improving the network flexibility and to implement a multi-scale network with patch-unshuffle, which is not technically sound. 2. The ablation study is not complete. The central idea of this paper is to improve DBSN used in AP-BSN with multi-scale architecture, however, there are existing multi-scale BSN architechture such as [1]. What's the performance of porposed network compare with [1]? 3. The competing methods is not complete. This paper shows good performance on real-world image denoising, but the latest self-supervised denoising methods [2][3] are ignored. [1] High-quality self-supervised deep image denoising. NIPS, 2019. [2] LG-BPN: Local and Global Blind-Patch Network for Self-Supervised Real-World Denoising. CVPR, 2023. [3] Spatially Adaptive Self-Supervised Learning for Real-World Image Denoising. CVPR, 2023

Questions

My major concern is weakness 2 and 3, please complete the necessary experiments.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors hav adequately addressed the limitations and potential negative societal impact.

Summary

This work addresses the problem of developing a self-supervised learning-based denoiser. To this end, this work proposes the concepts of patch shuffle/unshuffle operations to effectively downsample and upsample the features while ensuring that the J-invariance property holds with the resulting network. Using dilated attention blocks and patch shuffle/unshuffle, this work comes up with a variant of Unet architecture that can significantly improve J-invariance networks' spatial information aggregation ability. The proposed method improves the denoising performance on public benchmark datasets.

Strengths

1. The paper is written well. 2. The new notions of patch shuffle/unshuffle and dilated attention blocks to ensure the holding of J-invariance properly is interesting 3. The proposed method outperforms prior arts significantly

Weaknesses

1. A study is missing to elaborate more on what other network architectures the proposed ideas can find a perfect fit. There have been numerous variants of Unet proposed in the literature. Can those extensions boost the performance further remain unclear.

Questions

- Please address my comments under weaknesses. - Visual comparisons with other self supervised methods (example, Figure 6) reveals as if the proposed method lacks details while removing the noise completely whereas other methods retains more details. Is there an explanation on why so? Is it possible to control the denoising level of the proposed method to earn back the lost details?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations has been addressed adequately.

Summary

This work extends the field of blind spot networks (BSN) for self-supervised image denoising. They propose patch-unshuffle/shuffle a technique for downsampling/upsampling that preserve J-invariance, which allows to build U-Net like architectures expanding the receptive field and utilizing multi-scale representations in BSN. Second, the dilated attention block is introduced, a J-invariant channel attention mechanism incorporating global information. The method outperforms existing self-supervised methods by 1.6dB (SIDD dataset) and 0.7dB (DND dataset) in PSNR resulting also in improved perceived image quality for the depicted examples.

Strengths

1. The proposed method is interesting and alleviates the constrained architecture design imposed by J-invariance opening up possibilities for new designs of BSNs. 2. The experiments on the two most common benchmarks for self-supervised image denoising are extensive and the method is compared to a range of baseline methods. 3. The authors theoretically derive the J-invariance property of their proposed patch-unshuffle method.

Weaknesses

1. Section 3.2 introduces the dilated attention block (DAB) that preserve J-invariance. However, from the sentence "To fulfill this requirement we incorporate a d-dilated 3x3 depth-wise convolution (DDC) before gating and attention, taking inspiration from D-BSN" it is not clear to me how exactly J-invariance is preserved nor if or how the approach differs from the previous work. 2. From Section 2.2 it is not clear to me how pixel-shuffle downsampling (PD) works. From Figure 4 it seems as PD changes the dimension of the input image, however this is not discussed in Section 2.2 and the algorithm overview in Figure 3 shows the signal before and after PD to exhibit the same dimension. 3. I find that dropping the channel dimension in the proof in Section 3.1 is a bit confusing as now several input indices are mapped to the same output index. Maybe an illustration with an explicit example of how a set of indices is transformed might be helpful. 4. Summarizing the above points the clarity of the work could be improved.

Questions

1. Choosing the by far worst performing supervised baseline as the only visual reference for supervised training in Figure 1 seems a bit misleading with respect to the comparison of supervised and self-supervised methods. 2. The caption in Table 3 (3b with DAB) does not fit the description in the text (3b no DAB). 3. Line 256 refers to other work without citing it. 4. The paper uses a numeric citation style in the text but alphabetic in the references making it impossible to match citations to entries in the references.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors briefly touch on one limitation of their work.

Summary

This paper presents a method for self-supervised image denoising. Specifically, a patch-unshuffle (and shuffle) operation together with a dilated attention block was proposed to achieve the goal. Experimental results on two real noise datasets (SIDD and DND) show the effectiveness of the proposed method over other self-supervised methods.

Strengths

+ Real noisy image denoising is an important problem in the denoising community and the authors also alleviate the limitations from the $\mathcal{J}$-invariance to some extent. + The proposed method was shown to perform generally better than other self-supervised methods on the SIDD and DND real noise datasets. + The paper is generally well-written and easy to follow.

Weaknesses

- From the description, it seems the authors were motivated by the flexibility in network structures. But it is unclear why the flexibility in network structure design is necessary, or how would it benefit the denoising community. - The novelty of the proposed patch-unshuffle/shuffle is a bit limited, especially considering the previously proposed pixel-unshuffle/shuffle approach [47]. The key idea is very similar with the only difference as pixel vs. patch. Whereas the significance of such a change and novelty were not well clarified with convincing justifications. For example, what if replace the patch-unshuffle/shuffle blocks in the proposed method with the corresponding pixel-wise version? It is also unclear what is the contribution of the PD and PU processing at the input/output ends. What if remove these processing operations and only use the proposed patch-wise blocks? On the other hand, from what was shown in Fig.5 in [47], it seems that they already used a similar idea of patch-shuffle. - The proposed patch-unshuffle was motivated by the claimed issue of breaking the \mathcal{J}-invariance in pixel-unshuffle. But as also acknowledged by the authors, the proposed patch-unshuffle can still break it in some cases. - The novelty of the proposed dilated attention block is also a bit limited. As mentioned by the authors, it was based on the channel attention [7] being applied to U-Net, with inspirations from D-BSN [36] and SCA [7]. If there is a new design involved, the contribution and novelty are unclear, and it lacks convincing (experimental) validation, e.g. what if without the new design? how the performance relates to the proposed design? - It is unclear what dilation rate was used in the proposed DAB block. And unclear how this hyper-parameter d affects the model's performance. - The proposed method performs worse than the C2N+DIDN for the SSIM (Table 1). We know that SSIM usually represents the visual or structural quality of the reconstructed image. But there is no clarification for this. - In the ablation study (Table 2), the authors attribute the decreased performance when the level increased from 3 to 4, to the resolution of the latent features. To validate this, an experiment with a larger input image (thus a higher resolution for the latent features when level=4) should have been included. - Although LG-BPN [35] may not be published (and not published at CVPR) when the proposed method was submitted, the idea of local-global branches and the larger receptive fields through dilated convolutions is very similar to the proposed method. The novelty is as a result weakened. - The proposed method claimed the issues within existing methods about the \mathcal{J}-invariance, and motivated by this, claimed that the proposed method "successfully alleviates the constrained architecture design imposed" by this. But it is unclear how the constraints for architecture design were alleviated. This was not validated, unless other designs were shown to be effective with the proposed method. - Missing definition to p (L136-144). If it's the same as the p in L145, please specify (the first time it appears). - It is unclear what "PUCA" indicates, "Patch-Unshuffle Channel Attention", or "xxxUnetxxx"? Please clearly define it the first time it was introduced. - In the caption of Fig. 3, "During encoding...through patch shuffling..." should be "patch unshuffling"?

Questions

It would be helpful if the authors could address the concerns raised in the above Weaknesses section. For example, the main motivation of the proposed method and its validity; the technical novelty of the newly proposed components (i.e. the patch-unshuffle and the DAB blocks); those concerns about the experiments. The reviewer would be happy to change the rating if the concerns could be well addressed.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors clearly mention the limitations of their method (in the Conclusion) and potential societal impacts.

Summary

This paper introduces PUCA, a J-invariant U-Net for self-supervised image denoising. Specifically, the authors propose a patch-unshuffle and dilated attention block to allow the use of the multi-scale structure for enlarging the receptive field. Extensive experiments demonstrate that the proposed PUCA outperforms the existing method by a notable margin. There is also adequate analysis to illustrate the correctness and properties of the proposed method.

Strengths

1. Good Quality. The submission is technically sound in my opinion and the advantages and limitations of this work are discussed carefully and honestly. 2. Good Clarity. The submission is written with sufficient clear definitions and formulas. And the organizations are well-designed. 3. Good motivation. The authors observe that the commonly used simple structure, such as multi-scale structure, in image denoising violates the J-invariance and thus cannot be used in BSNs. 4. Solid solution. The authors propose patch unsuffle to achieve downsampling without violating j-invariance. In addition, proof is also provided to demonstrate this property of patch unshuffle theoretically. 5. Sufficient experiments. The authors conduct sufficient experiments and ablation studies to demonstrate the effectiveness of the proposed method. The results show that the proposed method outperforms all existing methods by a notable margin.

Weaknesses

No obvious weaknesses.

Questions

In L28, the authors emphasize the generalization issue of the supervised method and note that the unsupervised method can generalize better. However, this claim is not justified. Can the author show the cross-dataset validation of both the supervised method and the unsupervised method?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The paper contains an adequate discussion of social impacts and limitations.

Post-rebuttal comments

The rebuttal has addressed all my concerns. I retains my original rating.

Post-rebuttal Comments

Thanks for the authors' feedback. It has addressed my question. I'll maintain my original rating.

Re: rebuttal

Thanks to the authors' rebuttal. I have read the rebuttal and comments from other reviewers. It is good to see that most of my concerns were addressed, making the paper more clear and easier to understand. The authors are suggested to include these explanations into their final version. Although the technical novelty is still a bit limited to me, given the related prior works, I would raise my rating in response to the major concerns being addressed.