Normalization-Equivariant Neural Networks with Application to Image Denoising

Summary

The paper proposes normalization-equivariant CNNs, in which the convolutional kernels are forced to sum to 1 and the activation sorts pairs of neurons. This is shown to perform similarly to other models on image denoising when trained with the same noise levels, but generalizes much better to unseen noise levels.

Strengths

S1) The paper presents a simple and effective method to improve generalization in image denoising CNNs. S2) Results are strong, particularly the ones shown in Fig 4. S3) Section 5.2 presents insights about previously observed phenomena of bias-free/scale-invariant models performing well on denoising, and explains why the normalization-equivariance proposed is potentially superior.

Weaknesses

W2) Table 2 reports results on the non-blind experiments but I couldn't find whether the results in Fig 4 and 5 are blind or non-blind. I assume they are blind and since the models were trained on a single noise level. If they are not and if the noise level is known at inference time, I think the significance of the results is diminished since with a known noise level the ordinary CNNs can be perform quite well as shown in Fig. 5. Please clarify. W3) Fig 2 implies that the kernel constraint for convolutional kernels is along the channel axis, but it seems that it should be along both channel and spatial dimensions to preserve shift-equivariance. The text does not seem to specify it for the convolutional kernel since Eq 5 refers to a linear layer. Please clarify and include a precise definition of the convolutional kernel between two layers of shape H x W x C. W4) Some recent methods of denoising are missed [1, 2]. Zamir et [1] seems to report slightly better results than the baselines on denoising on the same datasets and also evaluate on different datasets. This is not a dealbreaker but the submission would be made stronger when evaluating on more datasets since the ones used are quite small. References: [1] Zamir et al, "Restormer: Efficient Transformer for High-Resolution Image Restoration", CVPR'22. [2] Chen et al, "Simple Baselines for Image Restoration", ECCV'22.

Questions

Minor: Typically in image processing, for an image f: Z^2 -> R, "scale" refers to resizing (f'(x) = f(ax)) and "shift" refers to moving pixels (f'(x) = f(x+a), in the sense that convolution is shift-equivariant). In the submission, it seems that "scale" means multiplying all pixel values by a constant and shift means adding a constant to all pixels. I recommend clarifying the difference to avoid confusion.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

W1) I couldn't find a discussion of limitations in the paper and I think this is a major weakness. Table 1 in the supplement shows that the norm-equivariant models need to be trained for about twice the number of iterations of the baselines and with MSE instead of L1 loss. For fairness, I think the ordinary and scale-eq baselines should be also evaluated under the same conditions and this possible limitation should be reported in the main text. It seems that the method is also incompatible with other common used layers such as max-pooling and batch normalization, which should also be discussed.

Summary

The authors of this paper propose an architectural change in neural networks, in a way that enforces the normalization equivariance property. The authors demonstrate how networks that satisfy this property can be used in the setting of image denoising.

Strengths

- The paper is clearly written overall. The intended goal of the architectural changes proposed is clear, and how these changes achieve this goal is also made explicit in the paper. The proposed architecture is also novel, as far as I am aware. - There is a clear benefit derived from the architectural changes proposed by the authors. The intended effect of normalization equivariance is to make the output of the network disregard potential alterations in the scale of the noise added to the model. This is made clear via Figure 4, which precisely shows that the alterations to the architecture achieve the intended effect. The authors also adequately explain in the paper how this equivariance with respect to this normalization makes their models robust to the amount of noise added to the input, namely in that adding different amounts of noise only corresponds to renormalization. - The idea that the authors propose regarding sort pool between channels is an interesting and novel way of pooling information from different channels in CNNs. This seems to be a novel inclusion in the architecture of a neural network, and I am interested in its potential applications beyond normalization equivariance.

Weaknesses

My main issues with this paper lie with the somewhat limited setting of the experiments performed to validate the abilities of the proposed architecture. - While interesting, the target problem for this paper is somewhat limited. The only application considered in this paper is that of image denoising, which is arguably a very limited setting. I believe that demonstrating the effects of enforcing normalization equivariance in other problems (such as image classification or other types of reconstruction e.g. superresolution) would improve the paper. - Even in the context of denoising, the authors of this paper only consider the setting of Gaussian noise being applied to an image. While generally accepted as a valid setting for denoising, It would be interesting to see if the normalization equivariance was able to help in the case where different types of noise are applied (e.g. salt-pepper noise, random masking of pixels, and so on). - Regarding baselines, the authors compare against techniques which use regular feedforward neural networks. While the comparisons support the points made by the authors, I believe that it would be interesting to compare against techniques that employ generative networks to solve the denoising problems e.g. via DPS [A]. Reference: [A] Chung et al., “Diffusion Posterior Sampling for General Noisy Inverse Problems”, ICLR 2023.

Questions

I would be interested to know if the authors have considered applying their architectural change not when training the models from scratch, but by altering select layers of a pretrained model (for example, by including the sorting layer as part of a pretrained architecture). If something like this is possible, then it would alleviate some of the issues I mentioned above, allowing the method to be more readily applied to a larger set of problems.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

2 fair

Limitations

I believe the authors should better clarify in the paper how currently their method is applicable mainly to image denoising as it stands (even though in principle normalization equivariance may be helpful in other tasks as well). Regarding negative societal impact, I do not see any immediate issues arising from this work.

Summary

In this paper, the authors propose an architectural design which guarantees shift and scale equivariance. The proposed method is called normalization equivariance in which scale equivariance is inherited from prior work by removing layers biases and shift equivariance is achieved through affine-constrained convolutions along with channel-wise sort pool (instead of ReLUs). They adopt two popular architectures to become normalization equivariance, and show that their variant outperforms both unconstrained, and scale-equivariance models in generalizing outside of the training range.

Strengths

The paper is written very clearly and has excellent flow. It is technically strong and propositions and claims are backed up both theoretically and empirically. The idea of enforcing shift equivariance by design is original and the authors nicely elucidate its necessity and the positive impact it has. In my opinion, this work offers a significant contribution in the area of useful inductive biases and also understanding neural networks trained for denoising.

Weaknesses

Generally the paper is written very well and all sections are carefully and throughly explained. Here are a few minor issues: I believe rotation needs to generally be an invariance (as opposed to equivariance) with respect to labels in image classification (line 52) Proof for proposition 2 can be included in the appendix. The derivation of eq 6 (from line 178 onward) is not clear. It would be useful if the authors re-write that paragraph. Number of possible states is still $2^S$ if ReLU activations are used instead of sorting (line 222)

Questions

It would be useful to see comparison of computational cost for training. Although sort pooling is restricted to n=2, I assume it is still most costly than using ReLUs. A numeral comparison can show the cost for gaining normalization-equivariance. It wasn't clear from the text how the affine convolution constraint was enforced during training. It would be useful to explain that in the main text since it is one of the two main design contributions.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

No major limitations or potential negative societal impacts.

Summary

This work proposes a normalization equivariant deep convolutional neural network for image denoising. The authors provide a mathematical construction of a normalization equivariant neural network and propose sorting nonlinearity. The proposed nonlinearity shows comparable performance with the baseline and shows robustness against different noise levels.

Strengths

1. The work provides a sound mathematical formulation for the Normalization equivariant Neural Network. 2. The proposed model is robust across different noise level 3. The paper is well written, nicely explained

Weaknesses

1. The effect of newly introduced sorting nonlinearity will be better understood through an ablation study, i.e., reporting the performance of a model with only affine convolution and Relu. 2. The effect of shift-equivariance is not well described and demonstrated. Even though it is addressed in the last section, its’ benefit over the scale-equivariant model is not well demonstrated and motivated. Given the novelty of the work is providing shift equivariance (in addition to scale-equivariance), it deserves better motivation and justification. 3. The proposed non-linearity performs point-wise swapping of the output of different filters at some points. So, it distorts the learned features. It has a significant possibility of affecting the performance of other computer vision tasks (e.g., segmentation), limiting its use only to denoising.

Questions

1. The work [1] shows a better generalization of scale equivariant neural networks on different noise scales compared to what is shown here. What might be different here that lead to this difference? 2. Line 282, “but this is much less true for λ < 1 (stretching) for the reason that it may exceed the bounds of the interval [0, 1]. This explains why scale-equivariant functions do not generalize well to noise levels lower than their training one”: how λ < 1 related to not generalizing to lower noise levels? 1.ROBUST AND INTERPRETABLE BLIND IMAGE DENOISING VIA BIAS-FREE CONVOLUTIONAL NEURAL NETWORKS

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Response to author rebuttal.

I thank the authors for their response to my concerns. From the general response, I see that my concern regarding the use of only Gaussian noise as a setting has been alleviated, with experiments showing robustness to other types of noise as well. I also appreciate the effort the authors made to extend their proposed method to settings beyond image denoising. While simple, the image classification experiments provided show some preliminary results on how the method can be used on tasks other than denoising. This somewhat alleviates the limitations of the paper. As such, I have raised my score slightly.

Response to the Rebuttal

I thank the authors for the response.