New Experimental Results
Dear reviewers,
We sincerely thank you for your time spent reviewing our work and responding to our rebuttal. We wanted to update you all to let you know that we have conducted experiments based on the comments from some reviewers. In particular, some reviewers expressed concern regarding our lack of empirical results to support our theory. To address these concerns, we have conducted two experiments: one on using the new $\alpha$-divergence loss to learn regularizers and the other on the requirement of positive homogeneity in the neural network layers.
**Denoising comparison of Hellinger and Adversarial Regularizers:** We wanted to test the performance of regularizers learned using the Hellinger-based loss of Eq (5) and those learned using the adversarial loss in Theorem 2.4. To do this, we consider denoising on the MNIST dataset. We take $1000$ random samples from the MNIST training set (constituting our $\\mathcal{D}_r$ distribution) and add Gaussian noise with variance $\\sigma^2 = 0.05$ (constituting our $\\mathcal{D}_n$ distribution). We then aim to reconstruct *test* samples from the MNIST dataset corrupted with Gaussian noise of the same variance seen during training. For our regularizers, we parameterized them with a deep convolutional neural network, similar to the construction in Lunz et al [51]. Specifically, the network has 8 convolutional layers with LeakyReLU activation functions and an additional 3-layer MLP with LeakyReLU activations and no bias terms. The final layer is the Euclidean $\\ell_2$ norm. This network implements a star-shaped regularizer, as outlined in the discussion of our Theorem 4.3. The regularizers were trained using the adversarial loss and Hellinger-based loss (Eq (5)). We also used the gradient penalty term from Lunz et al [51] for both losses.
After training, we then aim to reconstruct noisy *test* samples $y = x_0 + z$ where $x_0$ is a test digit and $z \sim \mathcal{N}(0,\sigma^2I)$. We do this by solving the following with gradient descent initialized at $y$:
$$\\min_{x} ||x - y||^2 + \\lambda \mathcal{R}(x).$$
Here, $\\mathcal{R}$ denotes our trained regularizer. For the adversarially trained network, we used $\lambda := 2 \cdot \tilde{\lambda}$ where $\tilde{\lambda} = \mathbb{E}_{z\sim\mathcal{N}(0,\sigma^2I)}||z||_2$, as described in [51]. For the Hellinger-based network, we found that $\\lambda = 10 \tilde{\lambda}^2$ gave better performance, so we used this for recovery.
We display the average PSNR and MSE over 20 test images for both networks below. We see that the Hellinger-based loss gives competitive performance as compared to the adversarially trained network. This suggests that these new $\alpha$-divergence based losses are potentially worth exploring from a practical perspective as well.
**Noisy image MSE, PSNR:** 0.0486, 13.13
**Hellinger MSE, PSNR:** 0.0046, 23.52
**Adversarial MSE, PSNR:** 0.005, 23.14
**The role of the activation function:** While our theory for neural networks is mainly limited to activation functions such as ReLU and LeakyReLU due to their positive homogeneity, we wanted to understand whether this was a limitation or beneficial from an empirical perspective. Hence we analyzed the influence of the activation function in the above network's performance in denoising. For this study, we changed all LeakyReLU activations in the network to be either the Exponential Linear Unit (ELU), the Gaussian Error Linear Unit (GELU), or the Tanh activation. We trained these networks using the adversarial loss as previously described and then used them for denoising on the same images from the previous experiment. As compared to the network with LeakyReLU activations, we see significant performance degradations when switching to these non-positively homogenous activations. This suggests that potentially positive homogeneity of the activations is a useful property in terms of performance of the neural network-based regularizer.
**LeakyReLU MSE, PSNR:** 0.005, 23.14
**ELU MSE, PSNR:** 0.042, 14.16
**GELU MSE, PSNR:** 0.031, 15.29
**Tanh MSE, PSNR:** 0.049, 13.14
We sincerely thank the reviewers again for their engagement and for raising these concerns. We believe that these additional experiments are valuable in providing support for our theory and will significantly improve the paper quality. We will be sure to include these in the updated version of manuscript.
Please let us know if you have any additional questions or concerns.