Summary
This paper addresses the issue of certified accuracy with deterministic approaches by utilizing the Lipschitz property of neural networks. The authors propose a novel layer called LiResNet, which enables easy computation of the Lipschitz constant. Additionally, they introduce a new loss function called Efficient Margin Maximization (EMMA), that stabilizes robust training by simultaneously penalizing worst-case adversarial examples from all classes. Finally, the authors conduct experiments on CIFAR-10/100, Tiny-ImageNet, and ImageNet datasets, demonstrating that their approach achieves state-of-the-art results in certified accuracy under $\ell_2$ perturbations and $\epsilon = 36/255$.
Weaknesses
While the experimental results are good and interesting. The paper suffers from overclaiming important contributions and not discussing important related works.
1. **Overclaim on Lipschitz Residual layer**: The claim that the paper is the first to propose a Lipschitz Residual layer is incorrect. The CPL layer [1], and more recently the SLL layer [2], are two previous works that have introduced a 1-Lischitz _Residual_ layer with ReLU activation. The authors mention these two papers in the related work without discussing their contribution. The authors should discuss these works.
2. **On the efficiency and tightness of the approach**: In the abstract, the authors have written: "We show that fast ways of bounding the Lipschitz constant for conventional ResNets are loose, and show how to address this by designing a new residual block"
- **On the efficiency of the approach**: I don't see how the approach is more efficient than previous works, since the authors just use the power iteration (verified in the code provided in the supplementary material as this is not mentioned in the paper or the appendix). The power iteration has been used in three previous works [3, 4, 1]. We can also note that the authors use a power iteration with 10 iterations (default value in the code) while [3,4,1] showed that using only 1 iteration was sufficient and more efficient. Therefore, I don't see how the author’s approach is "more efficient" than previous work.
- **On the tightness of the bound**: It is true that the value of the Lipschitz constant of LiResNet is tighter than the one from a Residual Layer with nonlinearity, in fact, the power iteration computes the _exact_ Lipschitz of LiResNet, since LiResNet is a _linear_ layer -- the PM computes the Lipschitz of the map $x \mapsto (I + W) x$. Computing the Lipschitz of a Residual layer with a nonlinearity leads to a looser bound, because the nonlinearity allows for a lower Lipschitz. The authors should clarify this.
3. **On the new EMMA loss**. The Efficient Margin MAximization (EMMA) loss introduced in the paper is very similar to the one provided by [3]. The authors discuss the difference between their loss and the one from [3] in Appendix A, arguing that the main difference is the use of the _Lipschitz constant for each margin_ while [3] uses the _global Lipschitz constant_ (I believe this paragraph deserves to be in the main paper). This is true, however, the use of the Lipschitz constant for each margin has been proposed twice before [5, 6] (the last layer normalization reduces to the same Lipschitz), therefore, the EMMA loss is a simple combination of two known approaches.
4. **On the experiments**: Table 1 presents results only for the perturbation level of $\epsilon = 36/255$. The authors should present results for a multitude of values to show the overall robustness of their approach (or make a graph with certified accuracy vs \epsilon). A large body of work [1, 2, 6, 7, 8, 9, 10] has presented the certified robustness of their model for at least 3 or 4 perturbation thresholds. I would like to see the same comparison. In Table 2, the authors talk about "VRA performance (%)". What does VRA performance (%) mean? Again the authors should provide VRA with respect to a specific perturbation and not assert that the overall robustness of a model can be computed with only one single perturbation threshold.
**Conclusion**: The paper feels like a patchwork of several existing ideas and looks more like engineering work than research work. The paper has combined the work of [3] and [5,6] for the loss, used the same algorithm (power iteration) as [3,4,1] to compute the Lipschitz, and proposed a new linear layer that seems to have good properties. They used several tricks (epsilon scheduler presented in Appendix B1, normalization trick presented in Appendix B2) without ablation study. With all this, they showed that it is possible to improve the certified accuracy for $\epsilon = 36/255$ and managed to achieve very good certified accuracy on ImageNet, which has not been done before. I find the results of this paper interesting because I think the overall problem is interesting and important, but I think the paper (in its current form) has little impact on the topic of certified accuracy with deterministic approaches. To improve the research value of the paper, the authors should provide a comprehensive ablation study and explain how and why these different techniques, when combined together, significantly improve certified accuracy.
[1] Meunier et al., A Dynamical System Perspective for Lipschitz Neural Networks ICML 2022
[2] Araujo et al., A Unified Algebraic Perspective on Lipschitz Neural Networks, ICLR 2023
[3] Tsuzuku et al., Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks, NeurIPS 2018
[4] Farnia et al., Generalizable Adversarial Training via Spectral Normalization, ICLR 2019
[5] Leino et al., Globally-Robust Neural Networks, ICML 2021
[6] Singla et al. Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100, ICLR 2022
[7] Trockman et al., Orthogonalizing Convolutional Layers with the Cayley Transform, ICLR 2021
[8] Singla et al, Skew Orthogonal Convolutions, ICML 2021
[9] Prach et al., Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networks, ECCV 2022
[10] Huang et al., Training certifiably robust neural networks with efficient local Lipschitz bounds, NeurIPS 2021
Questions
The authors have combined many different techniques and tricks to achieve their certified robustness. An ablation study would be interesting to identify those that increase the certified robustness.
- What is the performance of the author's architecture with an SLL layer that is 1 Lipschitz?
- What is the performance of the author's architecture with the loss from [3]? How does the EMMA loss improve the VRA?
The authors added some tricks for better training: the epsilon scheduler presented in Appendix B1 and the normalization trick presented in Appendix B2.
- How does the epsilon scheduler affect the final certified accuracy?
- How does the normalization trick affect the final certified accuracy?
- What is the certified accuracy for other perturbation thresholds? (e.g. $72/255$, $108/255$, $1$ to allow comparison with other work)
[3] Tsuzuku et al., Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks, NeurIPS 2018
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.