Summary
This paper proposes to scale the skip connection in diffusion model Unets by an exponential factor. The authors should that the feature norms of vanilla Unets oscillate across batches, and that their proposed method results in much smaller feature output oscillations. They conjecture that this method stabilizes training, and prove bounds on the oscillation which their method decreases. The proposed method shows faster convergence and shows improvements across multiple models, datasets, model sizes.
Strengths
1. The authors method follows prior work on scaling block output and/or the skip connection, where such scaling methods have been shown to increase stability/robustness and result in Lipschitz continuous models.
1. The learnt scales in LS method seems to somewhat mirror the author's theoretical exponential CS method, providing further validation.
1. The authors method results in faster convergence and shows improvements across multiple models, datasets, model sizes, and compares favourably to some other scaling methods.
1. The author's proposed method can be very easily adapted to existing models.
Weaknesses
1. The authors show that feature norms oscillate across samples, and they say this implies parameters must also be oscillating (line 47, line 172). While the authors demonstrate that feature norms do oscillate (and I am willing to agree the same may be true of gradient norms), this may not directly cause the parameters/updates to oscillate.
1. The authors works (and proposed exponential CS solution) is remarkably similar to [1], where a scaling of $b^l$ is proposed for the block output (compared to this paper's scaling of the skip connection by a similar value). An experimental comparison with this method should be added to Table 3, or perhaps the similarities/differences discussed.
1. Discussion of prior related works should be expanded. For example, [2] proposes scaling of the skip connection and/or the block output, and has similar exponential bounds.
1. The constants in theorem 3.1 are ignored (assumed to be 1) in line 270 to derive the values for the scaling parameter. If this constant is for example 10 or 0.1, that will dramatically change the proposed values for CS. Further investigation into this constant is required.
[1] Hanin, B., & Rolnick, D. (2018). How to Start Training: The Effect of Initialization and Architecture. Advances in Neural Information Processing Systems, 31, 571–581.
[2] Balduzzi, D., Frean, M., Leary, L., Lewis, J.P., Ma, K.W. & McWilliams, B.. (2017). The Shattered Gradients Problem: If resnets are the answer, then what is the question?. <i>Proceedings of the 34th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 70:342-350 Available from https://proceedings.mlr.press/v70/balduzzi17b.html.
Questions
1. The authors derive bounds for oscillations (eq 4), and they show that their methods decreases this bound, but how tight is this bound? Comparing the minimum and maximum norms predicted by this formula, with that in Figure 1., is important to show if this bound is indeed useful, or perhaps their proposed method has some other side effect that results in the performance improvements.
1. Direct evidence of parameter oscillation (mentioned in line 47, 172 can be shown), perhaps by comparing a per-parameter Adam style "m" and "v" across the baseline method and the author's method. If oscillation of parameters is indeed smaller, one would perhaps expect the norm of "m/v" to be larger in the author's case. Is this indeed the case?
1. Given that the exponential change in scaling with increasing total number of layers, does the performance of the method suffer compared to $\frac{1}{\sqrt{2}}-CS$ as number of layers increases to more depths (as large depth will cause the skip connection to essentially be scaled to zero)?
Minor presentation issues (authors are not expected to respond to these, as they are minor issues that can be fixed later):
1. Line 85 "serir" should perhaps be "series"?
1. In table 1, comparisons to author's method "x+LS/CS" should perhaps be placed directly below "x", to make it easier to compare scores.
1. The values for CS for table 1 are mentioned in the supplementary, but perhaps they should be mentioned at line 272 or in 315.
1. Quotes around 'CS.py' in supplementary above Figure 2 are inverted.
1. $\frac{1}{\sqrt{2}}-CS$ reads as "1 by square root 2 minus CS", and for a moment I was trying to think what $CS$ are the authors subtracting from $\frac{1}{\sqrt{2}}$. Perhaps a better notation for this may be something along the lines of $(CS)\frac{1}{\sqrt{2}}$?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.