Truly Scale-Equivariant Deep Nets with Fourier Layers

Summary

The authors of this study introduce architectural modifications to the conventional CNN framework by incorporating Fourier layers. Their aim is to achieve scale-equivariance, addressing the anti-aliasing problem that occurs when images are downscaled in the continuous domain. To tackle this issue, the authors propose a discrete domain downsampling formulation and propose corresponding changes to the network architecture. Through experiments on a few datasets, the authors demonstrate that their method achieves zero scale-equivariance error.

Strengths

- The paper is well-written, precise, and easy to follow. In particular, their motivation is stated clearly, they introduce the preliminaries succinctly, and the approach stated is coherent and structured. - To the best of my knowledge the proposed method is novel and the claims that the module modifications achieve scale-equivariance is correct. - I am unaware of any method but this that solves the chosen problem completely, i.e achieves zero scale-equivariance error when downsampling is done via anti-aliasing and subsampling ('ideal' subsampling). The results are also competent when they do 'non-ideal' subsampling when compared to standard baselines. - The motivation that images of higher-scale should achieve better classification accuracy than the ones of lower scales is reasonable and the corresponding proposed constraint is sound. Overall I think the ideas presented in this paper maybe of interest to the broad ML community.

Weaknesses

- Due to the complexity involved in the several DFT/Inverse-DFT related operations, the baselines they compare to which involve kernel resizing are certainly not as costly as the proposed method. I suggest the authors to discuss how their method compares to the baselines in terms of computational/time complexity. - For the same aforementioned reasons, I suspect that the method might not scale to high dimensional datasets like ImageNet or standard architectures like ResNet50. This I think would be a fundamental limitation, therefore I encourage the authors to discuss the challenges in scaling their method to the mentioned settings. I would be interested to know which modules will be particularly harder to scale. - Some minor text changes: - Line 119, Claim 1: 'even lower in frequencies' needs to be 'even in lower frequencies.' - Line 138, Claim 2: 'equivaraint' needs to be 'equivariant.' - Figure 1, Figtitle: “On the other hand, downsampling the high-res feature is guaranteed to achieve the same low-resolution feature.” -> Unclear formulation. It is not clear how the downsampled version of the features was achieved. The text suggests that it has merely been downsampled from the high-resolution version, but then it is not clear which role the new proposed architecture plays. Please rephrase for clarity. - Line 32: “Specifically, these works are derived using a continuous domain down-scaling operation, i.e., there is no need to consider anti-aliasing. However, when performing a down-scaling, the Nyquist theorem [21, 27] tells us that an anti-alias filter is necessary to avoid high-frequency content to alias into lower frequencies. (...) To address this gap, from prior work, we consider the down-scaling operation directly in the discrete domain taking into account of the anti-aliasing.” Several grammar errors and the paragraph is unclear to me. In the first sentence, the authors write that previous works “do not need to consider anti-aliasing”. But then, they aim to bridge the gap and take anti-aliasing into account. Please fix the formulation as it is currently very unclear whether previous works had an issue with anti-aliasing or not. - Line 103: The notation for the definition of g is unclear. It is unclear what is meant by that x is \in {R^1, R^2 … R^N}, which makes it hard to understand the following definitions. The notation of {..} suggests that x can be from R^1, but then how is the Fourier transform of one number even defined? The Fourier transform is needed for D_R(x). I am confused by this definition, please rewrite for clarity. - It would be helpful if the authors would state that small x refers to the spatial domain and big X always refers to the Fourier domain. While this is the common practice, it would still be nice to state it explicitly. - I don’t understand Fig. 2b. Are the yellow regions ones and the gray ones zero? - Line 142: “Element-wise non-linearities, e.g., ReLU, in the spatial domain are generally not scale-equivariant under the ideal downsampling operation DR.” The reasoning of this claim is not obvious to me, please explain / give some intuition.

Questions

I encourage the authors to respond to the points I raised on weaknesses and fix minor text errors. I am willing to raise my score if the authors adequately respond to my concerns.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors have not included a Broader Impact section and did not discuss the potential negative societal impact of their work. They also do not have a discussion section where some limitations of their work could have been discussed.

Summary

This paper introduces Truly Scale-Equivariant Deep Nets with Fourier Layers, which directly formulate down-scaling in the discrete domain to address the anti-aliasing problem. The proposed methods are validated on the MNIST-scale and STL-10 datasets, demonstrating good classification performance while preserving zero equivariance error.

Strengths

1. This paper addresses how to learn scale-equivariant representations in deep neural networks, overcoming the limitation of previous methods that did not account for anti-aliasing. 2. The paper proposes a method that can consider the down-scaling operation directly in the discrete domain with anti-aliasing and achieves good performance.

Weaknesses

1. This paper does not provide sufficient experiments to demonstrate the generality of the proposed method. It only tests on two small datasets and a single high-level vision task (image classification), which limits the impact of the proposed method. This paper should also evaluate the method on more larger-scale datasets, more high-level vision tasks (e.g., image segmentation), and more CNN architectures. 2. Why does the baseline use zero-padding to the original resolution as the input, instead of resizing to the original resolution? Does this affect baseline's representation learning and performance? 3. The paper lacks an analysis of the computational cost of the proposed method and other compared baseline during training and inference.

Questions

Please refer to the weakness part.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Not applicable.

Summary

The submitted paper explores the problem of scale-equivariant neural networks. To get the desired property by design, the authors propose new building blocks: a spatially local Fourier layer, a novel scale-equivariant pooling layer, and scale-equivariant way of applying activation functions. All the introduced modifications rely heavily on the Fourier transform of the input feature map. Additionally, a new loss function is proposed which aims to promote the natural consistency of predictions made at different scales. The effect of the new approach is demonstrated on the problem of image classification on two datasets: MNIST-scale and STL-10-scale.

Strengths

I find the proposed modifications well explained and motivated. To the best of my knowledge, the presented approach is original enough, and it may be interesting for the broad community. Quality of the results seems very satisfactory and I hope they can serve as a reliable baseline for further research on this topic.

Weaknesses

1. While the proposed results seem strong enough for the task of object recognition, the authors have not reported the computational or memory overhead introduced by their approach. Without this knowledge, it is hard to understand if their method can be directly scaled to more high-resolution data without any significant changes. 1. I admit that most prior papers also mostly report their performance for image classification, it is quite clear that the property of scale equivariance may be even more desirable for the dense prediction tasks, e.g. semantic segmentation. Are there any barriers to applying the same method to the common segmentation architectures? 1. Regarding the low performance of baseline models on STL-10-scale data (line 274), have the authors tried other padding schemes than zero-padding, e.g. so-called reflect/replicate/circular strategies (see the [link](https://pytorch.org/docs/stable/generated/torch.nn.functional.pad.html))? I find it suspicious that, e.g. the reported DISCO performance of 91.93% accuracy on STL-10 [1] drops to 47.68% on STL-10-scale (Tab. 4) in the provided evaluation results. I think, more discussion is needed on that matter. [1] Sosnovik et al. DISCO: accurate Discrete Scale Convolutions. In BMVC, 2021.

Questions

Please, address the aforementioned weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No additional inquiries.

Summary

This paper proposes a formulation of scale equivariant networks, which takes anti-aliasing into account. This is in contrast to previous works, in which the issue of aliasing is often disregarded. The proposed method achieves competitive classification accuracy on MNIST-scale and STL-10, while maintaining zero equivariance error.

Strengths

This work addresses a problem that has been thus far disregarded when talking about scale equivariance: the aliasing issue. The proposed methods and solutions are sound and are empirically evaluated with very good results.

Weaknesses

* My main concern is that, to the best of my understanding, the proposed framework does not consider inter-scale relationships. Instead, it considers multiple scales in parallel which are processed independently of one another. This brings into question whether this work should be defined as a work in group equivariance or rather a work on models aiming to be resolution-agnostic, i.e., able to work regardless of the resolution of the input, e.g., FlexConv [1], S4ND[2], \inf-Diffusion [3]. * This method relies heavily on using the Fourier and inverse Fourier transform for each of its layers. In addition, it is well-known that performing convolutions on the spatial domain is faster and requires less memory than performing them in the Fourier domain when convolutional kernels are small O(Nk) vs O(NlogN). However, the proposed method actually relies on small convolutional kernels. These two factors call into question the computational and memory complexity of the proposed method. I believe that this is the main practical limitation of the method, and I would like the authors to discuss this in the paper –in case it is indeed an issue–. * Connecting to the previous point, I believe that having a section in which the limitations of the work are clearly stated and discussed is vital.

Questions

* I find the definition of the convolutional kernels (Eq. 14) too convoluted. I would simply say that you define a kernel of size k in the spatial domain and send it to the Fourier domain next. * There are several works that tackle the issue of downsampling breaking equivariance (even in a general group setting) as well as tackling aliasing in many other settings, e.g.,[1,2,3,4,5,6,7] among many more. I believe that discussing these works would be beneficial to scope the paper better. * In line 213, you mention that you do the analysis of Sosnovik et al on trained models. Although I understand your reasoning, I actually believe that treating non-trained networks directly, i.e., random nets, makes more sense in order to evaluate the real capability of the networks to remove aliasing and preserve equivariance. Have you done this analysis as well? What were the results? Did you still get zero equivariance error? * I believe line 165 has an error. * The paper has some small typos. Please correct them.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

As mentioned previously, the authors have not listed any limitations of their work in the paper. ### Conclusion In conclusion, I think this is an interesting paper with important contributions to the field. However, I am not sure if this paper should be framed as a scale-equivariant paper or rather as a resolution-agnostic neural architecture. I believe –please do tell me if you don't agree– that the latter option is more accurate. Although it requires some rewriting, I believe it would be possible to still do this during the rebuttal period. #### References [1] Romero, D. W., Bruintjes, R. J., Tomczak, J. M., Bekkers, E. J., Hoogendoorn, M., & van Gemert, J. C. (2021). Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprint arXiv:2110.08059. [2] Nguyen, Eric, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. "S4nd: Modeling images and videos as multidimensional signals with state spaces." Advances in neural information processing systems 35 (2022): 2846-2861. [3] Bond-Taylor, S., & Willcocks, C. G. (2023). $\infty $-Diff: Infinite Resolution Diffusion with Subsampled Mollified States. arXiv preprint arXiv:2303.18242. [4] Karras, Tero, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. "Alias-free generative adversarial networks." Advances in Neural Information Processing Systems 34 (2021): 852-863. [5] Zhang, R., 2019, May. Making convolutional networks shift-invariant again. In International conference on machine learning (pp. 7324-7334). PMLR. [6] Xu, J., Kim, H., Rainforth, T. and Teh, Y., 2021. Group equivariant subsampling. Advances in Neural Information Processing Systems, 34, pp.5934-5946. [7] Rojas-Gomez, Renan A., Teck-Yian Lim, Alex Schwing, Minh Do, and Raymond A. Yeh. "Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks." Advances in Neural Information Processing Systems 35 (2022): 35755-35768.

Response to the authors' rebuttal

Dear authors, thank you for responding to my concerns. Most of my questions have been answered. Do you think this approach could be scaled to larger DNNs on larger datasets, such as e.g. a ResNet50 on ImageNet? I appreciate the complexity analysis and the statement in the general response: "From our understanding, modern GPUs are heavily optimized for regular convolution operation which can be done in place. On the other hand, the FFT algorithm does not get the full advantage of GPUs due to reasons like unique memory access patterns, moderate arithmetic intensities, etc." Given we had a narrow-purpose accelerator, such as a parallel processing unit which would be specialized to FFT operations instead of regular convolutions, could this method be straight-forwardly applied to larger DNNs or do you see any other fundamental issues in scaling it? Best, Reviewer JmYY

Response to Rebuttal

Dear authors, Thank you for responding to my questions. Your answers have addressed my concerns. As far as these answers are reflected in the camera-ready version of the paper, I am happy to support acceptance. Just as a final observation, regarding this response: > The interactions between the scales happen in two ways: (a) as stated in Claim 1, the “output’s frequency terms can only have dependencies on the terms in that are even lower in frequencies...” I indeed agree with the authors. However, I would like to note that in papers such as SESN, B-Spline G-CNNs, Wavelet Networks [2], Deep Scale Spaces (just to name a few), scale group convolutions can depend on all scales considered at every layer, i.e., not only on lower frequencies. Perhaps your method is actually better than making the group conv depend on all scales, as it avoids padding along the scale axis (which is problematic as mentioned by these methods), and simultaneously addresses aliasing. Perhaps a small analysis on this could also add to the paper. Best regards, Reviewer qJeg [1] Romero, D.W., Bekkers, E.J., Tomczak, J.M. and Hoogendoorn, M., 2020. Wavelet networks: Scale equivariant learning from raw waveforms. arXiv preprint arXiv:2006.05259.

Response to the authors' rebuttal

Dear authors, Thank you for responding to my questions. My concerns have been well addressed. I would like to increase my rating to Weak Accept. The reason why I don't give a higher score is that currently, this method still has computation and memory bottlenecks. Best regards, Reviewer 5utz