Scattering Vision Transformer: Spectral Mixing Matters

Summary

Assistant The article is about the Scattering Vision Transformer, or SVT, which is a new adaptation of transformers for computer vision tasks. Its unique feature is the use of a spectrally scattering network, which captures fine-grained information about an image and addresses the issue of information loss caused by down-sampling operations. The SVT achieves state-of-the-art performance on the ImageNet dataset and outperforms other transformers in other vision tasks.

Strengths

1. The use of DT-CWT, which is both shift-invariant and free from aliasing, is a robust and reliable approach. 2. The combination of tensor multiplication in low-frequency components and Einstein multiplication in high-frequency components is a compelling and efficient technique. 3. The experimental results demonstrate great promise for the proposed method.

Weaknesses

1. To enhance comprehension, a brief illustration of DT-CWT through either descriptive language or illustrative figures should be provided in the method or background section. 2. Not all layers of vision transformers are scatter layers. An explanation and study should be provided. 3. The paper would benefit from the inclusion of mathematical theorem proofs. I found it difficult to understand the benefits of DT-CWT, particularly the decoupling of low and high frequencies. The paper's methods read more like a technical report than a compelling story of motivation, possibly due to the omission of some logistic or preliminary details. 4. To further evaluate the performance, provide comparisions of speed and over-fitting tests on ImageNet V2. 5. There are some typos in the paper. For example, on line 260, "figure ??" is not clear. Additionally, the official reference uses brackets [xx], whereas parentheses are used in the text. Please ensure consistency and accuracy in all references.

Questions

1. Could you provide an explanation for why scatter layer is used in shallower layers but traditional attention layer is used in deeper layers? 2. Is there any evidence, such as references, mathematical induction, or related research studies, to support the idea of using tensor multiplication for low pass and Einstein multiplication for high pass?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

There appear to be no potential negative social impacts.

Summary

This paper proposed to use DTCWT in the transformer model in early stage, with the motivation of saving computation cost without information loss. The proposed scattering module has a low-pass and a high-pass branch. Tensor multiplication & Einstein multiplication are applied to the LF & HF branch respectively. The proposed module replaces the first 2 stages in hierarchical ViT models. The proposed SVT model achieved superior performance on image classification on the ImageNet1K dataset, and the downstream transfer learning results.

Strengths

Replacing low level transformer modules with the proposed DTCWT based module achieves better accuracy and latency trade-off. Applying DTCWT to transformer models may not be explored yet by prior work.

Weaknesses

DTCWT has been used in the previous CNN based deep learning method, e.g. Uses of Complex Wavelets in Deep Convolutional Neural Networks. Existing work already shows that replacing early transformer stage with convolutional layers could give better accuracy and latency trade-off.

Questions

How do we compare SVT with combining ViT with CNN, i.e. an architecture with CNN in early stages while transformer in late stages?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The proposed module is a highly hand engineered component. Adding manually designed components to the deep learning models may contradict with the motivation of removing inductive bias to the model and building generic learning based architectures.

Summary

This paper proposes to use Dual-time Complex Wavelet Transforms to decompose images into high and low frequency components in vision transformers. With this technique, it claims to address the problem of attention complexity without the loss of information as in Fourier or DWT based Transformers. This claim is based on the invertible property of Dual-time Complex Wavelet Transforms. This paper claims to achieve state-of-art image classification performance on the ImageNet dataset with a significant reduction in a number of parameters and FLOPS, and comparable results in instance segmentation. Qualitative and quantitative results are provided.

Strengths

1. The most attractive of this paper is the usage of Dual-time Complex Wavelet Transforms. The reviewer appreciates that the authors noticed the information loss in previous Fourier or DWT based Transformers, and proposed to adopt Dual-time Complex Wavelet Transforms to solve such a problem. It is invertible (though also redundant) and hence would not lead to a loss. 2. The paper provides a lot of technical details, and also the codes to reproduce the results. 3. Extensive experiments are conducted in the paper.

Weaknesses

1. Clarity of Technical Contribution The paper's presentation raises some concerns about clarity and comprehension. The key technical contribution of the work is obscured by convoluted explanations and distracting statements. For example, L81-L93, at the beginning of the method section, this paper detailedly talks about something like "In Vanilla SVT, Given an image, we split it into patches of size 16*16. We use a linear projection layer to get the embedding feature for each patch ......" These sentences seem to depict unique contributions of the proposed method but, upon closer inspection, merely explain standard operations in the Vision Transformer (ViT). Similarly, lines 102-109 are bewildering, appearing repetitive and unclear about the authors' intended message. Moreover, the paper repeatedly emphasizes "tensor multiplication" and "Einstein multiplication" across various sections. These appear to be mere implementation details, not substantial technical contributions. The manuscript's current style is closer a technical report rather than an academic paper. The authors are strongly encouraged to (a) explicitly state the key contribution of this work in the rebuttal and method section, and (b) condense lines 81-126 into a succinct, high-level introduction of the method, ideally within 10 lines. 2. Claims need support Some of the claims in the paper, while potentially plausible, lack sufficient support/proof. For example, L52, "the ability to separate low-frequency and high-frequency components of an image is also important". In L100, "SVT has improved robustness compared to most other transformers, which will also be established in the performance studies." These could be true, but the authors did not prove the robustness is improved. 3. Minors There are some minor typos/problems to be fixed. For example, L130, "Where" to "where". In page 7, some table captions are bold and some are not.

Questions

Overall the reviewer thinks using DTCWT in vision transformers is worth exploring, and this is why the reviewer still vote for borderline accept at this stage. If the authors cannot provide appropriate explanation/discussion about their contribution in the rebuttal period, the reviewer would downvote the score. Besides, the reviewer feel some details provided by the main paper are not interesting/exciting, e.g., Table 1. For future work, the reviewer would suggest the authors to explore more insightful things. For instance, DTCWT enjoys invertibility but has redundancy at the same time. How would the network deal with such a redundancy? Moreover, providing more quantitive or qualitative comparisons to Fourier or DWT based Transformers to analyse how the invertibility of DTCWT helps vision transformers understand visual contents, instead of just reporting the final numbers for the tasks like classification. ____________________________________________________ The author response solves the main concerns. Under the assumption that the authors would revise the manuscript as they promised, the reviewer agreed to raise the score to weak accept.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

N/A

Summary

The paper proposes SVT, a novel vision transformer model that addresses the challenges of attention complexity and capturing fine-grained information in images. SVT utilizes a spectral scattering network and the Dual-time Complex Wavelet Transforms (DTCWT) to decompose image features into low-frequency and high-frequency components. The paper also introduces an efficient feature mixing technique using Einstein multiplication in the high-frequency components and tensor multiplication in the low-frequency components. Experimental results demonstrate that SVT achieves state-of-the-art performance on the ImageNet dataset with reduced parameters and computational complexity.

Strengths

1.The proposed SVT model presents an innovative approach to addressing attention complexity and capturing fine-grained information in images. The use of spectral scattering and DTCWT decomposition enables efficient representation and separation of frequency components. 2.The feature mixing technique using Einstein multiplication is a novel contribution that efficiently combines token and channel features, leading to improved performance. 3.The experimental results on the ImageNet dataset and other vision tasks demonstrate the superiority of SVT compared to existing vision transformers, such as LiTv2 and iFormer. The significant reduction in parameters and computational complexity further enhances the practicality and scalability of SVT.

Weaknesses

1. The English writing in this paper needs to be carefully reviewed as there are several grammar errors. For example, line 79, lesser should be less, line 83, "Given" should be in lower case "given". And in Figure 4's illustration, some parts are missing. There are many more errors in your paper writing. 2. I find the approach in this paper to be interesting but overly tricky. As we have seen with LLMs, many highly technical improvements become less significant when supported by the large parameter size and data volume of such models. Moreover, in the experimental section, the performance improvement brought by SVT does not appear to be significant. From my perspective, simple and elegant techniques often yield more meaningful improvements compared to complex transformations.

Questions

In the experiments, why not compare your method with more popular CNN-based methods like Yolo and Faster-RCNN?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Author has presented some limitations of the proposed methods and future plan.

Summary

The paper introduces the Scattering Vision Transformer (SVT), which utilizes a spectrally scattering network to capture fine-grained information in images and addresses the invertibility issue. SVT incorporates a novel spectral mixing technique using Einstein multiplication for efficient channel and token mixing. The approach achieves state-of-the-art performance on the ImageNet dataset, significantly reducing the number of parameters and FLOPS. It also demonstrates competitive results in other vision tasks, including transfer learning on standard datasets.

Strengths

## Novelty The use of scattering networks and Fourier-like frequency processing is novel and innovative. The paper addresses a significant problem related to the texture processing ability of vision transformers, showcasing desirable novelty and originality. ## Quality & Clarity The paper is well-organized, providing clear preliminaries, assumptions, definitions, and solutions. The experiments are detailed and concrete. ## Significance SVT effectively separates low-frequency and high-frequency image components while reducing computational complexity through the Einstein multiplication-based mixing technique. It achieves state-of-the-art performance on image classification and instance segmentation tasks and shows comparable results in object detection tasks.

Weaknesses

The computational costs and complexity limit the number of directional orientations used in SVT. Currently, SVT employs six orientations, but increasing the number of orientations would capture more semantic information at the expense of higher computational complexity. Optimization possibilities should be explored. Additionally, SVT's performance in domains such as speech and NLP remains unexplored.

Questions

None

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

None

Thank you for the response

The response has addressed my concerns, so I am updating my rating to "accept."