Infinite-Dimensional Feature Interaction

Summary

This work proposes a novel approach for enhancing neural network performance by scaling feature interaction spaces to infinite dimensions using kernel methods. Recent advancements have introduced feature interaction spaces, but these are often limited to finite dimensions, primarily through element-wise multiplications. To overcome these limitations, the authors propose InfiNet, a model architecture leveraging the Radial Basis Function (RBF) kernel to enable infinite-dimensional feature interactions. Finally, the authors provide several empirical results on standard vision tasks.

Strengths

This work provides an interesting generalization of feature-feature interactions via kernels. For the best of my knowledge, this is a novel idea that appears to perform well in practice. However, I am not overly familiar with the current state of the field of deep learning for computer vision. It further provides several larger-scale experiments and interesting ablations.

Weaknesses

* there is no theoretical justification that increasing the dimension of the feature-feature interaction space will lead to better generalization. The paper does a good job analysing this question with ablations. However, this remains an open theoretical question. * I understand that the motivation for this work comes from applications in computer vision. However, since a major focus in this paper is on comparing the proposed approach to self attention, it would be interesting to not only test this method on images, but also on language. * the method is reported to have lower FLOPs on average than competing methods. Why is that? Is that a major drawback of this method? * performance improvement on ImageNet is only marginally. In many cases the proposed method even performs worse than competing methods. * paragraph starting in line 148: this is on over-claim and has to be removed or rigorously proved. It is not clear how a higher order of $k$ implies better generalization or training. Unless shown in this paper or referenced from another paper, this has to be removed. Minor: * line 28: more context for formulating self attention that way has to be provided. It is explained in more detail only at the end of section 3. * caption of figure 2: there is '?'. Moreover, a description of the presented images should be included. What is shown in Figure 2 on the right hand side? This is only explained in the main text,not the caption. This needs to be changed. * figure 2, first image on the left: hard to read -- text overlaps with drawing.

Questions

* what is meant in line 47 + 48? the current formulation is very cryptic. What exactly is linear in $k$? * figure 2: why does the addition and multiplication interactions reach the same accuracy on cifar10? Isn't that basically MLP vs self-attention? I would presume self attention to perform better.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

* The paper provides empirical results only on vision tasks. However, a major selling point of this paper is generalizing approaches like self attention in terms of feature-feature interactions. Therefore, comparisons with transformers on language tasks should be performed. * No theoretical analysis is provided proving that the proposed method leads to better generalization. * The method appears to have on average lower FLOPs than competing methods, while at the same time only marginally outperforming (or even performing worse than) competing methods on imageNet.

Summary

This paper studies placing a kernel function inside of a neural network architecture to facilitate interaction of features/dimensional expansion. They consider deep convolutional networks with parallel pathway features $x$ and $x'$ and a kernel function computed with both pathways' features as inputs $k(x,x')$. Standard kernel mathematics is used to explain feature expansion. The main novel results are empirical performance of these "InfiNet" architectures, which are shown to perform well in a number of computer vision tests.

Strengths

The idea of unifying different orders of interaction embodied in various neural network architectures, including Transformers is appealing and probably important. The accuracy of the InfiNet experiments is impressive, with a moderate reduction in FLOPs. The paper is easy to read and well-organized, although suggestions are given for how it could be improved.

Weaknesses

My main concerns with the paper are a lack of context for the approach as well as missing important explanations. I also think a good amount of the math that's included could be considered "filler" material that could go into the appendix, since it doesn't represent new results. (I am referring to sections 4.1 and 4.2, most of which can be found in most textbooks which cover kernel methods.) * Notation which is commonly used in the paper $\oplus$, $\otimes$, * is not explained. You should _explicitly_ define it somewhere, at least in the appendix (and refer people there). In particular, people may be confused by * for elementwise/Hadamard multiplication, since in convnet literature this is often the convolution operator. You call this the "Star Operation" in line 124, but I think it is just elementwise multiplication. * The authors seem to have missed the vast literature on the connections between random features, neural networks at init, and kernel methods. (CKNs are mentioned but without any discussion of the topics I mention here.) In particular, one way that you could approximate the InfiNet architecture would be to take the two feature streams and pass them each into the same wide, random network/layer and compute the dot product of features at the next level. That would only approximate the kernel function in the InfiNet architecture, and is likely less efficient, but it provides a way to perform dimensionality expansion with a more traditional layer. The authors should discuss these connections. * Different order of interactions have been studied in random feature and kernel settings already. In random features, interaction order is connected to the sparsity of weights, see e.g. https://arxiv.org/abs/2103.03191 and https://arxiv.org/abs/1909.02603. In kernels, this were referred to as additive kernels https://arxiv.org/abs/1602.00287, also studied in multiple kernel learning https://arxiv.org/abs/0809.1493 (these are just some examples among a larger literature). * The authors do not seem to want to release their code. They have said "Yes" on Question 5, stating that the code and data are open, but there is no link or indication in the text that the code is available or will be when the paper is published. That seems deceptive.

Questions

* There is a tension between dimensional expansion, which leads to expressivity in networks, and generalization, which is typically better in low-dimensional settings. Can you discuss this? * When queries and keys in a transformer are computed using a multilayer network with nonlinearities (rather than a single linear layer, as you've considered), aren't the effective order of interactions higher? * You claim that the kernel map applied to inputs with $C$ channels takes constant $O(1)$ time (section 4 intro). Wouldn't evaluating the kernel still take $O(C)$ i.e. linear time? * Can you please include the matrix/tensor shapes and layer sizes explicitly in section 5.1? They could be put into the appendix. It is unclear how many kernel evaluations are performed and on what shape input. Minor points: * Sentence lines 45-47 is confusing and should be reworded. Also, the combinatorial expression with the limit is unexplained, not obvious, and doesn't seem to contribute anything here. I suggest removing it. * Line 61, the expression for span of a certain space is unclear. The main point seems to be that this is an infinite-dimensional function space. Does using this math really add anything? * Line 61: "as low-overhead as exponential operations" is unclear. Do you mean "evaluating an exponential function"? * Line 91: "Kernel Method" -> "Kernel Methods" typo * Line 106: "isotropic" here is unclear to me, suggest removing * Line 110: "medium" for the intermediate layer connotes different size, suggest changing to "intermediate" or "middle" * Line 130: Without saying it, are you assuming that the image inputs span the pixel space vector? * Line 149: "two element-wise multiplication" typo -> "multiplications" * Notation $W_a \mathbf{x}$ is confusing: In equation (1) this seems to output a scalar. Is that the same in Eqn (6)? What are the shapes of the W matrices? * What is a "feature branch"? Unclear throughout. * You say the input is passed through "STEM" and refer people to the ConvNeXT paper https://arxiv.org/pdf/2201.03545. There is more than one "stem" in that paper. Can you be explicit about what you did?

Rating

7

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

I would strongly prefer that the limitations be included in the main text during the discussion. With movement of some of the standard math, there would be space. The authors only consider the squared exponential kernel with bandwidth parameter equal to 1. Other kernels might work better. In particular, the effective dimensionality of the RKHS (related to the kernel decay rates) would be higher with a "less smooth" kernel like the exponential/Laplace kernel. The results are likely not reproducible unless the authors release their code. The results are also limited only to supervised vision tasks, rather than other modalities or unsupervised settings.

Summary

The authors present a new architecture for computer vision applications that models high-order interactions between features. The architecture is similar to an attention block, but introduces an RBF Kernel layer that captures interactions of order higher than two. The resulting method has strong empirical performance across image classification tasks.

Strengths

- The idea of the paper is very interesitng and novel. - The empirical results show promising performance across involve multiple tasks against sophisticated methods

Weaknesses

- The presentation of the method seems overly complex in some places. For example, providing a clearer explanation of each new layer (perhaps in pseudocode) would help. While the Infiniblock definition is clear, the reader needs to go back to the previous section to understand the input/output shaped of the RBF layer, which takes work, and can be made simpler. Making clearer the intuition behind high-order interactions would be helpful as well. Showing examples of what the model learns would be helpful to make things concrete. - The empirical performance is reasonably similar to those of previous methods, hence the empirical improvement is not that large.

Questions

- What are some examples of features and interactions that help learning and that the new model can learn? - Is it possible to analyze or visualize what interactions the model learned?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

I do not see any ethical and societal implications of the work that need to be discussed.

Summary

The paper shifts the focus from traditional neural network design, which emphasizes feature representation space scaling, to feature interaction space scaling. It introduces a new model architecture, InfiNet, that enables feature interaction within an infinite-dimensional space using the RBF kernel, leading to state-of-the-art results. The paper also discusses the limitations of current models in capturing low-order interactions and proposes the use of classic kernel methods to engage features in an infinite-dimensional space.

Strengths

- The idea of the paper is simple, novel and well exposed. - The paper introduces InfiNet, a model architecture that leverages infinite-dimensional feature interactions using RBF kernels, which enhances model performance of traditional models. - InfiNet achieves new state-of-the-art performance in various tasks, demonstrating the effectiveness of infinite-dimensional interactions. - The paper includes extensive experiments on datasets like ImageNet and MS COCO, showing the scalability and efficiency of InfiNet.

Weaknesses

- the paper builds on the simple use of kernel methods. The novelty of the methods is minimal, in the end it is an RBF kernel. - the performance improvement of Infinet over other models is mostly marginal and no errors have been displayed. - the paper doesn't really have theoretical novelty

Questions

- How does InfiNet compare to other models in terms of training time and resource consumption? - Can the kernel methods used in InfiNet be applied to other types of neural network architectures beyond those discussed? - Can the authors quantify the increased dimensionality of the kernel methods over simpler operations (sum, product). If the authors take the simplest architecture for imagenet and look at the representations generated by means of using different kernels, can they quantify what is the actual increase in the intrinsic dimensionality of the representation upon training? It is not fully clear to me that the increase in performance is due to an increase in dimensionality. - the author mention the possibility of exploiting a learnable kernel in place of RBF. Could the author explain and discuss the ratio behind using RBF in place of others? Is it solely driven by the computational complexity. Would the results be different with a different kernel?

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

2

Limitations

None