Scaling MLPs: A Tale of Inductive Bias

Summary

This paper argues that MLP networks can perform well on challenging tasks when they are scaled up. Thus, they are able to compensate for their lack of inherent inductive bias when trained on a large amount of data with many model parameters. The authors show through experiments on popular image classification datasets that indeed, MLPs can perform surprisingly well when scaled up. Along the way, the authors make a number of other observations regarding the role of MLP architecture, data augmentations and scaling laws.

Strengths

**Originality** The question the authors aim to answer is quite interesting. As far as I can tell, few empirical studies have approached the question of whether MLPs can be made to perform competitively on standard benchmarks under large-scale data and model size (although studies have trained MLPs on standard benchmarks for other reasons). **Quality** The experiments are extensive and thorough. The authors train on ImageNet21k and evaluate on standard image classification benchmarks; in my view, these are exactly the most valuable experiments to run to investigate the question the authors aim to answer. **Clarity** The paper is quite well-written. The key motivation of the paper and background are made very clear, and figures are well-illustrated. As a minor suggestion, moving Table 1 and 2 closer to the experiment section would be helpful. **Significance** Overall, if the paper can demonstrate its claims properly, the paper would be quite significant to the field. Showing that the inductive bias of CNNs or Transformers can be compensated with scale will perhaps focus the attention of the field toward building inductive biases that can't as easily be compensated by scale, as well as validating (or invalidating) the idea of applying theory done on MLPs to other models.

Weaknesses

My main concern with the paper is with the interpretation of the experimental results; I'm concerned that the results are not enough to support the authors' hypothesis 3. First, the authors only conduct experiments on image classification tasks. If the authors wished to make a more general claim about inductive bias in MLPs vs other models, it would be best to conduct experiments on other domains. Alternatively, the authors may wish to restrict their claims to the image domain. Second, the empirical results on ImageNet do not appear quite strong. In Table 4, the authors find a maximum ImageNet top-1 accuracy of 41%; this does not seem very competitive. If the authors wanted to show that MLPs can indeed circumvent their lack of inductive bias, I would expect to see a result showing near state-of-the-art (or at least competitive results) on ImageNet. I may be missing some key empirical results though. Third, the authors find that data augmentation is critical to the performance of MLPs. Data augmentation, as the authors note, allows the model to learn invariances that would otherwise be built into a model like a CNN. Thus, data augmentation is a powerful inductive bias. However, this fact seems to undermine the authors' claim that scale alone is sufficient to compensate for lack of inductive bias. The authors may wish to reformulate their claim to suggest that architectural inductive biases can be compensated by a combination of scale and other inductive biases. Similarly, the authors find that using a specific inverted bottleneck MLP architecture is important for performance. The authors seem to view this as a minor addition to the MLP, but given the large performance improvement, this addition is actually quite significant. Indeed, in my view, architectural changes that are simple but lead to large performance gains are exactly those that are most valuable to study! Also, as a more minor point, the authors claim that MLPs are "completely free of any inductive bias." This seems to be too strong of a claim: they are often one of the simplest models to implement, but this doesn't mean they are free of any inductive bias. The claim quoted by the authors from the *Principles of Deep Learning Theory* book seems to be a more fair statement to me.

Questions

Can performance on ImageNet be improved to near state-of-the-art for MLPs? What is the relative significance of scale vs. data augmentation/model enhancements for MLPs?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

2 fair

Limitations

It would be good if the authors could mention some of the limitations listed above in the discussion section. No potential negative societal impacts.

Summary

This paper presents a comprehensive study on the scaling capabilities of MLPs, challenging the conventional belief that this architecture is limited in its performance on vision tasks. The authors employ recent advancements in deep learning architectures to modernize MLPs while preserving their core characteristics. The experimental results obtained in this study are remarkable, indicating that tasks previously deemed impossible for MLPs can now be accomplished. To achieve these results, the authors incorporate several crucial components of modern deep learning architectures into MLPs. They introduce normalization layers, skip connections, bottlenecks, and a large batch size, along with a modern optimizer. These additions ensure stable training, thus allowing to train on large datasets. Additionally, the authors demonstrate that despite the absence of inherent inductive bias in MLPs, data augmentation and sufficient data volume effectively compensate for this limitation. Consequently, scaled MLPs perform competitively with architectures possessing stronger inductive biases, such as CNNs or ViTs, on several datasets, while being much more computationally efficient at a fixed resolution. Finally, the authors assess the scaling properties of MLPs providing accurate estimates of their scaling laws in terms of data budget and compute budgets. Unlike other architectures, in scaling MLPs larger datasets are more important than compute budget. The findings open up new possibilities for utilizing MLPs in vision tasks and highlight the importance of reconsidering their scaling capabilities in the broader context of deep learning research.

Strengths

#### 1. **Addressing a clear research gap in the literature**: This paper successfully completes the research program started with the ViT paper and continued with the MLP-Mixer and ConvMixer that challenges common assumptions regarding the benefits of embedding inductive biases in architectures. By answering the question of whether inductive bias can be fully circumvented with more data affirmatively, this paper fills a significant void in the literature. Although the results might have been expected by many, the clear empirical validation of this hypothesis makes this paper an important contribution. #### 2. **Thorough experiments with solid results**: The execution of the experiments and the manner in which they are conducted are exemplary. The paper provides a comprehensive ablation analysis across various settings, thoroughly examining and validating each claim. The results obtained are both convincing and transparent. Notably, the authors openly discuss the relatively poorer performance on ImageNet and offer insightful explanations for this observation, enhancing the credibility of their findings. #### 3. **Outstanding presentation**: The paper exhibits outstanding writing quality, making it a pleasure to read. The clarity of the language and the structure of the paper contribute to its overall excellence. The authors effectively communicate their ideas, methods, and results, ensuring that readers can easily comprehend and appreciate the significance of their work.

Weaknesses

#### 1. **Missing discussion regarding the importance of processing images in patches** In my opinion, the paper is missing an important discussion regarding the critical factor that makes MLP-mixers superior to the presented full MLP architecture, i.e., the processing of images in patches. Although I understand a very in-depth empirical analysis of these differences might be outside of the scope of the paper, it is important to fully acknowledge the importance of this key design choice. In particular, I believe it is important to also reference the ["Patches are all you need?"](https://openreview.net/forum?id=rAnB7JSMXL) paper which also tried to fill a research gap in this sense prior to this work. #### 2. **Missing discussion on the effect of image resolution on the results** It seems that an important design decision in this paper has been the downscaling of all datasets to a 64x64 resolution. I believe this decision deserves further motivation in the text as well as some reflection on the fact that MLPs cannot be applied to inputs of different sizes.

Questions

I would encourage the authors to address the stated weaknesses in the rebuttal and in any future revision of the manuscript. I honestly believe doing so will make the paper better.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Overall, I believe the paper has addressed well most of its limitations and that the authors have been open and honest in describing the shortcomings of their results. However, as mentioned in Weakness 2, I believe it is important that they also discuss how resolution and different image sizes matter in their results.

Summary

This paper studies scaling of MLPs on various image classification datasets with and without pretraining and data augmentation.

Strengths

MLP scaling trends are a welcome data point for practitioners and theoreticians alike.

Weaknesses

The paper does not carefully tune the learning rate, weight decay, and other hyperparameters, so the trends shown in this work may not be a Pareto frontier.

Questions

The abstract states "Given the recent narrative less inductive bias is better ... it is natural to explore the limits of this hypothesis" in L4 but this claim is not supported. Rather, a weaker version stated in L52 is investigated in this work, which could be clarified. The paper states in L7-12 that while deep learning theory literature almost always studies MLPs, there are no empirical datapoints connecting to theory. This sounds like the paper would cite relevant theory works and try to confirm/refute them, but this is not the case. The paper explores scaling of MLPs. At least, the thought in L7-12 does not connect smoothly to the question raised in L12-13. Perhaps, this statement could be softened a bit. L15 Perhaps, it should be clarified that pretraining on ImageNet21k is used: "... drastically improves with scale [and pretraining ...]" In Table 3, the parameter count (in the first layer) is resolution dependent, but the resolution is not given. L194. MLP does indeed become better, but not enough to state that it's competitive with a ResNet. L231-232. It is common knowledge that FLOP = 2P for MLP L236. (5) doesn't have the coefficient c. Also, power-law is defined as `a * x ^ power + b` without the affine offset for `x`. L296. Discussion should be Conclusion. I don't fully understand why Block(z) = z + ... is called an inverted bottleneck MLP. Residual MLP seems more natural

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

No, the authors didn't write potential societal impact of their work.

Summary

This paper studies the abilities of Vanilla MLPs and a variant of the MLP with residual connections, LayerNorm, and inverted bottlenecks, performs when scaling in size, compute, batch size, and dataset size. Many learning theory papers study MLP-based settings and claim to motivate their results by deep learning results that are usually run on CNNs or transformers. Somewhat counterintuitively, there are not many clear results on MLPs that showcase all of the relevant behaviors of interest that are studied on more complicated architectures. To justify the theoretical avenues of research further, we would hope that scaling MLPs behaves similarly to scaling CNNs and transformers. Furthermore, one goal of this paper is to understand the claim that the transformer is better due to lack of inductive bias, being a move from a strong inductive bias of CNNs to a weaker inductive bias. Due to the lack of inductive bias, it is often claimed that transformers require further pretraining, but in this regime end up performing better. The authors study a variety of these claims as they pertain to MLPs and show that their “modern” version of the MLP can, indeed, showcase powerlaw generalization scaling with total compute, as well as improved performance with size of the model and dataset size, as well as the ability to be pretrained and effectively transferred to downstream tasks (all on visual recognition tasks). EDIT: Thank you for your rebuttal, I think with these changes and clarifications it is a good contribution to NeurIPS and I intend to keep my score and continue to lean towards accept rather than reject.

Strengths

This is a question I myself was excited to see studied. I bid on this paper as I am highly curious to what extent MLPs share the same behaviors as their more modern counterparts. Having seen many learning theory papers that study the MLP, it is a very natural question to know whether they share some of the interesting dynamics that CNNs and transformers show. I think the experiments are largely beginning to answer this question, though I do have many questions and some potential concerns about experiments and the presentation of results in the paper (which I have written in the questions section) and would appreciate more clarity in order to feel very confident in accepting this paper. I am generally impressed by the results that are shown, especially the transfer learning results, in that an MLP can, in some settings, achieve such accuracies on tasks like CIFAR-10 and CIFAR-100. In some of these settings I am not yet sure the authors have gone deep enough (discussed more below), but I am overall excited about this type of work and want to see more of it. It seems natural to have tried many of these experiments, and yet I can’t recall any paper that has done so. Clearly, the authors have identified a few key examples of settings in which model scaling is commonly studied, and they have given compelling evidence that some MLPs do indeed follow our existing intuitions. I would love to see a lot more evaluations and discussions, and I think the current results could be a launching pad for more thorough work. As such, I feel that this paper leaves a lot of questions to be answered, but in that sense I think it might still be a worthwhile contribution to the community in its current form, and will help motivate a much-needed bridge between theory and practice, and further inspire more detailed empirical research. However, a revision that carefully addresses the questions I’ve left below would go a long way to convince me that it is a very strong piece of work.

Weaknesses

The authors are motivated by the question of whether MLPs reflect the empirical observations of practical models, but I have some concern of how they qualify practical. These results show some settings of interest that to me seem fairly specific, and try to discuss what is practical or not. I think it is overall a little bit shallow in total experimentation, given it is a purely empirical paper and experiments are all using one GPU I think more can be done. Generally, I am averse to criticizing a paper for not enough experiments and this paper, I feel, has shown sufficient evidence to convince me there is something interesting. But I think that a lot more work has to be done to really show many of the dimensions by which MLPs scale, and how it relates and compares to modern architectures. I know the comment above may be a little vague, so I list below the various specific questions (in the questions section) that I have upon reading this paper, which I hope the authors will find helpful. I think the questions below subsume anything I could write on weaknesses, and I think the paper would be very strong if it has answers to these questions.

Questions

- Nitpicky: I am not sure I agree that MLP-Mixer has “arguable even less inductive bias” than a Vision Transformer, but this is a very subjective opinion and I could see an argument for it, but given that data is pre-processed into patches and there is mixing between patches and features, it feels like there is quite a similar level of “inductive bias” between the two, and so I am hesitant to accept this claim at face value, but I don’t mind if it is left in the paper either - The authors make the statement at least twice early on in the paper (e.g. top of page 3) that “MLPs invest their compute significantly more into dataset size compared to model size” and I don’t think I understand what this means, nor did I feel I had much clarity on this statement after reading the experiments. Is it saying that scaling dataset size is more important than scaling model size? If that is the case, I don’t think any of the current plots very clearly justify this point, perhaps you can find a way to make it more clearly - Why do you use LayerNorm and not BatchNorm? Do you compare to BatchNorm? Does it matter, and if it does how much? How does this relate to normalizations and choices in other models? E.g. does CNN or ViT have less susceptibility to be affected (in how it scales) by normalization layer choices? - Do you need to use a bottleneck embedding layer? This seems like a pretty significant requirement. Do these scalings not work without a bottleneck embedding? If that is the case I think it is important to be very clear that a specific MLP gives this scaling, but then if you were to take an MLP with wider layers than input dimension (no bottleneck embedding) these results don’t hold. This seems very important to emphasize in your study, as many learning theory papers (that I’ve come across) don’t necessarily distinguish that MLPs need to have a bottleneck embedding layer - Strong label smoothing also seems like a strong constraint, does this work without it? I think generally label smoothing I can believe is practical but not so aggressive as to use alpha=0.3. If these behaviors are only specific to bottleneck embeddings with strong label smoothing I think that should really change the tenor of the main claims, because you are studying a highly specific setting in which MLPs showcase such scalings, but that an average MLP trained on one-hot encoded labels with wide layers, for instance, does not do this (this seems very important to make clear) - Perhaps there is a citation or another line of work that gives precedent for aggressive label smoothing like this? You don’t provide any citation or justification of this choice - A bit related to the last point, you mention combatting overfitting and severe overfitting, but you haven’t reported any training accuracies nor generalization gaps. Throughout Table 1 it is hard for me to totally understand what is overfit vs. underfit when you only report test accuracies. - Furthermore, you don’t mention whether you use a validation set or any cross-validation when experimenting with different choices, and then report the final error on a held-out test set. Can you clarify your experimental methodology? - In Table 1 why do you train Vanilla at 100 and then inverted + DA at 1000 and 5000, what happened to Vanilla at 1000 and 5000? Plus Vanilla + DA, Inverted, and ResNet18 + DA don’t report how many epochs it is trained for. Maybe reporting train accuracies and/or generalization gap will help. - It seems like it would be useful, when comparing different settings, to stop model training at the same train accuracy to control for other factors, is this something you do? Or how do you justify the comparisons you are making? It is hard for me to get a total sense of all of the dynamics at play here purely from test accuracies. - One of your claims is that larger batches significantly boosts both up and downstream performance, and Figure 3 is plotting Error vs. batch size showing this but the line in page 6 says “we plot pre-training batch size against resulting linear downstream accuracy…” which I just found a bit confusing because when I read that and looked at the downward trend in Figure 3 my immediate thought was that accuracy was going down, but then I saw you had error on the y-axis and not accuracy, so maybe you can reword this line - In the “Role of augmentations” section you mention that data augmentation gives indirect inductive bias, but you don’t really qualify what this means nor give any citation or justification of this. I guess it reads just like a statement you can make because it’s vague, but I don’t know what exactly you are trying to say to me and why it is correct. Perhaps you can give some intuition on what is left implicit here

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A