On the Joint Interaction of Models, Data, and Features

Response to Reviewer PinD

Thank you for your thoughtful feedback! We are glad that you still find our work original and creative. > many assumptions went into the combinatorial analysis. We fully agree that many assumptions were made. The way we see it is that deep learning's complexity necessitates some level of simplification for tractable analysis. In this case, since the resulting framework is still useful for understanding empirical phenomena, we feel the assumptions are justified. > The framework is less useful for understanding the learning process of networks… This is correct. The framework as it stands does not describe the learning process, which is an extremely complicated subject itself. However, we postulate that techniques similar to [1] could be used to study the optimization of a feature model like ours in the future. > typos Thank you for the close reading and for catching these typos! We have fixed them in the revision. > have you considered modeling the rare-dominant variation as a spectrum instead of a binary? This is an excellent question! We have indeed considered the possibility of modeling it as a spectrum, which we discussed in Appendix D.1 and D.4. In our preliminary investigation, we found that modeling the feature spectrum may require introducing more free parameters (e.g., Zipf’s distribution), and can drastically increase the complexity of analysis. Given that this is our first foray into the topic, we opted for the simpler binary model. This is an important problem and we plan to explore it in more detail in future works. We hope these responses have addressed some of the questions and we are open to further discussion. Thank you again for your constructive feedback! **Reference** [1] Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. Allen-Zhu et al.

Response to Reviewer MLur (1/2)

Thank you for your detailed review and thoughtful feedback. We are glad that you find the problem we study timely and important. Below are our responses to your concerns: > I definitely agree with "the empirical phenomena of deep learning can be understood at many different layers of abstraction". However, I think the model proposed in this paper is too simplistic… We definitely agree that our model is unconventional and more abstract than most existing works but similar assumptions have been made in prior work [6]. As is the case for all theoretical models, simplifications are required to keep analysis tractable. Despite the simplification, our model already exhibits some interesting behaviors and offers insight into empirical phenomena so we believe that it adds value to the current discussion on feature learning. In the case of random feature models, we feel that it is a little unfair to say in hindsight that there was a very obvious way to move towards making it more realistic. When NTK first came out in 2018, it was thought to be very promising. Indeed, despite its simplification, it offers clear value in the analysis of some phenomena in deep networks. However, it only later became clear that there was a big gap between NTK and real deep learning (despite many efforts to close this gap). More recent works (many of which you already brought up) rely on quite distinct analysis tools than those used in NTK and, still, it’s not clear if any of them is close to real-world practice. As such, while we agree (and have acknowledged in Section 5, Appendix D.1 and D.4) that there is a gap between our proposed framework and practice, we feel it’s a tall order for the current state of deep learning theory to have no strong assumptions. Just like NTK analysis, we hope that our approach will ultimately prove valuable (albeit along different dimensions), despite its simplicity. > Although the model is based on some observations using the interaction tensor, I still find the model to be not very well motivated. Any insights on how this can relate to the training of deep nets? The primary goal of our work is to investigate the effect of different features being learned by different models on GDE and the calibration of deep ensembles, both of which are robust empirical observations. While our model does not directly provide insights into training, understanding these feature interactions may indirectly inform training strategies in the future. For example, [6] shows potential avenues for how observations like ours may transfer to training. > It is not very clear how the authors set the hyperparameters in their model (e.g., the thresholds). As detailed in Appendix F.10 (referred to at the end of Section 4), our conclusions are robust across a range of parameter settings so we believe that the settings used in our paper should be appropriate for most situations. > The notations are a bit confusing ($i, j, k$, etc.). I suggest authors avoid using these generic letters. Also, the paper will benefit greatly from a figure that summarizes all the notations ($p_d, p_r$, etc.). It will also help explain the method. We apologize for the confusion. There is already a centralized list of notations in Appendix A which we could not include in the main text due to space constraints (this is referred to in the second paragraph of section 5). Beyond this, could you be more specific about where the $i,j,k$ are causing confusion so we could better address the confusion? $i,j,k$ are used to denote an arbitrary member of the ordered collection so we feel this is in accordance with the standard usage. > But the model is very abstract/high-level and I'm not sure if the assumptions that they make are stronger or weaker than calibration. We want to clarify that we *do not* claim that the assumptions we make are weaker than calibration (since neither is a clear superset of the other), but we do believe that it sheds more light on the underlying reasons behind GDE. Only stating the model is calibrated does not tell us much about the model or the data, but our assumptions do. In fact, our framework tells us how one can break our assumption (and calibration) through data distribution interventions. Of course, it is an important open question why so many natural distributions seem to satisfy these assumptions.

Response to Reviewer MLur (2/2)

> The theoretical understanding of feature learning is not as rudimentary as the authors claim We sincerely apologize for the characterization. Perhaps “rudimentary” is not the best description. We have revised the phrase to be “still in early development” to better convey the sentiment. However, we do feel that it is reasonable to say that we still have much to understand about feature learning. To briefly address the specific works you mentioned: these models generally assume the input data are isotropic Gaussian and the function class is a 2 layer MLP. The feature is usually a linear function of the input and often special training algorithms are needed (e.g., layer-wise training). A notable exception is [3] which learns non-linear features but still requires layer-wise training. Most of these works are quite recent (and all came out after we started this project): [3,4,5] showed up in October 2023 and [2] was updated in November 2023. In general, it is hard to say how well they encapsulate the full complexity of deep learning, or how feasible it is to apply them to modern architectures and datasets. Nonetheless, the goal of this work is not to compare existing theoretical works on the topic or discuss their merits and shortcomings, but rather to identify what realistic (perhaps strong) modeling assumptions to make when studying feature learning. Given the recently increasing interest in the subject, we view these works and our approach as complementary components of a broader exploration into feature learning. We have added a discussion of these works to Appendix D.1, but would be happy to revise them if you have further suggestions. We hope these responses address your concerns and we are open to further discussion. Thank you again for your constructive feedback! **Reference** [6] Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. Allen-Zhu et al.

Response to Reviewer PxMC

Thank you for the support of our paper! We are glad that you find our work valuable. Regarding Figure 1, to better address your concerns, could you elaborate on which aspects of the figure you found unclear? This will help us make more targeted improvements. The goal of Figure 1 is to demonstrate that our definition of features can capture images with redundant information and images that contain uncommon instances of a given class. To make this clearer, we have updated the figure in the paper with texts to better reflect the core idea. Does this resolve your concerns? If not, we’d be happy to make more adjustments.

Response to Reviewer AyvM

Thank you for your supportive feedback on our work. We are glad that you find our new perspective on feature learning interesting. Below are our responses to your questions: > how important are these hyperparameters? How to properly set them? In Appendix F.10, we conducted an ablation study and found that our qualitative conclusions are robust to variations in these hyperparameters. Therefore, we believe the values used in our paper should be appropriate for most applications. > Currently theoretical framework analyzes binary classification, does the analysis extend to multi-class classification? We discuss the necessary steps for the extension to multi-class classification in Appendix D.3. >Does the framework allow for introducing distribution shifts and etc? This is a great question. We believe our framework can accommodate distribution shifts. For instance, the simplest form of covariate shift one could introduce is to add a new set of features that are not present in the training data with some probability at test time. Since the model has not seen these new features during training, we may assume that individual members of the ensemble will make random predictions (but the prediction of different models will be correlated) if the data points only contain the new features (the same assumption for features that the models did not learn). One can also introduce a covariate shift on existing features but that would require imposing a non-uniform distribution over the existing features. We leave the exploration of these directions to future works. We hope these answers clarify your question, and we are open to further discussion. Thank you again for your constructive feedback.