Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models

Summary

The authors investigate the infinite-width limit of a DEQ, and prove that the output converges to a Gaussian process. Their result importantly leverages the intermediary analysis of a finite-depth, finite-width DEQ. Their main technical result is that the limit of infinite width and infinite depth commute for such networks, which they build upon to establish the convergence to a Gaussian process. Numerical checks are presented to bolster the claim.

Strengths

The paper is very clearly written and easy to follow, with the main technical points being sufficiently discussed, and the relevant context being provided. Cautious numerical evidence is further provided to bolster the claims. Overall, the paper is mostly technical in nature, and altough it does not discuss the generalization properties of infinite-width DEQs, this result should be interesting to some in the NeurIPS machine learning theory community.

Weaknesses

I have not read the proof, and am not familiar with the literature of DEQs, and therefore give a low confidence score. The presentation is sound and I am convinced by the numerical checks. As a very minor remark, while I do understand discussion about the generalization ability of infinite-width DEQs is out of the scope of the present work, I do feel like the inclusion of some simple empirical comparisons with other infinite-width limits of neural networks (NNGPs and NTKs) would benefit the overal reach of the work. I have a number of questions, which I list below.

Questions

- The authors discuss how previous works show that for MLPs and ResNets, the infinite width and depth limits do not commute, while they show they do for DEQs. However, little discussion is provided as to why this difference arises: is it because of the share weights of the DEQ, or the input injection at each layer? I would find further intuition and comparison to MLPs helpful and insighful. - It is not clear why $\sigma_u$ does not enter in Lemma 4.2. Naively, the $\sigma_u \to 0$ limit should correspond to a MLP, for which the two limits do not commute. Is it the case that (14) holds for any $\sigma_u>0$? - (Minor) To my awareness, the recursions (7-13) for the infinite-width GP kernel of a DEQ are new. Could the authors provide more intuition as to how the kernel of a infinite-width but finite-depth DEQ qualitatively differs from the MLP GP kernel ($\sigma_u=0$)? For instance, a plot of the spectrum of the kernel for various $\sigma_u$ in the supplementary material would help build up intuition. - (Minor) l73: the sentence is written twice.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The paper is purely theoretical in nature and as such does not pose any foreseeable societal impact. The technical limitations of the work a clearly stated therein.

Summary

This paper focuses on the DEQ (deep equilibrium) model, an infinitely deep neural network with shared weight matrices across layers. The authors show that the network approaches a Gaussian process as the network width goes to infinity, and the limit of infinite width and infinite depth commute, also, the Gaussian vectors remain non-degenerate as network depth tends to infinity for any pairs of distinct inputs. These results do not hold for previously well-studied wide neural networks without shared weights.

Strengths

The paper presents meticulous analysis on the infinite width and infinite depth limits of the DEQ model (networks with tied weights) and specifically the rate of convergence of the two limits. The theoretical analysis is supported by numerical results.

Weaknesses

1. While technically sound, it is unclear what are the potential insights and contribution of this work to the field. I recommend that the authors add a paragraph in the conclusion section (or if possible include some numerics) on the potential implications of their main results (that the infinite depth and width limits commute for DEQ and that the structure of the limiting kernel is preserved). For example, does this explain why DEQ’s achieve competitive results with the state-of-the-art? How does the result shed light on the infinite width limit of vanilla RNNs? 2. This work focuses on properties of the DEQ at initialization, and there is no learning in the network.

Questions

1. I am not familiar with the line of work on DEQs, but based on the references I think it refers to the method of computing the fixed point of the infinitely deep networks with tied-weights, whereas in the paper it simply refers to the network structure? 2. Line 91: What are the conditions on A here (elements in A need to be subgaussian?) 3. Theorem 3.1: Here you introduce the covariance function, however, it may be useful to point out that the covariance function here is different than the covariance of f in NNGP and NTK, in NNGP the readout weights of the network is learned whereas in NTK all the weights are learned. If I understand correctly, here the covariance function is simply f at initialization with all the weights being Gaussian. As you also mention NTK in line 129 it may be helpful to stress the distinction. 4. Fig 4: why is there a large jump at the initial stage in the theory but not in simulation for the smallest eigenvalue over iterations? 5. Fig 4 the last panel: can you confirm this result with simulations?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors should add a paragraph on limitations in the conclusion section.

Summary

This paper examines the infinite width behaviour of DEQs, a kind of neural network architecture that can be viewed as an infinite-depth RNN. They show that contrary to regular MLPs, the limit of infinite width and infinite depth in DEQs commutes. They back their theory up with some numerical experiments.

Strengths

- The authors ask an interesting theoretical question and derive a surprising result (that the limits commute). - Paper is generally well written

Weaknesses

- The central importance of the commutation of the limits could perhaps be better argued for. As it stands, this seems interesting to me theoretically, but I don't see any major practical advances stemming from this result. It would be good to know if the authors see things differently. - I found some of the Figures hard to understand, particularly Fig 3 and Fig 4. Would appreciate more detail here.

Questions

- What is happening in the rightmost figure in Fig 3? It looks like there is an orange curve that goes all the way to the bottom, but the blue curve does not stay with it. I'm having trouble understanding the significance of this. - Why does the relative error bottom out at different values in Fig 1, left? - What is the legend in Fig 1 left? Is it width of the network?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes

Summary

This paper takes the theoretical tools for infinite width limits of fully connected deep neural networks (e.g. NNGP limits, tensor programs etc), and applies it to a deep equilibrium-type neural network. This model is seen as the depth \to infinity limit of a feedforward-type network. Convergence to a Gaussian process is proven for this model, and the typical type of recurrence relation for the covariance function is obtained. The theoretical results are validated with several experiments that confirm the findings, and shed light on how quick the convergence to the fixed point is.

Strengths

* The paper is well written and easy to follow. The proofs are explained well and seem to be free of any major errors. * The experiments do a good job validating the theory. They cover a broad range of possible questions one might ask about the model. * The fact that the infinite depth limit exists in a non-degenerate way is a nice finding (as opposed to other types of feedforward networks where the infinite depth limit may be degenerate in some way). This means that the idea of feeding the input $x$ into each layer directly may be a stabilizing force that can be used for very deep networks.

Weaknesses

* The main result of this paper is largely what one would expect using the infinite width technology, that is to say there is no "surprises" that happen along the way. That is not to say that something is lacking in the methods applied here, just that the result is more or less what you would expect. To really make this an outstanding paper, it would be nice to additionally see some explanation of why the resulting model is interesting...for example is there some way in which it outperforms ordinary feedforward nets or resnets or is there a way in which it is more efficient. Perhaps this kind of thing appears in the existing literature on deep equilibrium models (I am not an expert on this type of model) * Usually the big strength of the infinite width limits is that the NTK is a constant which enables a theoretical understanding of how training happens, not just initialization. This analysis of the training kernel is not carried out in this work. * The fact that the limit with or without shared weights is identical suggests that some important theoretical considerations may be missing here...surely there is some kind of important difference between these models that this infinite width limit is not capturing. In particular, I think the training kernel for the model with or without shared weights would be different even though the conjugate kernels studied here are the same. It would be interesting to compare/contrast these more, either by experiments or by more theory to get a handle on what this model is (or isn't) actually doing.

Questions

* Have you thought about the training kernel (NTK) for this model? Is there a reason the usual analysis would not go through? * Is there any analysis on the differences between the DEQ and the standard feedforward model in the wide width limit? What are the advantages/disadvantages? This may be found somewhere in the existing literature, but this was not clear to me and the fact that this DEQ model would have some advantages would be an important part of motivating the work done here.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

* One limitation that always comes up in these limiting type things is the question of how large real neural networks have to be for the theory to actually work. Some discussion or experiments specifically addressing this (e.g. showing the error in the predictions as a function of network size) could help explicitly address this. * Related to above: It would also be interesting to understand the fluctuations around the infinite width limit and how much they effect things (which may be quite different for this model vs for ordinary feedforward networks), but that is likely beyond the scope of this paper.

Acknowledgements

I thank the authors for the clarification, and do not change my scoring.

Acknowledgement

I acknowledge the rebuttal from the authors.

Reply to rebuttal

Sorry for the late response. I have read the rebuttal and I appreciate the clarification and new numerical validations. Correspondingly I would like to raise my score to 5. I do find the result that the limits commute very interesting, my concern is about the DEQ model itself. It does not seem to be a commonly used model and it is vague how the results may be applicable to other related more practical architectures.