Summary
This paper introduces two corrections to previous work that relates the representation cost of infinite-depth nonlinear networks to the bottleneck rank. These two corrections are obtained by Taylor expansion of the representation cost. In addition to the previously-known rank term $R^{(0)}(f)$ that is directly related to the bottleneck rank, the authors consider two corrections (1) $R^{(1)}(f)$, which bounds the pseudo-determinant of the Jacobian, and (2) $R^{(2)}(f)$, which plays a role in the intermediate representations in the hidden layer of the network. The authors specifically discuss several theoretical properties of $R^{(1)}(f)$. This term acts as a notion of regularity in the learning process of deep nonlinear networks, and it explains various interesting properties, such as how deep networks trained with conventional L2-regularized loss are uniquely determined (i.e., which $f$ among candidates with the same $R^{(0)}(f)$ is selected) and why the rank of the true function is not underestimated. The authors also briefly demonstrate the relationship between the curvature of the domain manifold and $R^{(1)}$ using the rank of the identity function, and discuss the connection between the learning rate and the Lipschitzness of learned deep networks with the results of the neural tangent kernel. Finally, the authors discuss the dynamics of hidden representations, which in the few initial layers the representations undergo changes to minimize $R^{(0)}$, then, as they move into the low-dimensional space, they change smoothly influenced by $R^{(1)}$ and $R^{(2)}$, i.e., exhibit the bottleneck structure of networks.
Strengths
This paper is a direct follow-up to [1], and since I am not very familiar with [1], my evaluation may not be very accurate.
Theoretical understanding of deep nonlinear networks is a highly significant field and holds considerable interest among the NeurIPS audience. For me, the theoretical results of this paper are highly novel and intriguing. The paper successfully modifies the results of [1], and provides a rigorous yet intuitive explanation of why neural networks trained with our L2-regularized loss do not underestimate the rank of the true function. Additionally, it demonstrates that under some conditions, the representation dynamics of deep nonlinear dynamics exhibit a bottleneck structure, which consists of a sequence of long bottleneck dimensional representations. This result is also noteworthy.
***
[1] Arthur Jacot. Implicit bias of large depth networks: a notion of rank for nonlinear functions. In The Eleventh International Conference on Learning Representations, 2023.
Weaknesses
I couldn't find any major drawbacks in this paper.
The frequent mention of $R^{(2)}$ in Remark 10, before its properties were clearly stated, was somewhat confusing to me, although it is not a critical issue.
There are just a few very minor typos:
In line 32, “… not control the the …”
In line 34, “… formalizes the a …”
In line 118, “… because the the …”
In line 156, “… $k^{*}$ of he task ..."
Questions
I couldn't find any major drawbacks in this paper. One small question is:
I would like to hear the authors’ thoughts on whether the optimal latent dimensionality of a well-trained auto-encoder $f$, which is likely to be similar to the identity $id$, might be associated with $R^{(0)}(f;\Omega)$ and $R^{(1)}(f;\Omega)$, for the case when $Rank_{J}(id;\Omega) = Rank_{BN}(id;\Omega)$, and if possible, when $Rank_{J}(id;\Omega) < Rank_{BN}(id;\Omega)$
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The authors do not explicitly address the limitations. However, in some of the theorems/propositions, they briefly mention the limitations of the proposed one, indicating what it can only explain or what constraints it has.