Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective

Summary

The paper analyzes the features of a deep equilibrium model (DEQ) using neural collapse metrics under class-balanced and imbalanced settings. In particular, it is shown that under imbalanced conditions, the class means of features in DEQ are relatively closer to a simplex ETF orientation than in the case of an explicit neural network model (such as ResNet).

Strengths

The paper takes an interesting theoretical view of analyzing the features of deep equilibrium models using the neural collapse metrics. In particular, the paper extends the unconstrained feature model to cases with implicit layers and analyzes the settings under which neural collapse is ideal (with respect to data balancedness).

Weaknesses

1. The paper suffers from major presentation issues. In particular, a lot of notation/formulation errors lead to unclear results. For instance: - In Eq (1) and the text below, the function $\phi()$ is assigned to a matrix without mentioning the class-wise arrangement of the features in matrix $H$. Similarly, in the definition of NC1, the class means vector has a different notation when compared with the one presented in Appendix A (line 482). - The NC2 formulation in Theorem 3.1 is wrong. - $\tau(i)$ in Theorem 3.1 is not defined. - Mismatch in the scale of NC1 values in Figure 3. 2. The author's claim about DEQ reaching lower loss when compared to explicit layers is not validated in the experiments. Especially, the (train) cross-entropy loss is not illustrated. Furthermore, based on the results in Figure 3, it is unclear when the ResNet/DEQ reaches the terminal phase of training (i.e 100% training accuracy). 3. The paper states that DEQ models can be memory efficient, but do not provide any numerical results on the computational benefits of employing them. Also, the experimental results rely on training with a single random seed which is an incomplete measure of good/bad performance.

Questions

1. The experimental setup with ResNet and the DEQ is unclear. Which part of the ResNet is being used as the image encoder? As of now, it seems like only the last ResNet block is reformulated as the DEQ, and all the previous layers are used as the image encoder. Is this observation correct? If that is the case, then what is the computational overhead of the DEQ layer? 2. Based on this setup of replacing only one ResNet block with the DEQ, how does this approach compare to simply fixing the last classifier layer of ResNet as a simplex ETF to address the class-imbalance issue? 3. What happens if we replace more than one block of the ResNet with a DEQ layer? How sensitive is the DEQ layer to learning rate and batch size?

Rating

5

Confidence

4

Soundness

3

Presentation

1

Contribution

3

Limitations

N/A

Summary

This paper studies the neural collapse (NC) phenomenon in Deep Equilibrium model (DEQ), a competitive implicit neural network model to standard explicit model. The authors compare the theoretical property of DEQ and NN on the layer-peeled model, a simplified model that include the last two layers only. It was shown that under a balanced sample setting, NC happens both in DEQ and standard NN, under an imbalanced sample setting, DEQ exhibits better robustness property in terms of the NC metrics. Experimental results are presented to justify the theoretical analysis on CIFAR10 and CIFAR100, it was shown that DEQ have better performance than standard NN across in various settings.

Strengths

The finding of this paper is novel and interesting, it provides a insightful perspective to understand the advantage of DEQ over the standard NN. The authors provide solid theoretical analysis and experimental results to support the main finding. Overall, the paper is well-organized and easy to follow.

Weaknesses

My major concern of this paper is its contribution and significance. Since the discovery of minority collapse, there has been tons of literature on how to mitigate the minority collapse. For example, Proposition 1 in [1] proves that simply reweighting the samples can mitigate the minority collapse. Therefore, my opinion is that neural collapse analysis can provide limited insight about the advantage of DEQ over standard NN, since the improvement is relatively marginal compared with simple techniques. I would suggest the authors compare the DEQ and standard NN with some standard techniques such as reweighting, and confirm that the advantage indeed have practical influence. [1] Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training

Questions

1. The experimental results are limited to ResNet-18 only, I would encourage the authors to perform more extensive evaluation on various architectures to support the finding. 2. The proof needs to be better organized. It will be helpful if the authors can provide an outline of proof in the appendix, restate the main result and break it into several propositions, and add some high level intuition of why DEQ outperforms standard NN. 3. Neural Collapse solutions do not have full rank in general, would that bring issues when writing down the inverse of gradient in equation (3)? 4. In your theoretical analysis, it was assumed that the DEQ is linear with a specific expansion form. The authors should highlight this with a hold out assumption and discuss when it can be satisfied. In particular, what is $W_{\text{DEQ}^i}$ in line 162? Are they being optimized in the programming? In the current form it seems to reduce to simply a linear function $Wx$.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have properly addressed the limitations.

Summary

This paper investigates the representation of Deep Equilibrium Models (DEQ), highlighting their memory efficiency and competitive performance. Using NC, it shows that DEQ exhibits NC under balanced conditions and maintains advantages in imbalanced settings. Theoretical findings are validated through experiments, demonstrating DEQ's superior handling of imbalanced datasets.

Strengths

1 The paper provides a theoretical analysis of the representation of Deep Equilibrium Models under both balanced and imbalanced conditions using the Neural Collapse phenomenon. This analysis demonstrating DEQ’s advantages over explicit neural networks under some mild conditions. 2 Validation of theoretical insights through experimental results on datasets like Cifar-10 and Cifar-100, enhances the credibility of the findings that the superior performance of DEQ, especially in handling imbalanced datasets. 3 The introduction of the Neural Collapse phenomenon offers a novel perspective on deep representations in implicit neural networks.

Weaknesses

1 The analysis is limited to simple imbalanced scenarios and DEQ models, restricting the generalizability of the findings to more complex real-world situations. 2 While the paper provides valuable theoretical insights and experimental validation in section 5, there may be a need for more in-depth quantitative analysis like statistical analysis to further support the claims and conclusions drawn. 3 The lack of broader experiment for $K_A\neq 3$ in Table 3 may hinder a comprehensive comparison of DEQ with Explicit NN.

Questions

Please see the weaknesses.

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have adequately addressed the limitations.

Summary

The author analyzes the DEQ from the prospective of Neural Collapse to demonstrate the reason why DEQ is effective. The Nerual collapse means that at the final phase of training (training error is close to zero), the feature and classifier vector converges to a simplex Equiangular Tight Frame. Neural collapse usually happens in balanced dataset since convergence in some classes in unbalanced dataset is difficult, which is called minority collapse. To analyze neural collapse in DEQ, the author starts from a layer-peeled model, which focus on last-layer features. And they prove that Neural Collapse also happens in DEQ. However, the lower bound of loss function is smaller compared with explicit neural network. In imbalanced dataset, the DEQ also show a smaller lower bound in loss function. One reason is that DEQ can be seen as an infinite layer network, which has better representation capacity. And the repeated iteration of DEQ can mitigate the problem in imbalanced dataset. However, minority collapse also happens in DEQ. The author provide some experiment to prove their claim.

Strengths

1: To analyze the advantage of DEQ from perspective of DEQ is interesting. This paper may bring up some new perspective to analyze DEQ. 2: The author provide analysis on both balanced and imbalanced dataset. 3: The author provide extensive experiment to prove their idea.

Weaknesses

1: The smaller lower bound of loss function may cause overfitting, which may affect the performance in the test phase. Add discussions on this part can be helpful. 2: Maybe add some experiments on dataset other than Cifar can make the conclusion more persuasive.

Questions

See weakness.

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Please let us know if any further questions!

Dear Reviewers and AC, Thank you for providing valuable comments. We have tried to address the concerns raised by the reviewers. We observe that, except for a3Sj, reviewers have no follow-up comments on our responses. Given the upcoming discussion stage deadline, please let us know if you have further questions. Besides, we would like to sincerely thank reviewer a3Sj for recognizing our contribution and raising their score from 5 to 7. Thanks and best regards. Authors